Question: What action leads to this outcome? Exploring naive physical action-effect forecasting.

A Comprehensive Neural Framework for Diverse Natural Language Processing Tasks

This paper introduces a flexible neural network architecture and learning methodology designed to tackle a wide range of natural language processing (NLP) tasks, such as part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. The framework's adaptability is achieved by eschewing task-specific engineering and the reliance on extensive prior knowledge. Instead of utilizing handcrafted input features tailored for each specific task, the system is designed to learn its own internal representations from a large corpus of mostly unlabeled data. This approach not only enhances the model's versatility but also reduces the need for manual feature engineering. The developed framework is further utilized to create an open-source tagging system that boasts high performance while maintaining low computational demands, making it accessible for a broad range of applications in the NLP domain.

Adults are extremely adept at recognizing social cues, such as eye direction or pointing gestures, that establish the basis of joint attention. These skills serve as the developmental basis for more complex forms of metaphor and analogy by allowing an infant to ground shared experiences and by assisting in the development of more complex communication skills. In this chapter, we review some of the evidence for the developmental course of these joint attention skills from developmental psychology, from disorders of social development such as autism, and from the evolutionary development of these social skills. We also describe an on-going research program aimed at testing existing models of joint attention development by building a human-like robot which communicates naturally with humans using joint attention. Our group has constructed an upper-torso humanoid robot, called Cog, in part to investigate how to build intelligent robotic systems by following a developmental progression of skills similar to that observed in human development. Just as a child learns social skills and conventions through interactions with its parents, our robot will learn to interact with people using natural social communication. We further consider the critical role that imitation plays in bootstrapping a system from simple visual behaviors to more complex social skills. We will present data from a face and eye finding system that serves as the basis of this developmental chain, and an example of how this system can imitate the head movements of an individual.

How does agency arise in biocognitive systems? Can the emergence of agency be explained by physical laws? We review these and related questions in philosophy, psychology, biology, and physics. Based on the review we ask the questions: i) Can machines have agency? and, if so, ii) How can we build machines with agency? We examine existing work in artificial intelligence and machine learning on selfmotivated, self-teaching, and self-developing systems with respect to the ontogenesis (“coming into being”) of agency in computational systems. The impact of these “autogenic” systems or machine agency on science, technology, and humanity will be discussed.

Research has shown that judgments tend to assimilate to irrelevant "anchors." We extend anchoring effects to show that anchors can even operate across modalities by, apparently, priming a general sense of magnitude that is not moored to any unit or scale. An initial study showed that participants drawing long "anchor" lines made higher numerical estimates of target lengths than did those drawing shorter lines. We then replicated this finding, showing that a similar pattern was obtained even when the target estimates were not in the dimension of length. A third study showed that an anchor's length relative to its context, and not its absolute length, is the key to predicting the anchor's impact on judgments. A final study demonstrated that magnitude priming (priming a sense of largeness or smallness) is a plausible mechanism underlying the reported effects. We conclude that the boundary conditions of anchoring effects may be much looser than previously thought, with anchors operating across modalities and dimensions to bias judgment.

Open domain causal learning involves learning and establishing causal connections between events directly from sensory experiences. It has been established in psychology that this often requires background knowledge. However, background knowledge has to be built from first experiences, which we term ground level causal learning, which basically involves observing temporal correlations. Subsequent knowledge level causal learning can then be based on this ground level causal knowledge. The causal connections between events, such as between lightning and thunder, are often hard to discern based on simple temporal correlations because there might be noise - e.g., wind, headlights, sounds of vehicles, etc. - that intervene between lightning and thunder. In this paper, we adopt the position that causal learning is inductive and pragmatic, and causal connections exist on a scale of graded strength. We describe a method that is able to filter away noise in the environment to obtain likely causal connections between events.

We study the problem of learning physical properties of objects from unlabeled videos. Humans can learn basic physical laws when they are very young, which suggests that such tasks may be important goals for computational vision systems. We consider various scenarios: objects sliding down an inclined surface and colliding; objects attached to a spring; objects falling onto various surfaces, etc. Many physical properties like mass, density, and coefficient of restitution influence the outcome of these scenarios, and our goal is to recover them automatically. We have collected 17,408 video clips containing 101 objects of various materials and appearances (shapes, colors, and sizes). Together, they form a dataset, named Physics 101, for studying object-centered physical properties. We propose an unsupervised representation learning model, which explicitly encodes basic physical laws into the structure and use them, with automatically discovered observations from videos, as supervision. Experiments demonstrate that our model can learn physical properties of objects from video. We also illustrate how its generative nature enables solving other tasks such as outcome prediction.

JackIn is a new human-human communication framework for connecting two or more people. With first-person view video streaming from a person (called Body) wearing a transparent head-mounted display and a head-mounted camera, the other person (called Ghost) participates in the shared first-person view. With JackIn, people's activities can be shared and assistance or guidance can be given through other peoples expertise. This can be applied to daily activities such as cooking lessons, shopping navigation, education in craft-work or electrical work, and sharing experiences of sporting and live events. For a better viewing experience with frist-person view, we developed the out-of-body view in which first-person images are integrated to construct a scene around a Body, and a Ghost can virtually control the viewpoint to look around the space surrounding the Body. We also developed a tele-pointing gesture interface. We conducted an experiment to evaluate how effective this framework is and found that Ghosts can understand the spatial situation of the Body.

Learning commonsense knowledge from natural language text is nontrivial due to reporting bias: people rarely state the obvious, e.g., “My house is bigger than me.” However, while rarely stated explicitly, this trivial everyday knowledge does influence the way people talk about the world, which provides indirect clues to reason about the world. For example, a statement like, “Tyler entered his house” implies that his house is bigger than Tyler. In this paper, we present an approach to infer relative physical knowledge of actions and objects along five dimensions (e.g., size, weight, and strength) from unstructured natural language text. We frame knowledge acquisition as joint inference over two closely related problems: learning (1) relative physical knowledge of object pairs and (2) physical implications of actions when applied to those object pairs. Empirical results demonstrate that it is possible to extract knowledge of actions and objects from language and that joint inference over different types of knowledge improves performance.

In three studies we found further evidence for a previously discovered Human-Robot (HR) asymmetry in moral judgments: that people blame robots more for inaction than action in a moral dilemma but blame humans more for action than inaction in the identical dilemma (where inaction allows four persons to die and action sacrifices one to save the four). Importantly, we found that people's representation of the “robot” making these moral decisions appears to be one of a mechanical robot. For when we manipulated the pictorial display of a verbally described robot, people showed the HR asymmetry only when making judgments about a mechanical-looking robot, not a humanoid robot. This is the first demonstration that robot appearance affects people's moral judgments about robots.

A study by Slater, et al., [1995] indicated that naive subjects in an immersive virtual environment experience a higher subjective sense of presence when they locomote by walking-in-place (virtual walking) than when they push-button-fly (along the floor plane). We replicated their study, adding real walking as a third condition. Our study confirmed their findings. We also found that real walking is significantly better than both virtual walking and flying in ease (simplicity, straightforwardness, naturalness) as a mode of locomotion. The greatest difference in subjective presence was between flyers and both kinds of walkers. In addition, subjective presence was higher for real walkers than virtual walkers, but the difference was statistically significant only in some models. Follow-on studies show virtual walking can be substantially improved by detecting footfalls with a head accelerometer. As in the Slater study, subjective presence significantly correlated with subjects’ degree of association with their virtual bodies (avatars). This, our strongest statistical result, suggests that substantial potential presence gains can be had from tracking all limbs and customizing avatar appearance. An unexpected by-product was that real walking through our enhanced version of Slater’s visual-cliff virtual environment (Figure 1) yielded a strikingly compelling virtual experience—the strongest we and most of our visitors have yet experienced. The most needed system improvement is the substitution of wireless technology for all links to the user. CR

This paper introduces a novel prosthetic hand control architecture using inertial information for grasp prediction in order to reduce the cognitive burden of amputees. A pair of inertial measurement sensors (IMUs) are fitted on the wrist and bicep to record arm trajectory when reaching to grasp an object. Each object class can be associated with different methods for grasping and manipulation. An observation experiment was conducted to find the most common grasping methods for generic object classes: Very Small (VS), Small (S), and Medium (M). A Cup (CP) class was also examined to find differences in grasping habits for pick and place, and drinking applications. The resulting grasps were used to test the discriminatory ability of inertial motion features in the upper limb for VS, S and CP object classes. Subject experiments demonstrated an average classification success rate of 90.8%, 69.2% and 88.1% for VS, S and CP classes respectively when using a k-nearest neighbors algorithm with a Euclidean distance metric. The results suggest that inertial motion features have great potential to predict the grasp pattern during reach, and to the authors' knowledge, is the first IMU-based control strategy to utilize natural motion that is aimed at hand prosthesis control.


Question: What activity leads to this? Progress in predicting basic physical action-outcome relationships.

This paper delves into the task of Visual Question Answering (VQA), focusing on the intricate challenge of predicting the effects of naive physical actions. Unlike conventional VQA approaches that deal with static images, this study introduces a novel framework specifically designed to understand the dynamic relationship between actions and their physical consequences. The proposed system employs a combination of visual and textual analysis to accurately predict the outcomes of various actions depicted in images. By incorporating an understanding of physics and object interactions, the model is able to anticipate the effects of actions that may not be immediately obvious from the visual content alone. This approach not only enriches the field of VQA but also paves the way for more sophisticated and context-aware image understanding systems. The effectiveness of the proposed method is validated through extensive experiments on datasets that test the model's ability to predict action effects, demonstrating significant improvements over baseline VQA models.

Personality is strongly influenced by motivation systems that organise responses to rewards and punishments and that drive approach and avoidance behavior. Neuropsychological research has identified: (a) two avoidance systems, one related to pure active avoidance and escape, and one to passive avoidance and behavioral inhibition produced by goal-conflict; and (b) two approach systems, one related to the actions of reward seeking and one to experience and behavior related to pleasure on receiving reward. These systems mediate fluid moment-by-moment reactions to changing stimuli, with relatively stable person-specific sensitivities to these stimuli manifested in personality traits. We review what is known about these motivational traits, integrating the theory-driven approach based on animal learning paradigms with the empirical tradition of the Big Five personality model. People differ from one another, and this fact is obvious to everyone. It is common to talk about people’s personalities using lexical terms to describe their characteristic ways of thinking, feeling and behaving (e.g., ‘bold’, ‘lazy’, ‘intelligent’), and we use these descriptors to infer people’s intentions and likely future behavior. Personality psychologists have long analyzed the ratings of large numbers of trait descriptive adjectives to produce the most widely used taxonomy of personality: the Big Five, which includes the dimensions of Extraversion, Neuroticism, Conscientiousness, Agreeableness, and Openness to Experience ⁄ Intellect (John, Naumann, & Soto, 2008). These Big Five traits also emerge from existing personality questionnaires that were not designed specifically to measure them (e.g., Markon, Krueger, & Watson, 2005), suggesting it is a good candidate for a consensual model in personality psychology. What the Big Five model does not immediately offer, however, is an explanation for the causal sources of personality traits. Why do people think, feel, and act in the ways that they do? People react to situations, of course; but different people react differently to the same situation, suggesting that they have different behavioral propensities. In order to answer this why question, we must discover what drives people’s actions and reactions. Inferring motivation from observed personality has been something of a dark art in psychology. However, one promising approach to this question is based on the biology of motivational control systems, studied by psychologists for over a century in non-human animals, and for somewhat less time in humans. This approach operates on the premise that stable individual differences in behavior (personality traits) must be due to relatively stable individual differences in the operation of brain systems that produce (state) behavior from moment-tomoment. From this perspective, each of our many traits reflects the operations of a set of brain systems that has evolved to respond to a different class of functional requirements (Denissen & Penke, 2008; McNaughton, 1989; Nettle, 2006; Pickering & Gray, 1999). Social and Personality Psychology Compass 7/3 (2013): 158–175, 10.1111/spc3.12016 a 2013 Blackwell Publishing Ltd In what follows, we focus on those motivational processes and personality traits most closely aligned with biological research on reactions to reward and punishment and associated approach and avoidance behavior. This focus is warranted both by the importance of these phenomena for motivation and by the existence of extensive research on them. Our aim is to offer an introduction for researchers wishing to explore the role of motivation in personality from the perspective of these underlying psychobiological systems. Only after a description of what is known about the operation of these systems do we branch out to consider the personality traits associated with them. Our major assumption is that most fundamental personality traits have a motivational core; and we aim to show that the descriptive personality research tradition, which produced the Big Five, can be integrated with the experimental research tradition that has focused on the sensitivities of basic motivation systems. In this review, we focus on systems related to approach and avoidance primarily at the level of explanation that Gray (1975) labeled ‘the conceptual nervous system’, which is based on analysis of behavior as well as neurobiology and attempts to describe important psychological processes without specifying their exact instantiation in the nervous system – this approach has afforded a detailed analysis of reactions to classes of motivationally significant stimuli and can be used to derive predictions concerning the functions of the real nervous system (e.g., in fMRI studies). Rather than going into extensive detail regarding the biological basis of the systems, we focus primarily on their functions, discussing biological evidence only when it is necessary for addressing some functional question. Approach-Avoidance Theories of Motivation and Their Relation to Personality The most important classes of motivational stimuli can be grouped into ‘rewards’ and ‘punishments’. Animals can be seen as cybernetic systems with attractors and repulsors (positive and negative goals) that have evolved to promote survival and reproduction (Carver & Scheier, 1998; DeYoung, 2010d). Without a tendency to approach beneficial stimuli (e.g., food, drink, and sexual mates) and to avoid aversive stimuli (e.g., predators and poisons) a species would not survive. ‘Reward’ and ‘punishment’ may seem straightforward concepts, but they hide some nonobvious complexities. For the classical behaviorist, rewards increase the frequency of the behavior leading to them, whereas punishments decrease the frequency of behavior leading to them. That is, a ‘reward’ is something a person will work to obtain; and a ‘punishment’ is something a person will work to avoid. But the behaviorist definition of ‘reward’ also includes a different class of stimuli, namely the termination or omission of expected punishment. The effect on behavior and emotion of the ‘hope’ of achieving a reward is similar to that of anticipated ‘relief’ through avoiding a punishment. Similarly, although a ‘punishment’ can be described as something people will work to avoid or escape from (or which they will attack defensively), the omission of an expected reward is experienced as punishing; an effect known as frustrative nonreward. Thus, ‘fear’ has important similarities with ‘frustration’. (For further discussion of this literature, see Corr & McNaughton, 2012.) These complexities can be understood straightforwardly from the cybernetic perspective, in which rewards are any stimuli that indicate progress toward or attainment of a goal, whereas punishments are any stimuli that disrupt progress toward a goal. However, in any experimental situation, it is necessary to confirm that the subject perceives stimuli as actually rewarding and punishing, as there are likely to be significant individual differences in how people react to the same stimuli (for further discussion of this point, see Corr, forthcoming). Motivation and Personality 159 a 2013 Blackwell Publishing Ltd Social and Personality Psychology Compass 7/3 (2013): 158–175, 10.1111/spc3.12016 Current approach-avoidance theories trace their origins to early researchers who posited that two motivation ⁄emotion processes underlie behavior (e.g., Konorski, 1967; Mowrer, 1960; Schneirla, 1959), one related to reward (approach behavior and positive emotions), and the other to punishment (avoidance behavior and negative emotions). Neuroscience measures, including pharmacological manipulation, assessment of neural activity, and neuroanatomical studies, have been used to investigate the neuropsychological systems that underlie reactions to these classes of stimuli, providing confirmation of the hypothesis that distinct systems underlie rewardand punishment-related motivation (Gray & McNaughton, 2000). This animal-based work migrated into personality psychology in the 1970s via Jeffrey A. Gray (e.g., 1970, 1972a,b, 1975, 1977), whose Reinforcement Sensitivity theory (RST) argued that the major traits of personality reflect long-term stabilities in systems that mediate reactions to different classes of reinforcing stimuli, generating emotion and shaping (‘motivating’) approach and avoidance behavior. The leap from understanding motivational systems to understanding personality traits requires the postulate that relatively stable individual differences exist in the operations of these brain-behavioral systems. A personality trait can be defined as a probabilistic constant in equations that predict the frequency and intensity with which individuals exhibit various motivational states, as well as the behavioral, emotional, and cognitive states that accompany these motivational states (DeYoung, 2010c; Fleeson, 2001; Fleeson & Gallagher, 2009). Note that this assumes exposure of the population to a normal range of situations. If situations are limited to prevent exposure to some trait-relevant class of situations, then individual differences in that trait may not be manifest. A neuropsychological approach to personality aims to understand both the biological systems that are responsible for the states associated with any given trait and the parameters of those systems that cause them to differ across individuals. The systems themselves will be present in every intact human brain, but the values of their parameters will vary from person to person. Thus, for example, all people have brain systems that respond to punishing stimuli, but in different individuals these systems respond differently to a given stimulus. It is the typical level of response of such a system in any given individual, averaged across different situations, that is associated with that individual’s score on the personality trait in question. This is not to imply that an individual wil


Question: Understanding the actions that lead to specific physical outcomes: exploring naive action-effect prediction.

This paper presents a novel method for training deep neural networks on noisy labels by integrating a consistency principle through bootstrapping. Traditional deep learning models for visual object recognition and detection rely heavily on supervised training with mechanisms like dropout to prevent overfitting. The efficacy of these models is significantly influenced by the quantity of labeled examples, with an implicit assumption that the labels are precise and unambiguous. However, this assumption often fails in real-world scenarios where class labels might be missing, objects in images may lack precise localization, or labeling can be subjective. To address these issues, we propose an innovative approach that introduces consistency to the prediction objective, ensuring that similar predictions are made for comparable inputs. This consistency is determined by the similarity of features extracted by deep networks from the input data. Our experiments on various datasets demonstrate the robustness of our method against label noise. For instance, on the MNIST dataset of handwritten digits, our model exhibits resilience to label corruption. On the Toronto Face Database, we showcase the model's ability to handle subjective labels in emotion recognition, achieving state-of-the-art results, and we also illustrate how unlabeled face images can be utilized without altering our method. On the ILSVRC2014 detection challenge, we confirm that our approach can be extended to very deep networks, high-resolution images, and structured outputs, leading to enhanced scalable detection capabilities.

People learning new concepts can often generalize successfully from just a single example, yet machine learning algorithms typically require tens or hundreds of examples to perform with similar accuracy. People can also use learned concepts in richer ways than conventional algorithms—for action, imagination, and explanation. We present a computational model that captures these human learning abilities for a large class of simple visual concepts: handwritten characters from the world’s alphabets. The model represents concepts as simple programs that best explain observed examples under a Bayesian criterion. On a challenging one-shot classification task, the model achieves human-level performance while outperforming recent deep learning approaches. We also present several “visual Turing tests” probing the model’s creative generalization abilities, which in many cases are indistinguishable from human behavior.

Much of the user-generated content on social media is provided by ordinary people telling stories about their daily lives. We develop and test a novel method for learning fine-grained common-sense knowledge from these stories about contingent (causal and conditional) relationships between everyday events. This type of knowledge is useful for text and story understanding, information extraction, question answering, and text summarization. We test and compare different methods for learning contingency relation, and compare what is learned from topic-sorted story collections vs. general-domain stories. Our experiments show that using topicspecific datasets enables learning finergrained knowledge about events and results in significant improvement over the baselines. An evaluation on Amazon Mechanical Turk shows 82% of the relations between events that we learn from topic-sorted stories are judged as contingent.

This paper introduces Push-Net, a deep recurrent neural network model, which enables a robot to push objects of unknown physical properties for re-positioning and re-orientation, using only visual camera images as input. The unknown physical properties is a major challenge for pushing. Push-Net overcomes the challenge by tracking a history of push interactions with an LSTM module and training an auxiliary objective function that estimates an object’s center of mass. We trained Push-Net entirely in simulation and tested it extensively on many different objects in both simulation and on two real robots, a Fetch arm and a Kinova MICO arm. Experiments suggest that Push-Net is robust and efficient. It achieved over 97% success rate in simulation on average and succeeded in all real robot experiments with a small number of pushes.

In this paper we first point out a fatal drawback that the widely used Granger causality (GC) needs to estimate the autoregressive model, which is equivalent to taking a series of backward recursive operations which are infeasible in many irreversible chemical reaction models. Thus, new causality (NC) proposed by Hu et al. (2011) is theoretically shown to be more sensitive to reveal true causality than GC. We then apply GC and NC to motor imagery (MI) which is an important mental process in cognitive neuroscience and psychology and has received growing attention for a long time. We study causality flow during MI using scalp electroencephalograms from nine subjects in Brain-computer interface competition IV held in 2008. We are interested in three regions: Cz (central area of the cerebral cortex), C3 (left area of the cerebral cortex), and C4 (right area of the cerebral cortex) which are considered to be optimal locations for recognizing MI states in the literature. Our results show that: 1) there is strong directional connectivity from Cz to C3/C4 during left- and right-hand MIs based on GC and NC; 2) during left-hand MI, there is directional connectivity from C4 to C3 based on GC and NC; 3) during right-hand MI, there is strong directional connectivity from C3 to C4 which is much clearly revealed by NC than by GC, i.e., NC largely improves the classification rate; and 4) NC is demonstrated to be much more sensitive to reveal causal influence between different brain regions than GC.

Verbal communication is a joint activity; however, speech production and comprehension have primarily been analyzed as independent processes within the boundaries of individual brains. Here, we applied fMRI to record brain activity from both speakers and listeners during natural verbal communication. We used the speaker's spatiotemporal brain activity to model listeners' brain activity and found that the speaker's activity is spatially and temporally coupled with the listener's activity. This coupling vanishes when participants fail to communicate. Moreover, though on average the listener's brain activity mirrors the speaker's activity with a delay, we also find areas that exhibit predictive anticipatory responses. We connected the extent of neural coupling to a quantitative measure of story comprehension and find that the greater the anticipatory speaker-listener coupling, the greater the understanding. We argue that the observed alignment of production- and comprehension-based processes serves as a mechanism by which brains convey information.

The mirror neuron system (MNS) has been proposed to play an important role in social cognition by providing a neural mechanism by which others' actions, intentions, and emotions can be understood. Here functional magnetic resonance imaging was used to directly examine the relationship between MNS activity and two distinct indicators of social functioning in typically-developing children (aged 10.1 years+/-7 months): empathy and interpersonal competence. Reliable activity in pars opercularis, the frontal component of the MNS, was elicited by observation and imitation of emotional expressions. Importantly, activity in this region (as well as in the anterior insula and amygdala) was significantly and positively correlated with established behavioral measures indexing children's empathic behavior (during both imitation and observation) and interpersonal skills (during imitation only). These findings suggest that simulation mechanisms and the MNS may indeed be relevant to social functioning in everyday life during typical human development.

In literary and drama criticism, emotions, and moral emotions in particular, have been pointed out as one of characterizing features of stories. In this paper, we propose to model story characters as value-based emotional agents, who appraise their own and others’ actions based on their desires and values, and feel the appropriate moral emotions in response to narrative situations that challenge their goals and values. In order to validate the appropriateness of the agent model for narrative characters, we ran an experiment with human participants aimed at comparing their expectations about characters’ emotions with the predictions of the value-based model of emotional agent. The results of the experiment show that the participants’ expectations meet the predictions of the model. 1998 ACM Subject Classification I.2.11 Intelligent Agents, I.2.0 Cognitive Simulation

As described in the main paper, we collected a dataset D = (x(n), e(n), a(n), π(n))500 n=1. Environment Complexity. Our environments are 3D scenarios consisting of complex objects such as fridge, microwave and television with many states. These objects can be in different spatial relations with respect to other objects. For example, “bag of chips” can be found behind the television. Figure 1 shows some sample environments from our dataset. For example, an object of category television consists of 6 channels, volume level and power status. An object can have different values of states in different environment and different environment consists of different set of objects and their placement. For example, television might be powered on in one environment and closed in another, microwave might have an object inside it or not in different environment, etc. Moreover, there are often more than one object of the same category. For example, our environment typically have two books, five couches, four pillows etc. Objects of the same category can have different appearance. For example, a book can have the cover of a math book or a Guinness book of world record; resulting in complex object descriptions such as in “throw the sticky stuff in the bowl”. They can also have the same appearance making people use spatial relations or other means while describing them such as in “get me the cup next to the microwave”. This dataset is significantly more challenging compared to the 2D navigation dataset or GUI actions in windows dataset considered earlier.

In this paper we propose a theoretical framework of distributed representations and a methodology of representational analysis for the study of distributed cognitive tasksÑtasks that require the processing of information distributed across the internal mind and the external environment. The basic principle of distributed representations is that the representational system of a distributed cognitive task is a set of internal and external representations, which together represent the abstract structure of the task. The basic strategy of representational analysis is to decompose the representation of a hierarchical task into its component levels so that the representational properties at each level can be independently examined. The theoretical framework and the methodology are used to analyze the hierarchical structure of the Tower of Hanoi problem. Based on this analysis, four experiments are designed to examine the representational properties of the Tower of Hanoi. Finally, the nature of external representations is discussed. This research was supported by a grant to Donald Norman and Edwin Hutchins from the Ames Research Center of the National Aeronautics & Space Agency, Grant NCC 2-591 in the Aviation Safety/Automation Program, technical monitor, Everett Palmer. Additional support was provided by funds from the Apple Computer Company and the Digital Equipment Corporation to the Affiliates of Cognitive Science at

Temporal language is often couched in spatial metaphors. English has been claimed to have two space-time metaphoric systems: the ego-moving metaphor, wherein the observer’s context progresses along the time-line towards the future, and the time-moving metaphor, wherein time is conceived of as a river or conveyor belt on which events are moving from the future to the past. In three experiments, we investigated the psychological status of these metaphors by asking subjects to carry out temporal inferences stated in terms of spatial metaphors. In Experiment 1, we found that subjects were slowed in their processing when the assertions shifted from one spatial metaphoric system to the other. In Experiment 2, we determined that this cost of shifting could not be attributed to local lexical factors. In Experiment 3, we again found this metaphor consistency effect in a naturalistic version of the study in which we asked commonsense time questions of passengers at an airport. The results of the three studies provide converging evidence that people use spatial metaphors in temporal reasoning. Implications for the status of metaphoric systems arc discussed.


Question: What action leads to this? Investigating basic physical action-outcome forecasting.

YAGO: A Semantic Knowledge Core and Its Applications

This paper discusses YAGO, a significant semantic knowledge core that has been developed to represent a vast array of facts about the world in a structured manner. YAGO, which stands for "Yet Another Great Ontology," is not merely a database but a sophisticated knowledge representation system that integrates information from various sources such as Wikipedia, WordNet, and GeoNames. Unlike other knowledge bases, YAGO goes beyond simply storing facts; it employs a semantic model that allows for the inference of additional knowledge based on the relationships between entities. This capability makes YAGO particularly useful for tasks that require a deep understanding of the relationships between concepts, such as question answering, information retrieval, and natural language processing. YAGO's design emphasizes the importance of context in interpreting facts, making it a valuable resource for applications that need to make sense of complex and nuanced information. Furthermore, YAGO's ability to infer knowledge from its core of facts enables it to provide more detailed and accurate answers to queries than would be possible with a simple fact database. The paper also discusses several applications of YAGO, demonstrating its versatility and the potential it holds for advancing the field of artificial intelligence.

The simple, but general formal theory of fun and intrinsic motivation and creativity (1990-2010) is based on the concept of maximizing intrinsic reward for the active creation or discovery of novel, surprising patterns allowing for improved prediction or data compression. It generalizes the traditional field of active learning, and is related to old, but less formal ideas in aesthetics theory and developmental psychology. It has been argued that the theory explains many essential aspects of intelligence including autonomous development, science, art, music, and humor. This overview first describes theoretically optimal (but not necessarily practical) ways of implementing the basic computational principles on exploratory, intrinsically motivated agents or robots, encouraging them to provoke event sequences exhibiting previously unknown, but learnable algorithmic regularities. Emphasis is put on the importance of limited computational resources for online prediction and compression. Discrete and continuous time formulations are given. Previous practical, but nonoptimal implementations (1991, 1995, and 1997-2002) are reviewed, as well as several recent variants by others (2005-2010). A simplified typology addresses current confusion concerning the precise nature of intrinsic motivation.

Human automation interaction (HAI) systems have thus far failed to live up to expectations mainly because human users do not always interact with the automation appropriately. Trust in automation (TiA) has been considered a central influence on the way a human user interacts with an automation; if TiA is too high there will be overuse, if TiA is too low there will be disuse. However, even though extensive research into TiA has identified specific HAI behaviors, or trust outcomes, a unique mapping between trust states and trust outcomes has yet to be clearly identified. Interaction behaviors have been intensely studied in the domain of HAI and TiA and this has led to a reframing of the issues of problems with HAI in terms of reliance and compliance. We find the behaviorally defined terms reliance and compliance to be useful in their functionality for application in real-world situations. However, we note that once an inappropriate interaction behavior has occurred it is too late to mitigate it. We therefore take a step back and look at the interaction decision that precedes the behavior. We note that the decision neuroscience community has revealed that decisions are fairly stereotyped processes accompanied by measurable psychophysiological correlates. Two literatures were therefore reviewed. TiA literature was extensively reviewed in order to understand the relationship between TiA and trust outcomes, as well as to identify gaps in current knowledge. We note that an interaction decision precedes an interaction behavior and believe that we can leverage knowledge of the psychophysiological correlates of decisions to improve joint system performance. As we believe that understanding the interaction decision will be critical to the eventual mitigation of inappropriate interaction behavior, we reviewed the decision making literature and provide a synopsis of the state of the art understanding of the decision process from a decision neuroscience perspective. We forward hypotheses based on this understanding that could shape a research path toward the ability to mitigate interaction behavior in the real world.

We investigated whether people can use analog imagery to model the behavior of a simple mechanical interaction. Subjects saw a static computer display of two touching gears that had different diameters. Their task was to determine whether marks on each gear would meet if the gears rotated inward. This task added a problem of coordination to the typical analog rotation task in that the gears had a physical interdependency; the angular velocity of one gear depended on the angular velocity of the other gear. In the first experiment, we found the linear relationship between response time and angular disparity that indicates analog imagery. In the second experiment, we found that people can also solve the problem through a non-analog, visual comparison. We also found that people of varying spatial ability could switch between analog and non-analog solutions if instructed to do so. In the third experiment, we examined whether the elicitation of physical knowledge would influence solution strategies. To do so, we manipulated the visual realism of the gear display. Subjects who saw the most realistic gears coordinated their transformations by using the surfaces of the gears, as though they were relying on the friction connecting the surfaces. Subjects who saw more schematic displays relied on analytic strategies, such as comparing the ratios made by the angles and/or diameters of the two gears. To explain the relationship between spatial and physical knowledge found in the experiments, we constructed a computer simulation of what we call depictive modeling. In a depictive model, general spatial knowledge and context-sensitive physical knowledge have the same ontology. This is different from prior simulations in which a non-analog representation would be needed to coordinate the analog behaviors of physical objects. In our simulation, the inference that coordinates the gear motions emerges from the analog rotations themselves. We suggest that mental depictions create a bridge between imagery and mental model research by positing the referent as the primary conceptual entity.

Advanced Driver Assistance Systems (ADAS) have made driving safer over the last decade. They prepare vehicles for unsafe road conditions and alert drivers if they perform a dangerous maneuver. However, many accidents are unavoidable because by the time drivers are alerted, it is already too late. Anticipating maneuvers beforehand can alert drivers before they perform the maneuver and also give ADAS more time to avoid or prepare for the danger. In this work we anticipate driving maneuvers a few seconds before they occur. For this purpose we equip a car with cameras and a computing device to capture the driving context from both inside and outside of the car. We propose an Autoregressive Input-Output HMM to model the contextual information alongwith the maneuvers. We evaluate our approach on a diverse data set with 1180 miles of natural freeway and city driving and show that we can anticipate maneuvers 3.5 seconds before they occur with over 80% F1-score in real-time.

Temporal language is often couched in spatial metaphors. English has been claimed to have two space-time metaphoric systems: the ego-moving metaphor, wherein the observer’s context progresses along the time-line towards the future, and the time-moving metaphor, wherein time is conceived of as a river or conveyor belt on which events are moving from the future to the past. In three experiments, we investigated the psychological status of these metaphors by asking subjects to carry out temporal inferences stated in terms of spatial metaphors. In Experiment 1, we found that subjects were slowed in their processing when the assertions shifted from one spatial metaphoric system to the other. In Experiment 2, we determined that this cost of shifting could not be attributed to local lexical factors. In Experiment 3, we again found this metaphor consistency effect in a naturalistic version of the study in which we asked commonsense time questions of passengers at an airport. The results of the three studies provide converging evidence that people use spatial metaphors in temporal reasoning. Implications for the status of metaphoric systems arc discussed.

Open domain causal learning involves learning and establishing causal connections between events directly from sensory experiences. It has been established in psychology that this often requires background knowledge. However, background knowledge has to be built from first experiences, which we term ground level causal learning, which basically involves observing temporal correlations. Subsequent knowledge level causal learning can then be based on this ground level causal knowledge. The causal connections between events, such as between lightning and thunder, are often hard to discern based on simple temporal correlations because there might be noise - e.g., wind, headlights, sounds of vehicles, etc. - that intervene between lightning and thunder. In this paper, we adopt the position that causal learning is inductive and pragmatic, and causal connections exist on a scale of graded strength. We describe a method that is able to filter away noise in the environment to obtain likely causal connections between events.


Question: Advancing 3D face recognition in real-world scenarios: A registration-free method employing fine-grained matching of 3D keypoint descriptors.

This paper presents a novel 3D face recognition system that leverages an ensemble of independently matched regions for enhanced accuracy. Through extensive experimentation, we find that utilizing 28 small, distinct regions on the face leads to optimal 3D face recognition performance. Our system employs score-based fusion techniques to combine the matching scores from these individual regions. Notably, the Borda count and consensus voting methods outperform traditional fusion rules such as sum, product, and minimum. Moreover, our algorithm's resilience is showcased through simulations involving significant image distortions and artifacts. To the best of our knowledge, this is the first study to utilize a substantial number of 3D face regions for high-performance matching. Our system achieves a rank one recognition rate of 97.2% and a verification rate of 93.2% at a 0.1% false accept rate, outperforming other published methods on the Face Recognition Grand Challenge v2 data set.

We study the question of feature sets for robust visual object recognition; adopting linear SVM based human detection as a test case. After reviewing existing edge and gradient based descriptors, we show experimentally that grids of histograms of oriented gradient (HOG) descriptors significantly outperform existing feature sets for human detection. We study the influence of each stage of the computation on performance, concluding that fine-scale gradients, fine orientation binning, relatively coarse spatial binning, and high-quality local contrast normalization in overlapping descriptor blocks are all important for good results. The new approach gives near-perfect separation on the original MIT pedestrian database, so we introduce a more challenging dataset containing over 1800 annotated human images with a large range of pose variations and backgrounds.

A novel minutiae-based fingerprint matching algorithm is proposed. A minutiae matching algorithm has to solve two problems: correspondence and similarity computation. For the correspondence problem, we assign each minutia two descriptors: texture-based and minutiae-based descriptors, and use an alignment-based greedy matching algorithm to establish the correspondences between minutiae. For the similarity computation, we extract a 17-D feature vector from the matching result, and convert the feature vector into a matching score using support vector classifier. The proposed algorithm is tested on FVC2002 databases and compared to all participators in FVC2002. According to equal error rate, the proposed algorithm ranks 1st on DB3, the most difficult database in FVC2002, and on the average ranks 2nd on all 4 databases. 2007 Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved.

Fast and reliable algorithms for estimating the head pose are essential for many applications and higher-level face analysis tasks. We address the problem of head pose estimation from depth data, which can be captured using the ever more affordable 3D sensing technologies available today. To achieve robustness, we formulate pose estimation as a regression problem. While detecting specific face parts like the nose is sensitive to occlusions, learning the regression on rather generic surface patches requires enormous amount of training data in order to achieve accurate estimates. We propose to use random regression forests for the task at hand, given their capability to handle large training datasets. Moreover, we synthesize a great amount of annotated training data using a statistical model of the human face. In our experiments, we show that our approach can handle real data presenting large pose changes, partial occlusions, and facial expressions, even though it is trained only on synthetic neutral face data. We have thoroughly evaluated our system on a publicly available database on which we achieve state-of-the-art performance without having to resort to the graphics card.

Binary descriptors are becoming increasingly popular as a means to compare feature points very fast while requiring comparatively small amounts of memory. The typical approach to creating them is to first compute floating-point ones, using an algorithm such as SIFT, and then to binarize them. In this paper, we show that we can directly compute a binary descriptor, which we call BRIEF, on the basis of simple intensity difference tests. As a result, BRIEF is very fast both to build and to match. We compare it against SURF and SIFT on standard benchmarks and show that it yields comparable recognition accuracy, while running in an almost vanishing fraction of the time required by either.

The accuracy of automated human face recognition algorithms can significantly degrade while recognizing same subjects under make-up and disguised appearances. Increasing constraints on enhanced security and surveillance requires enhanced accuracy from face recognition algorithms for faces under disguise and/or makeup. This paper presents a new database for face images under disguised and make-up appearances the development of face recognition algorithms under such covariates. This database has 2460 images from 410 different subjects and is acquired under real environment, focuses on make-up and disguises covariates and also provides ground truth (eye glass, goggle, mustache, beard) for every image. This can enable developed algorithms to automatically quantify their capability for identifying such important disguise attribute during the face recognition We also present comparative experimental results from two popular commercial matchers and from recent publications. Our experimental results suggest significant performance degradation in the capability of these matchers in automatically recognizing these faces. We also analyze face detection accuracy from these matchers. The experimental results underline the challenges in recognizing faces under these covariates. Availability of this new database in public domain will help to advance much needed research and development in recognizing make-up and disguised faces.

We present an efficient method of fitting a set of local feature models to an image within the popular Active Shape Model (ASM) framework [3]. We compare two different types of non-linear boosted feature models trained using GentleBoost [9]. The first type is a conventional feature detector classifier, which learns a discrimination function between the appearance of a feature and the local neighbourhood. The second local model type is a boosted regression predictor which learns the relationship between the local neighbourhood appearance and the displacement from the true feature location. At run-time the second regression model is much more efficient as only the current feature patch needs to be processed. We show that within the local iterative search of the ASM the local feature regression provides improved localisation on two publicly available human face test sets as well as increasing the search speed by a factor of eight.

A new transform is presented that utilizes local radial symmetry to highlight points of interest within a scene. Its lowcomputational complexity and fast runtimes makes this method well-suited for real-time vision applications. The performance of the transform is demonstrated on a wide variety of images and compared with leading techniques from the literature. Both as a facial feature detector and as a generic region of interest detector the new transform is seen to offer equal or superior performance to contemporary techniques at a relatively low-computational cost. A real-time implementation of the transform is presented running at over 60 frames per second on a standard Pentium III PC.

In this paper, weinvestigaterecognition of humanfaces in a meetingroom. The major challenges of identifying humanfacesin this environmentincludelow quality of input images,poor illumination,unrestrictedheadposesand continuouslychangingfacial expressionsandocclusion.In order to addresstheseproblemswe proposea novel algorithm, DynamicSpaceWarping (DSW).Thebasic idea of the algorithm is to combinelocal features under certain spatial constraints. We compare DSWwith the eigenface approachondatacollectedfromvariousmeetings.Wehave testedboth front and profile face imagesand imageswith two stagesof occlusion.Theexperimentalresultsindicate thattheDSWapproachoutperformstheeigenfaceapproach in bothcases.

Human pose estimation in real-time is a challenging problem in computer vision. In this paper, we present a novel approach to recover a 3D human pose in real-time from a single depth human silhouette using Principal Direction Analysis (PDA) on each recognized body part. In our work, the human body parts are first recognized from a depth human body silhouette via the trained Random Forests (RFs). On each recognized body part which is presented as a set of 3D points cloud, PDA is applied to estimate the principal direction of the body part. Finally, a 3D human pose gets recovered by mapping the principal directional vector to each body part of a 3D human body model which is created with a set of super-quadrics linked by the kinematic chains. In our experiments, we have performed quantitative and qualitative evaluations of the proposed 3D human pose reconstruction methodology. Our evaluation results show that the proposed approach performs reliably on a sequence of unconstrained poses and achieves an average reconstruction error of 7.46 degree in a few key joint angles. Our 3D pose recovery methodology should be applicable to many areas such as human computer interactions and human activity recognition.

In this paper we propose a new method for detecting multiple specific 3D objects in real time. We start from the template-based approach based on the LINE2D/LINEMOD representation introduced recently by Hinterstoisser et al., yet extend it in two ways. First, we propose to learn the templates in a discriminative fashion. We show that this can be done online during the collection of the example images, in just a few milliseconds, and has a big impact on the accuracy of the detector. Second, we propose a scheme based on cascades that speeds up detection. Since detection of an object is fast, new objects can be added with very low cost, making our approach scale well. In our experiments, we easily handle 10-30 3D objects at frame rates above 10fps using a single CPU core. We outperform the state-of-the-art both in terms of speed as well as in terms of accuracy, as validated on 3 different datasets. This holds both when using monocular color images (with LINE2D) and when using RGBD images (with LINEMOD). Moreover, we propose a challenging new dataset made of 12 objects, for future competing methods on monocular color images.


Question: Advancing 3D face recognition in real-world scenarios: A registration-free method employing fine-grained matching of 3D keypoint descriptors.

This comprehensive review article offers an updated examination of the field of face recognition, emphasizing advancements and challenges in both still and video-based recognition systems. The heightened interest in this domain can be attributed to two primary factors: the burgeoning demand for practical applications in commerce and law enforcement, and the recent accessibility of viable technologies following decades of extensive research. Despite the notable progress, contemporary machine recognition systems face significant limitations when confronted with the real-world conditions often encountered in practical applications. A notable example is the challenge of accurately recognizing faces in outdoor settings, where variations in illumination and pose significantly complicate the recognition process. This paper endeavors to address these issues by providing a critical survey of the latest research in face recognition. Two main objectives drive the compilation of this review: first, to offer an updated analysis of the existing literature, and second, to contribute insights into the ongoing studies of automated face recognition. To achieve a thorough review, we not only classify the various recognition techniques but also provide detailed explanations of the methodologies that exemplify each category. Furthermore, the review encompasses supplementary topics such as psychophysical studies, system evaluation criteria, and strategies for managing illumination and pose variations, which are pivotal to the advancement of face recognition technology.

We present an efficient method of fitting a set of local feature models to an image within the popular Active Shape Model (ASM) framework [3]. We compare two different types of non-linear boosted feature models trained using GentleBoost [9]. The first type is a conventional feature detector classifier, which learns a discrimination function between the appearance of a feature and the local neighbourhood. The second local model type is a boosted regression predictor which learns the relationship between the local neighbourhood appearance and the displacement from the true feature location. At run-time the second regression model is much more efficient as only the current feature patch needs to be processed. We show that within the local iterative search of the ASM the local feature regression provides improved localisation on two publicly available human face test sets as well as increasing the search speed by a factor of eight.

Matching people across nonoverlapping camera views at different locations and different times, known as person reidentification, is both a hard and important problem for associating behavior of people observed in a large distributed space over a prolonged period of time. Person reidentification is fundamentally challenging because of the large visual appearance changes caused by variations in view angle, lighting, background clutter, and occlusion. To address these challenges, most previous approaches aim to model and extract distinctive and reliable visual features. However, seeking an optimal and robust similarity measure that quantifies a wide range of features against realistic viewing conditions from a distance is still an open and unsolved problem for person reidentification. In this paper, we formulate person reidentification as a relative distance comparison (RDC) learning problem in order to learn the optimal similarity measure between a pair of person images. This approach avoids treating all features indiscriminately and does not assume the existence of some universally distinctive and reliable features. To that end, a novel relative distance comparison model is introduced. The model is formulated to maximize the likelihood of a pair of true matches having a relatively smaller distance than that of a wrong match pair in a soft discriminant manner. Moreover, in order to maintain the tractability of the model in large scale learning, we further develop an ensemble RDC model. Extensive experiments on three publicly available benchmarking datasets are carried out to demonstrate the clear superiority of the proposed RDC models over related popular person reidentification techniques. The results also show that the new RDC models are more robust against visual appearance changes and less susceptible to model overfitting compared to other related existing models.

In this paper, we describe the acquisition and contents of a large-scale Chinese face database: the CAS-PEAL face database. The goals of creating the CAS-PEAL face database include the following: 1) providing the worldwide researchers of face recognition with different sources of variations, particularly pose, expression, accessories, and lighting (PEAL), and exhaustive ground-truth information in one uniform database; 2) advancing the state-of-the-art face recognition technologies aiming at practical applications by using off-the-shelf imaging equipment and by designing normal face variations in the database; and 3) providing a large-scale face database of Mongolian. Currently, the CAS-PEAL face database contains 99 594 images of 1040 individuals (595 males and 445 females). A total of nine cameras are mounted horizontally on an arc arm to simultaneously capture images across different poses. Each subject is asked to look straight ahead, up, and down to obtain 27 images in three shots. Five facial expressions, six accessories, and 15 lighting changes are also included in the database. A selected subset of the database (CAS-PEAL-R1, containing 30 863 images of the 1040 subjects) is available to other researchers now. We discuss the evaluation protocol based on the CAS-PEAL-R1 database and present the performance of four algorithms as a baseline to do the following: 1) elementarily assess the difficulty of the database for face recognition algorithms; 2) preference evaluation results for researchers using the database; and 3) identify the strengths and weaknesses of the commonly used algorithms.

We propose a new framework for image recognition by selectively pooling local visual descriptors, and show its superior discriminative power on fine-grained image classification tasks. The representation is based on selecting the most confident local descriptors for nonlinear function learning using a linear approximation in an embedded higher dimensional space. The advantage of our Selective Pooling Vector over the previous state-of-the-art Super Vector and Fisher Vector representations, is that it ensures a more accurate learning function, which proves to be important for classifying details in fine-grained image recognition. Our experimental results corroborate this claim: with a simple linear SVM as the classifier, the selective pooling vector achieves significant performance gains on standard benchmark datasets for various fine-grained tasks such as the CMU Multi-PIE dataset for face recognition, the Caltech-UCSD Bird dataset and the Stanford Dogs dataset for fine-grained object categorization. On all datasets we outperform the state of the arts and boost the recognition rates to 96.4%, 48.9%, 52.0% respectively.

Face anti-spoofing plays a crucial role in protecting face recognition systems from various attacks. Previous modelbased and deep learning approaches achieve satisfactory performance for 2D face spoofs, but remain limited for more advanced 3D attacks such as vivid masks. In this paper, we address 3D face anti-spoofing via the proposed Hypergraph Convolutional Neural Networks (HGCNN). Firstly, we construct a computation-efficient and posture-invariant face representation with only a few key points on hypergraphs. The hypergraph representation is then fed into the designed HGCNN with hypergraph convolution for feature extraction, while the depth auxiliary is also exploited for 3D mask anti-spoofing. Further, we build a 3D face attack database with color, depth and infrared light information to overcome the deficiency of 3D face anti-spoofing data. Experiments show that our method achieves the state-of-theart performance over widely used 3D and 2D databases as well as the proposed one under various tests.

This work addresses the challenging problem of unconstrained 3D human pose estimation (HPE) from a novel perspective. Existing approaches struggle to operate in realistic applications, mainly due to their scene-dependent priors, such as background segmentation and multi-camera network, which restrict their use in unconstrained environments. We therfore present a framework which applies action detection and 2D pose estimation techniques to infer 3D poses in an unconstrained video. Action detection offers spatiotemporal priors to 3D human pose estimation by both recognising and localising actions in space-time. Instead of holistic features, e.g. silhouettes, we leverage the flexibility of deformable part model to detect 2D body parts as a feature to estimate 3D poses. A new unconstrained pose dataset has been collected to justify the feasibility of our method, which demonstrated promising results, significantly outperforming the relevant state-of-the-arts.

Person re-identification is about recognizing people who have passed by a sensor earlier. Previous work is mainly based on RGB data, but in this work we for the first time present a system where we combine RGB, depth, and thermal data for re-identification purposes. First, from each of the three modalities, we obtain some particular features: from RGB data, we model color information from different regions of the body, from depth data, we compute different soft body biometrics, and from thermal data, we extract local structural information. Then, the three information types are combined in a joined classifier. The tri-modal system is evaluated on a new RGB-D-T dataset, showing successful results in re-identification scenarios.

Morphometric measures and geometrical features are widely used to descript faces. Generally, they are extracted punctually from landmarks, namely anthropometric reference points. The aims are various, such as face recognition, facial expression recognition, face detection, study of changes in facial morphology due to growth, or dysmorphologies. Most of the time, landmarks were extracted with the help of an algorithm or manually located on the faces. Then, measures are computed or geometrical features are extracted to perform the scope of the study. This paper is intended as a survey collecting and explaining all these features, in order to provide a structured user database of the potential parameters and their charateristics. Firstly, facial soft-tissue landmarks are defined and contextualized; then the various morphometric measures are introduced and some results are given; lastly, the most important measures are compared to identify the best one for face recognition applications.

This paper describes a depth image based real-time skeleton fitting algorithm for the hand, using an object recognition by parts approach, and the use of this hand modeler in an American Sign Language (ASL) digit recognition application. In particular, we created a realistic 3D hand model that represents the hand with 21 different parts. Random decision forests (RDF) are trained on synthetic depth images generated by animating the hand model, which are then used to perform per pixel classification and assign each pixel to a hand part. The classification results are fed into a local mode finding algorithm to estimate the joint locations for the hand skeleton. The system can process depth images retrieved from Kinect in real-time at 30 fps. As an application of the system, we also describe a support vector machine (SVM) based recognition module for the ten digits of ASL based on our method, which attains a recognition rate of 99.9% on live depth images in real-time1.

We present an approach to nonrigid registration of 3D surfaces. We cast isometric embedding as MRF optimization and apply efficient global optimization algorithms based on linear programming relaxations. The Markov random field perspective suggests a natural connection with robust statistics and motivates robust forms of the intrinsic distortion functional. Our approach outperforms a large body of prior work by a significant margin, increasing registration precision on real data by a factor of 3.


Question: Advancing 3D face recognition in real-world scenarios: A registration-free method employing fine-grained matching of 3D keypoint descriptors.

A Novel 3D Face Recognition System: Combating Nonrigid Deformations and Pose Variations through Deformable Modeling

The utilization of 3D surface matching in face recognition offers a significant advantage over traditional 2D image-based systems, particularly in addressing issues related to pose and lighting changes. However, the 3D shape is not immune to nonrigid facial movements such as expressions. Storing multiple templates for various expressions for each individual in a large database is impractical. To address this challenge, we introduce a facial surface modeling and matching technique that effectively matches 2.5D facial scans under conditions of both nonrigid deformations and pose variations (multiview) to a stored 3D face model with a neutral expression.

Our method involves a hierarchical geodesic-based resampling strategy to extract landmarks for modeling facial surface deformations. By learning deformations from a small group of subjects (the control group), we can synthesize these deformations onto a 3D neutral model (not included in the control group), resulting in a deformed template. A personalized 3D deformable model is constructed for each subject in the gallery relative to the control group, integrating the templates with synthesized deformations.

The proposed approach enables the handling of expressions and pose changes concurrently by fitting this generative deformable model to a test scan. We have developed a fully automated prototype system for 3D face matching based on deformable modeling. Experimental outcomes on a subset of the FRGC v2.0 3D benchmark and the MSU multiview 3D face database, both featuring expression variations, illustrate that our deformation modeling technique significantly boosts 3D face matching accuracy by 7 and 10 percentage points, respectively, compared to matching with 3D neutral models. This underscores the effectiveness of our system in overcoming the limitations posed by nonrigid deformations and pose variations in 3D face recognition.

Matching people across nonoverlapping camera views at different locations and different times, known as person reidentification, is both a hard and important problem for associating behavior of people observed in a large distributed space over a prolonged period of time. Person reidentification is fundamentally challenging because of the large visual appearance changes caused by variations in view angle, lighting, background clutter, and occlusion. To address these challenges, most previous approaches aim to model and extract distinctive and reliable visual features. However, seeking an optimal and robust similarity measure that quantifies a wide range of features against realistic viewing conditions from a distance is still an open and unsolved problem for person reidentification. In this paper, we formulate person reidentification as a relative distance comparison (RDC) learning problem in order to learn the optimal similarity measure between a pair of person images. This approach avoids treating all features indiscriminately and does not assume the existence of some universally distinctive and reliable features. To that end, a novel relative distance comparison model is introduced. The model is formulated to maximize the likelihood of a pair of true matches having a relatively smaller distance than that of a wrong match pair in a soft discriminant manner. Moreover, in order to maintain the tractability of the model in large scale learning, we further develop an ensemble RDC model. Extensive experiments on three publicly available benchmarking datasets are carried out to demonstrate the clear superiority of the proposed RDC models over related popular person reidentification techniques. The results also show that the new RDC models are more robust against visual appearance changes and less susceptible to model overfitting compared to other related existing models.

This paper concerns the problem of facial landmark detection. We provide a unique new analysis of the features produced at intermediate layers of a convolutional neural network (CNN) trained to regress facial landmark coordinates. This analysis shows that while being processed by the CNN, face images can be partitioned in an unsupervised manner into subsets containing faces in similar poses (i.e., 3D views) and facial properties (e.g., presence or absence of eye-wear). Based on this finding, we describe a novel CNN architecture, specialized to regress the facial landmark coordinates of faces in specific poses and appearances. To address the shortage of training data, particularly in extreme profile poses, we additionally present data augmentation techniques designed to provide sufficient training examples for each of these specialized sub-networks. The proposed Tweaked CNN (TCNN) architecture is shown to outperform existing landmark detection methods in an extensive battery of tests on the AFW, ALFW, and 300W benchmarks. Finally, to promote reproducibility of our results, we make code and trained models publicly available through our project webpage.

Markerless Augmented Reality (AR) registration using the standard Homography matrix is unstable, and for image-based registration it has very low accuracy. In this paper, we present a new method to improve the stability and the accuracy of marker-less registration in AR. Based on the Visual Simultaneous Localization and Mapping (V-SLAM) framework, our method adds a three-dimensional dense cloud processing step to the state-of-the-art ORB-SLAM in order to deal with mainly the point cloud fusion and the object recognition. Our algorithm for the object recognition process acts as a stabilizer to improve the registration accuracy during the model to the scene transformation process. This has been achieved by integrating the Hough voting algorithm with the Iterative Closest Points(ICP) method. Our proposed AR framework also further increases the registration accuracy with the use of integrated camera poses on the registration of virtual objects. Our experiments show that the proposed method not only accelerates the speed of camera tracking with a standard SLAM system, but also effectively identifies objects and improves the stability of markerless augmented reality applications.

Probabilistic point-set registration methods have been gaining more attention for their robustness to noise, outliers and occlusions. However, these methods tend to be much slower than the popular iterative closest point (ICP) algorithms, which severely limits their usability. In this paper, we contribute a novel probabilistic registration method that achieves state-of-the-art robustness as well as substantially faster computational performance than modern ICP implementations. This is achieved using a rigorous yet computationally-efficient probabilistic formulation. Pointset registration is cast as a maximum likelihood estimation and solved using the EM algorithm. We show that with a simple augmentation, the E step can be formulated as a filtering problem, allowing us to leverage advances in efficient Gaussian filtering methods. We also propose a customized permutohedral filter [1] for improved efficiency while retaining sufficient accuracy for our task. Additionally, we present a simple and efficient twist parameterization that generalizes our method to the registration of articulated and deformable objects. For articulated objects, the complexity of our method is almost independent of the Degrees Of Freedom (DOFs), which makes it highly efficient even for high DOF systems. The results demonstrate the proposed method consistently outperforms many competitive baselines on a variety of registration tasks. The video demo and source code are available on our project page.

Recognition of human beings using shapes of their full facial surfaces is a difficult problem. Our approach is to approximate a facial surface using a collection of (closed) facial curves, and to compare surfaces by comparing their corresponding curves. The differences between shapes of curves are quantified using lengths of geodesic paths between them on a pre-defined curve shape space. The metric for comparing facial surfaces is a composition of the metric involving individual facial curves. These ideas are demonstrated in the context of face recognition using the nearest-neighbor classifier

In this paper, we present a novel approach for human action recognition with histograms of 3D joint locations (HOJ3D) as a compact representation of postures. We extract the 3D skeletal joint locations from Kinect depth maps using Shotton et al.'s method [6]. The HOJ3D computed from the action depth sequences are reprojected using LDA and then clustered into k posture visual words, which represent the prototypical poses of actions. The temporal evolutions of those visual words are modeled by discrete hidden Markov models (HMMs). In addition, due to the design of our spherical coordinate system and the robust 3D skeleton estimation from Kinect, our method demonstrates significant view invariance on our 3D action dataset. Our dataset is composed of 200 3D sequences of 10 indoor activities performed by 10 individuals in varied views. Our method is real-time and achieves superior results on the challenging 3D action dataset. We also tested our algorithm on the MSR Action 3D dataset and our algorithm outperforms Li et al. [25] on most of the cases.

Non-rigid registration of 3D shapes is an essential task of increasing importance as commodity depth sensors become more widely available for scanning dynamic scenes. Non-rigid registration is much more challenging than rigid registration as it estimates a set of local transformations instead of a single global transformation, and hence is prone to the overfitting issue due to underdetermination. The common wisdom in previous methods is to impose an ℓ2-norm regularization on the local transformation differences. However, the ℓ2-norm regularization tends to bias the solution towards outliers and noise with heavy-tailed distribution, which is verified by the poor goodness-of-fit of the Gaussian distribution over transformation differences. On the contrary, Laplacian distribution fits well with the transformation differences, suggesting the use of a sparsity prior. We propose a sparse non-rigid registration (SNR) method with an ℓ1-norm regularized model for transformation estimation, which is effectively solved by an alternate direction method (ADM) under the augmented Lagrangian framework. We also devise a multi-resolution scheme for robust and progressive registration. Results on both public datasets and our scanned datasets show the superiority of our method, particularly in handling large-scale deformations as well as outliers and noise.

We propose a modeland exemplar-based approach for face recognition. This problem has been previously tackled using either models or exemplars, with limited success. Our idea uses models to synthesize many more exemplars, which are then used in the learning stage of a face recognition system. To demonstrate this, we develop a statistical shape-fromshading model to recover face shape from a single image, and to synthesize the same face under new illumination. We then use this to build a simple and fast classifier that was not possible before because of a lack of training data.

This paper aims to present a review of recent as well as classic image registration methods. Image registration is the process of overlaying images (two or more) of the same scene taken at different times, from different viewpoints, and/or by different sensors. The registration geometrically align two images (the reference and sensed images). The reviewed approaches are classified according to their nature (areabased and feature-based) and according to four basic steps of image registration procedure: feature detection, feature matching, mapping function design, and image transformation and resampling. Main contributions, advantages, and drawbacks of the methods are mentioned in the paper. Problematic issues of image registration and outlook for the future research are discussed too. The major goal of the paper is to provide a comprehensive reference source for the researchers involved in image registration, regardless of particular application areas. q 2003 Elsevier B.V. All rights reserved.

In this paper, we propose two novel methods for face recognition under arbitrary unknown lighting by using spherical harmonics illumination representation, which require only one training image per subject and no 3D shape information. Our methods are based on the result which demonstrated that the set of images of a convex Lambertian object obtained under a wide variety of lighting conditions can be approximated accurately by a low-dimensional linear subspace. We provide two methods to estimate the spherical harmonic basis images spanning this space from just one image. Our first method builds the statistical model based on a collection of 2D basis images. We demonstrate that, by using the learned statistics, we can estimate the spherical harmonic basis images from just one image taken under arbitrary illumination conditions if there is no pose variation. Compared to the first method, the second method builds the statistical models directly in 3D spaces by combining the spherical harmonic illumination representation and a 3D morphable model of human faces to recover basis images from images across both poses and illuminations. After estimating the basis images, we use the same recognition scheme for both methods: we recognize the face for which there exists a weighted combination of basis images that is the closest to the test face image. We provide a series of experiments that achieve high recognition rates, under a wide range of illumination conditions, including multiple sources of illumination. Our methods achieve comparable levels of accuracy with methods that have much more onerous training data requirements. Comparison of the two methods is also provided.


Question: Advancing 3D Face Recognition in Reality: A Keypoint Descriptor Matching Method Without Registration, Focusing on Fine-Grained 3D Details.

This paper presents an innovative technique for the registration of 3D shapes, offering a significant advancement in the field of 3D object recognition. The proposed method, which focuses on the alignment of 3D shapes, is particularly useful in applications requiring precise matching of complex structures. By addressing the challenges associated with traditional registration techniques, this approach ensures a more accurate and efficient registration process. The method is designed to be adaptable to various 3D datasets, making it a versatile tool for researchers and practitioners working with 3D models. Its effectiveness is demonstrated through rigorous testing on a range of 3D shapes, showcasing its capability to handle diverse and intricate geometries. The registration method's robustness and reliability make it a valuable addition to the toolkit of professionals dealing with 3D data analysis and manipulation.

We propose multi-view and volumetric convolutional neural networks (ConvNets) for 3D shape recognition, which combines surface normal and height fields to capture local geometry and physical size of an object. This strategy helps distinguishing between objects with similar geometries but different sizes. This is especially useful for enhancing volumetric ConvNets and classifying 3D scans with insufficient surface details. Experimental results on CAD and real-world scan datasets showed that our technique outperforms previous approaches.

While 3D object representations are being revived in the context of multi-view object class detection and scene understanding, they have not yet attained wide-spread use in fine-grained categorization. State-of-the-art approaches achieve remarkable performance when training data is plentiful, but they are typically tied to flat, 2D representations that model objects as a collection of unconnected views, limiting their ability to generalize across viewpoints. In this paper, we therefore lift two state-of-the-art 2D object representations to 3D, on the level of both local feature appearance and location. In extensive experiments on existing and newly proposed datasets, we show our 3D object representations outperform their state-of-the-art 2D counterparts for fine-grained categorization and demonstrate their efficacy for estimating 3D geometry from images via ultra-wide baseline matching and 3D reconstruction.

This paper presents a novel combination of local-local information for an efficient finger-knuckle-print (FKP) based recognition system which is robust to scale and rotation. The non-uniform brightness of the FKP due to relatively curvature surface is corrected and texture is enhanced. The local features of the enhanced FKP are extracted using the scale invariant feature transform (SIFT) and the speeded up robust features (SURF). Corresponding features of the enrolled and the query FKPs are matched using nearest-neighbour-ratio method and then the derived SIFT and SURF matching scores are fused using weighted sum rule. The proposed system is evaluated using PolyU FKP database of 7920 images for both identification mode and verification mode. It is observed that the system performs with CRR of 100% and EER of 0.215%. Further, it is evaluated against various scales and rotations of the query image and is found to be robust for query images downscaled upto 60% and for any orientation of query image.

In this paper we introduce a 3-dimensional (3D) SIFT descriptor for video or 3D imagery such as MRI data. We also show how this new descriptor is able to better represent the 3D nature of video data in the application of action recognition. This paper will show how 3D SIFT is able to outperform previously used description methods in an elegant and efficient manner. We use a bag of words approach to represent videos, and present a method to discover relationships between spatio-temporal words in order to better describe the video data.

Understanding 3D object structure from a single image is an important but difficult task in computer vision, mostly due to the lack of 3D object annotations in real images. Previous work tackles this problem by either solving an optimization task given 2D keypoint positions, or training on synthetic data with ground truth 3D information. In this work, we propose 3D INterpreter Network (3D-INN), an endto-end framework which sequentially estimates 2D keypoint heatmaps and 3D object structure, trained on both real 2D-annotated images and synthetic 3D data. This is made possible mainly by two technical innovations. First, we propose a Projection Layer, which projects estimated 3D structure to 2D space, so that 3D-INN can be trained to predict 3D structural parameters supervised by 2D annotations on real images. Second, heatmaps of keypoints serve as an intermediate representation connecting real and synthetic data, enabling 3D-INN to benefit from the variation and abundance of synthetic 3D objects, without suffering from the difference between the statistics of real and synthesized images due to imperfect rendering. The network achieves state-of-the-art performance on both 2D keypoint estimation and 3D structure recovery. We also show that the recovered 3D information can be used in other vision applications, such as 3D rendering and image retrieval.

Face recognition is widely used in computer vision and in many other biometric applications where security is a major concern. The most common problem in recognizing a face arises due to pose variations, different illumination conditions and so on. The main focus of this paper is to recognize whether a given face input corresponds to a registered person in the database. Face recognition is done using Histogram of Oriented Gradients (HOG) technique in AT & T database with an inclusion of a real time subject to evaluate the performance of the algorithm. The feature vectors generated by HOG descriptor are used to train Support Vector Machines (SVM) and results are verified against a given test input. The proposed method checks whether a test image in different pose and lighting conditions is matched correctly with trained images of the facial database. The results of the proposed approach show minimal false positives and improved detection accuracy.

We present a hierarchical regression framework for estimating hand joint positions from single depth images based on local surface normals. The hierarchical regression follows the tree structured topology of hand from wrist to finger tips. We propose a conditional regression forest, i.e. the Frame Conditioned Regression Forest (FCRF) which uses a new normal difference feature. At each stage of the regression, the frame of reference is established from either the local surface normal or previously estimated hand joints. By making the regression with respect to the local frame, the pose estimation is more robust to rigid transformations. We also introduce a new efficient approximation to estimate surface normals. We verify the effectiveness of our method by conducting experiments on two challenging real-world datasets and show consistent improvements over previous discriminative pose estimation methods.

We propose a novel 3D neural network architecture for 3D hand pose estimation from a single depth image. Different from previous works that mostly run on 2D depth image domain and require intermediate or post process to bring in the supervision from 3D space, we convert the depth map to a 3D volumetric representation, and feed it into a 3D convolutional neural network(CNN) to directly produce the pose in 3D requiring no further process. Our system does not require the ground truth reference point for initialization, and our network architecture naturally integrates both local feature and global context in 3D space. To increase the coverage of the hand pose space of the training data, we render synthetic depth image by transferring hand pose from existing real image datasets. We evaluation our algorithm on two public benchmarks and achieve the state-ofthe-art performance. The synthetic hand pose dataset will be available.

We present a novel discriminative regression based approach for the Constrained Local Models (CLMs) framework, referred to as the Discriminative Response Map Fitting (DRMF) method, which shows impressive performance in the generic face fitting scenario. The motivation behind this approach is that, unlike the holistic texture based features used in the discriminative AAM approaches, the response map can be represented by a small set of parameters and these parameters can be very efficiently used for reconstructing unseen response maps. Furthermore, we show that by adopting very simple off-the-shelf regression techniques, it is possible to learn robust functions from response maps to the shape parameters updates. The experiments, conducted on Multi-PIE, XM2VTS and LFPW database, show that the proposed DRMF method outperforms state-of-the-art algorithms for the task of generic face fitting. Moreover, the DRMF method is computationally very efficient and is real-time capable. The current MATLAB implementation takes 1 second per image. To facilitate future comparisons, we release the MATLAB code and the pre-trained models for research purposes.

Extracting well-distributed, reliable, and precisely aligned point pairs for accurate image registration is a difficult task, particularly for multisource remote sensing images that have significant illumination, rotation, and scene differences. The scale-invariant feature transform (SIFT) approach, as a well-known feature-based image matching algorithm, has been successfully applied in a number of automatic registration of remote sensing images. Regardless of its distinctiveness and robustness, the SIFT algorithm suffers from some problems in the quality, quantity, and distribution of extracted features particularly in multisource remote sensing imageries. In this paper, an improved SIFT algorithm is introduced that is fully automated and applicable to various kinds of optical remote sensing images, even with those that are five times the difference in scale. The main key of the proposed approach is a selection strategy of SIFT features in the full distribution of location and scale where the feature qualities are quarantined based on the stability and distinctiveness constraints. Then, the extracted features are introduced to an initial cross-matching process followed by a consistency check in the projective transformation model. Comprehensive evaluation of efficiency, distribution quality, and positional accuracy of the extracted point pairs proves the capabilities of the proposed matching algorithm on a variety of optical remote sensing images.