Question: Online and batch training for pseudo-metric learning algorithms.

This paper delves into the theoretical underpinnings of statistical learning, a critical aspect of machine learning that deals with the analysis of data and the development of algorithms capable of making predictions based on that data. Unlike traditional programming approaches, statistical learning theory focuses on enabling machines to learn from data, improving their performance with experience. Central to this theory are concepts such as inductive bias, which refers to the set of assumptions a learner uses to predict outputs given inputs not seen in the training data. Another key component is the trade-off between bias and variance, where bias represents the error due to erroneous assumptions in the learning algorithm, and variance is the error from sensitivity to small fluctuations in the training set. The paper also discusses the role of regularization in controlling model complexity to prevent overfitting, ensuring that the learned model generalizes well to unseen data. Additionally, it explores the concept of PAC (Probably Approximately Correct) learning, which provides a framework for understanding the sample complexity required for learning a function to a certain level of accuracy with a high probability. The theory is not only applicable to supervised learning scenarios, where the algorithm learns from labeled data, but also to unsupervised learning, where patterns are discovered in unlabeled data, and semi-supervised learning, which combines both labeled and unlabeled data for training. Through rigorous mathematical analysis and empirical validation, statistical learning theory provides a solid foundation for the design and analysis of machine learning algorithms, enabling them to handle a wide range of learning tasks with varying degrees of complexity and uncertainty.

Learning-based hashing algorithms are “hot topics” because they can greatly increase the scale at which existing methods operate. In this paper, we propose a new learning-based hashing method called “fast supervised discrete hashing” (FSDH) based on “supervised discrete hashing” (SDH). Regressing the training examples (or hash code) to the corresponding class labels is widely used in ordinary least squares regression. Rather than adopting this method, FSDH uses a very simple yet effective regression of the class labels of training examples to the corresponding hash code to accelerate the algorithm. To the best of our knowledge, this strategy has not previously been used for hashing. Traditional SDH decomposes the optimization into three sub-problems, with the most critical sub-problem - discrete optimization for binary hash codes - solved using iterative discrete cyclic coordinate descent (DCC), which is time-consuming. However, FSDH has a closed-form solution and only requires a single rather than iterative hash code-solving step, which is highly efficient. Furthermore, FSDH is usually faster than SDH for solving the projection matrix for least squares regression, making FSDH generally faster than SDH. For example, our results show that FSDH is about 12-times faster than SDH when the number of hashing bits is 128 on the CIFAR-10 data base, and FSDH is about 151-times faster than FastHash when the number of hashing bits is 64 on the MNIST data-base. Our experimental results show that FSDH is not only fast, but also outperforms other comparative methods.

Positive and unlabelled learning (PU learning) has been investigated to deal with the situation where only the positive examples and the unlabelled examples are available. Most of the previous works focus on identifying some negative examples from the unlabelled data, so that the supervised learning methods can be applied to build a classifier. However, for the remaining unlabelled data, which can not be explicitly identified as positive or negative (we call them ambiguous examples), they either exclude them from the training phase or simply enforce them to either class. Consequently, their performance may be constrained. This paper proposes a novel approach, called similarity-based PU learning (SPUL) method, by associating the ambiguous examples with two similarity weights, which indicate the similarity of an ambiguous example towards the positive class and the negative class, respectively. The local similarity-based and global similarity-based mechanisms are proposed to generate the similarity weights. The ambiguous examples and their similarity-weights are thereafter incorporated into an SVM-based learning phase to build a more accurate classifier. Extensive experiments on real-world datasets have shown that SPUL outperforms state-of-the-art PU learning methods.

We consider noisy crowdsourced assessments and their impact on learning-to-rank algorithms. Starting with EM-weighted assessments, we modify LambdaMART in order to use smoothed probabilistic preferences over pairs of documents, directly as input to the ranking algorithm.

We develop a parameter-free face recognition algorithm which is insensitive to large variations in lighting, expression, occlusion, and age using a single gallery sample per subject. We take advantage of the observation that equidistant prototypes embedding is an optimal embedding that maximizes the minimum one-against-the-rest margin between the classes. Rather than preserving the global or local structure of the training data, our method, called linear regression analysis (LRA), applies least-square regression technique to map gallery samples to the equally distant locations, regardless of the true structure of training data. Further, a novel generic learning method, which maps the intra-class facial differences of the generic faces to the zero vectors, is incorporated to enhance the generalization capability of LRA. Using this novel method, learning based on only a handful of generic classes can largely improve the face recognition performance, even when the generic data are collected from a different database and camera set-up. The incremental learning based on the Greville algorithm makes the mapping matrix efficiently updated from the newly coming gallery classes, training samples, or generic variations. Although it is fairly simple and parameter-free, LRA, combined with commonly used local descriptors, such as Gabor representation and local binary patterns, outperforms the state-of-the-art methods for several standard experiments on the Extended Yale B, CMU PIE, AR, and ∗Corresponding author. Tel:+86 10 62283059 Fax: +86 10 62285019 Email address: whdeng@bupt.edu.cn (Weihong Deng) Preprint submitted to Elsevier March 28, 2014

Example-basedmethodsareeffectivefor parameterestimationproblemswhentheunderlyingsystemis simpleor thedimensionalityof the input is low. For complex andhigh-dimensional problemssuch asposeestimation,thenumberof required examplesand the computationalcomplexity rapidly becmeprohibitivelyhigh. We introducea new algorithm that learnsa setof hashingfunctionsthat efficiently index examplesrelevant to a particular estimationtask. Our algorithm extendsa recentlydevelopedmethodfor locality-sensitivehashing, which findsapproximateneighborsin timesublinearin thenumber of examples.Thismethoddependscritically on thechoiceof hashfunctions;weshowhowto find thesetof hashfunctions thatare optimallyrelevantto a particular estimationproblem.Experimentsdemonstr atethat theresultingalgorithm,which wecall Parameter -SensitiveHashing, canrapidlyandaccuratelyestimatethearticulatedposeof humanfiguresfroma large databaseof exampleimages. 0Part of thiswork wasdonewhenG.S.andP.V. werewith MitsubishiElectricResearchLabs,Cambridge,MA.

We consider visual category recognition in the framework of measuring similarities, or equivalently perceptual distances, to prototype examples of categories. This approach is quite flexible, and permits recognition based on color, texture, and particularly shape, in a homogeneous framework. While nearest neighbor classifiers are natural in this setting, they suffer from the problem of high variance (in bias-variance decomposition) in the case of limited sampling. Alternatively, one could use support vector machines but they involve time-consuming optimization and computation of pairwise distances. We propose a hybrid of these two methods which deals naturally with the multiclass setting, has reasonable computational complexity both in training and at run time, and yields excellent results in practice. The basic idea is to find close neighbors to a query sample and train a local support vector machine that preserves the distance function on the collection of neighbors. Our method can be applied to large, multiclass data sets for which it outperforms nearest neighbor and support vector machines, and remains efficient when the problem becomes intractable for support vector machines. A wide variety of distance functions can be used and our experiments show state-of-the-art performance on a number of benchmark data sets for shape and texture classification (MNIST, USPS, CUReT) and object recognition (Caltech- 101). On Caltech-101 we achieved a correct classification rate of 59.05%(±0.56%) at 15 training images per class, and 66.23%(±0.48%) at 30 training images.

In various application domains, including image recognition, it is natural to represent each example as a set of vectors. With a base kernel we can implicitly map these vectors to a Hilbert space and fit a Gaussian distribution to the whole set using Kernel PCA. We define our kernel between examples as Bhattacharyya’s measure of affinity between such Gaussians. The resulting kernel is computable in closed form and enjoys many favorable properties, including graceful behavior under transformations, potentially justifying the vector set representation even in cases when more conventional representations also exist.

Proceedings of the 1998 IEEE International Conference on Computer Vision, Bombay, India We introduce a new distance between two distributions that we call the Earth Mover’s Distance (EMD), which reflects the minimal amount of work that must be performed to transform one distribution into the other by moving “distribution mass” around. This is a special case of the transportation problem from linear optimization, for which efficient algorithms are available. The EMD also allows for partial matching. When used to compare distributions that have the same overall mass, the EMD is a true metric, and has easy-to-compute lower bounds. In this paper we focus on applications to image databases, especially color and texture. We use the EMD to exhibit the structure of color-distribution and texture spaces by means of MultiDimensional Scaling displays. We also propose a novel approach to the problem of navigating through a collection of color images, which leads to a new paradigm for image database search.

Compatibility between items, such as clothes and shoes, is a major factor among customer’s purchasing decisions. However, learning “compatibility” is challenging due to (1) broader notions of compatibility than those of similarity, (2) the asymmetric nature of compatibility, and (3) only a small set of compatible and incompatible items are observed. We propose an end-to-end trainable system to embed each item into a latent vector and project a query item into K compatible prototypes in the same space. These prototypes reflect the broad notions of compatibility. We refer to both the embedding and prototypes as “Compatibility Family”. In our learned space, we introduce a novel Projected Compatibility Distance (PCD) function which is differentiable and ensures diversity by aiming for at least one prototype to be close to a compatible item, whereas none of the prototypes are close to an incompatible item. We evaluate our system on a toy dataset, two Amazon product datasets, and Polyvore outfit dataset. Our method consistently achieves state-of-the-art performance. Finally, we show that we can visualize the candidate compatible prototypes using a Metric-regularized Conditional Generative Adversarial Network (MrCGAN), where the input is a projected prototype and the output is a generated image of a compatible item. We ask human evaluators to judge the relative compatibility between our generated images and images generated by CGANs conditioned directly on query items. Our generated images are significantly preferred, with roughly twice the number of votes as others.

Boosted decision trees are among the most popular learning techniques in use today. While exhibiting fast speeds at test time, relatively slow training renders them impractical for applications with real-time learning requirements. We propose a principled approach to overcome this drawback. We prove a bound on the error of a decision stump given its preliminary error on a subset of the training data; the bound may be used to prune unpromising features early in the training process. We propose a fast training algorithm that exploits this bound, yielding speedups of an order of magnitude at no cost in the final performance of the classifier. Our method is not a new variant of Boosting; rather, it is used in conjunction with existing Boosting algorithms and other sampling methods to achieve even greater speedups.


Question: DeepMimic: Demonstration-driven deep reinforcement learning for physics-based character ability acquisition.

This paper presents the EPOpt algorithm, a novel approach for learning robust neural network policies through the use of model ensembles and adversarial training in reinforcement learning. Addressing the critical issues of sample complexity and safety, especially pertinent when employing deep neural networks for policy representation in real-world tasks, EPOpt leverages a simulated source domain to approximate the real-world target domain. This strategy enables the augmentation of real data with simulated data, aiming to overcome the challenges associated with discrepancies between the source and target domains.

In EPOpt, an ensemble of simulated source domains is utilized, coupled with adversarial training techniques, to cultivate policies that are resilient and capable of generalizing across a wide spectrum of potential target domains. This includes scenarios where unmodeled effects are present. The algorithm further enhances its adaptability by dynamically adjusting the probability distribution over the source domains in the ensemble, incorporating data from the target domain and utilizing approximate Bayesian methods. This adaptation process facilitates a continuous refinement of the ensemble, making it an increasingly accurate representation of the target domain. Consequently, EPOpt offers a dual advantage: robustness in policy learning and the capacity for continuous learning and adaptation.

In recent years, deep generative models have been shown to ‘imagine’ convincing high-dimensional observations such as images, audio, and even video, learning directly from raw data. In this work, we ask how to imagine goal-directed visual plans – a plausible sequence of observations that transition a dynamical system from its current configuration to a desired goal state, which can later be used as a reference trajectory for control. We focus on systems with high-dimensional observations, such as images, and propose an approach that naturally combines representation learning and planning. Our framework learns a generative model of sequential observations, where the generative process is induced by a transition in a low-dimensional planning model, and an additional noise. By maximizing the mutual information between the generated observations and the transition in the planning model, we obtain a low-dimensional representation that best explains the causal nature of the data. We structure the planning model to be compatible with efficient planning algorithms, and we propose several such models based on either discrete or continuous states. Finally, to generate a visual plan, we project the current and goal observations onto their respective states in the planning model, plan a trajectory, and then use the generative model to transform the trajectory to a sequence of observations. We demonstrate our method on imagining plausible visual plans of rope manipulation3.

This review investigates two recent developments in artificial intelligence and neural computation: learning from imitation and the development of humanoid robots. It is postulated that the study of imitation learning offers a promising route to gain new insights into mechanisms of perceptual motor control that could ultimately lead to the creation of autonomous humanoid robots. Imitation learning focuses on three important issues: efficient motor learning, the connection between action and perception, and modular motor control in the form of movement primitives. It is reviewed here how research on representations of, and functional connections between, action and perception have contributed to our understanding of motor acts of other beings. The recent discovery that some areas in the primate brain are active during both movement perception and execution has provided a hypothetical neural basis of imitation. Computational approaches to imitation learning are also described, initially from the perspective of traditional AI and robotics, but also from the perspective of neural network models and statistical-learning research. Parallels and differences between biological and computational approaches to imitation are highlighted and an overview of current projects that actually employ imitation learning for humanoid robots is given.

The Programming by Demonstration (PbD) technique aims at teaching a robot to accomplish a task by learning from a human demonstration. In a manipulation context, recognizing the demonstrator's hand gestures, specifically when and how objects are grasped, plays a significant role. Here, a system is presented that uses both hand shape and contact-point information obtained from a data glove and tactile sensors to recognize continuous human-grasp sequences. The sensor fusion, grasp classification, and task segmentation are made by a hidden Markov model recognizer. Twelve different grasp types from a general, task-independent taxonomy are recognized. An accuracy of up to 95% could be achieved for a multiple-user system.

Recent advances in deep learning have allowed artificial agents to rival human-level performance on a wide range of complex tasks; however, the ability of these networks to learn generalizable strategies remains a pressing challenge. This critical limitation is due in part to two factors: the opaque information representation in deep neural networks and the complexity of the task environments in which they are typically deployed. Here we propose a novel Hierarchical Q-Network (HQN), motivated by theories of the hierarchical organization of the human prefrontal cortex, that attempts to identify lower dimensional patterns in the value landscape that can be exploited to construct an internal model of rules in simple environments. We draw on combinatorial games, where there exists a single optimal strategy for winning that generalizes across other features of the game, to probe the strategy generalization of the HQN and other reinforcement learning (RL) agents using variations of Wythoff’s game. Traditional RL approaches failed to reach satisfactory performance on variants of Wythoff’s Game; however, the HQN learned heuristic-like strategies that generalized across changes in board configuration. More importantly, the HQN allowed for transparent inspection of the agent’s internal model of the game following training. Our results show how a biologically inspired hierarchical learner can facilitate learning abstract rules to promote robust and flexible action policies in simplified training environments with clearly delineated optimal strategies. 1 ar X iv :1 80 1. 06 68 9v 1 [ cs .A I] 2 0 Ja n 20 18

In this work, we explore the possibility of training agents which are able to show intelligent behaviour in many different scenarios. We present the effectiveness of different machine learning algorithms in the StarCraft II Learning Environment [1] and their performance in different scenarios and compare them. In the end, we recreate DeepMind’s FullyConv Agent with slightly better results.

We present a framework for autonomous driving which can learn from human demonstrations, and we apply it to the longitudinal control of an autonomous car. Offline, we model car-following strategies from a set of example driving sequences. Online, the model is used to compute accelerations which replicate what a human driver would do in the same situation. This reference acceleration is tracked by a predictive controller which enforces a set of comfort and safety constraints before applying the final acceleration. The controller is designed to be robust to the uncertainty in the predicted motion of the preceding vehicle. In addition, we estimate the confidence of the driver model predictions and use it in the cost function of the predictive controller. As a result, we can handle cases where the training data used to learn the driver model does not provide sufficient information about how a human driver would handle the current driving situation. The approach is validated using a combination of simulations and experiments on our autonomous vehicle.

We present an approach to learn the inverse kinematics of the “bionic handling assistant”-an elephant trunk robot. This task comprises substantial challenges including high dimensionality, restrictive and unknown actuation ranges, and nonstationary system behavior. We use a recent exploration scheme, online goal babbling, which deals with these challenges by bootstrapping and adapting the inverse kinematics on the fly. We show the success of the method in extensive real-world experiments on the nonstationary robot, including a novel combination of learning and traditional feedback control. Simulations further investigate the impact of nonstationary actuation ranges, drifting sensors, and morphological changes. The experiments provide the first substantial quantitative real-world evidence for the success of goal-directed bootstrapping schemes, moreover with the challenge of nonstationary system behavior. We thereby provide the first functioning control concept for this challenging robot platform.

We propose to combine recent Convolutional Neural Networks (CNN) models with depth imaging to obtain a reliable and fast multi-person pose estimation algorithm applicable to Human Robot Interaction (HRI) scenarios. Our hypothesis is that depth images contain less structures and are easier to process than RGB images while keeping the required information for human detection and pose inference, thus allowing the use of simpler networks for the task. Our contributions are threefold. (i) we propose a fast and efficient network based on residual blocks (called RPM) for body landmark localization from depth images; (ii) we created a public dataset DIH comprising more than 170k synthetic images of human bodies with various shapes and viewpoints as well as real (annotated) data for evaluation; (iii) we show that our model trained on synthetic data from scratch can perform well on real data, obtaining similar results to larger models initialized with pre-trained networks. It thus provides a good trade-off between performance and computation. Experiments on real data demonstrate the validity of our approach.

We present a system that transforms a monocular video of a soccer game into a moving 3D reconstruction, in which the players and field can be rendered interactively with a 3D viewer or through an Augmented Reality device. At the heart of our paper is an approach to estimate the depth map of each player, using a CNN that is trained on 3D player data extracted from soccer video games. We compare with state of the art body pose and depth estimation techniques, and show results on both synthetic ground truth benchmarks, and real YouTube soccer footage.

Interpreting decision making logic in demonstration videos is key to collaborating with and mimicking humans. To empower machines with this ability, we propose a neural program synthesizer that is able to explicitly synthesize underlying programs from behaviorally diverse and visually complicated demonstration videos. We introduce a summarizer module as part of our model to improve the network’s ability to integrate multiple demonstrations varying in behavior. We also employ a multi-task objective to encourage the model to learn meaningful intermediate representations for end-to-end training. We show that our model is able to reliably synthesize underlying programs as well as capture diverse behaviors exhibited in demonstrations. The code is available at https://shaohua0116.github.io/demo2program.


Question:
DeepMimic:示范引导的深度强化学习，用于掌握基于物理的角色技能。

(Note: I've kept the length and complexity of the text similar, but have translated the text into Chinese to maintain semantic style, as the original example does not specify a particular style. If a specific style is desired, please let me know.)

Alternatively, if you prefer an English rephrased version:

DeepMimic: A demonstration-guided deep reinforcement learning approach for mastering physics-based character skills.

DeepStride: Hierarchical Deep Reinforcement Learning for Dynamic and Adaptive Locomotion

In this research, we present DeepStride, a framework that utilizes hierarchical deep reinforcement learning to teach physics-based characters a wide range of environment-aware locomotion skills, requiring minimal prior knowledge. Our approach is structured in two stages. Initially, we train low-level controllers that operate at a granular timescale, designed to achieve stable and versatile walking gaits that align with specific stepping objectives and stylistic preferences. These controllers are robust against external forces, terrain irregularities, and style transitions. Subsequently, high-level controllers are developed, functioning at the pace of individual steps. They direct the low-level controllers by setting desired step targets, basing decisions on rich, high-dimensional inputs like terrain maps or contextual environmental representations. Both tiers of the control policy are honed through deep reinforcement learning. Our experiments are conducted on a simulated 3D biped model. The low-level controllers are shown to be adaptable across various motion styles and resilient to disturbances, terrain changes, and style blending. The high-level controllers exhibit the ability to track paths through diverse terrains, guide a soccer ball to a designated spot, and successfully navigate through static and moving obstacles. This dual-level approach showcases the effectiveness of deep reinforcement learning in endowing characters with sophisticated and adaptive locomotion capabilities.

We propose a novel training algorithm for reinforcement learning which combines the strength of deep Q-learning with a constrained optimization approach to tighten optimality and encourage faster reward propagation. Our novel technique makes deep reinforcement learning more practical by drastically reducing the training time. We evaluate the performance of our approach on the 49 games of the challenging Arcade Learning Environment, and report significant improvements in both training time and accuracy.

Human 3D pose estimation from a single image is a challenging task with numerous applications. Convolutional Neural Networks (CNNs) have recently achieved superior performance on the task of 2D pose estimation from a single image, by training on images with 2D annotations collected by crowd sourcing. This suggests that similar success could be achieved for direct estimation of 3D poses. However, 3D poses are much harder to annotate, and the lack of suitable annotated training images hinders attempts towards end-to-end solutions. To address this issue, we opt to automatically synthesize training images with ground truth pose annotations. Our work is a systematic study along this road. We find that pose space coverage and texture diversity are the key ingredients for the effectiveness of synthetic training data. We present a fully automatic, scalable approach that samples the human pose space for guiding the synthesis procedure and extracts clothing textures from real images. Furthermore, we explore domain adaptation for bridging the gap between our synthetic training images and real testing photos. We demonstrate that CNNs trained with our synthetic images out-perform those trained with real photos on 3D pose estimation tasks.

A typical goal for transfer learning algorithms is to utilize knowledge gained in a source task to learn a target task faster. Recently introduced transfer methods in reinforcement learning settings have shown considerable promise, but they typically transfer between pairs of very similar tasks. This work introduces Rule Transfer, a transfer algorithm that first learns rules to summarize a source task policy and then leverages those rules to learn faster in a target task. This paper demonstrates that Rule Transfer can effectively speed up learning in Keepaway, a benchmark RL problem in the robot soccer domain, based on experience from source tasks in the gridworld domain. We empirically show, through the use of three distinct transfer metrics, that Rule Transfer is effective across these domains.

While Reinforcement Learning (RL) is not traditionally designed for interactive supervisory input from a human teacher, several works in both robot and software agents have adapted it for human input by letting a human trainer control the reward signal. In this work, we experimentally examine the assumption underlying these works, namely that the human-given reward is compatible with the traditional RL reward signal. We describe an experimental platform with a simulated RL robot and present an analysis of real-time human teaching behavior found in a study in which untrained subjects taught the robot to perform a new task. We report three main observations on how people administer feedback when teaching a robot a task through Reinforcement Learning: (a) they use the reward channel not only for feedback, but also for future-directed guidance; (b) they have a positive bias to their feedback — possibly using the signal as a motivational channel; and (c) they change their behavior as they develop a mental model of the robotic learner. Given this, we made specific modifications to the simulated RL robot, and analyzed and evaluated its learning behavior in four additional experiments with human trainers. We report significant improvements on several learning measures. This work demonstrates the importance of understanding the human-teacher/robot-learner partnership in order to design algorithms that support how people want to teach while simultaneously improving the robot’s learning behavior.

The Programming by Demonstration (PbD) technique aims at teaching a robot to accomplish a task by learning from a human demonstration. In a manipulation context, recognizing the demonstrator's hand gestures, specifically when and how objects are grasped, plays a significant role. Here, a system is presented that uses both hand shape and contact-point information obtained from a data glove and tactile sensors to recognize continuous human-grasp sequences. The sensor fusion, grasp classification, and task segmentation are made by a hidden Markov model recognizer. Twelve different grasp types from a general, task-independent taxonomy are recognized. An accuracy of up to 95% could be achieved for a multiple-user system.

In this work, we propose a novel robot learning framework called Neural Task Programming (NTP), which bridges the idea of few-shot learning from demonstration and neural program induction. NTP takes as input a task specification (e.g., video demonstration of a task) and recursively decomposes it into finer sub-task specifications. These specifications are fed to a hierarchical neural program, where bottom-level programs are callable subroutines that interact with the environment. We validate our method in three robot manipulation tasks. NTP achieves strong generalization across sequential tasks that exhibit hierarchal and compositional structures. The experimental results show that NTP learns to generalize well towards unseen tasks with increasing lengths, variable topologies, and changing objectives.stanfordvl.github.io/ntp/.

Deep learning algorithms have recently appeared that pretrain hidden layers of neural networks in unsupervised ways, leading to state-of-the-art performance on large classification problems. These methods can also pretrain networks used for reinforcement learning. However, this ignores the additional information that exists in a reinforcement learning paradigm via the ongoing sequence of state, action, new state tuples. This paper demonstrates that learning a predictive model of state dynamics can result in a pretrained hidden layer structure that reduces the time needed to solve reinforcement learning problems.

Reinforcement Learning (RL) is a learning paradigm concerned with learning to control a system so as to maximize an objective over the long term. This approach to learning has received immense interest in recent times and success manifests itself in the form of human-level performance on games like Go. While RL is emerging as a practical component in reallife systems, most successes have been in Single Agent domains. This report will instead specifically focus on challenges that are unique to Multi-Agent Systems interacting in mixed cooperative and competitive environments. The report concludes with advances in the paradigm of training Multi-Agent Systems called Decentralized Actor, Centralized Critic, based on an extension of MDPs called Decentralized Partially Observable MDPs, which has seen a renewed interest lately. 1 ar X iv :1 80 7. 09 42 7v 1 [ cs .A I] 2 5 Ju l 2 01 8

Humans are able to understand and perform complex tasks by strategically structuring the tasks into incremental steps or subgoals. For a robot attempting to learn to perform a sequential task with critical subgoal states, such states can provide a natural opportunity for interaction with a human expert. This paper analyzes the benefit of incorporating a notion of subgoals into Inverse Reinforcement Learning (IRL) with a Human-In-The-Loop (HITL) framework. The learning process is interactive, with a human expert first providing input in the form of full demonstrations along with some subgoal states. These subgoal states define a set of subtasks for the learning agent to complete in order to achieve the final goal. The learning agent queries for partial demonstrations corresponding to each subtask as needed when the agent struggles with the subtask. The proposed Human Interactive IRL (HI-IRL) framework is evaluated on several discrete path-planning tasks. We demonstrate that subgoal-based interactive structuring of the learning task results in significantly more efficient learning, requiring only a fraction of the demonstration data needed for learning the underlying reward function with the baseline IRL model.

One of the primary challenges of developmental robotics is the question of how to learn and represent increasingly complex behavior in a self-motivated, open-ended way. Barto, Singh, and Chentanez (Barto, Singh, & Chentanez 2004; Singh, Barto, & Chentanez 2004) have recently presented an algorithm for intrinsically motivated reinforcement learning that strives to achieve broad competence in an environment in a tasknonspecific manner by incorporating internal reward to build a hierarchical collection of skills. This paper suggests that with its emphasis on task-general, self-motivated, and hierarchical learning, intrinsically motivated reinforcement learning is an obvious choice for organizing behavior in developmental robotics. We present additional preliminary results from a gridworld abstraction of a robot environment and advocate a lay-ion of a robot environment and advocate a layered learning architecture for applying the algorithm on a physically embodied system.


Question:
DeepMimic:示范引导的深度强化学习以掌握基于物理的角色技能。

(Note: I kept the length and semantic style the same, but due to the nature of the sentence and the technical terms used, the complexity may not be exactly the same in translation. The meaning of the sentence is preserved.)

This paper presents an innovative method for enabling simulated characters to develop terrain-adaptive locomotion skills through the application of deep reinforcement learning (DeepRL). Traditionally, reinforcement learning necessitates the use of sparse, manually engineered features, which can be limiting. However, by leveraging advancements in DeepRL, we propose a novel Mixture of Actor-Critic Experts (MACE) framework. MACE is capable of rapidly learning dynamic locomotion skills that are adaptive to various terrains. It achieves this by processing high-dimensional state and terrain data as inputs, and generating parameterized leaps or steps as output actions. Compared to a single actor-critic model, MACE exhibits accelerated learning and fosters the emergence of specialized actor-critic experts. Our approach also incorporates Boltzmann exploration and initializes actor biases to promote specialization, further enhancing learning efficiency. The effectiveness of our solution is showcased through demonstrations involving a variety of planar characters and different terrain types.

What would it take to teach a computer to play a game entirely through language and sketching? In this paper, we present an implemented program through which an instructor can teach the rules of simple board games using such input. We describe the architecture, information flow, and vocabulary of instructional events and walk through an annotated example. In our approach, the instructional and communication events guide abductive reasoning for language interpretation and help to integrate information from sketching and language. Having a general target representation enables the learning process to be viewed more as translation and problem solving than as induction. Lastly, learning by demonstration complements and extends instruction, resulting in concrete, operational rules.

We present and discuss four important yet underserved research questions critical to the future of sharedenvironment human-robot collaboration. We begin with a brief survey of research surrounding individual components required for a complete collaborative robot control system, discussing the current state of the art in Learning from Demonstration, active learning, adaptive planning systems, and intention recognition. We motivate the exploration of the presented research questions by relating them to existing work and representative use cases from the domains of construction and cooking.

Reinforcement learning offers one of the most general framework to take traditional robotics towards true autonomy and versatility. However, applying reinforcement learning to high dimensional movement systems like humanoid robots remains an unsolved problem. In this paper, we discuss different approaches of reinforcement learning in terms of their applicability in humanoid robotics. Methods can be coarsely classified into three different categories, i.e., greedy methods, ‘vanilla’ policy gradient methods, and natural gradient methods. We discuss that greedy methods are not likely to scale into the domain humanoid robotics as they are problematic when used with function approximation. ‘Vanilla’ policy gradient methods on the other hand have been successfully applied on real-world robots including at least one humanoid robot [3]. We demonstrate that these methods can be significantly improved using the natural policy gradient instead of the regular policy gradient. A derivation of the natural policy gradient is provided, proving that the average policy gradient of Kakade [10] is indeed the true natural gradient. A general algorithm for estimating the natural gradient, the Natural Actor-Critic algorithm, is introduced. This algorithm converges to the nearest local minimum of the cost function with respect to the Fisher information metric under suitable conditions. The algorithm outperforms non-natural policy gradients by far in a cart-pole balancing evaluation, and for learning nonlinear dynamic motor primitives for humanoid robot control. It offers a promising route for the development of reinforcement learning for truly high-dimensionally continuous state-action systems.

Though successful in high-dimensional domains, deep reinforcement learning exhibits high sample complexity and suffers from stability issues as reported by researchers and practitioners in the field. These problems hinder the application of such algorithms in real-world and safety-critical scenarios. In this paper, we take steps towards stable and efficient reinforcement learning by following a model-based approach that is known to reduce agent-environment interactions. Namely, our method augments deep Q-networks (DQNs) with model predictions for transitions, rewards, and termination flags. Having the model at hand, we then conduct a rigorous theoretical study of our algorithm and show, for the first time, convergence to a stationary point. En route, we provide a counterexample showing that ’vanilla’ DQNs can diverge confirming practitioners’ and researchers’ experiences. Our proof is novel in its own right and can be extended to other forms of deep reinforcement learning. In particular, we believe exploiting the relation between reinforcement (with deep function approximators) and online learning can serve as a recipe for future proofs in the domain. Finally, we validate our theoretical results in 20 games from the Atari benchmark. Our results show that following the proposed model-based learning approach not only ensures convergence but leads to a reduction in sample complexity and superior performance. Introduction Model-free deep reinforcement learning methods have recently demonstrated impressive performance on a range of complex learning tasks (Hessel et al. 2018; Lillicrap et al. 2017; Jaderberg et al. 2017). Deep Q-Networks (DQNs) (Mnih et al. 2015), in particular, stand out as a versatile and effective tool for a wide range of applications. DQNs offer an end-to-end solution for many reinforcement learning problems by generalizing tabular Q-learning to highdimensional input spaces. Unfortunately, DQNs still suffer from a number of important drawbacks that hinder wide-spread adoption. There is a general lack of theoretical understanding to guide the use of deep Q-learning algorithms. Combining non-linear function approximation with off-policy reinforcement learning is known to lead to possible divergence issues, even in very simple tasks (Boyan and Preliminary work. Moore 1995). Despite improvements introduced by deep Qnetworks (Mnih et al. 2015), Q-learning approaches based on deep neural networks still exhibit occasional instability during the learning process, see for example (van Hasselt, Guez, and Silver 2016). In this paper, we aim to address the instability of current deep RL approaches by augmenting model-free value learning with a predictive model. We propose a hybrid model and value learning objective, which stabilises the learning process and reduces sample complexity when compared to model-free algorithms. We attribute this reduction to the richer feedback signal provided by our method compared to standard deep Q-networks. In particular, our model provides feedback at every time step based on the current prediction error, which in turn eliminates one source of sample inefficiency caused by sparse rewards. These conclusions are also in accordance with previous research conducted on linear function approximation in reinforcement learning (Parr et al. 2008; Sutton et al. 2012; Song et al. 2016). While linear function approximation results provide a motivation for model-based stabilisation, such theories fail to generalise to deep non-linear architectures (Song et al. 2016). To close this gap in the literature and demonstrate stability of our method, we prove convergence in the general deep RL setting with deep value function approximators. Theoretically analysing deep RL algorithms, however, is challenging due to nonlinear functional dependencies and objective non-convexities prohibiting convergence to globally optimal policies. As such, the best one would hope for is a stationary point (e.g., a first-order one) that guarantees vanishing gradients. Even understanding gradient behavior for deep networks in RL can be difficult due to explorationbased policies and replay memory considerations. To alleviate some of these problems, we map deep reinforcement learning to a mathematical framework explicitly designed for understanding the exploration/exploitation trade-off. Namely, we formalise the problem as an online learning game the agent plays against an adversary (i.e., the environment). Such a link allows us to study a more general problem combining notions of regret, reinforcement learning, and optimisation. Given such a link, we prove that for any > 0, regret vanishes as O ( T 1−2 lnT ) with T being the total number of rounds. This, in turn, guarantees convergence to a stationary point. ar X iv :1 80 9. 01 90 6v 1 [ cs .L G ] 6 S ep 2 01 8 Guided by the above theoretical results, we validate that our algorithm leads to faster learning by conducting experiments on 20 Atari games. Due to the high computational and budgeting demands, we chose a benchmark that is closest to our setting. Concretely, we picked DQNs as our template algorithm to improve on both theoretically and empirically. It is worth noting that extending our theoretical results to more complex RL algorithms is an orthogonal future direction. Background: Reinforcement Learning We consider a discrete time, infinite horizon, discounted Markov Decision Process (MDP) 〈 S,A, P , P , γ 〉 . Here S denotes the state space, A the set of possible actions, P (R) : S × A × R → [0, 1] is the reward function, P (S) : S × A × S → [0, 1] is the state transition function, and γ ∈ (0, 1) is a discount factor. An agent, being in state St ∈ S at time step t, executes an action at ∈ A sampled from its policy π(at|St), a conditional probability distribution where π : S × A → [0, 1]. The agent’s action elicits from the environment a reward signal rt ∈ R indicating instantaneous reward, a terminal flag ft ∈ {0, 1} indicating a terminal event that restarts the environment, and a transition to a successor state St+1 ∈ S. We assume that the sets S, A, and R are discrete. The reward rt is sampled from the conditional probability distribution P (rt|St,at). Similarly, ft ∼ P (F (ft|St,at) with P (F ) : S × A × {0, 1} → [0, 1], where a terminal event (ft = 1) restarts the environment according to some initial state distribution S0 ∼ P (S) 0 (S0). The state transition to a successor state is determined by a stochastic state transition function according to St+1 ∼ P (St+1|St,at). The agent’s goal is to maximise future cumulative reward E P (S) 0 ,π,P (R),P (F ),P (S) [ ∞ ∑ t=0 ( t ∏ t′=0 (1− ft′−1) ) γrt ] with respect to the policy π. An important quantity in RL are Q-values Q(St,at), which are defined as the expected future cumulative reward value when executing action at in state St and subsequently following policy π. Qvalues enable us to conveniently phrase the RL problem as maxπ ∑ S p (S) ∑ a π(a|S)Q(S,a), where p(S) is the stationary state distribution obtained when executing π in the environment (starting from S0 ∼ P (S) 0 (S0)). Deep Q-Networks: Value-based reinforcement learning approaches identify optimal Q-values directly using parametric function approximators Qθ(S,a), where θ ∈ Θ represents the parameters (Watkins 1989; Busoniu et al. 2010). Optimal Q-value estimates then correspond to an optimal policy π(a|S) = δa,argmaxa′Qθ(S,a′). Deep Q-networks (Mnih et al. 2015) learn a deep neural network based Qvalue approximation by performing stochastic gradient descent on the following training objective: L(θ) = ES,a,r,f,S′ [( r + (1− f)γmax a′ Qθ−(S ′,a′) −Qθ(S,a) )2] . (1) The expectation ranges over transitions S,a, r, f,S′ sampled from an experience replay memory (S′ denotes the state at the next time step). Use of this replay memory, together with the use of a separate target network Qθ− (with different parameters θ− ∈ Θ) for calculating the bootstrap values maxa′ Qθ−(S ′,a′), helps stabilise the learning process. Background: Online Learning In this paper, we employ a special form of regret minimisation games that we briefly review here. A regret minimisation game is a triple 〈Θ,F , T 〉, where Θ is a non-empty decision set, F is the set of moves of the adversary which contains bounded convex functions from R to R and T is the total number of rounds. The game commences in rounds, where at round j = 1, . . . , T , the agent chooses a prediction θj ∈ Θ and the adversary a loss function Lj ∈ F . At the end of the round, the adversary reveals its choice and the agent suffers a loss Lj(θj). In this paper, we are concerned with the full-information case where the agent can observe the complete loss function Lj at the end of each round. The goal of the game is for the agent to make successive predictions to minimise cumulative regret defined as:

The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Until now neural networks have not been capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks that they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on a hand-written digit dataset and by learning several Atari 2600 games sequentially.

This paper addresses the challenge of learning to play many different video games with little domain-specific knowledge. Specifically, it introduces a neuroevolution approach to general Atari 2600 game playing. Four neuroevolution algorithms were paired with three different state representations and evaluated on a set of 61 Atari games. The neuroevolution agents represent different points along the spectrum of algorithmic sophistication - including weight evolution on topologically fixed neural networks (conventional neuroevolution), covariance matrix adaptation evolution strategy (CMA-ES), neuroevolution of augmenting topologies (NEAT), and indirect network encoding (HyperNEAT). State representations include an object representation of the game screen, the raw pixels of the game screen, and seeded noise (a comparative baseline). Results indicate that direct-encoding methods work best on compact state representations while indirect-encoding methods (i.e., HyperNEAT) allow scaling to higher dimensional representations (i.e., the raw game screen). Previous approaches based on temporal-difference (TD) learning had trouble dealing with the large state spaces and sparse reward gradients often found in Atari games. Neuroevolution ameliorates these problems and evolved policies achieve state-of-the-art results, even surpassing human high scores on three games. These results suggest that neuroevolution is a promising approach to general video game playing (GVGP).

Affordances encode relationships between actions, objects, and effects. They play an important role on basic cognitive capabilities such as prediction and planning. We address the problem of learning affordances through the interaction of a robot with the environment, a key step to understand the world properties and develop social skills. We present a general model for learning object affordances using Bayesian networks integrated within a general developmental architecture for social robots. Since learning is based on a probabilistic model, the approach is able to deal with uncertainty, redundancy, and irrelevant information. We demonstrate successful learning in the real world by having an humanoid robot interacting with objects. We illustrate the benefits of the acquired knowledge in imitation games.

The ability to use a 2D map to navigate a complex 3D environment is quite remarkable, and even difficult for many humans. Localization and navigation is also an important problem in domains such as robotics, and has recently become a focus of the deep reinforcement learning community. In this paper we teach a reinforcement learning agent to read a map in order to find the shortest way out of a random maze it has never seen before. Our system combines several state-of-theart methods such as A3C and incorporates novel elements such as a recurrent localization cell. Our agent learns to localize itself based on 3D first person images and an approximate orientation angle. The agent generalizes well to bigger mazes, showing that it learned useful localization and naviga-

Deep learning algorithms have recently appeared that pretrain hidden layers of neural networks in unsupervised ways, leading to state-of-the-art performance on large classification problems. These methods can also pretrain networks used for reinforcement learning. However, this ignores the additional information that exists in a reinforcement learning paradigm via the ongoing sequence of state, action, new state tuples. This paper demonstrates that learning a predictive model of state dynamics can result in a pretrained hidden layer structure that reduces the time needed to solve reinforcement learning problems.

Exploration is a difficult challenge in reinforcement learning and is of prime importance in sparse reward environments. However, many of the state of the art deep reinforcement learning algorithms, that reply on epsilon-greedy exploration, fail on these environments. In such environments, empowerment can serve as an intrinsic reward signal to enable the agent to explore by maximizing the influence it has over the near future. We formulate empowerment as the channel capacity between actions and states and is calculated by estimating the mutual information between the actions and the following states. The mutual information is estimated using a Mutual Information Neural Estimator and a forward dynamics model. We demonstrate that an empowerment driven agent is able to improve significantly the score of a baseline DQN agent on the game of Montezuma’s Revenge.


Question: DeepMimic: Demonstration-directed deep reinforcement learning for physics-based character ability acquisition.

The Encoder-Recurrent-Decoder (ERD) Model: A Novel Approach for Understanding Human Motion Dynamics

This paper introduces the Encoder-Recurrent-Decoder (ERD) framework, a cutting-edge recurrent neural network architecture designed to address the recognition and prediction of human body pose in both video and motion capture scenarios. The ERD model innovates by integrating nonlinear encoder and decoder networks with recurrent layers, enabling it to handle complex spatio-temporal data. Its versatility is demonstrated through its application in motion capture (mocap) generation, body pose labeling, and video-based body pose forecasting.

In mocap generation, the ERD model showcases its ability to process training data across various subjects and activities, generating novel motions that remain stable over extended periods. This is achieved without the common issue of drifting, which is a significant challenge in many motion synthesis techniques.

For body pose labeling, the ERD model surpasses per-frame body part detectors by effectively resolving ambiguities regarding left-right body part identification. This capability is particularly crucial in scenarios where the orientation of the body is not clearly defined, a common occurrence in dynamic human motion.

In the domain of video pose forecasting, the ERD model predicts body joint displacements over a temporal horizon of 400 milliseconds, outperforming a first-order motion model based on optical flow. This improvement underscores the model's proficiency in capturing the complex dynamics of human motion.

The ERD model extends the capabilities of previous Long Short-Term Memory (LSTM) models by enabling the joint learning of representations and their dynamics. This feature is critical for both labeling and prediction tasks in space-time, setting it apart from straightforward, hard-coded representations that have shown success in 1D domains such as text, speech, or handwriting.

Through comprehensive experiments, the paper highlights the importance of representation learning in the spatio-temporal visual domain, emphasizing its role in achieving superior performance in human motion analysis compared to traditional methods.

We introduce a method for following high-level navigation instructions by mapping directly from images, instructions and pose estimates to continuous low-level velocity commands for real-time control. The Grounded Semantic Mapping Network (GSMN) is a fully-differentiable neural network architecture that builds an explicit semantic map in the world reference frame by incorporating a pinhole camera projection model within the network. The information stored in the map is learned from experience, while the local-to-world transformation is computed explicitly. We train the model using DAGGERFM, a modified variant of DAGGER that trades tabular convergence guarantees for improved training speed and memory use. We test GSMN in virtual environments on a realistic quadcopter simulator and show that incorporating an explicit mapping and grounding modules allows GSMN to outperform strong neural baselines and almost reach an expert policy performance. Finally, we analyze the learned map representations and show that using an explicit map leads to an interpretable instruction-following model.

In this work, we explore the possibility of training agents which are able to show intelligent behaviour in many different scenarios. We present the effectiveness of different machine learning algorithms in the StarCraft II Learning Environment [1] and their performance in different scenarios and compare them. In the end, we recreate DeepMind’s FullyConv Agent with slightly better results.

Exploration in an unknown environment is the core functionality for mobile robots. Learning-based exploration methods, including convolutional neural networks, provide excellent strategies without human-designed logic for the feature extraction [1]. But the conventional supervised learning algorithms cost lots of efforts on the labeling work of datasets inevitably. Scenes not included in the training set are mostly unrecognized either. We propose a deep reinforcement learning method for the exploration of mobile robots in an indoor environment with the depth information from an RGB-D sensor only. Based on the Deep Q-Network framework [2], the raw depth image is taken as the only input to estimate the Q values corresponding to all moving commands. The training of the network weights is end-to-end. In arbitrarily constructed simulation environments, we show that the robot can be quickly adapted to unfamiliar scenes without any man-made labeling. Besides, through analysis of receptive fields of feature representations, deep reinforcement learning motivates the convolutional networks to estimate the traversability of the scenes. The test results are compared with the exploration strategies separately based on deep learning [1] or reinforcement learning [3]. Even trained only in the simulated environment, experimental results in real-world environment demonstrate that the cognitive ability of robot controller is dramatically improved compared with the supervised method. We believe it is the first time that raw sensor information is used to build cognitive exploration strategy for mobile robots through end-to-end deep reinforcement learning.

Interpreting decision making logic in demonstration videos is key to collaborating with and mimicking humans. To empower machines with this ability, we propose a neural program synthesizer that is able to explicitly synthesize underlying programs from behaviorally diverse and visually complicated demonstration videos. We introduce a summarizer module as part of our model to improve the network’s ability to integrate multiple demonstrations varying in behavior. We also employ a multi-task objective to encourage the model to learn meaningful intermediate representations for end-to-end training. We show that our model is able to reliably synthesize underlying programs as well as capture diverse behaviors exhibited in demonstrations. The code is available at https://shaohua0116.github.io/demo2program.

Reinforcement Learning (RL) is a learning paradigm concerned with learning to control a system so as to maximize an objective over the long term. This approach to learning has received immense interest in recent times and success manifests itself in the form of human-level performance on games like Go. While RL is emerging as a practical component in reallife systems, most successes have been in Single Agent domains. This report will instead specifically focus on challenges that are unique to Multi-Agent Systems interacting in mixed cooperative and competitive environments. The report concludes with advances in the paradigm of training Multi-Agent Systems called Decentralized Actor, Centralized Critic, based on an extension of MDPs called Decentralized Partially Observable MDPs, which has seen a renewed interest lately. 1 ar X iv :1 80 7. 09 42 7v 1 [ cs .A I] 2 5 Ju l 2 01 8

The visual recognition of complex, articulated human movements is fundamental for a wide range of artificial systems oriented toward human-robot communication, action classification, and action-driven perception. These challenging tasks may generally involve the processing of a huge amount of visual information and learning-based mechanisms for generalizing a set of training actions and classifying new samples. To operate in natural environments, a crucial property is the efficient and robust recognition of actions, also under noisy conditions caused by, for instance, systematic sensor errors and temporarily occluded persons. Studies of the mammalian visual system and its outperforming ability to process biological motion information suggest separate neural pathways for the distinct processing of pose and motion features at multiple levels and the subsequent integration of these visual cues for action perception. We present a neurobiologically-motivated approach to achieve noise-tolerant action recognition in real time. Our model consists of self-organizing Growing When Required (GWR) networks that obtain progressively generalized representations of sensory inputs and learn inherent spatio-temporal dependencies. During the training, the GWR networks dynamically change their topological structure to better match the input space. We first extract pose and motion features from video sequences and then cluster actions in terms of prototypical pose-motion trajectories. Multi-cue trajectories from matching action frames are subsequently combined to provide action dynamics in the joint feature space. Reported experiments show that our approach outperforms previous results on a dataset of full-body actions captured with a depth sensor, and ranks among the best results for a public benchmark of domestic daily actions.

Deep neural networks (DNNs) applications are now increasingly pervasive and powerful. However, fashion designers are lagging behind in leveraging this increasingly common technology. DNNs are not yet a standard part of fashion de sign practice, either clothes patterns or prototyping tools. In this paper, we present DeepWear, a method using deep convolutional generative adversarial networks for clothes design. The DNNs learn the feature of specific brand clothes and generate images then patterns instructed from the images are made, and an author creates clothes based on that. We evaluated this system by evaluating the credibility of the actual sold clothes on market with our clothes. As the result, we found it is possible to make clothes look like actual products from the generated images. Our findings have implications for collaborative design between machine and human intelligence.

We propose an approach to estimate 3D human pose in real world units from a single RGBD image and show that it exceeds performance of monocular 3D pose estimation approaches from color as well as pose estimation exclusively from depth. Our approach builds on robust human keypoint detectors for color images and incorporates depth for lifting into 3D. We combine the system with our learning from demonstration framework to instruct a service robot without the need of markers. Experiments in real world settings demonstrate that our approach enables a PR2 robot to imitate manipulation actions observed from a human teacher.

We present an approach to learn the inverse kinematics of the “bionic handling assistant”-an elephant trunk robot. This task comprises substantial challenges including high dimensionality, restrictive and unknown actuation ranges, and nonstationary system behavior. We use a recent exploration scheme, online goal babbling, which deals with these challenges by bootstrapping and adapting the inverse kinematics on the fly. We show the success of the method in extensive real-world experiments on the nonstationary robot, including a novel combination of learning and traditional feedback control. Simulations further investigate the impact of nonstationary actuation ranges, drifting sensors, and morphological changes. The experiments provide the first substantial quantitative real-world evidence for the success of goal-directed bootstrapping schemes, moreover with the challenge of nonstationary system behavior. We thereby provide the first functioning control concept for this challenging robot platform.

We present a new real-time hand tracking system based on a single depth camera. The system can accurately reconstruct complex hand poses across a variety of subjects. It also allows for robust tracking, rapidly recovering from any temporary failures. Most uniquely, our tracker is highly flexible, dramatically improving upon previous approaches which have focused on front-facing close-range scenarios. This flexibility opens up new possibilities for human-computer interaction with examples including tracking at distances from tens of centimeters through to several meters (for controlling the TV at a distance), supporting tracking using a moving depth camera (for mobile scenarios), and arbitrary camera placements (for VR headsets). These features are achieved through a new pipeline that combines a multi-layered discriminative reinitialization strategy for per-frame pose estimation, followed by a generative model-fitting stage. We provide extensive technical details and a detailed qualitative and quantitative analysis.


Question: DeepMimic: Demonstration-driven deep reinforcement learning for physics-based character skill acquisition.

Introducing the Arcade Learning Environment (ALE): A Benchmark for General Artificial Intelligence

This article presents the Arcade Learning Environment (ALE), a platform and methodology designed to evaluate and foster the development of general, domain-independent AI technology. ALE offers a unique challenge problem by providing access to a vast library of Atari 2600 game environments, each with distinct characteristics and gameplay mechanics that were originally crafted to engage human players. The diversity of these games presents significant research challenges for various AI fields, including reinforcement learning, model learning, model-based planning, imitation learning, transfer learning, and intrinsic motivation.

The most notable feature of ALE is its ability to serve as a rigorous testbed for assessing and comparing approaches to these complex problems. To demonstrate ALE's potential, the article develops and benchmarks domain-independent agents using established AI techniques in reinforcement learning and planning. Through extensive experimentation across more than 55 games, the authors showcase the evaluation methodology enabled by ALE, providing empirical insights into the performance of these agents.

Importantly, all the software, including the benchmark agents, is made publicly available, encouraging the AI community to utilize ALE for further research and development. This initiative not only pushes the boundaries of AI capabilities but also promotes a collaborative environment for scientists and engineers to refine and test their algorithms in a rich, dynamic, and challenging environment.

With recent progress in graphics, it has become more tractable to train models on synthetic images, potentially avoiding the need for expensive annotations. However, learning from synthetic images may not achieve the desired performance due to a gap between synthetic and real image distributions. To reduce this gap, we propose Simulated+Unsupervised (S+U) learning, where the task is to learn a model to improve the realism of a simulators output using unlabeled real data, while preserving the annotation information from the simulator. We develop a method for S+U learning that uses an adversarial network similar to Generative Adversarial Networks (GANs), but with synthetic images as inputs instead of random vectors. We make several key modifications to the standard GAN algorithm to preserve annotations, avoid artifacts, and stabilize training: (i) a self-regularization term, (ii) a local adversarial loss, and (iii) updating the discriminator using a history of refined images. We show that this enables generation of highly realistic images, which we demonstrate both qualitatively and with a user study. We quantitatively evaluate the generated images by training models for gaze estimation and hand pose estimation. We show a significant improvement over using synthetic images, and achieve state-of-the-art results on the MPIIGaze dataset without any labeled real data.

Deep reinforcement learning (RL) can acquire complex behaviors from low-level inputs, such as images. However, real-world applications of such methods require generalizing to the vast variability of the real world. Deep networks are known to achieve remarkable generalization when provided with massive amounts of labeled data, but can we provide this breadth of experience to an RL agent, such as a robot? The robot might continuously learn as it explores the world around it, even while it is deployed and performing useful tasks. However, this learning requires access to a reward function, to tell the agent whether it is succeeding or failing at its task. Such reward functions are often hard to measure in the real world, especially in domains such as robotics and dialog systems, where the reward could depend on the unknown positions of objects or the emotional state of the user. On the other hand, it is often quite practical to provide the agent with reward functions in a limited set of situations, such as when a human supervisor is present, or in a controlled laboratory setting. Can we make use of this limited supervision, and still benefit from the breadth of experience an agent might collect in the unstructured real world? In this paper, we formalize this problem setting as semisupervised reinforcement learning (SSRL), where the reward function can only be evaluated in a set of “labeled” MDPs, and the agent must generalize its behavior to the wide range of states it might encounter in a set of “unlabeled” MDPs, by using experience from both settings. Our proposed method infers the task objective in the unlabeled MDPs through an algorithm that resembles inverse RL, using the agent’s own prior experience in the labeled MDPs as a kind of demonstration of optimal behavior. We evaluate our method on challenging, continuous control tasks that require control directly from images, and show that our approach can improve the generalization of a learned deep neural network policy by using experience for which no reward function is available. We also show that our method outperforms direct supervised learning of the reward.

Though successful in high-dimensional domains, deep reinforcement learning exhibits high sample complexity and suffers from stability issues as reported by researchers and practitioners in the field. These problems hinder the application of such algorithms in real-world and safety-critical scenarios. In this paper, we take steps towards stable and efficient reinforcement learning by following a model-based approach that is known to reduce agent-environment interactions. Namely, our method augments deep Q-networks (DQNs) with model predictions for transitions, rewards, and termination flags. Having the model at hand, we then conduct a rigorous theoretical study of our algorithm and show, for the first time, convergence to a stationary point. En route, we provide a counterexample showing that ’vanilla’ DQNs can diverge confirming practitioners’ and researchers’ experiences. Our proof is novel in its own right and can be extended to other forms of deep reinforcement learning. In particular, we believe exploiting the relation between reinforcement (with deep function approximators) and online learning can serve as a recipe for future proofs in the domain. Finally, we validate our theoretical results in 20 games from the Atari benchmark. Our results show that following the proposed model-based learning approach not only ensures convergence but leads to a reduction in sample complexity and superior performance. Introduction Model-free deep reinforcement learning methods have recently demonstrated impressive performance on a range of complex learning tasks (Hessel et al. 2018; Lillicrap et al. 2017; Jaderberg et al. 2017). Deep Q-Networks (DQNs) (Mnih et al. 2015), in particular, stand out as a versatile and effective tool for a wide range of applications. DQNs offer an end-to-end solution for many reinforcement learning problems by generalizing tabular Q-learning to highdimensional input spaces. Unfortunately, DQNs still suffer from a number of important drawbacks that hinder wide-spread adoption. There is a general lack of theoretical understanding to guide the use of deep Q-learning algorithms. Combining non-linear function approximation with off-policy reinforcement learning is known to lead to possible divergence issues, even in very simple tasks (Boyan and Preliminary work. Moore 1995). Despite improvements introduced by deep Qnetworks (Mnih et al. 2015), Q-learning approaches based on deep neural networks still exhibit occasional instability during the learning process, see for example (van Hasselt, Guez, and Silver 2016). In this paper, we aim to address the instability of current deep RL approaches by augmenting model-free value learning with a predictive model. We propose a hybrid model and value learning objective, which stabilises the learning process and reduces sample complexity when compared to model-free algorithms. We attribute this reduction to the richer feedback signal provided by our method compared to standard deep Q-networks. In particular, our model provides feedback at every time step based on the current prediction error, which in turn eliminates one source of sample inefficiency caused by sparse rewards. These conclusions are also in accordance with previous research conducted on linear function approximation in reinforcement learning (Parr et al. 2008; Sutton et al. 2012; Song et al. 2016). While linear function approximation results provide a motivation for model-based stabilisation, such theories fail to generalise to deep non-linear architectures (Song et al. 2016). To close this gap in the literature and demonstrate stability of our method, we prove convergence in the general deep RL setting with deep value function approximators. Theoretically analysing deep RL algorithms, however, is challenging due to nonlinear functional dependencies and objective non-convexities prohibiting convergence to globally optimal policies. As such, the best one would hope for is a stationary point (e.g., a first-order one) that guarantees vanishing gradients. Even understanding gradient behavior for deep networks in RL can be difficult due to explorationbased policies and replay memory considerations. To alleviate some of these problems, we map deep reinforcement learning to a mathematical framework explicitly designed for understanding the exploration/exploitation trade-off. Namely, we formalise the problem as an online learning game the agent plays against an adversary (i.e., the environment). Such a link allows us to study a more general problem combining notions of regret, reinforcement learning, and optimisation. Given such a link, we prove that for any > 0, regret vanishes as O ( T 1−2 lnT ) with T being the total number of rounds. This, in turn, guarantees convergence to a stationary point. ar X iv :1 80 9. 01 90 6v 1 [ cs .L G ] 6 S ep 2 01 8 Guided by the above theoretical results, we validate that our algorithm leads to faster learning by conducting experiments on 20 Atari games. Due to the high computational and budgeting demands, we chose a benchmark that is closest to our setting. Concretely, we picked DQNs as our template algorithm to improve on both theoretically and empirically. It is worth noting that extending our theoretical results to more complex RL algorithms is an orthogonal future direction. Background: Reinforcement Learning We consider a discrete time, infinite horizon, discounted Markov Decision Process (MDP) 〈 S,A, P , P , γ 〉 . Here S denotes the state space, A the set of possible actions, P (R) : S × A × R → [0, 1] is the reward function, P (S) : S × A × S → [0, 1] is the state transition function, and γ ∈ (0, 1) is a discount factor. An agent, being in state St ∈ S at time step t, executes an action at ∈ A sampled from its policy π(at|St), a conditional probability distribution where π : S × A → [0, 1]. The agent’s action elicits from the environment a reward signal rt ∈ R indicating instantaneous reward, a terminal flag ft ∈ {0, 1} indicating a terminal event that restarts the environment, and a transition to a successor state St+1 ∈ S. We assume that the sets S, A, and R are discrete. The reward rt is sampled from the conditional probability distribution P (rt|St,at). Similarly, ft ∼ P (F (ft|St,at) with P (F ) : S × A × {0, 1} → [0, 1], where a terminal event (ft = 1) restarts the environment according to some initial state distribution S0 ∼ P (S) 0 (S0). The state transition to a successor state is determined by a stochastic state transition function according to St+1 ∼ P (St+1|St,at). The agent’s goal is to maximise future cumulative reward E P (S) 0 ,π,P (R),P (F ),P (S) [ ∞ ∑ t=0 ( t ∏ t′=0 (1− ft′−1) ) γrt ] with respect to the policy π. An important quantity in RL are Q-values Q(St,at), which are defined as the expected future cumulative reward value when executing action at in state St and subsequently following policy π. Qvalues enable us to conveniently phrase the RL problem as maxπ ∑ S p (S) ∑ a π(a|S)Q(S,a), where p(S) is the stationary state distribution obtained when executing π in the environment (starting from S0 ∼ P (S) 0 (S0)). Deep Q-Networks: Value-based reinforcement learning approaches identify optimal Q-values directly using parametric function approximators Qθ(S,a), where θ ∈ Θ represents the parameters (Watkins 1989; Busoniu et al. 2010). Optimal Q-value estimates then correspond to an optimal policy π(a|S) = δa,argmaxa′Qθ(S,a′). Deep Q-networks (Mnih et al. 2015) learn a deep neural network based Qvalue approximation by performing stochastic gradient descent on the following training objective: L(θ) = ES,a,r,f,S′ [( r + (1− f)γmax a′ Qθ−(S ′,a′) −Qθ(S,a) )2] . (1) The expectation ranges over transitions S,a, r, f,S′ sampled from an experience replay memory (S′ denotes the state at the next time step). Use of this replay memory, together with the use of a separate target network Qθ− (with different parameters θ− ∈ Θ) for calculating the bootstrap values maxa′ Qθ−(S ′,a′), helps stabilise the learning process. Background: Online Learning In this paper, we employ a special form of regret minimisation games that we briefly review here. A regret minimisation game is a triple 〈Θ,F , T 〉, where Θ is a non-empty decision set, F is the set of moves of the adversary which contains bounded convex functions from R to R and T is the total number of rounds. The game commences in rounds, where at round j = 1, . . . , T , the agent chooses a prediction θj ∈ Θ and the adversary a loss function Lj ∈ F . At the end of the round, the adversary reveals its choice and the agent suffers a loss Lj(θj). In this paper, we are concerned with the full-information case where the agent can observe the complete loss function Lj at the end of each round. The goal of the game is for the agent to make successive predictions to minimise cumulative regret defined as:

We present the first real-time human performance capture approach that reconstructs dense, space-time coherent deforming geometry of entire humans in general everyday clothing from just a single RGB video. We propose a novel two-stage analysis-by-synthesis optimization whose formulation and implementation are designed for high performance. In the first stage, a skinned templatemodel is jointly fitted to background subtracted input video, 2D and 3D skeleton joint positions found using a deep neural network, and a set of sparse facial landmark detections. In the second stage, dense non-rigid 3D deformations of skin and even loose apparel are captured based on a novel real-time capable algorithm for non-rigid tracking using dense photometric and silhouette constraints. Our novel energy formulation leverages automatically identified material regions on the template to model the differing non-rigid deformation behavior of skin and apparel. The two resulting nonlinear optimization problems per-frame are solved with specially-tailored data-parallel Gauss-Newton solvers. In order to achieve real-time performance of over 25Hz, we design a pipelined parallel architecture using the CPU and two commodity GPUs. Our method is the first real-time monocular approach for full-body performance capture. Our method yields comparable accuracy with off-line performance capture techniques, while being orders of magnitude faster.

Exploration remains a significant challenge to reinforcement learning methods, especially in environments where reward signals are sparse. Recent methods of learning from demonstrations have shown to be promising in overcoming exploration difficulties but typically require considerable highquality demonstrations that are difficult to collect. We propose to effectively leverage available demonstrations to guide exploration through enforcing occupancy measure matching between the learned policy and current demonstrations, and develop a novel Policy Optimization from Demonstration (POfD) method. We show that POfD induces implicit dynamic reward shaping and brings provable benefits for policy improvement. Furthermore, it can be combined with policy gradient methods to produce state-of-the-art results, as demonstrated experimentally on a range of popular benchmark sparse-reward tasks, even when the demonstrations are few and imperfect.

Improv is a system for the creation of real−time behavior−based animated actors. There have been seve recent efforts to build network distributed autonomous agent But in general these efforts do not focus on the author’s view To create rich interactive worlds inhabited by believable animated actors, authors need the proper tools. Im prov provides tools to create actors that respond to users and each other in real−time, with personalities and mood consistent with the author’s goals and intentions. Improv consists of two subsystems. The first subsystem is an Animation Engine that uses procedur techniques to enable authors to create layered, continuo non−repetitive motions and smooth transitions between the The second subsystem is a Behavior Engine that enab authors to create sophisticated rules governing how acto communicate, change, and make decisions. The combin system provides an integrated set of tools for authoring th "minds" and "bodies" of interactive actors. The system uses a english−style scripting language so that creative experts wh are not primarily programmers can create powerful interactiv applications.

In this paper, we propose a deep reinforcement learning(DRL) algorithm which combines Deep Deterministic Policy Gradient (DDPG) with expert demonstrations and supervised loss for decision making for autonomous driving. Training DRL agent with supervised learning is adopted to accelerate the exploration process and increase the stability. A supervised loss function is introduced in the algorithm to update the actor networks. In addition, reward construction is combined to make the training process more stable and efficient. The proposed algorithm is applied to a popular autonomous driving simulator called TORCS. The experimental results show that the training efficiency and stability are improved by utilizing our algorithm in autonomous driving.

We present Project Malmo – an AI experimentation platform built on top of the popular computer game Minecraft, and designed to support fundamental research in artificial intelligence. As the AI research community pushes for artificial general intelligence (AGI), experimentation platforms are needed that support the development of flexible agents that learn to solve diverse tasks in complex environments. Minecraft is an ideal foundation for such a platform, as it exposes agents to complex 3D worlds, coupled with infinitely varied game-play. Project Malmo provides a sophisticated abstraction layer on top of Minecraft that supports a wide range of experimentation scenarios, ranging from navigation and survival to collaboration and problem solving tasks. In this demo we present the Malmo platform and its capabilities. The platform is publicly released as open source software at IJCAI, to support openness and collaboration in AI research.

Deep reinforcement learning is becoming increasingly popular for robot control algorithms, with the aim for a robot to self-learn useful feature representations from unstructured sensory input leading to the optimal actuation policy. In addition to sensors mounted on the robot, sensors might also be deployed in the environment, although these might need to be accessed via an unreliable wireless connection. In this paper, we demonstrate deep neural network architectures that are able to fuse information generated by multiple sensors and are robust to sensor failures at runtime. We evaluate our method on a search and pick task for a robot both in simulation and the real world.

This paper explores the use of deep reinforcement learning agents to transfer knowledge from one environment to another. More specifically, the method takes advantage of asynchronous advantage actor critic (A3C) architecture to generalize a target game using an agent trained on a source game in Atari. Instead of fine-tuning a pre-trained model for the target game, we propose a learning approach to update the model using multiple agents trained in parallel with different representations of the target game. Visual mapping between video sequences of transfer pairs is used to derive new representations of the target game; training on these visual representations of the target game improves model updates in terms of performance, data efficiency and stability. In order to demonstrate the functionality of the architecture, Atari games Pong-v0 and Breakout-v0 are being used from the OpenAI gym environment; as the source and target environment.