Question: Speed versus accuracy: Creating an optimal ASR system for spontaneous non-native speech in real-time applications.

This paper explores the enhancement of neural networks through a technique aimed at preventing co-adaptation among feature detectors, thereby improving their generalizability. When training large neural networks with small datasets, overfitting often occurs, resulting in poor performance on unseen data. The proposed method, termed "dropout," addresses this issue by randomly omitting half of the feature detectors during each training instance. This strategy prevents intricate co-adaptations, where feature detectors only function optimally in the presence of specific other detectors. Instead, each neuron learns to detect features that are universally beneficial for accurate predictions, regardless of the internal context in which they operate. Dropout significantly enhances performance on benchmark tasks, setting new records in speech and object recognition. The concept of dropout can be seen as an efficient model averaging technique for neural networks. Typically, reducing test set error involves averaging predictions from numerous distinct networks, a process that is computationally intensive. Dropout, however, enables the training of a vast number of different networks in an efficient manner, with each training instance potentially representing a unique network configuration. Despite this diversity, all networks share weights for the active hidden units, facilitating a streamlined training and testing process. The training of dropout neural networks utilizes standard stochastic gradient descent on mini-batches of training cases, with a modified penalty term to restrict weight growth. Instead of penalizing the entire weight vector's squared length (L2 norm), an upper bound is set on the L2 norm of the incoming weight vector for each hidden unit. If a weight update exceeds this limit, the weights are renormalized by division, preventing excessive growth. This constraint-based approach ensures that weights remain manageable, regardless of the magnitude of proposed updates, allowing for a controlled training process. We begin by

We present EMPHASIS, an emotional phoneme-based acoustic model for speech synthesis system. EMPHASIS includes a phoneme duration prediction model and an acoustic parameter prediction model. It uses a CBHG-based regression network to model the dependencies between linguistic features and acoustic features. We modify the input and output layer structures of the network to improve the performance. For the linguistic features, we apply a feature grouping strategy to enhance emotional and prosodic features. The acoustic parameters are designed to be suitable for the regression task and waveform reconstruction. EMPHASIS can synthesize speech in real-time and generate expressive interrogative and exclamatory speech with high audio quality. EMPHASIS is designed to be a multi-lingual model and can synthesize Mandarin-English speech for now. In the experiment of emotional speech synthesis, it achieves better subjective results than other real-time speech synthesis systems.

The Development Workflow There are two major stages within isolated word recognition: a training stage and a testing stage. Training involves “teaching” the system by building its dictionary, an acoustic model for each word that the system needs to recognize. In our example, the dictionary comprises the digits ‘zero’ to ‘nine’. In the testing stage we use acoustic models of these digits to recognize isolated words using a classification algorithm. The development workflow consists of three steps: Speech acquisition • Speech analysis • User interface development •

We provide a speech coding scheme employing a generative model based on SampleRNN that, while operating at significantly lower bitrates, matches or surpasses the perceptual quality of state-of-theart classic wide-band codecs. Moreover, it is demonstrated that the proposed scheme can provide a meaningful rate-distortion trade-off without retraining. We evaluate the proposed scheme in a series of listening tests and discuss limitations of the approach.

Speech recognition is largely taking advantage of deep learning, showing that substantial benefits can be obtained by modern Recurrent Neural Networks (RNNs). The most popular RNNs are Long Short-Term Memory (LSTMs), which typically reach state-of-the-art performance in many tasks thanks to their ability to learn long-term dependencies and robustness to vanishing gradients. Nevertheless, LSTMs have a rather complex design with three multiplicative gates, that might impair their efficient implementation. An attempt to simplify LSTMs has recently led to Gated Recurrent Units (GRUs), which are based on just two multiplicative gates. This paper builds on these efforts by further revising GRUs and proposing a simplified architecture potentially more suitable for speech recognition. The contribution of this work is two-fold. First, we suggest to remove the reset gate in the GRU design, resulting in a more efficient single-gate architecture. Second, we propose to replace tanh with ReLU activations in the state update equations. Results show that, in our implementation, the revised architecture reduces the per-epoch training time with more than 30% and consistently improves recognition performance across different tasks, input features, and noisy conditions when compared to a standard GRU.

The importance of automatically recognizing emotions from human speech has grown with the increasing role of spoken language interfaces in human-computer interaction applications. This paper explores the detection of domain-specific emotions using language and discourse information in conjunction with acoustic correlates of emotion in speech signals. The specific focus is on a case study of detecting negative and non-negative emotions using spoken language data obtained from a call center application. Most previous studies in emotion recognition have used only the acoustic information contained in speech. In this paper, a combination of three sources of information-acoustic, lexical, and discourse-is used for emotion recognition. To capture emotion information at the language level, an information-theoretic notion of emotional salience is introduced. Optimization of the acoustic correlates of emotion with respect to classification error was accomplished by investigating different feature sets obtained from feature selection, followed by principal component analysis. Experimental results on our call center data show that the best results are obtained when acoustic and language information are combined. Results show that combining all the information, rather than using only acoustic information, improves emotion classification by 40.7% for males and 36.4% for females (linear discriminant classifier used for acoustic information).

Assistive speech-based technologies can improve the quality of life for people affected with dysarthria, a motor speech disorder. In this paper, we explore multiple ways to improve Gaussian mixture model and deep neural network (DNN) based hidden Markov model (HMM) automatic speech recognition systems for TORGO dysarthric speech database. This work shows significant improvements over the previous attempts in building such systems in TORGO. We trained speaker-specific acoustic models by tuning various acoustic model parameters, using speaker normalized cepstral features and building complex DNN-HMM models with dropout and sequence-discrimination strategies. The DNN-HMM models for severe and severe-moderate dysarthric speakers were further improved by leveraging specific information from dysarthric speech to DNN models trained on audio files from both dysarthric and normal speech, using generalized distillation framework. To the best of our knowledge, this paper presents the best recognition accuracies for TORGO database till date.

Current dialogue systems generally operate in a pipelined, modular fashion on one complete utterance at a time. Evidence from human language understanding shows that human understanding operates incrementally and makes use of multiple sources of information during the parsing process, including traditionally “later” components such as pragmatics. In this paper we describe a spoken dialogue system that understands language incrementally, provides visual feedback on possible referents during the course of the user’s utterance, and allows for overlapping speech and actions. We further present findings from an empirical study showing that the resulting dialogue system is faster overall than its nonincremental counterpart. Furthermore, the incremental system is preferred to its nonincremental counterpart – beyond what is accounted for by factors such as speed and accuracy. These results indicate that successful incremental understanding systems will improve both performance and usability.

Recently Long Short-Term Memory (LSTM) Recurrent Neural Networks (RNN) acoustic models have demonstrated superior performance over deep neural networks (DNN) models in speech recognition and many other tasks. Although a lot of work have been reported on DNN model adaptation, very little has been done on LSTM model adaptation. In this paper we present our extensive studies of speaker adaptation of LSTM-RNN models for speech recognition. We investigated different adaptation methods combined with KL-divergence based regularization, where and which network component to adapt, supervised versus unsupervised adaptation and asymptotic analysis. We made a few distinct and important observations. In a large vocabulary speech recognition task, by adapting only 2.5% of the LSTM model parameters using 50 utterances per speaker, we obtained 12.6% WERR on the dev set and 9.1% WERR on the evaluation set over a strong LSTM baseline model.

Distant speech recognition (DSR) holds out the promise of providing a natural human computer interface in that it enables verbal interactions with computers without the necessity of donning intrusive body- or head-mounted devices. Recognizing distant speech robustly, however, remains a challenge. This paper provides a overview of DSR systems based on microphone arrays. In particular, we present recent work on acoustic beamforming for DSR, along with experimental results verifying the effectiveness of the various algorithms described here; beginning from a word error rate (WER) of 14.3% with a single microphone of a 64-channel linear array, our state-of-the-art DSR system achieved a WER of 5.3%, which was comparable to that of 4.2% obtained with a lapel microphone. Furthermore, we report the results of speech recognition experiments on data captured with a popular device, the Kinect [1]. Even for speakers at a distance of four meters from the Kinect, our DSR system achieved acceptable recognition performance on a large vocabulary task, a WER of 24.1%, beginning from a WER of 42.5% with a single array channel.

This paper describes connectionist techniques for recogni ti n of Broadcast News. The fundamental difference between connectionist systems and more convent ional mixture-of-Gaussian systems is that connectionist models directly estimate posterior probabi lities as opposed to likelihoods. Access to posterior probabilities has enabled us to develop a number of no vel approaches to confidence estimation, pronunciation modelling and search. In addition we have inv stigated a new feature extraction technique based on the modulation-filtered spectrogram, and methods f or combining multiple information sources. We have incorporated all of these techniques into a system fo r the transcription of Broadcast News, and we present results on the 1998 DARPA Hub-4E Broadcast News ev aluation data.


Question: An alternating optimization and quadrature technique for robust reinforcement learning algorithms.

This paper presents PILCO, an innovative and data-efficient method for model-based policy search. PILCO is designed to mitigate model bias, a significant challenge in model-based reinforcement learning, through a systematic approach. By constructing a probabilistic dynamics model and explicitly accounting for model uncertainty in long-term planning, PILCO is capable of handling minimal data and enables learning from the initial stages with just a few trials. The evaluation of policies is executed in closed form, leveraging cutting-edge approximate inference techniques. Additionally, policy gradients are calculated analytically to enhance policy refinement. The paper showcases the exceptional learning efficiency of PILCO on complex, high-dimensional control tasks, outperforming previous benchmarks.

**Question**: A Direct Search Method to solve Economic Dispatch Problem with Valve-Point Effect

**Documents**: An Improved Cuckoo Search Algorithm for Global Optimization
The cuckoo search (CS) algorithm is a recent optimization algorithm developed based on the obligate brood parasitism of some cuckoo species and the Lévy flight behavior of some birds and insects. In this paper, an improved cuckoo search algorithm is proposed to solve global optimization problems. The proposed algorithm, called the Improved Cuckoo Search (ICS) algorithm, modifies the original CS algorithm by introducing a new search strategy and an adaptive adjustment mechanism for the parameters. The new search strategy combines the advantages of the original CS algorithm and the differential evolution (DE) algorithm to enhance the search ability and the convergence speed of the algorithm. The adaptive adjustment mechanism adjusts the parameters according to the search progress to maintain a good balance between exploration and exploitation. The performance of the ICS algorithm is tested on a set of benchmark functions and compared with other state-of-the-art optimization algorithms. The experimental results show that the ICS algorithm outperforms the original CS algorithm and other optimization algorithms in terms of the solution quality and the convergence speed.

**Similar Documents**: This paper presents an Enhanced Cuckoo Search (ECS) algorithm, a refined version of the cuckoo search (CS) optimization method. The original CS algorithm, inspired by the brood parasitism of certain cuckoo species and the Lévy flight patterns observed in birds and insects, is improved in the ECS algorithm through the introduction of a novel search strategy and an adaptive parameter adjustment mechanism. The ECS algorithm integrates the strengths of the CS algorithm with elements from the differential evolution (DE) algorithm to bolster both the search capability and the rate of convergence. The adaptive parameter adjustment mechanism ensures that the algorithm dynamically balances exploration and exploitation by adjusting parameters in accordance with

We prove a quantitative connection between the expected sum of rewards of a policy and binary classification performance on created subproblems. This connection holds without any unobservable assumptions (no assumption of independence, small mixing time, fully observable states, or even hidden states) and the resulting statement is independent of the number of states or actions. The statement is critically dependent on the size of the rewards and prediction performance of the created classifiers.We also provide some general guidelines for obtaining good classification performance on the created subproblems. In particular, we discuss possible methods for generating training examples for a classifier learning algorithm.

TheSGD-QN algorithm is a stochastic gradient descent algorithm that m akes careful use of secondorder information and splits the parameter update into inde pendently scheduled components. Thanks to this design, SGD-QN iterates nearly as fast as a first-order stochastic gradient escent but requires less iterations to achieve the same accuracy. This algorith m won the “Wild Track” of the first PASCAL Large Scale Learning Challenge (Sonnenburg et al., 2008 ).

The key approaches for machine learning, particularly learning in unknown probabilistic environments, are new representations and computation mechanisms. In this paper, a novel quantum reinforcement learning (QRL) method is proposed by combining quantum theory and reinforcement learning (RL). Inspired by the state superposition principle and quantum parallelism, a framework of a value-updating algorithm is introduced. The state (action) in traditional RL is identified as the eigen state (eigen action) in QRL. The state (action) set can be represented with a quantum superposition state, and the eigen state (eigen action) can be obtained by randomly observing the simulated quantum state according to the collapse postulate of quantum measurement. The probability of the eigen action is determined by the probability amplitude, which is updated in parallel according to rewards. Some related characteristics of QRL such as convergence, optimality, and balancing between exploration and exploitation are also analyzed, which shows that this approach makes a good tradeoff between exploration and exploitation using the probability amplitude and can speedup learning through the quantum parallelism. To evaluate the performance and practicability of QRL, several simulated experiments are given, and the results demonstrate the effectiveness and superiority of the QRL algorithm for some complex problems. This paper is also an effective exploration on the application of quantum computation to artificial intelligence.

Robotics controllers often consist of expert-designed heuristics, which can be hard to tune in higher dimensions. Simulation can aid in optimizing these controllers if parameters learned in simulation transfer to hardware. Unfortunately, this is often not the case in legged locomotion, necessitating learning directly on hardware. This motivates using data-efficient learning techniques like Bayesian Optimization (BO) to minimize collecting expensive data samples. BO is a black-box data-efficient optimization scheme, though its performance typically degrades in higher dimensions. We aim to overcome this problem by incorporating domain knowledge, with a focus on bipedal locomotion. In our previous work, we proposed a feature transformation that projected a 16-dimensional locomotion controller to a 1-dimensional space using knowledge of human walking. When optimizing a human-inspired neuromuscular controller in simulation, this feature transformation enhanced sample efficiency of BO over traditional BO with a Squared Exponential kernel. In this paper, we present a generalized feature transform applicable to non-humanoid robot morphologies and evaluate it on the ATRIAS bipedal robot, in both simulation and hardware. We present three different walking controllers and two are evaluated on the real robot. Our results show that this feature transform captures important aspects of walking and accelerates learning on hardware and simulation, as compared to traditional BO.

While the need for hierarchies within control systems is apparent, it is also clear to many researchers that such hierarchies should be learned. Learning both the structure and the component behaviors is a di cult task. The bene t of learning the hierarchical structures of behaviors is that the decomposition of the control structure into smaller transportable chunks allows previously learned knowledge to be applied to new but related tasks. Presented in this paper are improvements to Nested Q-learning (NQL) that allow more realistic learning of control hierarchies in reinforcement environments. Also presented is a simulation of a simple robot performing a series of related tasks that is used to compare both hierarchical and non-hierarchal learning techniques.

We present the tool Q UASY, a quantitative synthesis tool. Q UASY takes qualitative and quantitative specifications and automatic ally onstructs a system that satisfies the qualitative specification and optimizes the qu antitative specification, if such a system exists. The user can choose between a system that satisfies and optimi zes the specifications (a) under all possible environment behaviors or (b) under th most-likely environment behaviors given as a probability distribution on the po ssible input sequences. QUASY solves these two quantitative synthesis problems by reduct ion to instances of 2-player games and Markov Decision Processes (MDPs) with qua ntitative winning objectives. QUASY can also be seen as a game solver for quantitative games. Most notable, it can solve lexicographic mean-payoff games with 2 players, MDPs with meanpayoff objectives, and ergodic MDPs with mean-payoff parit y objectives.

This paper presents the HDG learning algorithm, which uses a hierarchical decomposition of the state space to make learning to achieve goals more efficient with a small penalty in path quality. Special care must be taken when performing hierarchical planning and learning in stochastic domains, because macro-operators cannot be executed ballistically. The HDG algorithm, which is a descendent of Watkins’ Q-learning algorithm, is described here and preliminary empirical results are presented.

Reinforcement learning agents have traditionally been evaluated on small toy problems. With advances in computing power and the advent of the Arcade Learning Environment, it is now possible to evaluate algorithms on diverse and difficult problems within a consistent framework. We discuss some challenges posed by the arcade learning environment which do not manifest in simpler environments. We then provide a comparison of model-free, linear learning algorithms on this challenging problem set.

Motivated by a real-world problem, we study a novel budgeted optimization problem where the goal is to optimize an unknown function f(·) given a budget by requesting a sequence of samples from the function. In our setting, however, evaluating the function at precisely specified points is not practically possible due to prohibitive costs. Instead, we can only request constrained experiments. A constrained experiment, denoted by Q, specifies a subset of the input space for the experimenter to sample the function from. The outcome of Q includes a sampled experiment x, and its function output f(x). Importantly, as the constraints of Q become looser, the cost of fulfilling the request decreases, but the uncertainty about the location x increases. Our goal is to manage this trade-off by selecting a set of constrained experiments that best optimize f(·) within the budget. We study this problem in two different settings, the non-sequential (or batch) setting where a set of constrained experiments is selected at once, and the sequential setting where experiments are selected one at a time. We evaluate our proposed methods for both settings using synthetic and real functions. The experimental results demonstrate the efficacy of the proposed methods.

Learning, planning, and representing knowledge at multiple levels of temporal abstraction are key, longstanding challenges for AI. In this paper we consider how these challenges can be addressed within the mathematical framework of reinforcement learning and Markov decision processes (MDPs). We extend the usual notion of action in this framework to include options—closed-loop policies for taking action over a period of time. Examples of options include picking up an object, going to lunch, and traveling to a distant city, as well as primitive actions such as muscle twitches and joint torques. Overall, we show that options enable temporally abstract knowledge and action to be included in the reinforcement learning framework in a natural and general way. In particular, we show that options may be used interchangeably with primitive actions in planning methods such as dynamic programming and in learning methods such as Q-learning. Formally, a set of options defined over an MDP constitutes a semi-Markov decision process (SMDP), and the theory of SMDPs provides the foundation for the theory of options. However, the most interesting issues concern the interplay between the underlying MDP and the SMDP and are thus beyond SMDP theory. We present results for three such cases: (1) we show that the results of planning with options can be used during execution to interrupt options and thereby perform even better than planned, (2) we introduce new intra-option methods that are able to learn about an option from fragments of its execution, and (3) we propose a notion of subgoal that can be used to improve the options themselves. All of these results have precursors in the existing literature; the contribution of this paper is to establish them in a simpler and more general setting with fewer changes to the existing reinforcement learning framework. In particular, we show that these results can be obtained without committing to (or ruling out) any particular approach to state abstraction, hierarchy, function approximation, or the macroutility problem. ! 1999 Published by Elsevier Science B.V. All rights reserved. ∗ Corresponding author. 0004-3702/99/$ – see front matter ! 1999 Published by Elsevier Science B.V. All rights reserved. PII: S0004-3702(99)00052 -1 182 R.S. Sutton et al. / Artificial Intelligence 112 (1999) 181–211


Question: An alternating optimization and quadrature technique for robust reinforcement learning.

This study delves into the experimental assessment of model-based techniques for enhancing the performance of parameterized randomized algorithms, specifically focusing on Gaussian process models, which are the most prevalent in this domain. Among the methods explored, Sequential Parameter Optimization (SPO) [4] emerged as the most reliable and effective. To further refine the SPO methodology, we scrutinized critical design choices within its framework, elucidating the performance implications of each. Inspired by these insights, we introduce an enhanced variant of SPO, termed SPO+, which incorporates an innovative intensification strategy and utilizes log-transformed response values. Our empirical results in a domain where comparative data from other parameter optimization methods (that are model-free) exist, showcase that SPO+ delivers cutting-edge performance, positioning it as a leading approach in this field.

We propose a new approach to value function approximation which combines linear temporal difference reinforcement learning with subspace identification. In practical applications, reinforcement learning (RL) is complicated by the fact that state is either high-dimensional or partially observable. Therefore, RL methods are designed to work with features of state rather than state itself, and the success or failure of learning is often determined by the suitability of the selected features. By comparison, subspace identification (SSID) methods are designed to select a feature set which preserves as much information as possible about state. In this paper we connect the two approaches, looking at the problem of reinforcement learning with a large set of features, each of which may only be marginally useful for value function approximation. We introduce a new algorithm for this situation, called Predictive State Temporal Difference (PSTD) learning. As in SSID for predictive state representations, PSTD finds a linear compression operator that projects a large set of features down to a small set that preserves the maximum amount of predictive information. As in RL, PSTD then uses a Bellman recursion to estimate a value function. We discuss the connection between PSTD and prior approaches in RL and SSID. We prove that PSTD is statistically consistent, perform several experiments that illustrate its properties, and demonstrate its potential on a difficult optimal stopping problem.

Bayesian optimization is a prominent method for optimizing expensive-to-evaluate black-box functions that is widely applied to tuning the hyperparameters of machine learning algorithms. Despite its successes, the prototypical Bayesian optimization approach – using Gaussian process models – does not scale well to either many hyperparameters or many function evaluations. Attacking this lack of scalability and flexibility is thus one of the key challenges of the field. We present a general approach for using flexible parametric models (neural networks) for Bayesian optimization, staying as close to a truly Bayesian treatment as possible. We obtain scalability through stochastic gradient Hamiltonian Monte Carlo, whose robustness we improve via a scale adaptation. Experiments including multi-task Bayesian optimization with 21 tasks, parallel optimization of deep neural networks and deep reinforcement learning show the power and flexibility of this approach.

We address the problem of inverse reinforcement learning in Markov decision processes where the agent is risk-sensitive. We derive a risk-sensitive reinforcement learning algorithm with convergence guarantees that employs convex risk metrics and models of human decisionmaking deriving from behavioral economics. The risk-sensitive reinforcement learning algorithm provides the theoretical underpinning for a gradient-based inverse reinforcement learning algorithm that minimizes a loss function defined on observed behavior of a risk-sensitive agent. We demonstrate the performance of the proposed technique on two examples: (i) the canonical Grid World example and (ii) a Markov decision process modeling ride-sharing passengers’ decisions given price changes. In the latter, we use pricing and travel time data from a ride-sharing company to construct the transition probabilities and rewards of the Markov decision process.

Classical methods to control heating systems are often marred by suboptimal performance, inability to adapt to dynamic conditions and unreasonable assumptions e.g. existence of building models. This paper presents a novel deep reinforcement learning algorithm which can control space heating in buildings in a computationally efficient manner, and benchmarks it against other known techniques. The proposed algorithm outperforms rule based control by between 5-10% in a simulation environment for a number of price signals. We conclude that, while not optimal, the proposed algorithm offers additional practical advantages such as faster computation times and increased robustness to non-stationarities in building dynamics.

This paper presents the HDG learning algorithm, which uses a hierarchical decomposition of the state space to make learning to achieve goals more efficient with a small penalty in path quality. Special care must be taken when performing hierarchical planning and learning in stochastic domains, because macro-operators cannot be executed ballistically. The HDG algorithm, which is a descendent of Watkins’ Q-learning algorithm, is described here and preliminary empirical results are presented.

Exploration remains a significant challenge to reinforcement learning methods, especially in environments where reward signals are sparse. Recent methods of learning from demonstrations have shown to be promising in overcoming exploration difficulties but typically require considerable highquality demonstrations that are difficult to collect. We propose to effectively leverage available demonstrations to guide exploration through enforcing occupancy measure matching between the learned policy and current demonstrations, and develop a novel Policy Optimization from Demonstration (POfD) method. We show that POfD induces implicit dynamic reward shaping and brings provable benefits for policy improvement. Furthermore, it can be combined with policy gradient methods to produce state-of-the-art results, as demonstrated experimentally on a range of popular benchmark sparse-reward tasks, even when the demonstrations are few and imperfect.

Deep learning algorithms have recently appeared that pretrain hidden layers of neural networks in unsupervised ways, leading to state-of-the-art performance on large classification problems. These methods can also pretrain networks used for reinforcement learning. However, this ignores the additional information that exists in a reinforcement learning paradigm via the ongoing sequence of state, action, new state tuples. This paper demonstrates that learning a predictive model of state dynamics can result in a pretrained hidden layer structure that reduces the time needed to solve reinforcement learning problems.

Stochastic games provides a theoretical framework to multiagent reinforcement learning. Based on the framework, a multiagent reinforcement learning algorithm for zero-sum stochastic games was proposed by Littman and it was extended to general-sum games by Hu and Wellman. Given a stochastic game, if all agents learn with their algorithm, we can expect that the policies of the agents converge to a Nash equilibrium. However, agents with their algorithm always try to converge to a Nash equilibrium independent of the policies used by the other agents. In addition, in case there are multiple Nash equilibria, agents must agree on the equilibrium where they want to reach. Thus, their algorithm lacks adaptability in a sense. In this paper, we propose a multiagent reinforcement learning algorithm. The algorithm uses the extended optimal response which we introduce in this paper. It will converge to a Nash equilibrium when other agents are adaptable, otherwise it will make an optimal response. We also provide some empirical results in three simple stochastic games, which show that the algorithm can realize what we intend.

We establish geometric and topological properties of the space of value functions in finite stateaction Markov decision processes. Our main contribution is the characterization of the nature of its shape: a general polytope (Aigner et al., 2010). To demonstrate this result, we exhibit several properties of the structural relationship between policies and value functions including the line theorem, which shows that the value functions of policies constrained on all but one state describe a line segment. Finally, we use this novel perspective to introduce visualizations to enhance the understanding of the dynamics of reinforcement learning algorithms.

We consider learning in a Markov decision process where we are not explicitly given a reward function, but where instead we can observe an expert demonstrating the task that we want to learn to perform. This setting is useful in applications (such as the task of driving) where it may be difficult to write down an explicit reward function specifying exactly how different desiderata should be traded off. We think of the expert as trying to maximize a reward function that is expressible as a linear combination of known features, and give an algorithm for learning the task demonstrated by the expert. Our algorithm is based on using "inverse reinforcement learning" to try to recover the unknown reward function. We show that our algorithm terminates in a small number of iterations, and that even though we may never recover the expert's reward function, the policy output by the algorithm will attain performance close to that of the expert, where here performance is measured with respect to the expert's unknown reward function.


Question: An alternating optimization and quadrature technique for robust reinforcement learning.

This study explores the efficacy of neuroevolutionary reinforcement learning in addressing the complex challenge of generalized simulated helicopter hovering, a critical test case for reinforcement learning methodologies. Despite the natural affinity of neuroevolution for handling intricate transition dynamics and the high dimensionality of state and action spaces, the imperative for efficient exploration and on-line learning presents significant hurdles. To tackle these challenges, the study introduces and evaluates a range of techniques tailored to three progressively more demanding task variations, including the method that secured the top position in the 2008 Reinforcement Learning Competition. The findings reveal several key insights: (1) neuroevolutionary reinforcement learning is a potent tool for complex on-line learning tasks, exemplified by the mastery of generalized helicopter hovering, (2) neuroevolution is particularly adept at discovering effective hovering strategies but less so in learning helicopter models, (3) the complexity of model learning restricts model-based approaches to helicopter hovering tasks to scenarios where domain expertise can facilitate the design of an appropriate model representation, and (4) recent advancements in efficient resampling techniques have empowered neuroevolution to confront a broader spectrum of generalized reinforcement learning challenges with greater success.

After the recent groundbreaking results of AlphaGo and AlphaZero, we have seen strong interests in deep reinforcement learning and artificial general intelligence (AGI) in game playing. However, deep learning is resource-intensive and the theory is not yet well developed. For small games, simple classical table-based Q-learning might still be the algorithm of choice. General Game Playing (GGP) provides a good testbed for reinforcement learning to research AGI. Q-learning is one of the canonical reinforcement learning methods, and has been used by (Banerjee & Stone, IJCAI 2007) in GGP. In this paper we implement Q-learning in GGP for three small-board games (Tic-Tac-Toe, Connect Four, Hex), to allow comparison to Banerjee et al.. We find that Q-learning converges to a high win rate in GGP. For the -greedy strategy, we propose a first enhancement, the dynamic algorithm. In addition, inspired by (Gelly & Silver, ICML 2007) we combine online search (Monte Carlo Search) to enhance offline learning, and propose QM-learning for GGP. Both enhancements improve the performance of classical Q-learning. In this work, GGP allows us to show, if augmented by appropriate enhancements, that classical table-based Q-learning can perform well in small games.

Stochastic bandit problems have been analyzed from two different perspectives: a frequentist view, where the parameter is a deterministic unknown quantity, and a Bayesian approach, where the parameter is drawn from a prior distribution. We show in this paper that methods derived from this second perspective prove optimal when evaluated using the frequentist cumulated regret as a measure of performance. We give a general formulation for a class of Bayesian index policies that rely on quantiles of the posterior distribution. For binary bandits, we prove that the corresponding algorithm, termed BayesUCB, satisfies finite-time regret bounds that imply its asymptotic optimality. More generally, Bayes-UCB appears as an unifying framework for several variants of the UCB algorithm addressing different bandit problems (parametric multi-armed bandits, Gaussian bandits with unknown mean and variance, linear bandits). But the generality of the Bayesian approach makes it possible to address more challenging models. In particular, we show how to handle linear bandits with sparsity constraints by resorting to Gibbs sampling.

We describe a method to use discrete human feedback to enhance the performance of deep learning agents in virtual three-dimensional environments by extending deep-reinforcement learning to model the confidence and consistency of human feedback. This enables deep reinforcement learning algorithms to determine the most appropriate time to listen to the human feedback, exploit the current policy model, or explore the agent’s environment. Managing the trade-off between these three strategies allows DRL agents to be robust to inconsistent or intermittent human feedback. Through experimentation using a synthetic oracle, we show that our technique improves the training speed and overall performance of deep reinforcement learning in navigating three-dimensional environments using Minecraft. We further show that our technique is robust to highly innacurate human feedback and can also operate when no human feedback is given.

A central problem in artificial intelligence is that of planning to maximize future reward under uncertainty in a partially observable environment. In this paper we propose and demonstrate a novel algorithm which accurately learns a model of such an environment directly from sequences of action-observation pairs. We then close the loop from observations to actions by planning in the learned model and recovering a policy which is near-optimal in the original environment. Specifically, we present an efficient and statistically consistent spectral algorithm for learning the parameters of a Predictive State Representation (PSR). We demonstrate the algorithm by learning a model of a simulated high-dimensional, vision-based mobile robot planning task, and then perform approximate point-based planning in the learned PSR. Analysis of our results shows that the algorithm learns a state space which efficiently captures the essential features of the environment. This representation allows accurate prediction with a small number of parameters, and enables successful and efficient planning.

We propose a new approach to the problem of searching a space of policies for a Markov decision process (MDP) or a partially observable Markov decision process (POMDP), given a model. Our approach is based on the following observation: Any (PO)MDP can be transformed into an “equivalent” POMDP in which all state transitions (given the current state and action) are deterministic. This reduces the general problem of policy search to one in which we need only consider POMDPs with deterministic transitions. We give a natural way of estimating the value of all policies in these transformed POMDPs. Policy search is then simply performed by searching for a policy with high estimated value. We also establish conditions under which our value estimates will be good, recovering theoretical results similar to those of Kearns, Mansour and Ng [7], but with “sample complexity” bounds that have only a polynomial rather than exponential dependence on the horizon time. Our method applies to arbitrary POMDPs, including ones with infinite state and action spaces. We also present empirical results for our approach on a small discrete problem, and on a complex continuous state/continuous action problem involving learning to ride a bicycle.

We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.

In this paper, we consider the important problem of safe exploration in reinforcement learning. While reinforcement learning is well-suited to domains with complex transition dynamics and high-dimensional state-action spaces, an additional challenge is posed by the need for safe and efficient exploration. Traditional exploration techniques are not particularly useful for solving dangerous tasks, where the trial and error process may lead to the selection of actions whose execution in some states may result in damage to the learning system (or any other system). Consequently, when an agent begins an interaction with a dangerous and high-dimensional state-action space, an important question arises; namely, that of how to avoid (or at least minimize) damage caused by the exploration of the state-action space. We introduce the PI-SRL algorithm which safely improves suboptimal albeit robust behaviors for continuous state and action control tasks and which efficiently learns from the experience gained from the environment. We evaluate the proposed method in four complex tasks: automatic car parking, pole-balancing, helicopter hovering, and business management.

Balancing the ratio between exploration and exploitation is an important problem in reinforcement learning. This paper evaluates four different exploration strategies combined with Q-learning using random stochastic mazes to investigate their performances. We will compare: UCB-1, softmax, ∈-greedy, and pursuit. For this purpose we adapted the UCB-1 and pursuit strategies to be used in the Q-learning algorithm. The mazes consist of a single optimal goal state and two suboptimal goal states that lie closer to the starting position of the agent, which makes efficient exploration an important part of the learning agent. Furthermore, we evaluate two different kinds of reward functions, a normalized one with rewards between 0 and 1, and an unnormalized reward function that penalizes the agent for each step with a negative reward. We have performed an extensive grid-search to find the best parameters for each method and used the best parameters on novel randomly generated maze problems of different sizes. The results show that softmax exploration outperforms the other strategies, although it is harder to tune its temperature parameter. The worst performing exploration strategy is ∈-greedy.

In this paper, a simple and efficient method is proposed using one of the basic reinforcement learning tool Q-learning. This method is to achieve the aim of mobile robot maples navigation with collision-free. Thus, an end-to-end navigation model is built, which uses lidar sensor massage as input and moving command of mobile robot as output, so as to simplify the process of environmental perception and decision making on action. Through simulation and experiments, its effectiveness is proved. After trained by this method, mobile robot can safely reach the navigation target in an unknown environment without any prior demonstration. In addition, an extensive quantitative and qualitative evaluation of this method is presented by the comparison with traditional path planning method based on foregone global environment map.

In recent years impressive results have been presented illustrating the potential of quadrotors to solve challenging tasks. Generally, the derivation of the controllers involve complex analytical manipulation of the dynamics and are very specific to the task at hand. In addition, most approaches construct a trajectory and then design a stabilizing controller in a separate step, whereas a fully optimal solution requires finding both simultaneously. In this paper, a generalized approach is presented using an iterative optimal control algorithm. A series of complex tasks are thus solved using the same algorithm without the need for manual manipulation of the system dynamics, heuristic simplifications, or manual trajectory generation. First, aggressive maneuvers are performed by requiring the quadrotor to pass with a slung load through a window not high enough for the load to pass while hanging straight down. Second, go-to-goal tasks with single and double rotor failure are demonstrated. The adaptability and applicability of this unified approach to such diverse tasks with a nonlinear, underactuated, constrained, and in the case of the slung load, hybrid quadrotor systems is thus shown.


Question: An alternating optimization and quadrature technique for robust reinforcement learning.

This paper introduces an innovative method for automatic gait optimization in quadrupedal and bipedal robots using Gaussian Process Regression (GPR). Gait optimization, a fundamental yet complex task for robotics, often relies on automation techniques that suffer from limitations. These techniques, primarily based on local function optimization, are susceptible to local optima, fail to utilize past gait evaluations effectively, and do not account for noise in gait assessment. These shortcomings lead to increased demand for gait evaluations, making the optimization process time-consuming, data inefficient, and labor-intensive. Our proposed Bayesian approach, grounded in GPR, addresses these issues by employing a global search strategy. This strategy builds on a posterior model derived from all individual, potentially noisy, gait evaluations. The method's efficacy is demonstrated on a quadruped robot, where it optimizes two critical aspects: speed and smoothness. In both cases, our technique significantly reduces the number of gait evaluations required compared to leading local gradient-based approaches, showcasing its superiority in efficiency and performance.

Reinforcement Learning (RL) is a learning paradigm concerned with learning to control a system so as to maximize an objective over the long term. This approach to learning has received immense interest in recent times and success manifests itself in the form of human-level performance on games like Go. While RL is emerging as a practical component in reallife systems, most successes have been in Single Agent domains. This report will instead specifically focus on challenges that are unique to Multi-Agent Systems interacting in mixed cooperative and competitive environments. The report concludes with advances in the paradigm of training Multi-Agent Systems called Decentralized Actor, Centralized Critic, based on an extension of MDPs called Decentralized Partially Observable MDPs, which has seen a renewed interest lately. 1 ar X iv :1 80 7. 09 42 7v 1 [ cs .A I] 2 5 Ju l 2 01 8

Can one parallelize complex exploration– exploitation tradeoffs? As an example, consider the problem of optimal highthroughput experimental design, where we wish to sequentially design batches of experiments in order to simultaneously learn a surrogate function mapping stimulus to response and identify the maximum of the function. We formalize the task as a multiarmed bandit problem, where the unknown payoff function is sampled from a Gaussian process (GP), and instead of a single arm, in each round we pull a batch of several arms in parallel. We develop GP-BUCB, a principled algorithm for choosing batches, based on the GP-UCB algorithm for sequential GP optimization. We prove a surprising result; as compared to the sequential approach, the cumulative regret of the parallel algorithm only increases by a constant factor independent of the batch size B. Our results provide rigorous theoretical support for exploiting parallelism in Bayesian global optimization. We demonstrate the effectiveness of our approach on two real-world applications.

Cooperative games are those in which both agents share the same payoff structure. Valuebased reinforcement-learning algorithms, such as variants of Q-learning, have been applied to learning cooperative games, but they only apply when the game state is completely observable to both agents. Policy search methods are a reasonable alternative to value-based methods for partially observable environments. In this paper, we provide a gradient-based distributed policysearch method for cooperative games and compare the notion of local optimum to that of Nash equilibrium. We demonstrate the effectiveness of this method experimentally in a small, partially observable simulated soccer domain.

The limited nature of robot sensors make many important robo tics problems partially observable. These problems may require the s yst m to perform complex information-gathering operations. One approach to so lving these problems is to create plans inbelief-space , the space of probability distributions over the underlying state of the system. The belief-space plan encodes a st rategy for performing a task while gaining information as necessary. Most approach es to belief-space planning rely upon representing belief state in a particular way (typically as a Gaussian). Unfortunately, this can lead to large errors between the assumed density representation of belief state and the true belief state. Th is paper proposes a new sample-based approach to belief-space planning that has fix ed computational complexity while allowing arbitrary implementations of Bayes filtering to be used to track belief state. The approach is illustrated in the conte xt of a simple example and compared to a prior approach. Then, we propose an applicatio n of the technique to an instance of the grasp synthesis problem where a robot mu st simultaneously localize and grasp an object given initially uncertain obje ct parameters by planning information-gathering behavior. Experimental results ar e presented that demonstrate the approach to be capable of actively localizing and graspi ng boxes that are presented to the robot in uncertain and hard-to-localize config urations.

In this paper, we consider the important problem of safe exploration in reinforcement learning. While reinforcement learning is well-suited to domains with complex transition dynamics and high-dimensional state-action spaces, an additional challenge is posed by the need for safe and efficient exploration. Traditional exploration techniques are not particularly useful for solving dangerous tasks, where the trial and error process may lead to the selection of actions whose execution in some states may result in damage to the learning system (or any other system). Consequently, when an agent begins an interaction with a dangerous and high-dimensional state-action space, an important question arises; namely, that of how to avoid (or at least minimize) damage caused by the exploration of the state-action space. We introduce the PI-SRL algorithm which safely improves suboptimal albeit robust behaviors for continuous state and action control tasks and which efficiently learns from the experience gained from the environment. We evaluate the proposed method in four complex tasks: automatic car parking, pole-balancing, helicopter hovering, and business management.

In recent years impressive results have been presented illustrating the potential of quadrotors to solve challenging tasks. Generally, the derivation of the controllers involve complex analytical manipulation of the dynamics and are very specific to the task at hand. In addition, most approaches construct a trajectory and then design a stabilizing controller in a separate step, whereas a fully optimal solution requires finding both simultaneously. In this paper, a generalized approach is presented using an iterative optimal control algorithm. A series of complex tasks are thus solved using the same algorithm without the need for manual manipulation of the system dynamics, heuristic simplifications, or manual trajectory generation. First, aggressive maneuvers are performed by requiring the quadrotor to pass with a slung load through a window not high enough for the load to pass while hanging straight down. Second, go-to-goal tasks with single and double rotor failure are demonstrated. The adaptability and applicability of this unified approach to such diverse tasks with a nonlinear, underactuated, constrained, and in the case of the slung load, hybrid quadrotor systems is thus shown.

In this paper we propose to do portfolio management using reinforcement learning (RL) and independent factor model. Factors in independent factor model are mutually independent and exhibit better predictability. RL is applied to each factor to capture temporal dependence and provide investment suggestion on factor. Optimal weights on factors are found by portfolio optimization method subject to the investment suggestions and general portfolio constraints. Experimental results and analysis are given to show that the proposed method has better performance when compare to two alternative portfolio management systems.

Reinforcement Learning (RL) is learning through direct experimentation. It does not assume the existence of a teacher that provides examples upon which learning of a task takes place. Instead, in RL experience is the only teacher. With historical roots on the study of conditioned reflexes, RL soon attracted the interest of Engineers and Computer Scientists because of its theoretical relevance and potential applications in fields as diverse as Operational Research and Robotics. Computationally, RL is intended to operate in a learning environment composed by two subjects: the learner and a dynamic process. At successive time steps, the learner makes an observation of the process state, selects an action and applies it back to the process. The goal of the learner is to find out an action policy that controls the behavior of this dynamic process, guided by signals (reinforcements) that indicate how well it is performing the required task. These signals are usually associated to some dramatic condition — e.g., accomplishment of a subtask (reward) or complete failure (punishment), and the learner’s goal is to optimize its behavior based on some performance measure (a function of the received reinforcements). The crucial point is that in order to do that, the learner must evaluate the conditions (associations between observed states and chosen actions) that lead to rewards or punishments. In other words, it must learn how to assign credit to past actions and states by correctly estimating costs associated to these events. Starting from basic concepts, this tutorial presents the many flavors of RL algorithms, develops the corresponding mathematical tools, assess their practical limitations and discusses alternatives that have been proposed for applying RL to realistic tasks, such as those involving large state spaces or partial observability. It relies on examples and diagrams to illustrate the main points, and provides many references to the specialized literature and to Internet sites where relevant demos and additional information can be obtained.

In this paper, we propose a method for robust kernel density estimation. We interpret a KDE with Gaussian kernel as the inner product between a mapped test point and the centroid of mapped training points in kernel feature space. Our robust KDE replaces the centroid with a robust estimate based on M-estimation (P. Huber, 1981), The iteratively re-weighted least squares (IRWLS) algorithm for M-estimation depends only on inner products, and can therefore be implemented using the kernel trick. We prove the IRWLS method monotonically decreases its objective value at every iteration for a broad class of robust loss functions. Our proposed method is applied to synthetic data and network traffic volumes, and the results compare favorably to the standard KDE.

In the model-based policy search approach to reinforcement learning (RL), policies are found using a model (or "simulator") of the Markov decision process. However, for high-dimensional continuous-state tasks, it can be extremely difficult to build an accurate model, and thus often the algorithm returns a policy that works in simulation but not in real-life. The other extreme, model-free RL, tends to require infeasibly large numbers of real-life trials. In this paper, we present a hybrid algorithm that requires only an approximate model, and only a small number of real-life trials. The key idea is to successively "ground" the policy evaluations using real-life trials, but to rely on the approximate model to suggest local changes. Our theoretical results show that this algorithm achieves near-optimal performance in the real system, even when the model is only approximate. Empirical results also demonstrate that---when given only a crude model and a small number of real-life trials---our algorithm can obtain near-optimal performance in the real system.


Question: An alternating optimization and quadrature technique for robust reinforcement learning algorithms.

This paper delves into a response surface methodology tailored for the optimization of costly, black-box functions that are common in engineering scenarios. In such settings, the scarcity of function evaluations is a critical constraint, often due to the time-consuming or expensive nature of the evaluations themselves. Traditional global optimization techniques frequently fall short under these conditions, necessitating an innovative approach. The methodology we present is designed to tackle the complexity of nonlinear, multimodal functions typically encountered in engineering by constructing response surfaces from a limited number of function evaluations. These surfaces serve as surrogate models for visualization, tradeoff analysis, and optimization purposes.

A key component of our methodology is the efficient global optimization algorithm, which includes a reliable stopping rule. The algorithm's effectiveness hinges on balancing two objectives: exploiting the approximating surface (by sampling at its minimum) and enhancing the approximation's accuracy (by sampling at points where prediction error is likely to be significant). This balance is achieved by solving auxiliary problems that were once deemed computationally infeasible. We demonstrate how these hurdles can be overcome, leading to a robust optimization strategy that can navigate the intricacies of expensive black-box functions while minimizing the number of required evaluations.

The methodology's utility is underscored by its ability to handle the dual challenges of exploiting the response surface for optimization and improving its predictive capabilities. By addressing these auxiliary problems, we pave the way for more efficient and accurate global optimization in the face of costly function evaluations, making it particularly suitable for complex engineering optimization tasks.

Our goal in this work is to make high level decisions for mobile robots. In particular, given a queue of prioritized object delivery tasks, we wish to find a sequence of actions in real time to accomplish these tasks efficiently. We introduce a novel reinforcement learning algorithm called Smoothed Sarsa that learns a good policy for these delivery tasks by delaying the backup reinforcement step until the uncertainty in the state estimate improves. The state space is modeled by a Dynamic Bayesian Network and updated using a Region-based Particle Filter. We take advantage of the fact that only discrete (topological) representations of entity locations are needed for decision-making, to make the tracking and decision making more efficient. Our experiments show that policy search leads to faster task completion times as well as higher total reward compared to a manually crafted policy. Smoothed Sarsa learns a policy orders of magnitude faster than previous policy search algorithms. We demonstrate our results on the Player/Stage simulator and on the Pioneer robot.

Though successful in high-dimensional domains, deep reinforcement learning exhibits high sample complexity and suffers from stability issues as reported by researchers and practitioners in the field. These problems hinder the application of such algorithms in real-world and safety-critical scenarios. In this paper, we take steps towards stable and efficient reinforcement learning by following a model-based approach that is known to reduce agent-environment interactions. Namely, our method augments deep Q-networks (DQNs) with model predictions for transitions, rewards, and termination flags. Having the model at hand, we then conduct a rigorous theoretical study of our algorithm and show, for the first time, convergence to a stationary point. En route, we provide a counterexample showing that ’vanilla’ DQNs can diverge confirming practitioners’ and researchers’ experiences. Our proof is novel in its own right and can be extended to other forms of deep reinforcement learning. In particular, we believe exploiting the relation between reinforcement (with deep function approximators) and online learning can serve as a recipe for future proofs in the domain. Finally, we validate our theoretical results in 20 games from the Atari benchmark. Our results show that following the proposed model-based learning approach not only ensures convergence but leads to a reduction in sample complexity and superior performance. Introduction Model-free deep reinforcement learning methods have recently demonstrated impressive performance on a range of complex learning tasks (Hessel et al. 2018; Lillicrap et al. 2017; Jaderberg et al. 2017). Deep Q-Networks (DQNs) (Mnih et al. 2015), in particular, stand out as a versatile and effective tool for a wide range of applications. DQNs offer an end-to-end solution for many reinforcement learning problems by generalizing tabular Q-learning to highdimensional input spaces. Unfortunately, DQNs still suffer from a number of important drawbacks that hinder wide-spread adoption. There is a general lack of theoretical understanding to guide the use of deep Q-learning algorithms. Combining non-linear function approximation with off-policy reinforcement learning is known to lead to possible divergence issues, even in very simple tasks (Boyan and Preliminary work. Moore 1995). Despite improvements introduced by deep Qnetworks (Mnih et al. 2015), Q-learning approaches based on deep neural networks still exhibit occasional instability during the learning process, see for example (van Hasselt, Guez, and Silver 2016). In this paper, we aim to address the instability of current deep RL approaches by augmenting model-free value learning with a predictive model. We propose a hybrid model and value learning objective, which stabilises the learning process and reduces sample complexity when compared to model-free algorithms. We attribute this reduction to the richer feedback signal provided by our method compared to standard deep Q-networks. In particular, our model provides feedback at every time step based on the current prediction error, which in turn eliminates one source of sample inefficiency caused by sparse rewards. These conclusions are also in accordance with previous research conducted on linear function approximation in reinforcement learning (Parr et al. 2008; Sutton et al. 2012; Song et al. 2016). While linear function approximation results provide a motivation for model-based stabilisation, such theories fail to generalise to deep non-linear architectures (Song et al. 2016). To close this gap in the literature and demonstrate stability of our method, we prove convergence in the general deep RL setting with deep value function approximators. Theoretically analysing deep RL algorithms, however, is challenging due to nonlinear functional dependencies and objective non-convexities prohibiting convergence to globally optimal policies. As such, the best one would hope for is a stationary point (e.g., a first-order one) that guarantees vanishing gradients. Even understanding gradient behavior for deep networks in RL can be difficult due to explorationbased policies and replay memory considerations. To alleviate some of these problems, we map deep reinforcement learning to a mathematical framework explicitly designed for understanding the exploration/exploitation trade-off. Namely, we formalise the problem as an online learning game the agent plays against an adversary (i.e., the environment). Such a link allows us to study a more general problem combining notions of regret, reinforcement learning, and optimisation. Given such a link, we prove that for any > 0, regret vanishes as O ( T 1−2 lnT ) with T being the total number of rounds. This, in turn, guarantees convergence to a stationary point. ar X iv :1 80 9. 01 90 6v 1 [ cs .L G ] 6 S ep 2 01 8 Guided by the above theoretical results, we validate that our algorithm leads to faster learning by conducting experiments on 20 Atari games. Due to the high computational and budgeting demands, we chose a benchmark that is closest to our setting. Concretely, we picked DQNs as our template algorithm to improve on both theoretically and empirically. It is worth noting that extending our theoretical results to more complex RL algorithms is an orthogonal future direction. Background: Reinforcement Learning We consider a discrete time, infinite horizon, discounted Markov Decision Process (MDP) 〈 S,A, P , P , γ 〉 . Here S denotes the state space, A the set of possible actions, P (R) : S × A × R → [0, 1] is the reward function, P (S) : S × A × S → [0, 1] is the state transition function, and γ ∈ (0, 1) is a discount factor. An agent, being in state St ∈ S at time step t, executes an action at ∈ A sampled from its policy π(at|St), a conditional probability distribution where π : S × A → [0, 1]. The agent’s action elicits from the environment a reward signal rt ∈ R indicating instantaneous reward, a terminal flag ft ∈ {0, 1} indicating a terminal event that restarts the environment, and a transition to a successor state St+1 ∈ S. We assume that the sets S, A, and R are discrete. The reward rt is sampled from the conditional probability distribution P (rt|St,at). Similarly, ft ∼ P (F (ft|St,at) with P (F ) : S × A × {0, 1} → [0, 1], where a terminal event (ft = 1) restarts the environment according to some initial state distribution S0 ∼ P (S) 0 (S0). The state transition to a successor state is determined by a stochastic state transition function according to St+1 ∼ P (St+1|St,at). The agent’s goal is to maximise future cumulative reward E P (S) 0 ,π,P (R),P (F ),P (S) [ ∞ ∑ t=0 ( t ∏ t′=0 (1− ft′−1) ) γrt ] with respect to the policy π. An important quantity in RL are Q-values Q(St,at), which are defined as the expected future cumulative reward value when executing action at in state St and subsequently following policy π. Qvalues enable us to conveniently phrase the RL problem as maxπ ∑ S p (S) ∑ a π(a|S)Q(S,a), where p(S) is the stationary state distribution obtained when executing π in the environment (starting from S0 ∼ P (S) 0 (S0)). Deep Q-Networks: Value-based reinforcement learning approaches identify optimal Q-values directly using parametric function approximators Qθ(S,a), where θ ∈ Θ represents the parameters (Watkins 1989; Busoniu et al. 2010). Optimal Q-value estimates then correspond to an optimal policy π(a|S) = δa,argmaxa′Qθ(S,a′). Deep Q-networks (Mnih et al. 2015) learn a deep neural network based Qvalue approximation by performing stochastic gradient descent on the following training objective: L(θ) = ES,a,r,f,S′ [( r + (1− f)γmax a′ Qθ−(S ′,a′) −Qθ(S,a) )2] . (1) The expectation ranges over transitions S,a, r, f,S′ sampled from an experience replay memory (S′ denotes the state at the next time step). Use of this replay memory, together with the use of a separate target network Qθ− (with different parameters θ− ∈ Θ) for calculating the bootstrap values maxa′ Qθ−(S ′,a′), helps stabilise the learning process. Background: Online Learning In this paper, we employ a special form of regret minimisation games that we briefly review here. A regret minimisation game is a triple 〈Θ,F , T 〉, where Θ is a non-empty decision set, F is the set of moves of the adversary which contains bounded convex functions from R to R and T is the total number of rounds. The game commences in rounds, where at round j = 1, . . . , T , the agent chooses a prediction θj ∈ Θ and the adversary a loss function Lj ∈ F . At the end of the round, the adversary reveals its choice and the agent suffers a loss Lj(θj). In this paper, we are concerned with the full-information case where the agent can observe the complete loss function Lj at the end of each round. The goal of the game is for the agent to make successive predictions to minimise cumulative regret defined as:

We discsus Hinton’s (1989) relative payoff procedure (RPP), a static reinforcement learning algorithm whose foundation is not stochastic gradient ascent. We show circumstances under which applying the RPP is guaranteed to increase the mean return, even though it can make large changes in the values of the parameters. The proof is based on a mapping between the RPP and a form of the expectation-maximisation procedure of Dempster, Laird & Rubin (1976).

We propose a model-based approach to hierarchical reinforcement learning that exploits shared knowledge and selective execution at different levels of abstraction, to efficiently solve large, complex problems. Our framework adopts a new transition dynamics learning algorithm that identifies the common action-feature combinations of the subtasks, and evaluates the subtask execution choices through simulation. The framework is sample efficient, and tolerates uncertain and incomplete problem characterization of the subtasks. We test the framework on common benchmark problems and complex simulated robotic environments. It compares favorably against the stateof-the-art algorithms, and scales well in very large problems.

Model-based dual heuristic dynamic programming (MB-DHP) is a popular approach in approximating optimal solutions in control problems. Yet, it usually requires offline training for the model network, and thus resulting in extra computational cost. In this brief, we propose a model-free DHP (MF-DHP) design based on finite-difference technique. In particular, we adopt multilayer perceptron with one hidden layer for both the action and the critic networks design, and use delayed objective functions to train both the action and the critic networks online over time. We test both the MF-DHP and MB-DHP approaches with a discrete time example and a continuous time example under the same parameter settings. Our simulation results demonstrate that the MF-DHP approach can obtain a control performance competitive with that of the traditional MB-DHP approach while requiring less computational resources.

In this paper, we introduce pilco, a practical, data-efficient model-based policy search method. Pilco reduces model bias, one of the key problems of model-based reinforcement learning, in a principled way. By learning a probabilistic dynamics model and explicitly incorporating model uncertainty into long-term planning, pilco can cope with very little data and facilitates learning from scratch in only a few trials. Policy evaluation is performed in closed form using state-ofthe-art approximate inference. Furthermore, policy gradients are computed analytically for policy improvement. We report unprecedented learning efficiency on challenging and high-dimensional control tasks.

Due to the complex and high nonlinear structure of the quadruped robots, the dynamic control of quadruped robots has long been a big challenge for the researchers. In this paper, a guaranteed performance adaptive control algorithm is proposed for tracking control of a quadruped robot with parameter uncertainties. The controller is designed based on the sliding mode theory, and incorporates a transformed error, which includes the performance indices. Parameter uncertainties are estimated by a linear network. By choosing a proper Lyapunov function, the prescribed performance with the proposed controller is proven in the scenes of the parameter uncertainties and the environment constraint. Numerical simulations demonstrate the effectiveness and superiority of the proposed controller.

Conditional Value at Risk (CVaR) is a prominent risk measure that is being used extensively in various domains. We develop a new formula for the gradient of the CVaR in the form of a conditional expectation. Based on this formula, we propose a novel sampling-based estimator for the gradient of the CVaR, in the spirit of the likelihood-ratio method. We analyze the bias of the estimator, and prove the convergence of a corresponding stochastic gradient descent algorithm to a local CVaR optimum. Our method allows to consider CVaR optimization in new domains. As an example, we consider a reinforcement learning application, and learn a risksensitive controller for the game of Tetris.

Most policy search algorithms require thousands of training episodes to find an effective policy, which is often infeasible with a physical robot. This survey article focuses on the extreme other end of the spectrum: how can a robot adapt with only a handful of trials (a dozen) and a few minutes? By analogy with the word “big-data”, we refer to this challenge as “micro-data reinforcement learning”. We show that a first strategy is to leverage prior knowledge on the policy structure (e.g., dynamic movement primitives), on the policy parameters (e.g., demonstrations), or on the dynamics (e.g., simulators). A second strategy is to create data-driven surrogate models of the expected reward (e.g., Bayesian optimization) or the dynamical model (e.g., model-based policy search), so that the policy optimizer queries the model instead of the real system. Overall, all successful micro-data algorithms combine these two strategies by varying the kind of model and prior knowledge. The current scientific challenges essentially revolve around scaling up to complex robots (e.g., humanoids), designing generic priors, and optimizing the computing time.

This paper describes several ensemble methods that combine multiple different reinforcement learning (RL) algorithms in a single agent. The aim is to enhance learning speed and final performance by combining the chosen actions or action probabilities of different RL algorithms. We designed and implemented four different ensemble methods combining the following five different RL algorithms: Q-learning, Sarsa, actor-critic (AC), QV-learning, and AC learning automaton. The intuitively designed ensemble methods, namely, majority voting (MV), rank voting, Boltzmann multiplication (BM), and Boltzmann addition, combine the policies derived from the value functions of the different RL algorithms, in contrast to previous work where ensemble methods have been used in RL for representing and learning a single value function. We show experiments on five maze problems of varying complexity; the first problem is simple, but the other four maze tasks are of a dynamic or partially observable nature. The results indicate that the BM and MV ensembles significantly outperform the single RL algorithms.


Question: Pros and Cons of Immersive Participatory Augmented Reality Simulations in Education and Learning Processes

In recent years, a growing body of research has reevaluated the impact of various media on learning outcomes, leading to a rethinking of conventional wisdom regarding the benefits of specific media in educational settings. Meta-analyses and other studies have consistently shown that the choice of media does not inherently offer learning advantages. Gains in performance or time savings attributed to one medium over another are often due to factors other than the medium itself, such as instructional method or novelty effects.

A critical examination of media attribute and symbol system theories has revealed shortcomings, prompting a call for more promising research directions. Traditionally, media comparison studies have sought to identify the optimal mix of medium, student, subject matter, and learning task to enhance learning. However, current evidence suggests that media serve merely as delivery vehicles for instruction, akin to a truck delivering groceries without affecting nutrition. The choice of medium might influence the cost or scope of instruction distribution, but the content of the instruction, not the medium, determines learning outcomes.

Despite occasional reports of significant learning improvements linked to the introduction of a new medium, as observed in the case of television in El Salvador, these gains are typically attributed to accompanying curricular reforms rather than the medium itself. The belief that newer media confer a slight learning advantage over conventional methods is subject to rival hypotheses, including methodological artifacts, confounding variables, and publication bias favoring studies that show larger effect sizes for newer media.

A review of past meta-analyses on media research further supports the notion that media effects on learning are confounded by uncontrolled variables such as novelty and instructional method. The size of effect attributed to media treatments is often negligible, as evidenced by recent meta-analyses conducted by James Kulik and his colleagues. These studies have employed meta-analysis techniques to provide more accurate estimates of media treatment effects than previous reviews, which often relied on simplistic "box scores" of significant findings.

The distinction between box scores and effect sizes is crucial for understanding the true impact of media on learning. For instance, a review of audiotutorial instruction studies by Kulik et al. found that while a majority of studies favored audiotutorial instruction, only a small fraction reported statistically significant differences. Moreover, the effect sizes of media treatments are typically small, often measured in fractions of a standard deviation or minor improvements in test scores.

These findings challenge the notion that media inherently enhance learning and suggest that the focus should shift from medium to content and instructional methods. Future research should explore how instructional strategies and content can be optimized to improve learning outcomes across different media platforms

This study investigates the effects of a story-making app called Our Story and a selection of other educational apps on the learning engagement of forty-one Spanish 4–5-year-olds. Children were observed interacting in small groups with the story-making app and this was compared to their engagement with a selection of construction and drawing apps. Children’s engagement was analysed in two ways: it was categorised using Bangert-Drowns and Pyke’s taxonomy for individual hands-on engagement with educational software, and using the concept of exploratory talk as developed by Mercer et al. to analyse peer engagement. For both approaches, quantitative and qualitative indices of children’s engagement were considered. The overall findings suggested that in terms of the BangertDrowns and Pyke taxonomy, the quality of children’s individual engagement was higher with the OS app in contrast to their engagement with other app software. The frequency of children’s use of exploratory talk was similar with the OS and colouring and drawing apps, and a detailed qualitative analysis of the interaction transcripts revealed several instances of the OS and drawing apps supporting joint problem-solving and collaborative engagement. We suggest that critical indices of an app’s educational value are the extent to which the app supports opportunities for open-ended content and children’s independent use of increasingly difficult features. 2013 Elsevier Ltd. All rights reserved.

This paper explores a user interface technique which augments an immersive head tracked display with a hand-held miniature copy of the virtual environment. We call this interface technique the Worlds in Miniature (WIM) metaphor. In addition to the first-person perspective offered by a virtual reality system, a World in Miniature offers a second dynamic viewport onto the virtual environment. Objects may be directly manipulated either through the immersive viewport or through the three-dimensional viewport offered by the WIM. In addition to describing object manipulation, this paper explores ways in which Worlds in Miniature can act as a single unifying metaphor for such application independent interaction techniques as object selection, navigation, path planning, and visualization. The WIM metaphor offers multiple points of view and multiple scales at which the user can operate, without requiring explicit modes or commands. Informal user observation indicates that users adapt to the Worlds in Miniature metaphor quickly and that physical props are helpful in manipulating the WIM and other objects in the environment.

Pedagogy has emphasized that physical representations and tangible interactive objects benefit learning especially for young students. There are many tangible hardware platforms for introducing computer programming to children, but there is limited comparative evaluation of them in the context of a formal classroom. In this work, we explore the benefits of learning to code for tangible computers, such as robots and wearable computers, in comparison to programming for the desktop computer. For this purpose, 36 students participated in a within-groups study that involved three types of target computer platform tangibility: (1) desktop, (2) wearable, and (3) robotic. We employed similar blocks-based visual programming environments, and we measured emotional engagement, attitudes, and computer programming performance. We found that students were more engaged by and had a higher intention of learning programming with the robotic rather than the desktop computer. Furthermore, tangible computing platforms, either robot or wearable, did not affect the students’ performance in learning basic computational concepts (e.g., sequence, repeat, and decision). Our findings suggest that computer programming should be introduced through multiple target platforms (e.g., robots, smartphones, wearables) to engage children.

been developed during the last 20 years have proven to be quite successful, particularly in the domains of mathematics, science, and technology. They produce significant learning gains beyond classroom environments. They are capable of engaging most students’ attention and interest for hours. We have been working on a new generation of intelligent tutoring systems that hold mixedinitiative conversational dialogues with the learner. The tutoring systems present challenging problems and questions to the learner, the learner types in answers in English, and there is a lengthy multiturn dialogue as complete solutions or answers evolve. This article presents the tutoring systems that we have been developing. AUTOTUTOR is a conversational agent, with a talking head, that helps college students learn about computer literacy. ANDES, ATLAS, AND WHY2 help adults learn about physics. Instead of being mere information-delivery systems, our systems help students actively construct knowledge through conversations.

Creating a design theory for Constructivist Gaming Learning Environment necessitates, among other things, the establishment of design principles. These principles have the potential to help designers produce games, where users achieve higher levels of learning. This paper focuses on twelve design principles: Probing, Distributed, Multiple Routes, Practice, Psychosocial Moratorium, Regime of Competence, Self-Knowledge, Collective Knowledge, Engaging, User Interface Ease of Use, On Demand and Just-in-Time Tutorial, and Achievement. We report on two pilot studies of a qualitative nature in which we test our design principles. Game play testing and observations were carried out on five Massively Multiplayer Online Games (MMOGs): RuneScape, GuildWars, Ragnarok, World of WarCraft, and Final Fantasy XI. Two educational games, Carabella Goes to College and Outbreak at WatersEdge were also observed. Our findings indicate that not all of the popular MMOGs and educational games support all of these principles.

Teachers have tried to teach their students by introducing text books along with verbal instructions in traditional education system. However, teaching and learning methods could be changed for developing Information and Communication Technology (ICT). It's time to adapt students with interactive learning system so that they can improve their learning, catching, and memorizing capabilities. It is indispensable to create high quality and realistic leaning environment for students. Visual learning can be easier to understand and deal with their learning. We developed visual learning materials (an overview of solar system) in the form of video for students of primary level using different multimedia application tools. The objective of this paper is to examine the impact of student’s abilities to acquire new knowledge or skills through visual learning materials and blended leaning that is integration of visual learning materials with teacher’s instructions. We visited a primary school in Dhaka city for this study and conducted teaching with three different groups of students (i) teacher taught students by traditional system on same materials and marked level of student’s ability to adapt by a set of questions (ii) another group was taught with only visual learning material and assessment was done with 15 questionnaires, (iii) the third group was taught with the video of solar system combined with teacher’s instructions and assessed with the same questionnaires. This integration of visual materials (solar system) with verbal instructions is a blended approach of learning. The interactive blended approach greatly promoted students ability of acquisition of knowledge and skills. Students response and perception were very positive towards the blended technique than the other two methods. This interactive blending leaning system may be an appropriate method especially for school children.

Computer-based instruction (CBI) is becoming increasingly popular in the classroom, particularly because the latest technological advancements allow for visually rich and interactive environments. While the inherent nature of CBIs is often thought to engage learners, research examining the role of motivation in learning with these environments has resulted in mixed findings. These findings are further complicated by unique design characteristics of distinct CBIs. This literature review synthesizes research that has examined the role of theoretically-grounded constructs of motivation in the context of three popular CBIs, multimedia, hypermedia, and hypertext. Specifically, this literature review considered empirical studies that examined the effect of these CBIs on motivation, in addition to the effect of motivation on learning outcomes and the learning process within the context of these environments. The literature review concludes with a theoretical consideration of previous research and a discussion of a framework for future directions. 2010 Published by Elsevier Ltd.

During the Ottoman period, Taslihan was the largest accommodation complex in Sarajevo, Bosnia and Herzegovina. Today, only one wall remains as a memento of its existence. In this paper, we compare user appreciation of an interactive digital story about this building and of a serious game about Taslihan to see which application offers more knowledge and immersion while bringing this monument to life in the collective memory of the people.

This paper presents the results of the wearable computing prototype supporting trainingand qualification activities at the SKODA production facilities in Czech Republic. The emerged prototype is based upon the first of the 2 main “Variant Production Showcases” (training and assembly-line) which are to be implemented in the WearIT@work project (EC IP 004216). As an introduction, the authors of this paper investigate current training processes at Skoda, and derive the potential benefits and risks of applying wearable computing technology. Accordingly, the approach of creating the wearable prototypes, via usability experiments at the Skoda production site, is explained in detail. As a preliminary result, the first functional prototypes, including a task recognition prototype, based upon the components of the European Wearable Computing Platform, are described. The paper is rounded up by providing a short outlook regarding the second envisaged test case, which is focussed upon selected assembly line operations of blue collar workers.

As the frequency with which Classroom Response Systems (CRSs) are used is increasing, it becomes more and more important to define the affordances and limitations of these tools. Currently existing literature is largely either anecdotal or focuses on comparing CRS and non-CRS environments that are unequal in other aspects as well. In addition, the literature primarily describes situations in which the CRS is used to provide an individual as opposed to a group response. This article points to the need for a concerted research effort, one that rigorously explores conditions of use across diverse settings and pedagogies.