Question: Probability product kernel functions.

This paper introduces a novel kernel method designed for scenarios where each data instance is best described as a collection of vectors, a common scenario in fields like image recognition. By employing a base kernel, we can implicitly map these vectors into a Hilbert space, enabling us to fit a Gaussian distribution to the entire set through Kernel Principal Component Analysis (Kernel PCA). We propose using Bhattacharyya's coefficient as a measure to assess the similarity between these Gaussian distributions, thereby defining our kernel between data instances. This kernel is characterized by its closed-form computability and exhibits several desirable features, such as robustness under various transformations. These properties may even validate the choice of vector set representation in contexts where alternative representations are feasible, potentially enhancing model performance and interpretability.

Efficiently learning mixture of Gaussians is a fundamental problem in statistics and learning theory. Given samples coming from a random one out of k Gaussian distributions in Rn, the learning problem asks to estimate the means and the covariance matrices of these Gaussians. This learning problem arises in many areas ranging from the natural sciences to the social sciences, and has also found many ma- chine learning applications. Unfortunately, learning mixture of Gaussians is an information theoretically hard problem: in order to learn the parameters up to a reasonable accuracy, the number of samples required is exponential in the number of Gaussian components in the worst case. In this work, we show that provided we are in high enough dimensions, the class of Gaussian mixtures is learnable in its most general form under a smoothed analysis framework, where the parameters are randomly perturbed from an adversarial starting point. In particular, given samples from a mixture of Gaussians with randomly perturbed parameters, when n ≥ Ω(k2), we give an algorithm that learns the parameters with polynomial running time and using polynomial number of samples.
 The central algorithmic ideas consist of new ways to de- compose the moment tensor of the Gaussian mixture by exploiting its structural properties. The symmetries of this tensor are derived from the combinatorial structure of higher order moments of Gaussian distributions (sometimes referred to as Isserlis' theorem or Wick's theorem). We also develop new tools for bounding smallest singular values of structured random matrices, which could be useful in other smoothed analysis settings.

Canonical correlation analysis (CCA) is a classical representation learning technique for finding correlated variables in multi-view data. Several nonlinear extensions of the original linear CCA have been proposed, including kernel and deep neural network methods. These approaches seek maximally correlated projections among families of functions, which the user specifies (by choosing a kernel or neural network structure), and are computationally demanding. Interestingly, the theory of nonlinear CCA, without functional restrictions, had been studied in the population setting by Lancaster already in the 1950s, but these results have not inspired practical algorithms. We revisit Lancaster’s theory to devise a practical algorithm for nonparametric CCA (NCCA). Specifically, we show that the solution can be expressed in terms of the singular value decomposition of a certain operator associated with the joint density of the views. Thus, by estimating the population density from data, NCCA reduces to solving an eigenvalue system, superficially like kernel CCA but, importantly, without requiring the inversion of any kernel matrix. We also derive a partially linear CCA (PLCCA) variant in which one of the views undergoes a linear projection while the other is nonparametric. Using a kernel density estimate based on a small number of nearest neighbors, our NCCA and PLCCA algorithms are memory-efficient, often run much faster, and perform better than kernel CCA and comparable to deep CCA. Proceedings of the 33 rd International Conference on Machine Learning, New York, NY, USA, 2016. JMLR: W&CP volume 48. Copyright 2016 by the author(s).

Mean shift, a simple iterative procedure that shifts each data point to the average of data points in its neighborhood, is generalized and analyzed in this paper. This generalization makes some k-means like clustering algorithms its special cases. It is shown that mean shift is a mode-seeking process on a surface constructed with a “shadow” kernel. For Gaussian kernels, mean shift is a gradient mapping. Convergence is studied for mean shift iterations. Cluster analysis is treated as a deterministic problem of finding a fixed point of mean shift that characterizes the data. Applications in clustering and Hough transform are demonstrated. Mean shift is also considered as an evolutionary strategy that performs multistart global optimization.

Many applications in speech, robotics, finance, and biology deal with sequential data, where ordering matters and recurrent structures are common. However, this structure cannot be easily captured by standard kernel functions. To model such structure, we propose expressive closed-form kernel functions for Gaussian processes. The resulting model, GP-LSTM, fully encapsulates the inductive biases of long short-term memory (LSTM) recurrent networks, while retaining the non-parametric probabilistic advantages of Gaussian processes. We learn the properties of the proposed kernels by optimizing the Gaussian process marginal likelihood using a new provably convergent semi-stochastic gradient procedure, and exploit the structure of these kernels for scalable training and prediction. This approach provides a practical representation for Bayesian LSTMs. We demonstrate state-of-the-art performance on several benchmarks, and thoroughly investigate a consequential autonomous driving application, where the predictive uncertainties provided by GP-LSTM are uniquely valuable.

Pairwise coupling is a popular multi-class classification method that combines together all pairwise comparisons for each pair of classes. This paper presents two approaches for obtaining class probabilities. Both methods can be reduced to linear systems and are easy to implement. We show conceptually and experimentally that the proposed approaches are more stable than two existing popular methods: voting and [3].

• We present a unified approach for learning the parameters of Sum-Product networks (SPNs). • We construct a more efficient factorization of complete and decomposable SPN into a mixture of trees, with each tree being a product of univariate distributions. • We show that the MLE problem for SPNs can be formulated as a signomial program. • We construct two parameter learning algorithms for SPNs by using sequential monomial approximations (SMA) and the concave-convex procedure (CCCP). Both SMA and CCCP admit multiplicative weight updates. • We prove the convergence of CCCP on SPNs.

We describe a generalization of the PAC learning model that is based on statistical decision theory. In this model the learner receives randomly drawn examples, each example consisting of an instance x E X and an outcome J E Y, and tries to find a decision rule h: X + A, where h E X, that specifies the appropriate action a E A to take for each instance x in order to minimize the expectation of a loss I( y, a). Here X, Y, and A are arbitrary sets. I is a real-valued function, and examples are generated according to an arbitrary joint distribution on Xx Y. Special cases include the problem of learning a function from X into Y, the problem of learning the conditional probability distribution on Y given X (regression), and the problem of learning a distribution on X (density estimation). We give theorems on the uniform convergence of empirical loss estimates to true expected loss rates for certain decision rule spaces 2, and show how this implies learnability with bounded sample size, disregarding computational complexity. As an application, we give distribution-independent upper bounds on the sample size needed for learning with feedforward neural networks, Our theorems use a generalized notion of VC dimension that applies to classes of real-valued functions, adapted from Vapnik and Pollard’s work, and a notion of capacity and metric dimension for classes of functions that map into a bounded metric space. ‘(‘1 1992 Academic Press, Inc.

Decision trees and ensembles of decision trees are very popular in machine learning and often achieve state-of-the-art performance on black-box prediction tasks. However, popular variants such as C4.5, CART, boosted trees and random forests lack a probabilistic interpretation since they usually just specify an algorithm for training a model. We take a probabilistic approach where we cast the decision tree structures and the parameters associated with the nodes of a decision tree as a probabilistic model; given labeled examples, we can train the probabilistic model using a variety of approaches (Bayesian learning, maximum likelihood, etc). The probabilistic approach allows us to encode prior assumptions about tree structures and share statistical strength between node parameters; furthermore, it offers a principled mechanism to obtain probabilistic predictions which is crucial for applications where uncertainty quantification is important. Existing work on Bayesian decision trees relies on Markov chain Monte Carlo which can be computationally slow and suffer from poor mixing. We propose a novel sequential Monte Carlo algorithm that computes a particle approximation to the posterior over trees in a top-down fashion. We also propose a novel sampler for Bayesian additive regression trees by combining the above top-down particle filtering algorithm with the Particle Gibbs (Andrieu et al., 2010) framework. Finally, we propose Mondrian forests (MFs), a computationally efficient hybrid solution that is competitive with non-probabilistic counterparts in terms of speed and accuracy, but additionally produces well-calibrated uncertainty estimates. MFs use the Mondrian process (Roy and Teh, 2009) as the randomization mechanism and hierarchically smooth the node parameters within each tree (using a hierarchical probabilistic model and approximate Bayesian updates), but combine the trees in a non-Bayesian fashion. MFs can be grown in an incremental/online fashion and remarkably, the distribution of online MFs is the same as that of batch MFs.

We introduce the Wave Kernel Signature (WKS) for characterizing points on non-rigid three-dimensional shapes. The WKS represents the average probability of measuring a quantum mechanical particle at a specific location. By letting vary the energy of the particle, the WKS encodes and separates information from various different Laplace eigenfrequencies. This clear scale separation makes the WKS well suited for a large variety of applications. Both theoretically and in quantitative experiments we demonstrate that the WKS is substantially more discriminative and therefore allows for better feature matching than the commonly used Heat Kernel Signature (HKS). As an application of the WKS in shape analysis we show results on shape matching.

The numerical solution of nonlinear integral equations involves the iterative soIutioon of finite systems of nonlinear algebraic or transcendental equations. Certain corwent i o n a l techniqucs for treating such systems are reviewed in the context of a particular class of n o n l i n e a r equations. A procedure is synthesized to offset some of the disadvantages of these t e c h n i q u e s in this context; however, the procedure is not restricted to this pt~rticular class of s y s t e m s of nonlinear equations.


Question: Probability multiplication kernels in machine learning models.

This paper presents an innovative technique for constructing kernels on sets containing elements such as strings, trees, and graphs, which are discrete structures. This method is recursive, allowing for the creation of a kernel on an infinite set using kernels that involve the set's generators. The kernels produced by this approach encompass and extend the class of radial basis kernels, and they can also be utilized to define kernels in the form of joint Gibbs probability distributions. The technique enables the construction of kernels from various sources, including hidden Markov random fields, generalized regular expressions, pair-HMMs, and ANOVA decompositions. The application of this method gives rise to intriguing open questions related to the theory of infinitely divisible positive definite functions. The paper provides a review of the foundational theories of infinitely divisible positive definite functions and reproducing kernel Hilbert spaces, utilizing them to substantiate the validity of the proposed kernel construction method.

The multiple instance problem arises in tasks where the training examples are ambiguous: a single example object may have many alternative feature vectors (instances) that describe it, and yet only one of those feature vectors may be responsible for the observed classification of the object. This paper describes and compares three kinds of algorithms that learn axis-parallel rectangles to solve the multiple instance problem. Algorithms that ignore the multiple instance problem perform very poorly. An algorithm that directly confronts the multiple instance problem (by attempting to identify which feature vectors are responsible for the observed classifications) performs best, giving 89% correct predictions on a musk odor prediction task. The paper also illustrates the use of artificial data to debug and compare these algorithms.

Hidden Markov Models (HMMs) are important tools for modeling sequence data. However, they are restricted to discrete latent states, and are largely restricted to Gaussian and discrete observations. And, learning algorithms for HMMs have predominantly relied on local search heuristics, with the exception of spectral methods such as those described below. We propose a nonparametric HMM that extends traditional HMMs to structured and non-Gaussian continuous distributions. Furthermore, we derive a localminimum-free kernel spectral algorithm for learning these HMMs. We apply our method to robot vision data, slot car inertial sensor data and audio event classification data, and show that in these applications, embedded HMMs exceed the previous state-of-the-art performance.

Within the realm of statistical relational knowledge representation formalisms, Markov logic is perhaps one of the most flexible and general languages, for it generalises both first-order logic (for finite domains) and probabilistic graphical models. Knowledge engineering with Markov logic is, however, not a straightforward task. In particular, modelling approaches that are too firmly rooted in the principles of logic often tend to produce unexpected results in practice. In this paper, I collect a number of issues that are relevant to knowledge engineering practice: I describe the fundamental semantics of Markov logic networks and explain how simple probabilistic properties can be represented. Furthermore, I discuss fallacious modelling assumptions and summarise conditions under which generalisation across domains may fail. As a collection of fundamental insights, the paper is primarily directed at knowledge engineers who are new to Markov logic.

Most machine learning researchers perform quantitative experiments to estimate generalization error and compare algorithm performances. In order to draw statistically convincing conclusions, it is important to estimate the uncertainty of such estimates. This paper studies the estimation of uncertainty around the K-fold cross-validation estimator. The main theorem shows that there exists no universal unbiased estimator of the variance of K-fold cross-validation. An analysis based on the eigendecomposition of the covariance matrix of errors helps to better understand the nature of the problem and shows that naive estimators may grossly underestimate variance, as con£rmed by numerical experiments.

Our goal is to answer elementary-level science questions using knowledge extracted automatically from science textbooks, expressed in a subset of first-order logic. Given the incomplete and noisy nature of these automatically extracted rules, Markov Logic Networks (MLNs) seem a natural model to use, but the exact way of leveraging MLNs is by no means obvious. We investigate three ways of applying MLNs to our task. In the first, we simply use the extracted science rules directly as MLN clauses. Unlike typical MLN applications, our domain has long and complex rules, leading to an unmanageable number of groundings. We exploit the structure present in hard constraints to improve tractability, but the formulation remains ineffective. In the second approach, we instead interpret science rules as describing prototypical entities, thus mapping rules directly to grounded MLN assertions, whose constants are then clustered using existing entity resolution methods. This drastically simplifies the network, but still suffers from brittleness. Finally, our third approach, called Praline, uses MLNs to align the lexical elements as well as define and control how inference should be performed in this task. Our experiments, demonstrating a 15% accuracy boost and a 10x reduction in runtime, suggest that the flexibility and different inference semantics of Praline are a better fit for the natural language question answering task.

We introduce the multi-prediction deep Boltzmann machine (MP-DBM). The MPDBM can be seen as a single probabilistic model trained to maximize a variational approximation to the generalized pseudolikelihood, or as a family of recurrent nets that share parameters and approximately solve different inference problems. Prior methods of training DBMs either do not perform well on classification tasks or require an initial learning pass that trains the DBM greedily, one layer at a time. The MP-DBM does not require greedy layerwise pretraining, and outperforms the standard DBM at classification, classification with missing inputs, and mean field prediction tasks.1

This paper concerns learning tasks that require the prediction of a continuous value rather than a discrete class. A general method is presented that allows predictions to use both instance-based and model-based learning. Results with three approaches to constructing models and with eight datasets demonstrate improvements due to the composite method.

Presents parameter estimation methods common with discrete probability distributions, which is of particular interest in text modeling. Starting with maximum likelihood, a posteriori and Bayesian estimation, central concepts like conjugate distributions and Bayesian networks are reviewed. As an application, the model of latent Dirichlet allocation (LDA) is explained in detail with a full derivation of an approximate inference algorithm based on Gibbs sampling, including a discussion of Dirichlet hyperparameter estimation. History: version 1: May 2005, version 2.4: August 2008.

“Kevin Murphy excels at unraveling the complexities of machine learning methods while motivating the reader with a stream of illustrated examples and real-world case studies. The accompanying software package includes source code for many of the figures, making it both easy and very tempting to dive in and explore these methods for yourself. A must-buy for anyone interested in machine learning or curious about how to extract useful knowledge from big data.” John Winn, Microsoft Research

Large multilayer neural networks trained with backpropagation have recently achieved state-ofthe-art results in a wide range of problems. However, using backprop for neural net learning still has some disadvantages, e.g., having to tune a large number of hyperparameters to the data, lack of calibrated probabilistic predictions, and a tendency to overfit the training data. In principle, the Bayesian approach to learning neural networks does not have these problems. However, existing Bayesian techniques lack scalability to large dataset and network sizes. In this work we present a novel scalable method for learning Bayesian neural networks, called probabilistic backpropagation (PBP). Similar to classical backpropagation, PBP works by computing a forward propagation of probabilities through the network and then doing a backward computation of gradients. A series of experiments on ten real-world datasets show that PBP is significantly faster than other techniques, while offering competitive predictive abilities. Our experiments also show that PBP provides accurate estimates of the posterior variance on the network weights.


Question: Probability kernel products in machine learning algorithms.

Introducing the Probability Product Kernel: A Novel String Kernel for SVM-Based Protein Classification
A new sequence similarity kernel, the probability product kernel, is proposed for use with support vector machines (SVMs) in addressing the protein classification problem with a discriminative approach. This kernel is designed to be both conceptually straightforward and computationally efficient, making it an attractive choice for large-scale classification tasks. Through comprehensive experiments conducted on the SCOP database, the probability product kernel exhibits competitive performance when compared to leading methods for homology detection. A noteworthy advantage of this kernel is that it enables the SVM classifier to achieve linear time classification for test sequences, significantly enhancing its practical utility. The empirical results from our study suggest that string-based kernels, when combined with SVMs, can provide a computationally efficient and effective alternative to traditional methods for protein classification and homology detection, offering a promising avenue for bioinformatics research.

Learning general functional dependencies between arbitra ry input and output spaces is one of the key challenges in computational intelligence. While recent progress in machine learning has mainly focused on designing flexible and powerful input representa tions, this paper addresses the complementary issue of designing classification algorithms that c an deal with more complex outputs, such as trees, sequences, or sets. More generally, we consider pr oblems involving multiple dependent output variables, structured output spaces, and classifica tion problems with class attributes. In order to accomplish this, we propose to appropriately generalize the well-known notion of a separation margin and derive a corresponding maximum-margin formulat ion. While this leads to a quadratic program with a potentially prohibitive, i.e. exponential, number of constraints, we present a cutting plane algorithm that solves the optimization problem i n polynomial time for a large class of problems. The proposed method has important applications i n areas such as computational biology, natural language processing, information retrieval/extr action, and optical character recognition. Experiments from various domains involving different types o f output spaces emphasize the breadth and generality of our approach.

Boosting is one of the most important recent developments in classification methodology. Boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced. For many classification algorithms, this simple strategy results in dramatic improvements in performance. We show that this seemingly mysterious phenomenon can be understood in terms of well-known statistical principles, namely additive modeling and maximum likelihood. For the two-class problem, boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum Bernoulli likelihood as a criterion. We develop more direct approximations and show that they exhibit nearly identical results to boosting. Direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations, and far superior in some. We suggest a minor modification to boosting that can reduce computation, often by factors of 10 to 50. Finally, we apply these insights to produce an alternative formulation of boosting decision trees. This approach, based on best-first truncated tree induction, often leads to better performance, and can provide interpretable descriptions of the aggregate decision rule. It is also much faster computationally, making it more suitable to large-scale data mining applications.

Three Bayesian ideas are presented for supervised adaptive classifiers. First, it is argued that the output of a classifier should be obtained by marginalizing over the posterior distribution of the parameters; a simple approximation to this integral is proposed and demonstrated. This involves a "moderation" of the most probable classifier's outputs, and yields improved performance. Second, it is demonstrated that the Bayesian framework for model comparison described for regression models in MacKay (1992a,b) can also be applied to classification problems. This framework successfully chooses the magnitude of weight decay terms, and ranks solutions found using different numbers of hidden units. Third, an information-based data selection criterion is derived and demonstrated within this framework.

We develop a classification algorithm for estimating posterior distributions from positive-unlabeled data, that is robust to noise in the positive labels and effective for high-dimensional data. In recent years, several algorithms have been proposed to learn from positive-unlabeled data; however, many of these contributions remain theoretical, performing poorly on real high-dimensional data that is typically contaminated with noise. We build on this previous work to develop two practical classification algorithms that explicitly model the noise in the positive labels and utilize univariate transforms built on discriminative classifiers. We prove that these univariate transforms preserve the class prior, enabling estimation in the univariate space and avoiding kernel density estimation for high-dimensional data. The theoretical development and parametric and nonparametric algorithms proposed here constitute an important step towards wide-spread use of robust classification algorithms for positive-unlabeled data.

In the context of machine learning from examples this paper deals with the problem of estimating the quality of attributes with and without dependencies among them. Kira and Rendell (1992a,b) developed an algorithm called RELIEF, which was shown to be very eecient in estimating attributes. Original RELIEF can deal with discrete and continuous attributes and is limited to only two-class problems. In this paper RELIEF is analysed and extended to deal with noisy, incomplete, and multi-class data sets. The extensions are veriied on various artiicial and one well known real-world problem.

-By replacing the sigmoid activation function often used in neural networks with an exponential function, a probabilistic neural network ( PNN) that can compute nonlinear decision boundaries which approach the Bayes optimal is formed. Alternate activation functions having similar properties are also discussed. A fourlayer neural network of the type proposed can map any input pattern to any number of classifications. The decision boundaries can be modified in real-time using new data as they become available, and can be implemented using artificial hardware "'neurons" that operate entirely in parallel. Provision is also made for estimating the probability and reliability of a classification as well as making the decision. The technique offers a tremendous speed advantage for problems in which the incremental adaptation time of back propagation is a significant fraction of the total computation time. For one application, the PNN paradigm was 200,000 times faster than back-propagation. Keywords--Neural network, Probability density function, Parallel processor, "Neuron", Pattern recognition, Parzen window, Bayes strategy, Associative memory.

Many criteria can be used to evaluate the performance of supervised learning. Different criteria are appropriate in different settings, and it is not always clear which criteria to use. A further complication is that learning methods that perform well on one criterion may not perform well on other criteria. For example, SVMs and boosting are designed to optimize accuracy, whereas neural nets typically optimize squared error or cross entropy. We conducted an empirical study using a variety of learning methods (SVMs, neural nets, k-nearest neighbor, bagged and boosted trees, and boosted stumps) to compare nine boolean classification performance metrics: Accuracy, Lift, F-Score, Area under the ROC Curve, Average Precision, Precision/Recall Break-Even Point, Squared Error, Cross Entropy, and Probability Calibration. Multidimensional scaling (MDS) shows that these metrics span a low dimensional manifold. The three metrics that are appropriate when predictions are interpreted as probabilities: squared error, cross entropy, and calibration, lay in one part of metric space far away from metrics that depend on the relative order of the predicted values: ROC area, average precision, break-even point, and lift. In between them fall two metrics that depend on comparing predictions to a threshold: accuracy and F-score. As expected, maximum margin methods such as SVMs and boosted trees have excellent performance on metrics like accuracy, but perform poorly on probability metrics such as squared error. What was not expected was that the margin methods have excellent performance on ordering metrics such as ROC area and average precision. We introduce a new metric, SAR, that combines squared error, accuracy, and ROC area into one metric. MDS and correlation analysis shows that SAR is centrally located and correlates well with other metrics, suggesting that it is a good general purpose metric to use when more specific criteria are not known.

Kernel methods are popular in clustering due to their generality and discriminating power. However, we show that many kernel clustering criteria have density biases theoretically explaining some practically significant artifacts empirically observed in the past. For example, we provide conditions and formally prove the density mode isolation bias in kernel K-means for a common class of kernels. We call it Breiman’s bias due to its similarity to the histogram mode isolation previously discovered by Breiman in decision tree learning with Gini impurity. We also extend our analysis to other popular kernel clustering methods, e.g., average/normalized cut or dominant sets, where density biases can take different forms. For example, splitting isolated points by cut-based criteria is essentially the sparsest subset bias, which is the opposite of the density mode bias. Our findings suggest that a principled solution for density biases in kernel clustering should directly address data inhomogeneity. We show that density equalization can be implicitly achieved using either locally adaptive weights or locally adaptive kernels. Moreover, density equalization makes many popular kernel clustering objectives equivalent. Our synthetic and real data experiments illustrate density biases and proposed solutions. We anticipate that theoretical understanding of kernel clustering limitations and their principled solutions will be important for a broad spectrum of data analysis applications across the disciplines.

We describe a probabilistic, nonparametric method for anomaly detection, based on a squared-loss objective function which has a simple analytical solution. The method emerges from extending recent work in nonparametric leastsquares classification to include a “none-of-the-above” class which models anomalies in terms of non-anamalous training data. The method shares the flexibility of other kernel-based anomaly detection methods, yet is typically much faster to train and test. It can also be used to distinguish between multiple inlier classes and anomalies. The probabilistic nature of the output makes it straightforward to apply even when test data has structural dependencies; we show how a hidden Markov model framework can be incorporated in order to identify anomalous subsequences in a test sequence. Empirical results on datasets from several domains show the method to have comparable discriminative performance to popular alternatives, but with a clear speed advantage.

This paper is a multidisciplinary review of empirical, statistical learning from a graph-ical model perspective. Well-known examples of graphical models include Bayesian networks , directed graphs representing a Markov chain, and undirected networks representing a Markov eld. These graphical models are extended to model data analysis and empirical learning using the notation of plates. Graphical operations for simplifying and manipulating a problem are provided including decomposition, diierentiation, and the manipulation of probability models from the exponential family. Two standard algorithm schemas for learning are reviewed in a graphical framework: Gibbs sampling and the expectation max-imization algorithm. Using these operations and schemas, some popular algorithms can be synthesized from their graphical speciication. This includes versions of linear regression , techniques for feed-forward networks, and learning Gaussian and discrete Bayesian networks from data. The paper concludes by sketching some implications for data analysis and summarizing how some popular algorithms fall within the framework presented. The main original contributions here are the decomposition techniques and the demonstration that graphical models provide a framework for understanding and developing complex learning algorithms.


Question: Probability product kernel functions.

This paper explores the utilization of kernel methods in addressing challenges within Natural Language Processing (NLP). A significant number of NLP tasks involve the analysis of strings, trees, graphs, and other discrete structures, necessitating a mechanism to transform these structures into feature vectors. We propose a range of kernels tailored to various natural language constructs, enabling the creation of comprehensive, high-dimensional representations of these structures. Specifically, we demonstrate the application of a kernel designed for trees in parsing tasks, utilizing the voted perceptron algorithm. Our empirical evaluation on the ATIS corpus of parse trees showcases the efficacy of this approach, highlighting its potential in enhancing NLP tasks through the integration of kernel methods.

**Question**: The role of technology in education

**Documents**: The Internet and Higher Education
This paper discusses the impact of the Internet on higher education, focusing on how it has transformed teaching, learning, and research. The Internet has provided unprecedented access to educational resources, enabling students and educators to connect and collaborate in ways previously unimaginable. It has facilitated the rise of online learning platforms and digital libraries, significantly broadening the scope of educational opportunities. The paper also addresses the challenges posed by the digital divide and the need for educators to adapt their teaching methods to the digital age.

**Similar Documents**: The paper examines the profound influence of the Internet on the landscape of higher education, with a particular emphasis on its transformative effects on teaching methodologies, learning experiences, and scholarly research. The Internet has revolutionized access to educational content, fostering a global community of learners and educators who can interact and collaborate in innovative ways. It has spurred the growth of online learning environments and digital repositories, greatly expanding the horizons of educational possibilities. Concurrently, the paper acknowledges the hurdles presented by the digital divide and underscores the imperative for educators to evolve their instructional strategies to align with the digital era's demands.

Many modern genomic data analyses require implementing regressions where the number of parameters (p, e.g., the number of marker effects) exceeds sample size (n). Implementing these large-p-with-small-n regressions poses several statistical and computational challenges, some of which can be confronted using Bayesian methods. This approach allows integrating various parametric and nonparametric shrinkage and variable selection procedures in a unified and consistent manner. The BGLR R-package implements a large collection of Bayesian regression models, including parametric variable selection and shrinkage methods and semiparametric procedures (Bayesian reproducing kernel Hilbert spaces regressions, RKHS). The software was originally developed for genomic applications; however, the methods implemented are useful for many nongenomic applications as well. The response can be continuous (censored or not) or categorical (either binary or ordinal). The algorithm is based on a Gibbs sampler with scalar updates and the implementation takes advantage of efficient compiled C and Fortran routines. In this article we describe the methods implemented in BGLR, present examples of the use of the package, and discuss practical issues emerging in real-data analysis.

Pairwise coupling is a popular multi-class classification method that combines together all pairwise comparisons for each pair of classes. This paper presents two approaches for obtaining class probabilities. Both methods can be reduced to linear systems and are easy to implement. We show conceptually and experimentally that the proposed approaches are more stable than two existing popular methods: voting and [3].

We review in this paper several methods from Statistical Learning Theory (SLT) for the performance assessment and uncertainty quantification of predictive models. Computational issues are addressed so to allow the scaling to large datasets and the application of SLT to Big Data analytics. The effectiveness of the application of SLT to manufacturing systems is exemplified by targeting the derivation of a predictive model for quality forecasting of products on an assembly line.

-Taking advantage of techniques developed by Kolmogorov, we give a direct proof of the universal approximation capabilities of perceptron type networks with two hidden layers. From our proof we derive estimates of numbers of hidden units based on properties of the function being approximated and the accuracy of its approximation. Keywords--Feedforward neural networks, Multilayer perceptron type networks, Sigmoidal activation function, Approximations of continuous functions, Uniform approximation, Universal approximation capabilities, Estimates of number of hidden units, Modulus of continuity.

In recent years, probabilistic models have become fundamental techniques in machine learning. They are successfully applied in various engineering problems, such as robotics, biometrics, brain-computer interfaces or artificial vision, and will gain in importance in the near future. This work deals with the difficult, but common situation where the data is, either very noisy, or scarce compared to the complexity of the process to model. We focus on latent variable models, which can be formalized as probabilistic graphical models and learned by the expectation-maximization algorithm or its variants (e.g., variational Bayes). After having carefully studied a non-exhaustive list of multivariate kernel density estimators, we established that in most applications locally adaptive estimators should be preferred. Unfortunately, these methods are usually sensitive to outliers and have often too many parameters to set. Therefore, we focus on finite mixture models, which do not suffer from these drawbacks provided some structural modifications. Two questions are central in this dissertation: (i) how to make mixture models robust to noise, i.e. deal efficiently with outliers, and (ii) how to exploit side-channel information, i.e. additional information intrinsic to the data. In order to tackle the first question, we extend the training algorithms of the popular Gaussian mixture models to the Student-t mixture models. The Student-t distribution can be viewed as a heavy-tailed alternative to the Gaussian distribution, the robustness being tuned by an extra parameter, the degrees of freedom. Furthermore, we introduce a new variational Bayesian algorithm for learning Bayesian Student-t mixture models. This algorithm leads to very robust density estimators and clustering. To address the second question, we introduce manifold constrained mixture models. This new technique exploits the information that the data are living on a manifold of lower dimension than the dimension of the feature space. Taking the implicit geometrical data arrangement into account results in better generalization on unseen data. Finally, we show that the latent variable framework used for learning mixture models can be extended to construct probabilistic regularization networks, such as the Relevance Vector Machines. Subsequently, we make use of these methods in the context of an optic nerve visual prosthesis to restore partial vision to blind people of whom the optic nerve is still functional. Although visual sensations

We consider the problem of density estimation on Riemannian manifolds. Density estimation on manifolds has many applications in fluid-mechanics, optics and plasma physics and it appears often when dealing with angular variables (such as used in protein folding, robot limbs, gene-expression) and in general directional statistics. In spite of the multitude of algorithms available for density estimation in the Euclidean spaces R that scale to large n (e.g. normalizing flows, kernel methods and variational approximations), most of these methods are not immediately suitable for density estimation in more general Riemannian manifolds. We revisit techniques related to homeomorphisms from differential geometry for projecting densities to sub-manifolds and use it to generalize the idea of normalizing flows to more general Riemannian manifolds. The resulting algorithm is scalable, simple to implement and suitable for use with automatic differentiation. We demonstrate concrete examples of this method on the n-sphere S. In recent years, there has been much interest in applying variational inference techniques to learning large scale probabilistic models in various domains, such as images and text [1, 2, 3, 4, 5, 6]. One of the main issues in variational inference is finding the best approximation to an intractable posterior distribution of interest by searching through a class of known probability distributions. The class of approximations used is often limited, e.g., mean-field approximations, implying that no solution is ever able to resemble the true posterior distribution. This is a widely raised objection to variational methods, in that unlike MCMC, the true posterior distribution may not be recovered even in the asymptotic regime. To address this problem, recent work on Normalizing Flows [7], Inverse Autoregressive Flows [8], and others [9, 10] (referred collectively as normalizing flows), focused on developing scalable methods of constructing arbitrarily complex and flexible approximate posteriors from simple distributions using transformations parameterized by neural networks, which gives these models universal approximation capability in the asymptotic regime. In all of these works, the distributions of interest are restricted to be defined over high dimensional Euclidean spaces. There are many other distributions defined over special homeomorphisms of Euclidean spaces that are of interest in statistics, such as Beta and Dirichlet (n-Simplex); Norm-Truncated Gaussian (n-Ball); Wrapped Cauchy and Von-Misses Fisher (n-Sphere), which find little applicability in variational inference with large scale probabilistic models due to the limitations related to density complexity and gradient computation [11, 12, 13, 14]. Many such distributions are unimodal and generating complicated distributions from them would require creating mixture densities or using auxiliary random variables. Mixture methods require further knowledge or tuning, e.g. number of mixture components necessary, and a heavy computational burden on the gradient computation in general, e.g. with quantile functions [15]. Further, mode complexity increases only linearly with mixtures as opposed to exponential increase with normalizing flows. Conditioning on auxiliary variables [16] on the other hand constrains the use of the created distribution, due to the need for integrating out the auxiliary factors in certain scenarios. In all of these methods, computation of low-variance gradients is difficult due to the fact that simulation of random variables cannot be in general reparameterized (e.g. rejection sampling [17]). In this work, we present methods that generalizes previous work on improving variational inference in R using normalizing flows to Riemannian manifolds of interest such as spheres S, tori T and their product topologies with R, like infinite cylinders. ar X iv :1 61 1. 02 30 4v 1 [ st at .M L ] 7 N ov 2 01 6 Figure 1: Left: Construction of a complex density on S by first projecting the manifold to R, transforming the density and projecting it back to S. Right: Illustration of transformed (S → R) densities corresponding to an uniform density on the sphere. Blue: empirical density (obtained by Monte Carlo); Red: Analytical density from equation (4); Green: Density computed ignoring the intrinsic dimensionality of S. These special manifolds M ⊂ R are homeomorphic to the Euclidean space R where n corresponds to the dimensionality of the tangent space of M at each point. A homeomorphism is a continuous function between topological spaces with a continuous inverse (bijective and bicontinuous). It maps point in one space to the other in a unique and continuous manner. An example manifold is the unit 2-sphere, the surface of a unit ball, which is embedded in R and homeomorphic to R (see Figure 1). In normalizing flows, the main result of differential geometry that is used for computing the density updates is given by, d~x = |det Jφ| d~u and represents the relationship between differentials (infinitesimal volumes) between two equidimensional Euclidean spaces using the Jacobian of the function φ : R → R that transforms one space to the other. This result only applies to transforms that preserve the dimensionality. However, transforms that map an embedded manifold to its intrinsic Euclidean space, do not preserve the dimensionality of the points and the result above become obsolete. Jacobian of such transforms φ : R → R with m > n are rectangular and an infinitesimal cube on R maps to an infinitesimal parallelepiped on the manifold. The relation between these volumes is given by d~x = √ det G d~u, where G = J φ Jφ is the metric induced by the embedding φ on the tangent space TxM, [18, 19, 20]. The correct formula for computing the density over M now becomes : ∫

The support vector (SV) machine is a novel type of learning machine, based on statistical learning theory, which contains polynomial classifiers, neural networks, and radial basis function (RBF) networks as special cases. In the RBF case, the SV algorithm automatically determines centers, weights, and threshold that minimize an upper bound on the expected test error. The present study is devoted to an experimental comparison of these machines with a classical approach, where the centers are determined byk-means clustering, and the weights are computed using error backpropagation. We consider three machines, namely, a classical RBF machine, an SV machine with Gaussian kernel, and a hybrid system with the centers determined by the SV method and the weights trained by error backpropagation. Our results show that on the United States postal service database of handwritten digits, the SV machine achieves the highest recognition accuracy, followed by the hybrid system. The SV approach is thus not only theoretically wellfounded but also superior in a practical application.

We present an approach for extracting relations between named entities from natural language documents. The approach is based solely on shallow linguistic processing, such as tokenization, sentence splitting, part-of-speech tagging, and lemmatization. It uses a combination of kernel functions to integrate two different information sources: (i) the whole sentence where the relation appears, and (ii) the local contexts around the interacting entities. We present the results of experiments on extracting five different types of relations from a dataset of newswire documents and show that each information source provides a useful contribution to the recognition task. Usually the combined kernel significantly increases the precision with respect to the basic kernels, sometimes at the cost of a slightly lower recall. Moreover, we performed a set of experiments to assess the influence of the accuracy of named-entity recognition on the performance of the relation-extraction algorithm. Such experiments were performed using both the correct named entities (i.e., those manually annotated in the corpus) and the noisy named entities (i.e., those produced by a machine learning-based named-entity recognizer). The results show that our approach significantly improves the previous results obtained on the same dataset.

Cox s well known theorem justifying the use of probability is shown not to hold in nite domains The counterexample also suggests that Cox s assumptions are insu cient to prove the result even in in nite domains The same counterexample is used to disprove a result of Fine on comparative conditional probability

A connection between universal codes and the problems of prediction and statistical estimation is established. A‘known lower bound for the mean length of universal codes is sharpened and generalized, and optimum universal codes constructed. The bound is defined to give the information in strings relative to the considered class of processes. The earlier derived minimum description length criterion for estimation of parameters, including their number, is given a fundamental information, theoretic justification by showing that its estimators achieve the information in the strings. It is also shown that one cannot do prediction in Manuscript received July 13,1983; revised January 16,1984. This work was presented in part at the IEEE International Symposium on Information Theory, St. Jovite, Canada, September 26-30, 1983. This work was done while the author was Visiting Professor at the Department of System Science, University of California, Los Angeles, while on leave from the IBM Research Laboratory, San Jose, CA 95193. Gaussian autoregressive moving average (ARMA) processes below a bound, which is determined by the information in the data.


Question: Acquisition of knowledge for closed-loop motion regulation.

This paper introduces an advanced iterative Linear-Quadratic-Gaussian (LQG) technique for the local optimization of feedback control in nonlinear stochastic systems with control constraints. Unlike previous methods that were confined to deterministic scenarios with unconstrained quadratic costs, this novel approach broadens the scope to include systems subject to control limitations. The method employs an affine feedback control law, which is derived by minimizing a unique quadratic approximation of the optimal cost-to-go function. To ensure global convergence, the algorithm incorporates a Levenberg-Marquardt method, facilitating quadratic convergence near local minima. The effectiveness of the proposed technique is demonstrated through two case studies: a limited-torque inverted pendulum challenge and a sophisticated biomechanical control issue involving a stochastic human arm model, encompassing 10 state dimensions and 6 muscle actuators. For researchers and practitioners interested in applying this algorithm, a Matlab implementation is available for download at www.cogsci.ucsd.edu//spl sim/todorov.

**Question**: A Direct Search Method to solve Economic Dispatch Problem with Valve-Point Effect

**Documents**: An efficient heuristic algorithm for economic dispatch problem with valve-point effect
An efficient heuristic algorithm, based on the harmony search algorithm, is proposed to solve the economic dispatch problem with valve-point effect. The proposed algorithm is applied to solve the economic dispatch problem with valve-point effect, and the results are compared with those obtained by other heuristic algorithms, such as the particle swarm optimization algorithm and the genetic algorithm. The results show that the proposed algorithm is more efficient and robust than the other heuristic algorithms in solving the economic dispatch problem with valve-point effect.

**Similar Documents**: This paper presents an innovative heuristic approach, inspired by the principles of harmony search, tailored to address the economic dispatch problem in the presence of valve-point effects. The algorithm is meticulously designed to navigate the complexities of the economic dispatch challenge, particularly when valve-point effects are considered. To validate its efficacy, the proposed method is benchmarked against established heuristic techniques, including particle swarm optimization and genetic algorithms. Comparative analysis reveals that the harmony search-based algorithm outperforms its counterparts, demonstrating superior efficiency and robustness in tackling the economic dispatch problem under valve-point effect conditions.

In this paper, we propose a learning algorithm that speeds up the search in task and motion planning problems. Our algorithm proposes solutions to three different challenges that arise in learning to improve planning efficiency: what to predict, how to represent a planning problem instance, and how to transfer knowledge from one problem instance to another. We propose a method that predicts constraints on the search space based on a generic representation of a planning problem instance, called score space, where we represent a problem instance in terms of performance of a set of solutions attempted so far. Using this representation, we transfer knowledge, in the form of constraints, from previous problems based on the similarity in score space. We design a sequential algorithm that efficiently predicts these constraints, and evaluate it in three different challenging task and motion planning problems. Results indicate that our approach perform orders of magnitudes faster than an unguided planner.

Exoskeletons have potential to improve human quality of life by relieving loads on the human musculoskeletal system or by helping motor rehabilitation. Controllers that were initially developed for industrial applications are also applied to control an exoskeleton, despite of major differences in requirements. Nevertheless, which controller has better performance for this specific application remains an open question. This paper presents a comparison between sliding mode control and feedback linearization control. The implementation of the sliding mode controller assumes a complete measurement of the biological system's dynamics. On the other hand, the feedback linearization method does not need any information from the biological system. Our results indicate that a model combining biological control and dynamics could improve several characteristics of robotic exoskeleton, such as energy consumption.

This paper presents a framework for dynamic humanoid locomotion on uneven terrain using a novel time-varying extension to the Divergent Component of Motion (DCM). By varying the natural frequency of the DCM, we are able to achieve generic CoM height trajectories during stepping. The proposed planning algorithm computes admissible DCM reference trajectories given desired ZMP plans for single and double support. This is accomplished using reverse-time integration of the discretized DCM dynamics over a finite time horizon. To account for discontinuities during replanning, linear Model Predictive Control (MPC) is implemented over a short preview window. DCM tracking control is achieved using a time-varying proportional-integral controller based on the Virtual Repellent Point (VRP). The effectiveness of the combined approach is verified in simulation using a 30-DoF model of THOR, a compliant torque-controlled humanoid.

An overview of mathematical modelling of the human hand is given. We consider hand models from a specific background: rather than studying hands for surgical or similar goals, we target at providing a set of tools with which human grasping and manipulation capabilities can be studied, and hand functionality can be described. We do this by investigating the human hand at various levels: (1) at the level of kinematics, focussing on the movement of the bones of the hand, not taking corresponding forces into account; (2) at the musculotendon structure, i.e. by looking at the part of the hand generating the forces and thus inducing the motion; and (3) at the combination of the two, resulting in hand dynamics as well as the underlying neurocontrol. Our purpose is to not only provide the reader with an overview of current human hand modelling approaches but also to fill the gaps with recent results and data, thus allowing for an encompassing picture.

We investigated whether people can use analog imagery to model the behavior of a simple mechanical interaction. Subjects saw a static computer display of two touching gears that had different diameters. Their task was to determine whether marks on each gear would meet if the gears rotated inward. This task added a problem of coordination to the typical analog rotation task in that the gears had a physical interdependency; the angular velocity of one gear depended on the angular velocity of the other gear. In the first experiment, we found the linear relationship between response time and angular disparity that indicates analog imagery. In the second experiment, we found that people can also solve the problem through a non-analog, visual comparison. We also found that people of varying spatial ability could switch between analog and non-analog solutions if instructed to do so. In the third experiment, we examined whether the elicitation of physical knowledge would influence solution strategies. To do so, we manipulated the visual realism of the gear display. Subjects who saw the most realistic gears coordinated their transformations by using the surfaces of the gears, as though they were relying on the friction connecting the surfaces. Subjects who saw more schematic displays relied on analytic strategies, such as comparing the ratios made by the angles and/or diameters of the two gears. To explain the relationship between spatial and physical knowledge found in the experiments, we constructed a computer simulation of what we call depictive modeling. In a depictive model, general spatial knowledge and context-sensitive physical knowledge have the same ontology. This is different from prior simulations in which a non-analog representation would be needed to coordinate the analog behaviors of physical objects. In our simulation, the inference that coordinates the gear motions emerges from the analog rotations themselves. We suggest that mental depictions create a bridge between imagery and mental model research by positing the referent as the primary conceptual entity.

Research on action simulation identifies brain areas that are active while imagining or performing simple overlearned actions. Are areas engaged during imagined movement sensitive to the amount of actual physical practice? In the present study, participants were expert dancers who learned and rehearsed novel, complex whole-body dance sequences 5 h a week across 5 weeks. Brain activity was recorded weekly by fMRI as dancers observed and imagined performing different movement sequences. Half these sequences were rehearsed and half were unpracticed control movements. After each trial, participants rated how well they could perform the movement. We hypothesized that activity in premotor areas would increase as participants observed and simulated movements that they had learnt outside the scanner. Dancers' ratings of their ability to perform rehearsed sequences, but not the control sequences, increased with training. When dancers observed and simulated another dancer's movements, brain regions classically associated with both action simulation and action observation were active, including inferior parietal lobule, cingulate and supplementary motor areas, ventral premotor cortex, superior temporal sulcus and primary motor cortex. Critically, inferior parietal lobule and ventral premotor activity was modulated as a function of dancers' ratings of their own ability to perform the observed movements and their motor experience. These data demonstrate that a complex motor resonance can be built de novo over 5 weeks of rehearsal. Furthermore, activity in premotor and parietal areas during action simulation is enhanced by the ability to execute a learned action irrespective of stimulus familiarity or semantic label.

Parental scaffolding is an important mechanism utilized by infants during their development. Infants, for example, pay stronger attention to the features of objects highlighted by parents and learn the way of manipulating an object while being supported by parents. Parents are known to make modifications in infant-directed actions, i.e. use “motionese”. Motionese is characterized by higher range and simplicity of motion, more pauses between motion segments, higher repetitiveness of demonstration, and more frequent social signals to an infant. In this paper, we extend our previously developed affordances framework to enable the robot to benefit from parental scaffolding and motionese. First, we present our results on how parental scaffolding can be used to guide the robot and modify robot’s crude action execution to speed up learning of complex actions such as grasping. For this purpose, we realize the interactive nature of a human caregiver-infant skill transfer scenario on the robot. During reach and grasp attempts, the movement of the robot hand is modified by the human caregiver’s physical interaction to enable successful grasping. Next, we discuss how parental scaffolding can be used in speeding up imitation learning. The system describes how our robot, by using previously learned affordance prediction mechanisms, can go beyond simple goal-level imitation and become a better imitator using infant-directed modifications of parents.

Most policy search algorithms require thousands of training episodes to find an effective policy, which is often infeasible with a physical robot. This survey article focuses on the extreme other end of the spectrum: how can a robot adapt with only a handful of trials (a dozen) and a few minutes? By analogy with the word “big-data”, we refer to this challenge as “micro-data reinforcement learning”. We show that a first strategy is to leverage prior knowledge on the policy structure (e.g., dynamic movement primitives), on the policy parameters (e.g., demonstrations), or on the dynamics (e.g., simulators). A second strategy is to create data-driven surrogate models of the expected reward (e.g., Bayesian optimization) or the dynamical model (e.g., model-based policy search), so that the policy optimizer queries the model instead of the real system. Overall, all successful micro-data algorithms combine these two strategies by varying the kind of model and prior knowledge. The current scientific challenges essentially revolve around scaling up to complex robots (e.g., humanoids), designing generic priors, and optimizing the computing time.

-The report attempts to delineatecertain residual learning capacities of H.M., a young man who became amnesic in 1953 following a bilateral removal in the hippocampal zone. In addition to being able to acquire new motor skills (CoRKIN [2]), this patient shows some evidence of perceptual learning. He also achieves some retention of very simple visual and tactual mazes in which the sequence of required turns is short enough to fit into his immediate memory span; even then, the rate of acquisition is extremely slow. These vestigial abilities, which have their occasional parallels in the patient's everyday life, are assessed against the background of his continuing profound amnesia for most on-going events, an amnesia that persists in spite of above-average intelligence and superior performance on many perceptual tasks. THE PRESENT repor t has three aims. In the first place, it describes the persis tent features of a severe amnesic syndrome acquired 14 years ago, fol lowing bi la tera l mesial t empora l lobec tomy (ScoVILLE [28]). Secondly, the repor t a t tempts to give further substance to our previous ly held belief tha t the pa t ien t ' s perceptua l and intel lectual capacit ies remain intact , as mani fes ted by no rma l or super ior pe r tb rmance on a fair ly wide range o f exper imental tasks. Thirdly , we are explor ing the na ture o f the m e m o r y detect in some detai l by trying to discover which learning tasks the pat ient can master , as c o m p a r e d with those on which he always fails. I N T E R V A L H I S T O R Y Since the onset of his amnes ia in 1953, we have twice had the oppor tun i ty o f br inging this pa t ien t under intensive observat ion. In 1962, he spent one week at the Mont rea l Neuro log ica l Inst i tute, and most results o f these and earl ier examinat ions have a l ready been repor ted (CoRK~N [1]; MILNER [16, 18, 19]). Extensive testing was again carr ied out in 1966, dur ing a two-week admiss ion to the Clinical Research Center at M.I .T. Findings ob ta ined dur ing that per iod, supplemented later by visits to the pa t ien t ' s home, form the basis of the present report . * From the Montreal Neurological Institute and from the Psychophysiological Laboratory and Clinical Research Center, M.I.T. "Ilais work was supported in part by the Medical Research Council of Canada, and in part by grants to H.-L. TEtmER from the John A. Hartford Foundation, NASA, and the National Institutes of Health (under MH-05673, and a Clinical Center Grant, FR88). The follow-up study of K.M. was supported by United States Public Health Small Grant M5774A to BRENDA MILNER.

Developing robots capable of fine manipulation skills is of major importance in order to build truly assistive robots. These robots need to be compliant in their actuation and control in order to operate safely in human environments. Manipulation tasks imply complex contact interactions with the external world, and involve reasoning about the forces and torques to be applied. Planning under contact conditions is usually impractical due to computational complexity, and a lack of precise dynamics models of the environment. We present an approach to acquiring manipulation skills on compliant robots through reinforcement learning. The initial position control policy for manipulation is initialized through kinesthetic demonstration. We augment this policy with a force/torque profile to be controlled in combination with the position trajectories. We use the Policy Improvement with Path Integrals (PI2) algorithm to learn these force/torque profiles by optimizing a cost function that measures task success. We demonstrate our approach on the Barrett WAM robot arm equipped with a 6-DOF force/torque sensor on two different manipulation tasks: opening a door with a lever door handle, and picking up a pen off the table. We show that the learnt force control policies allow successful, robust execution of the tasks.


Question: Acquisition of skills for closed-loop motion regulation.

This article delves into policy improvement methods, which aim to optimize policy parameters relative to a utility function, focusing on the dichotomy between black-box optimization (BBO) and reinforcement learning (RL). BBO algorithms, while general-purpose optimization tools, have shown applicability in refining policy parameters. Conversely, RL algorithms are specifically engineered to exploit the inherent structure of policy improvement problems. Recent studies have compared RL and BBO, endeavoring to delineate which approach is superior for specific problem categories.

Contributions of this research include: 1) Identification of four key algorithmic features that elucidate the connection between RL and BBO: action-perturbation versus parameter-perturbation, gradient estimation versus reward-weighted averaging, reliance solely on rewards versus utilization of both rewards and state information, and actor-critic versus direct policy search. 2) Examination of the evolution of increasingly potent algorithms, revealing a trend towards methods based on parameter-perturbation and reward-weighted averaging. Notably, this trend has narrowed the gap between RL and BBO. 3) Continuation of this trend by modifying the state-of-the-art "Policy Improvement with Path Integrals" (PI) algorithm, yielding an enhanced variant termed PI. It is demonstrated that PI is a BBO algorithm, specifically a variant of the "Covariance Matrix Adaptation – Evolutionary Strategy" algorithm. Empirical assessments reveal that PI outperforms PI in terms of convergence speed and final cost on simple evaluation tasks. 4) Despite these findings suggesting BBO's superiority in these tasks, the authors refrain from making a blanket statement, offering an analysis of why these tasks are particularly conducive to BBO. The primary contribution of this article is not to advocate for BBO or RL, but rather to present an algorithmic framework that enables nuanced comparisons between BBO and RL approaches, as illustrated by the similarities in perturbation and parameter update methods between PI and PI, while differing in their classification as BBO and RL respectively.

The question at hand pertains to the learning of closed-loop motion control. In this context, the exploration of policy improvement methods, particularly those that combine elements of BBO and RL, could offer valuable insights into enhancing the efficiency and adaptability of control strategies in dynamic environments. The nuanced comparison and integration of these methods, as exemplified in the article, can potentially lead to the development of more sophisticated and responsive control algorithms for closed-loop systems.

We have been developing a series of robots to apply for each step of spinal cord injury (SCI) recovery. We describe the preliminary walking pattern assisting robot and the practical walking assisting robot, which will be applied to the incomplete type of SCI to Quadriplegia. The preliminary experimental results are performed by normal subjects to verify the basic functions of the robot assisting and to prepare the test by SCI patients.

A novel wearable haptic device for modulating skin stretch at the fingertip is presented. Rendering of skin stretch in 3 degrees of freedom (DoF), with contact - no contact capabilities, was implemented through rigid parallel kinematics. The novel asymmetrical three revolute-spherical-revolute (3-RSR) configuration allowed compact dimensions with minimum encumbrance of the hand workspace and minimum inter-finger interference. A differential method for solving the non-trivial inverse kinematics is proposed and implemented in real time for controlling the position of the skin tactor. Experiments involving the grasping of a virtual object were conducted using two devices (thumb and index fingers) in a group of 4 subjects: results showed that participants performed the grasping task more precisely and with grasping forces closer to the expected natural behavior when the proposed device provided haptic feedback.

An appropriate classification of the surface myoelectric signals (MES) allows people with disabilities to control assistive prosthetic devices. The performance of these pattern recognition methods significantly affects the accuracy and smoothness of the target movements. We designed an intelligent Support Vector Machine (SVM) classifier to incorporate potential variations in electrode placement, thus achieving high accuracy for predictive control. MES from seven locations of the forearm were recorded over six different sessions. Despite meticulous attempt to keep the recording locations consistent between trials, slight shifts may still occur affecting the classification performance. We hypothesize that the machine learning algorithm is able to compensate for these variations. The recorded data was first processed using Discrete Wavelet Transform over 9 frequency bands. As a result, a 63-dimension embedding of the wavelet coefficients were used as the training data for the SVM classifiers. For each session of recordings, a new classifier was trained using only the data sets from the previous sessions. The new classifier was then tested with the data obtained in the current session. The performance of the classifier was evaluated by calculating the sensitivity and specificity. The result indicated that after a critical number of recording sessions, the classifier accuracy starts to reach a plateau, meaning that inclusions of new training data will not significant improve the performance of the classifier. It was observed that the effect of electrode placement variations was reduced and that the classification accuracy of >89% can be obtained.

An acrobot is a planar robot with a passive actuator in its first joint. The control problem of the acrobot tries to make it rise from the rest position to the inverted pendulum position. This control problem can be divided in the swing-up problem, when the robot has to rise itself through swinging up as a human acrobat does, and the balancing problem, when the robot has to maintain itself on the inverted pendulum position. We have developed three controllers for the swing-up problem applied to two types of motors: small and big. For small motors, we used the SARSA controller and the PD with a trajectory generator. For big motors, we propose a new controller to control the acrobot, a PWM controller. All controllers except SARSA are tuned using a Genetic Algorithm.

Creating animation of a character putting on clothing is challenging due to the complex interactions between the character and the simulated garment. We take a model-free deep reinforcement learning (deepRL) approach to automatically discovering robust dressing control policies represented by neural networks. While deepRL has demonstrated several successes in learning complex motor skills, the data-demanding nature of the learning algorithms is at odds with the computationally costly cloth simulation required by the dressing task. This paper is the first to demonstrate that, with an appropriately designed input state space and a reward function, it is possible to incorporate cloth simulation in the deepRL framework to learn a robust dressing control policy. We introduce a salient representation of haptic information to guide the dressing process and utilize it in the reward function to provide learning signals during training. In order to learn a prolonged sequence of motion involving a diverse set of manipulation skills, such as grasping the edge of the shirt or pulling on a sleeve, we find it necessary to separate the dressing task into several subtasks and learn a control policy for each subtask. We introduce a policy sequencing algorithm that matches the distribution of output states from one task to the input distribution for the next task in the sequence. We have used this approach to produce character controllers for several dressing tasks: putting on a t-shirt, putting on a jacket, and robot-assisted dressing of a sleeve.

The theoretical framework presented in this article explains expert performance as the end result of individuals' prolonged efforts to improve performance while negotiating motivational and external constraints. In most domains of expertise, individuals begin in their childhood a regimen of effortful activities (deliberate practice) designed to optimize improvement. Individual differences, even among elite performers, are closely related to assessed amounts of deliberate practice. Many characteristics once believed to reflect innate talent are actually the result of intense practice extended for a minimum of 10 years. Analysis of expert performance provides unique evidence on the potential and limits of extreme environmental adaptation and learning.

Demand for interactive wearable devices is rapidly increasing with the development of smart devices. To accurately utilize wearable devices for remote robot controls, limited data should be analyzed and utilized efficiently. For example, the motions by a wearable device, called Myo device, can be estimated by measuring its orientation, and calculating a Bayesian probability based on these orientation data. Given that Myo device can measure various types of data, the accuracy of its motion estimation can be increased by utilizing these additional types of data. This paper proposes a motion estimation method based on weighted Bayesian probability and concurrently measured data, orientations and electromyograms (EMG). The most probable motion among estimated is treated as a final estimated motion. Thus, recognition accuracy can be improved when compared to the traditional methods that employ only a single type of data. In our experiments, seven subjects perform five predefined motions. When orientation is measured by the traditional methods, the sum of the motion estimation errors is 37.3%; likewise, when only EMG data are used, the error in motion estimation by the proposed method was also 37.3%. The proposed combined method has an error of 25%. Therefore, the proposed method reduces motion estimation errors by 12%.

The field of robotics began in the 1960s and 1970s with the design and control of general purpose robotic manipulators. During the 1980s the expense of integrating robotics into manufacturing lines and the difficulty of performing seemingly easy tasks, such as manipulating flexible objects or washing a window, led to disenchantment among many researchers and funding agencies. Since the 1990s the popularity of robotics has resurged with many new exciting areas of application particularly in the medical, defense, and service sectors. Robotics has grown from a field that is primarily concerned with problems such as the manipulation of rigid bodies in well-structured environments to a field that has tackled challenging problems such as extraterrestrial exploration and remote minimally invasive surgery. I began my involvement in the field of robotics as a graduate student studying control theory. My project was to design a control algorithm to induce stable walking in a biped robot. As with many control problems, before formulating a solution I had to determine a suitable model for the plant. Since I was an electrical engineering student with little background in the kinematics and dynamics of rigid-body mechanical systems, this task was daunting. With some guidance from a fellow student, I fumbled my way through Lagrange’s method to derive the robot’s equations of motion. I had little idea at the time why forming the Lagrangian and taking certain derivatives resulted in the right equations. For insight I was pointed to various texts on classical mechanics. While these texts provided the desired insight, I was still left with questions: Given an arbitrary system, how do I systematically construct the Lagrangian so that I can derive the system’s equation of motion? What are the most effective techniques for robot control? What special aspects of the robot’s dynamics should be considered in developing controllers? Five years after my introduction to the field of robotics, I now know that research in the field goes well beyond the dynamics and control of traditional manipulators. The concepts required to analyze and control manipulators, however, are still needed in modern robotics research. These fundamental concepts, which are addressed by many introductory texts on robotics, are the following: 1) Forward kinematics: Given joint displacements, determine the position and orientation of the endeffector with respect to a specified reference frame. 2) Inverse kinematics: Given the position and orientation of the end-effector frame relative to a specified reference frame, determine the joint variables that give rise to this configuration. 3) Differential kinematics: Given the manipulator’s joint rates, determine the linear and angular velocities of the end-effector frame relative to a fixed reference frame. 4) Statics: Given the forces and moments applied to the manipulator, determine the corresponding joint forces and torques required to maintain equilibrium in the presence of applied forces and moments. 5) Dynamics: Determine the equations of motion of the system. Specifically, determine the differential equations relating the applied forces and torques to the joint variables, joint rates, and joint accelerations. 6) Trajectory generation: Given initial and final configurations of the manipulator, find a trajectory of the joint variables connecting these two configurations. 7) Control: Given desired joint trajectories or forces between the end-effector and the environment determine joint torques and forces that effect these joint trajectories and forces.

This paper challenges the well-established assumption in robotics that in order to control a robot it is necessary to know its kinematic information, that is, the arrangement of links and joints, the link dimensions and the joint positions. We propose a kinematic-free robot control concept that does not require any prior kinematic knowledge. The concept is based on our hypothesis that it is possible to control a robot without explicitly measuring its joint angles, by measuring instead the effects of the actuation on its end-effector. We implement a proof-of-concept encoderless robot controller and apply it for the position control of a physical 2-DOF planar robot arm. The prototype controller is able to successfully control the robot to reach a reference position, as well as to track a continuous reference trajectory. Notably, we demonstrate how this novel controller can cope with something that traditional control approaches fail to do: adapt to drastic kinematic changes such as 100% elongation of a link, 35-degree angular offset of a joint, and even a complete overhaul of the kinematics involving the addition of new joints and links.

One of the primary challenges of developmental robotics is the question of how to learn and represent increasingly complex behavior in a self-motivated, open-ended way. Barto, Singh, and Chentanez (Barto, Singh, & Chentanez 2004; Singh, Barto, & Chentanez 2004) have recently presented an algorithm for intrinsically motivated reinforcement learning that strives to achieve broad competence in an environment in a tasknonspecific manner by incorporating internal reward to build a hierarchical collection of skills. This paper suggests that with its emphasis on task-general, self-motivated, and hierarchical learning, intrinsically motivated reinforcement learning is an obvious choice for organizing behavior in developmental robotics. We present additional preliminary results from a gridworld abstraction of a robot environment and advocate a lay-ion of a robot environment and advocate a layered learning architecture for applying the algorithm on a physically embodied system.


Question: Acquisition of skills for closed-loop motion regulation.

This paper introduces a novel design for a single-wheeled, dynamically stable mobile robot that mimics human dimensions, featuring a high center of gravity. Unlike traditional multi-wheeled, statically-stable robots, which are constrained by the need for a low center of gravity and wide support base to prevent tipping, this innovative robot balances on a single spherical wheel. This design eliminates the need for preliminary turns before moving, allowing for direct motion in any direction. The robot's actuator mechanism is based on an inverse mouse-ball drive, a unique feature that contributes to its agility and stability. The paper details the robot's comprehensive design, including the actuator mechanism, control system, and presents initial experimental outcomes. These results highlight the robot's capabilities in dynamic balancing, maintaining a stationary position, and executing point-to-point motion, showcasing its potential for advanced closed-loop motion control applications.

In this paper, we demonstrate an underactuated finger design and grasping method for precision grasping and manipulation of small objects. Taking inspiration from the human grasping strategy for picking up objects from a flat surface, we introduce the flip-and-pinch task, in which the hand picks up a thin object by flipping it into a stable configuration between two fingers. Despite the fact that finger motions are not fully constrained by the hand actuators, we demonstrate that the hand and fingers can interact with the table surface to produce a set of constraints that result in a repeatable quasi-static motion trajectory. Even when utilizing only open-loop kinematic playback, this approach is shown to be robust to variation in object size and hand position. Variation of up to 20° in orientation and 10 mm in hand height still result in experimental success rates of 80% or higher. These results suggest that the advantages of underactuated, adaptive robot hands can be carried over from basic grasping tasks to more dexterous tasks.

Biped walking improves the circulation of blood as well as bone density of the lower limbs, thereby enhancing the quality of life (QOL). It is significant not only to healthy people but also to physically challenged persons such as complete spinal cord injury (SCI) patients. The purpose of this paper is to propose an estimation algorithm that infers the intention related to the forward leg-swing in order to support the gait for complete SCI patients wearing an exoskeleton system called a Hybrid Assistive Limb (HAL), and to verify the effectiveness of the proposed algorithm through a clinical trial. The proposed algorithm infers the patient's intention in synchronization with the deviation of the center of the ground reaction force (CoGRF) that is observed immediately before a person starts walking. The patient conveys this intention by inducing the deviation of the CoGRF, using crutches or handrails with both of his/her arms. In the clinical trial, we confirmed that the algorithm inferred the patient's intention to swing the leg forward, and achieved a smooth gait in synchronization with it. As a result, the gait speed and cadence of the SCI patient with HAL during the 10-meter walking test increased to 6.67 [m/min] and 20 [steps/min], respectively after several trials.

This study presents a novel myoelectric pattern recognition strategy towards restoration of hand function after incomplete cervical spinal cord Injury (SCI). High density surface electromyogram (EMG) signals comprised of 57 channels were recorded from the forearm of nine subjects with incomplete cervical SCI while they tried to perform six different hand grasp patterns. A series of pattern recognition algorithms with different EMG feature sets and classifiers were implemented to identify the intended tasks of each SCI subject. High average overall accuracies (>; 97%) were achieved in classification of seven different classes (six intended hand grasp patterns plus a hand rest pattern), indicating that substantial motor control information can be extracted from partially paralyzed muscles of SCI subjects. Such information can potentially enable volitional control of assistive devices, thereby facilitating restoration of hand function. Furthermore, it was possible to maintain high levels of classification accuracy with a very limited number of electrodes selected from the high density surface EMG recordings. This demonstrates clinical feasibility and robustness in the concept of using myoelectric pattern recognition techniques toward improved function restoration for individuals with spinal injury.

Autonomous helicopter flight represents a challenging cont rol problem, with complex, noisy, dynamics. In this paper, we describe a s uccessful application of reinforcement learning to autonomous helic opter flight. We first fit a stochastic, nonlinear model of the helicopter dy namics. We then use the model to learn to hover in place, and to fly a number of maneuvers taken from an RC helicopter competition.

Recent results suggest that compliance and non-linearity in physical bodies of soft robots may not be disadvantageous properties with respect to control, but rather of advantage. In the context of morphological computation one could see such complex structures as potential computational resources. In this study, we implement and exploit this view point in a spine-driven quadruped robot called Kitty by using its flexible spine as a computational resource. The spine is an actuated multi-joint structure consisting of a sequence of soft silicone blocks. Its complex dynamics are captured by a set of force sensors and used to construct a closed-loop to drive the motor commands. We use simple static, linear readout weights to combine the sensor values to generate multiple gait patterns (bounding, trotting, turning behavior). In addition, we demonstrate the robustness of the setup by applying strong external perturbations in form of additional loads. The system is able to fully recover to its nominal gait patterns (which are encoded in the linear readout weights) after the perturbation has vanished.

THIS PAPER is concerned with the nature of the tutorial process; the means whereby an adult or "expert" helps somebody who is less adult or less expert. Though its aim is general, it is expressed in terms of a particular task: a tutor seeks to teach children aged 3, 4 and 5 yr to build a particular three-dimensional structure that requires a degree of skill that is initially beyond them. It is the usual type of tutoring situation in which one member "knows the answer" and the other does not, rather like a "practical" in which only the instructor "knows how". The changing interaction of tutor and children provide our data. A great deal of early problem solving by the developing child is of this order. Although from the earliest months of life he is a "natural" problem solver in his own right (e.g. Bruner, 1973) it is often the ease that his efforts are assisted and fostered by others who are more skilful than he is (Kaye, 1970). Whether he is learning the procedures that constitute the skills of attending, communicating, manipulating objects, locomoting, or, indeed, a more effective problem solving procedure itself, there are usually others in attendance who help him on his way. Tutorial interactions are, in short, a crucial feature of infancy and childhood. Our species, moreover, appears to be the only one in which any "intentional" tutoring goes on (Bruner, 1972; Hinde, 1971). For although it is true that many of the higher primate species learn by observation of their elders (Hamburg, 1968; van Lawick-Goodall, 1968), there is no evidence that those elders do anything to instruct their charges in the performance of the skill in question. What distinguishes man as a species is not only his capacity for learning, but for teaching as well. It is the main aim of this paper to examine some of the major implications of this interactive, instructional relationship between the developing child and his elders for the study of skill acquisition and problem solving. The acquisition of skill in the human child can be fruitfully conceived as a hierarchical program in which component skills are combined into "higher skills" by appropriate orchestration to meet new, more complex task requirements (Bruner, 1973). The process is analogous to problem solving in which mastery of "lower order" or constituent problems in a sine qua non for success with a larger jjroblcm, each level influencing the other—as with reading where the deciphering of words makes possible the deciphering of sentences, and sentences then aid in the deciphering of particular words (F. Smith, 1971). Given persistent intention in the young learner, given a "lexicon" of constituent skills, the crucial task is often one of com-

This study examined how individuals with and without neck pain performed exercises under the influence of altered visual feedback in virtual reality. Chronic neck pain (n = 9) and asymptomatic (n = 10) individuals were recruited for this cross-sectional study. Participants performed head rotations while receiving programmatically manipulated visual feedback from a head-mounted virtual reality display. The main outcome measure was the control-display gain (ratio between actual head rotation angle and visual rotation angle displayed) recorded at the just-noticeable difference. Actual head rotation angles were measured for different gains. Detection of the manipulated visual feedback was affected by gain. The just-noticeable gain for asymptomatic individuals, below and above unity gain, was 0.903 and 1.159, respectively. Head rotation angle decreased or increased 5.45° for every 0.1 increase or decrease in gain, respectively. The just-noticeable gain for chronic pain individuals, below unity gain, was 0.950. The head rotation angle increased 4.29° for every 0.1 decrease in gain. On average, chronic pain individuals reported that neck rotation was feasible for 84% of the unity gain trials, 66% of the individual just-noticeable difference trials, and 50% of the “nudged” just-noticeable difference trials. This research demonstrated that virtual reality may be useful for promoting the desired outcome of increased range of motion in neck rehabilitation exercises by altering visual feedback.

Information theory has recently been employed to specify more precisely than has hitherto been possible man's capacity in certain sensory, perceptual, and perceptual-motor functions (5, 10, 13, 15, 17, 18). The experiments reported in the present paper extend the theory to the human motor system. The applicability of only the basic concepts, amount of information, noise, channel capacity, and rate of information transmission, will be examined at this time. General familiarity with these concepts as formulated by recent writers (4,11, 20, 22) is assumed. Strictly speaking, we cannot study man's motor system at the behavioral level in isolation from its associated sensory mechanisms. We can only analyze the behavior of the entire receptor-neural-effector system. How-

Real-time functional MRI (rtfMRI) has been used as a basis for brain-computer interface (BCI) due to its ability to characterize region-specific brain activity in real-time. As an extension of BCI, we present an rtfMRI-based brain-machine interface (BMI) whereby 2-dimensional movement of a robotic arm was controlled by the regulation (and concurrent detection) of regional cortical activations in the primary motor areas. To do so, the subjects were engaged in the right- and/or left-hand motor imagery tasks. The blood oxygenation level dependent (BOLD) signal originating from the corresponding hand motor areas was then translated into horizontal or vertical robotic arm movement. The movement was broadcasted visually back to the subject as a feedback. We demonstrated that real-time control of the robotic arm only through the subjects' thought processes was possible using the rtfMRI-based BMI trials.

This paper presents preliminary results from a virtual reality (VR)-based system for hand rehabilitation that uses a CyberGlove and a Rutgers Master II-ND haptic glove. This computerized system trains finger range of motion, finger flexion speed, independence of finger motion, and finger strength using specific VR simulation exercises. A remote Web-based monitoring station was developed to allow telerehabilitation interventions. The remote therapist observes simplified versions of the patient exercises that are updated in real time. Patient data is stored transparently in an Oracle database, which is also Web accessible through a portal GUI. Thus the remote therapist or attending physician can graph exercise outcomes and thus evaluate patient outcomes at a distance. Data from the VR simulations is complemented by clinical measurements of hand function and strength. Eight chronic post-stroke subjects participated in a pilot study of the above system. In keeping with variability in both their lesion size and site and in their initial upper extremity function, each subject showed improvement on a unique combination of movement parameters in VR training. Importantly, these improvements transferred to gains on clinical tests, as well as to significant reductions in task-completion times for the prehension of real objects. These results are indicative of the potential feasibility of this exercise system for rehabilitation in patients with hand dysfunction resulting from neurological impairment.