Question: Applying the principle of maximum causal entropy to model goal-directed adaptive behavior.

This paper explores advanced techniques for Inverse Reinforcement Learning (IRL), a critical domain in machine learning that aims to decode the reward functions behind observed behaviors. Unlike conventional approaches, the algorithms presented here are designed to understand and model purposeful adaptive behavior by leveraging the principle of maximum causal entropy. This principle allows for the inference of complex reward structures that can explain the observed actions of agents in various environments. By maximizing the causal entropy, these algorithms ensure a diverse set of potential behaviors while still adhering to the observed data, thereby capturing the full spectrum of possible decision-making processes. This approach not only enhances the accuracy of behavior prediction but also provides a deeper insight into the underlying motivations and preferences that drive these actions. The algorithms are rigorously tested and compared against traditional methods, demonstrating their superior ability to model and predict purposeful adaptive behavior across different scenarios. This work is particularly relevant in the context of autonomous systems, where understanding and predicting human-like behavior is crucial for safe and effective interaction.

The emulation theory of representation is developed and explored as a framework that can revealingly synthesize a wide variety of representational functions of the brain. The framework is based on constructs from control theory (forward models) and signal processing (Kalman filters). The idea is that in addition to simply engaging with the body and environment, the brain constructs neural circuits that act as models of the body and environment. During overt sensorimotor engagement, these models are driven by efference copies in parallel with the body and environment, in order to provide expectations of the sensory feedback, and to enhance and process sensory information. These models can also be run off-line in order to produce imagery, estimate outcomes of different actions, and evaluate and develop motor plans. The framework is initially developed within the context of motor control, where it has been shown that inner models running in parallel with the body can reduce the effects of feedback delay problems. The same mechanisms can account for motor imagery as the off-line driving of the emulator via efference copies. The framework is extended to account for visual imagery as the off-line driving of an emulator of the motor-visual loop. I also show how such systems can provide for amodal spatial imagery. Perception, including visual perception, results from such models being used to form expectations of, and to interpret, sensory input. I close by briefly outlining other cognitive functions that might also be synthesized within this framework, including reasoning, theory of mind phenomena, and language.

Intelligent computer agents are both the original goal and the ultimate goal of artificial intelligence research. In striving toward that goal, our community has followed a practical research strategy of "divide-and-conquer," with different sub-communities attacking important component functions of intelligence, such as planning, search, knowledge representation, vision, and natural language. This strategy has been almost too successful, yielding both challenging theoretical problems that have come to dominate the inquiries of many researchers and a spate of practical techniques that have enabled other researchers to move into profitable commercial enterprises. While each of these pursuits is worthy in its own right, together they have had the unfortunate sideeffect of fragmenting the field and almost completely diverting it from efforts to build intelligent agents.

This paper describes an architecture that combines the complementary strengths of probabilistic graphical models and declarative programming to enable robots to represent and reason with qualitative and quantitative descriptions of uncertainty and domain knowledge. An action language is used for the architecture’s low-level (LL) and high-level (HL) system descriptions, and the HL definition of recorded history is expanded to allow prioritized defaults. For any given objective, tentative plans created in the HL using commonsense reasoning are implemented in the LL using probabilistic algorithms, and the corresponding observations are added to the HL history. Tight coupling between the levels helps automate the selection of relevant variables and the generation of policies in the LL for each HL action, and supports reasoning with violation of defaults, noisy observations and unreliable actions in complex domains. The architecture is evaluated in simulation and on robots moving objects in indoor domains.

Over the last two decades, much attention has been paid to the area of goal-oriented requirements engineering (GORE), where goals are used as a useful conceptualization to elicit, model, and analyze requirements, capturing alternatives and conflicts. Goal modeling has been adapted and applied to many sub-topics within requirements engineering (RE) and beyond, such as agent orientation, aspect orientation, business intelligence, model-driven development, and security. Despite extensive efforts in this field, the RE community lacks a recent, general systematic literature review of the area. In this work, we present a systematic mapping study, covering the 246 top-cited GORE-related conference and journal papers, according to Scopus. Our literature map addresses several research questions: we classify the types of papers (e.g., proposals, formalizations, meta-studies), look at the presence of evaluation, the topics covered (e.g., security, agents, scenarios), frameworks used, venues, citations, author networks, and overall publication numbers. For most questions, we evaluate trends over time. Our findings show a proliferation of papers with new ideas and few citations, with a small number of authors and papers dominating citations; however, there is a slight rise in papers which build upon past work (implementations, integrations, and extensions). We see a rise in papers concerning adaptation/variability/evolution and a slight rise in case studies. Overall, interest in GORE has increased. We use our analysis results to make recommendations concerning future GORE research and make our data publicly available.

Narrative, and in particular storytelling, is an important part of the human experience. Consequently, computational systems that can reason about narrative can be more effective communicators, entertainers, educators, and trainers. One of the central challenges in computational narrative reasoning is narrative generation, the automated creation of meaningful event sequences. There are many factors – logical and aesthetic – that contribute to the success of a narrative artifact. Central to this success is its understandability. We argue that the following two attributes of narratives are universal: (a) the logical causal progression of plot, and (b) character believability. Character believability is the perception by the audience that the actions performed by characters do not negatively impact the audience’s suspension of disbelief. Specifically, characters must be perceived by the audience to be intentional agents. In this article, we explore the use of refinement search as a technique for solving the narrative generation problem – to find a sound and believable sequence of character actions that transforms an initial world state into a world state in which goal propositions hold. We describe a novel refinement search planning algorithm – the Intent-based Partial Order Causal Link (IPOCL) planner – that, in addition to creating causally sound plot progression, reasons about character intentionality by identifying possible character goals that explain their actions and creating plan structures that explain why those characters commit to their goals. We present the results of an empirical evaluation that demonstrates that narrative plans generated by the IPOCL algorithm support audience comprehension of character intentions better than plans generated by conventional partial-order planners.

A connection between universal codes and the problems of prediction and statistical estimation is established. A‘known lower bound for the mean length of universal codes is sharpened and generalized, and optimum universal codes constructed. The bound is defined to give the information in strings relative to the considered class of processes. The earlier derived minimum description length criterion for estimation of parameters, including their number, is given a fundamental information, theoretic justification by showing that its estimators achieve the information in the strings. It is also shown that one cannot do prediction in Manuscript received July 13,1983; revised January 16,1984. This work was presented in part at the IEEE International Symposium on Information Theory, St. Jovite, Canada, September 26-30, 1983. This work was done while the author was Visiting Professor at the Department of System Science, University of California, Los Angeles, while on leave from the IBM Research Laboratory, San Jose, CA 95193. Gaussian autoregressive moving average (ARMA) processes below a bound, which is determined by the information in the data.

In biomimetic engineering, we may take inspiration from the products of biological evolution: we may instantiate biologically realistic neural architectures and algorithms in robots, or we may construct robots with morphologies that are found in nature. Alternatively, we may take inspiration from the process of evolution: we may evolve populations of robots in simulation and then manufacture physical versions of the most interesting or more capable robots that evolve. If we follow this latter approach and evolve both the neural and morphological subsystems of machines, we can perform controlled experiments that provide unique insight into how bodies and brains can work together to produce adaptive behavior, regardless of whether such bodies and brains are instantiated in a biological or technological substrate. In this paper, we review selected projects that use such methods to investigate the synergies and tradeoffs between neural architecture, morphology, action, and adaptive behavior.

Causation is a psychological tool of humans to understand the world and it is projected in natural language. Causation relates two events, so in order to understand the causal relation of those events and the causal reasoning of humans, the study of causality classification is required. We claim that the use of linguistic features may restrict the representation of causality, and dense vector spaces can provide a better encoding of the causal meaning of an utterance. Herein, we propose a neural network architecture only fed with word embeddings for the task of causality classification. Our results show that our claim holds, and we outperform the state-of-the-art on the AltLex corpus. The source code of our experiments is publicly available.1

Probabilities describe degrees of belief, and probabilistic inference describes rational reasoning under uncertainty. It is no wonder, then, that probabilistic models have exploded onto the scene of modern artificial intelligence, cognitive science, and applied statistics: these are all sciences of inference under uncertainty. But as probabilistic models have become more sophisticated, the tools to formally describe them and to perform probabilistic inference have wrestled with new complexity. Just as programming beyond the simplest algorithms requires tools for abstraction and composition, complex probabilistic modeling requires new progress in model representation—probabilistic programming languages. These languages provide compositional means for describing complex probability distributions; implementations of these languages provide generic inference engines: tools for performing efficient probabilistic inference over an arbitrary program. In their simplest form, probabilistic programming languages extend a well-specified deterministic programming language with primitive constructs for random choice. This is a relatively old idea, with foundational work by Giry, Kozen, Jones, Moggi, SahebDjahromi, Plotkin, and others [see e.g. 7]. Yet it has seen a resurgence thanks to new tools for probabilistic inference and new complexity of probabilistic modeling applications. There are a number of recent probabilistic programming languages [e.g. 8, 9, 11–17], embodying different tradeoffs in expressivity, efficiency, and perspicuity. We will focus on the probabilistic programming language Church [6] for simplicity, but the design of probabilistic languages to best support complex model representation and efficient inference is an active and important topic. Church extends (the purely functional subset of) Scheme with elementary random primitives, such as flip (a bernoulli), multinomial, and gaussian. In addition, Church includes language constructs that simplify modeling. For instance, mem, a higher-order procedure that memoizes its input function, is useful for describing persistent random properties and lazy model construction. (Interestingly, memoization has a semantic effect in probabilistic languages.) If we view the semantics of the underlying deterministic language as a map from programs to executions of the program, the semantics of the probabilistic language will be a map from programs to distributions over executions. When the program halts

A fuzzy cognitive map is a graphical means of representing arbitrarily complex models of interrelations between concepts. The purpose of this paper is to describe a dynamic fuzzy cognitive map based on the random neural network model. Previously, we have developed a random fuzzy cognitive map and illustrated its application in the modeling of processes. The adaptive fuzzy cognitive map changes its fuzzy causal web as causal patterns change and as experts update their causal knowledge. Our model carries out inferences via numerical calculation instead of symbolic deduction. We show how the dynamic random fuzzy cognitive map can reveal implications of models composed of dynamic processes. Copyright c ©2003 Yang’s Scientific Research Institute, LLC. All rights reserved.


Question: A comprehensive guide for implementing stochastic gradient MCMC algorithms.

This paper introduces a novel probabilistic generative model for collections of discrete data, with a particular focus on text, that expands upon and enhances several existing models such as naive Bayes/unigram, mixture of unigrams, and Hofmann's aspect model, often referred to as probabilistic latent semantic indexing (pLSI). Our proposed model assumes that each document is a product of a mixture of topics, with the proportions of these topics being continuous and distributed as a latent Dirichlet random variable. This approach provides a more nuanced understanding of text structure by modeling the probabilistic distribution of topics within each document. To handle the complexities of inference and learning in this model, we employ efficient variational algorithms. The efficacy of our model is demonstrated through a series of empirical studies, which cover a range of applications including text modeling, collaborative filtering, and text classification tasks. These results highlight the model's versatility and improved performance over previous methods in capturing the latent structure within discrete data collections.

We provide a new unifying view, including all existing prope r probabilistic sparse approximations for Gaussian process regression. Our approach relies on exp ressing theeffective priorwhich the methods are using. This allows new insights to be gained, and highlights the relationship between existing methods. It also allows for a clear theoretically j ustified ranking of the closeness of the known approximations to the corresponding full GPs. Finall y we point directly to designs of new better sparse approximations, combining the best of the exi sting strategies, within attractive computational constraints.

This paper presents a tutorial introduction to the use of variational methods for inference and learning in graphical models (Bayesian networks and Markov random fields). We present a number of examples of graphical models, including the QMR-DT database, the sigmoid belief network, the Boltzmann machine, and several variants of hidden Markov models, in which it is infeasible to run exact inference algorithms. We then introduce variational methods, which exploit laws of large numbers to transform the original graphical model into a simplified graphical model in which inference is efficient. Inference in the simpified model provides bounds on probabilities of interest in the original model. We describe a general framework for generating variational transformations based on convex duality. Finally we return to the examples and demonstrate how variational algorithms can be formulated in each case.

Decision trees and ensembles of decision trees are very popular in machine learning and often achieve state-of-the-art performance on black-box prediction tasks. However, popular variants such as C4.5, CART, boosted trees and random forests lack a probabilistic interpretation since they usually just specify an algorithm for training a model. We take a probabilistic approach where we cast the decision tree structures and the parameters associated with the nodes of a decision tree as a probabilistic model; given labeled examples, we can train the probabilistic model using a variety of approaches (Bayesian learning, maximum likelihood, etc). The probabilistic approach allows us to encode prior assumptions about tree structures and share statistical strength between node parameters; furthermore, it offers a principled mechanism to obtain probabilistic predictions which is crucial for applications where uncertainty quantification is important. Existing work on Bayesian decision trees relies on Markov chain Monte Carlo which can be computationally slow and suffer from poor mixing. We propose a novel sequential Monte Carlo algorithm that computes a particle approximation to the posterior over trees in a top-down fashion. We also propose a novel sampler for Bayesian additive regression trees by combining the above top-down particle filtering algorithm with the Particle Gibbs (Andrieu et al., 2010) framework. Finally, we propose Mondrian forests (MFs), a computationally efficient hybrid solution that is competitive with non-probabilistic counterparts in terms of speed and accuracy, but additionally produces well-calibrated uncertainty estimates. MFs use the Mondrian process (Roy and Teh, 2009) as the randomization mechanism and hierarchically smooth the node parameters within each tree (using a hierarchical probabilistic model and approximate Bayesian updates), but combine the trees in a non-Bayesian fashion. MFs can be grown in an incremental/online fashion and remarkably, the distribution of online MFs is the same as that of batch MFs.

In this paper we study how to learn stochastic, multimodal transition dynamics in reinforcement learning (RL) tasks. We focus on evaluating transition function estimation, while we defer planning over this model to future work. Stochasticity is a fundamental property of many task environments. However, discriminative function approximators have difficulty estimating multimodal stochasticity. In contrast, deep generative models do capture complex high-dimensional outcome distributions. First we discuss why, amongst such models, conditional variational inference (VI) is theoretically most appealing for model-based RL. Subsequently, we compare different VI models on their ability to learn complex stochasticity on simulated functions, as well as on a typical RL gridworld with multimodal dynamics. Results show VI successfully predicts multimodal outcomes, but also robustly ignores these for deterministic parts of the transition dynamics. In summary, we show a robust method to learn multimodal transitions using function approximation, which is a key preliminary for model-based RL in stochastic domains.

We consider Bayesian methods for multi-information source optimization (MISO), in which we seek to optimize an expensive-to-evaluate black-box objective function while also accessing cheaper but biased and noisy approximations (“information sources”). We present a novel algorithm that outperforms the state of the art for this problem by using a Gaussian process covariance kernel better suited to MISO than those used by previous approaches, and an acquisition function based on a one-step optimality analysis supported by efficient parallelization. We also provide a novel technique to guarantee the asymptotic quality of the solution provided by this algorithm. Experimental evaluations demonstrate that this algorithm consistently finds designs of higher value at less cost than previous approaches.

A combination of distributed computation, positive feedback and constructive greedy heuristic is proposed as a new approach to stochastic optimization and problem solving. Positive feedback accounts for rapid discovery of very good solutions, distributed computation avoids premature convergence, and greedy heuristic helps the procedure to find acceptable solutions in the early stages of the search process. An application of the proposed methodology to the classical travelling salesman problem shows that the system can rapidly provide very good, if not optimal, solutions. We report on many simulation results and discuss the working of the algorithm. Some hints about how this approach can be applied to a variety of optimization problems are also given.

Image processing pipelines combine the challenges of stencil computations and stream programs. They are composed of large graphs of different stencil stages, as well as complex reductions, and stages with global or data-dependent access patterns. Because of their complex structure, the performance difference between a naive implementation of a pipeline and an optimized one is often an order of magnitude. Efficient implementations require optimization of both parallelism and locality, but due to the nature of stencils, there is a fundamental tension between parallelism, locality, and introducing redundant recomputation of shared values.
 We present a systematic model of the tradeoff space fundamental to stencil pipelines, a schedule representation which describes concrete points in this space for each stage in an image processing pipeline, and an optimizing compiler for the Halide image processing language that synthesizes high performance implementations from a Halide algorithm and a schedule. Combining this compiler with stochastic search over the space of schedules enables terse, composable programs to achieve state-of-the-art performance on a wide range of real image processing pipelines, and across different hardware architectures, including multicores with SIMD, and heterogeneous CPU+GPU execution. From simple Halide programs written in a few hours, we demonstrate performance up to 5x faster than hand-tuned C, intrinsics, and CUDA implementations optimized by experts over weeks or months, for image processing applications beyond the reach of past automatic compilers.

The predominant methodology in training deep learning advocates the use of stochastic gradient descent methods (SGDs). Despite its ease of implementation, SGDs are difficult to tune and parallelize. These problems make it challenging to develop, debug and scale up deep learning algorithms with SGDs. In this paper, we show that more sophisticated off-the-shelf optimization methods such as Limited memory BFGS (L-BFGS) and Conjugate gradient (CG) with line search can significantly simplify and speed up the process of pretraining deep algorithms. In our experiments, the difference between LBFGS/CG and SGDs are more pronounced if we consider algorithmic extensions (e.g., sparsity regularization) and hardware extensions (e.g., GPUs or computer clusters). Our experiments with distributed optimization support the use of L-BFGS with locally connected networks and convolutional neural networks. Using L-BFGS, our convolutional network model achieves 0.69% on the standard MNIST dataset. This is a state-of-theart result on MNIST among algorithms that do not use distortions or pretraining.

BayesOpt is a library with state-of-the-art Bayesian optimization methods to solve nonlinear optimization, stochastic bandits or sequential experimental design problems. Bayesian optimization is sample efficient by building a posterior distribution to capture the evidence and prior knowledge for the target function. Built in standard C++, the library is extremely efficient while being portable and flexible. It includes a common interface for C, C++, Python, Matlab and Octave.

Graphics processing units (GPUs) are of increasing interest because they offer massive parallelism for high-throughput computing. While GPUs promise high peak performance, their challenge is a less-familiar programming model with more complex and irregular performance trade-offs than traditional CPUs or CMPs. In particular, modest changes in software or hardware characteristics can lead to large or unpredictable changes in performance. In response to these challenges, our work proposes, evaluates, and offers usage examples of Stargazer1, an automated GPU performance exploration framework based on stepwise regression modeling. Stargazer sparsely and randomly samples parameter values from a full GPU design space and simulates these designs. Then, our automated stepwise algorithm uses these sampled simulations to build a performance estimator that identifies the most significant architectural parameters and their interactions. The result is an application-specific performance model which can accurately predict program runtime for any point in the design space. Because very few initial performance samples are required relative to the extremely large design space, our method can drastically reduce simulation time in GPU studies. For example, we used Stargazer to explore a design space of nearly 1 million possibilities by sampling only 300 designs. For 11 GPU applications, we were able to estimate their runtime with less than 1.1% average error. In addition, we demonstrate several usage scenarios of Stargazer.


Question: A comprehensive guide for implementing stochastic gradient MCMC algorithms.

This paper delves into the adaptation of Hamiltonian Monte Carlo (HMC) methods for scenarios involving large datasets or continuous data streams, where traditional HMC's reliance on gradient computations becomes impractical. HMC algorithms are celebrated for their ability to propose distant states with high acceptance rates within a Metropolis-Hastings framework, thus facilitating more effective exploration of the state space than conventional random-walk proposals. However, their utility is curtailed in contexts with substantial data volumes or streaming data, due to the computational infeasibility of gradient calculations. To circumvent this challenge, a stochastic gradient estimate, derived from a data subset, is employed. Yet, this straightforward adaptation can lead to suboptimal outcomes, potentially undermining the algorithm's performance. To counteract the adverse effects of noisy gradient estimates, this paper proposes a modified approach that incorporates second-order Langevin dynamics, complete with a friction component. This addition serves to mitigate the impact of gradient noise, ensuring the desired target distribution remains invariant. Experimental results on synthetic datasets corroborate the theoretical foundations of this stochastic gradient HMC variant. Furthermore, the paper illustrates practical applications of this methodology, including its deployment in a neural network classification task and in the context of online Bayesian matrix factorization.

Training deep neural network is a high dimensional and a highly non-convex optimization problem. In this paper, we revisit Hessian-free optimization method for deep networks with negative curvature direction detection. We also develop its distributed variant and demonstrate superior scaling potential to SGD, which allows more efficiently utilizing larger computing resources thus enabling large models and faster time to obtain desired solution. We show that these techniques accelerate the training process for both the standard MNIST dataset and also the TIMIT speech recognition problem, demonstrating robust performance with upto an order of magnitude larger batch sizes. This increased scaling potential is illustrated with near l inear speed-up on upto 32 CPU nodes for a simple 4-layer network.

The use of machine learning algorithms frequently involves careful tuning of learning parameters and model hyperparameters. Unfortunately, this tuning is often a “black art” requiring expert experience, rules of thumb, or sometimes bruteforce search. There is therefore great appeal for automatic approaches that can optimize the performance of any given learning algorithm to the problem at hand. In this work, we consider this problem through the framework of Bayesian optimization, in which a learning algorithm’s generalization performance is modeled as a sample from a Gaussian process (GP). We show that certain choices for the nature of the GP, such as the type of kernel and the treatment of its hyperparameters, can play a crucial role in obtaining a good optimizer that can achieve expertlevel performance. We describe new algorithms that take into account the variable cost (duration) of learning algorithm experiments and that can leverage the presence of multiple cores for parallel experimentation. We show that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization for many algorithms including latent Dirichlet allocation, structured SVMs and convolutional neural networks.

Gaussian processes (GP) are powerful tools for probabilistic modeling purposes. They can be used to define prior distributions over latent functions in hierarchical Bayesian models. The prior over functions is defined implicitly by the mean and covariance function, which determine the smoothness and variability of the function. The inference can then be conducted directly in the function space by evaluating or approximating the posterior process. Despite their attractive theoretical properties GPs provide practical challenges in their implementation. GPstuff is a versatile collection of computational tools for GP models compatible with Linux and Windows MATLAB and Octave. It includes, among others, various inference methods, sparse approximations and tools for model assessment. In this work, we review these tools and demonstrate the use of GPstuff in several models.

Large scale realistic epidemic simulations have recently become an increasingly important application of high-performance computing. We propose a parallel algorithm, EpiFast, based on a novel interpretation of the stochastic disease propagation in a contact network. We implement it using a master-slave computation model which allows scalability on distributed memory systems.
 EpiFast runs extremely fast for realistic simulations that involve: (i) large populations consisting of millions of individuals and their heterogeneous details, (ii) dynamic interactions between the disease propagation, the individual behaviors, and the exogenous interventions, as well as (iii) large number of replicated runs necessary for statistically sound estimates about the stochastic epidemic evolution. We find that EpiFast runs several magnitude faster than another comparable simulation tool while delivering similar results.
 EpiFast has been tested on commodity clusters as well as SGI shared memory machines. For a fixed experiment, if given more computing resources, it scales automatically and runs faster. Finally, EpiFast has been used as the major simulation engine in real studies with rather sophisticated settings to evaluate various dynamic interventions and to provide decision support for public health policy makers.

We study the resilience to Byzantine failures of distributed implementations of Stochastic Gradient Descent (SGD). So far, distributed machine learning frameworks have largely ignored the possibility of failures, especially arbitrary (i.e., Byzantine) ones. Causes of failures include software bugs, network asynchrony, biases in local datasets, as well as attackers trying to compromise the entire system. Assuming a set of n workers, up to f being Byzantine, we ask how resilient can SGD be, without limiting the dimension, nor the size of the parameter space. We first show that no gradient aggregation rule based on a linear combination of the vectors proposed by the workers (i.e, current approaches) tolerates a single Byzantine failure. We then formulate a resilience property of the aggregation rule capturing the basic requirements to guarantee convergence despite f Byzantine workers. We propose Krum, an aggregation rule that satisfies our resilience property, which we argue is the first provably Byzantine-resilient algorithm for distributed SGD. We also report on experimental evaluations of Krum.

Graphical models bring together graph theory and probability theory in a powerful formalism for multivariate statistical modeling. In statistical signal processing— as well as in related fields such as communication theory, control theory and bioinformatics—statistical models have long been formulated in terms of graphs, and algorithms for computing basic statistical quantities such as likelihoods and marginal probabilities have often been expressed in terms of recursions operating on these graphs. Examples include hidden Markov models, Markov random fields, the forward-backward algorithm and Kalman filtering [ Rabiner and Juang (1993); Pearl (1988); Kailath et al. (2000)]. These ideas can be understood, unified and generalized within the formalism of graphical models. Indeed, graphical models provide a natural framework for formulating variations on these classical architectures, and for exploring entirely new families of statistical models. The recursive algorithms cited above are all instances of a general recursive algorithm known as the junction tree algorithm [ Lauritzen and Spiegelhalter, 1988]. The junction tree algorithm takes advantage of factorization properties of the joint probability distribution that are encoded by the pattern of missing edges in a graphical model. For suitably sparse graphs, the junction tree algorithm provides a systematic and practical solution to the general problem of computing likelihoods and other statistical quantities associated with a graphical model. Unfortunately, many graphical models of practical interest are not " suitably sparse, " so that the junction tree algorithm no longer provides a viable computational solution to the problem of computing marginal probabilities and other expectations. One popular source of methods for attempting to cope with such cases is the Markov chain Monte Carlo (MCMC) framework, and indeed there is a significant literature on

Many statistical models can be simulated forwards but have intractable likelihoods. Approximate Bayesian Computation (ABC) methods are used to infer properties of these models from data. Traditionally these methods approximate the posterior over parameters by conditioning on data being inside an -ball around the observed data, which is only correct in the limit →0. Monte Carlo methods can then draw samples from the approximate posterior to approximate predictions or error bars on parameters. These algorithms critically slow down as →0, and in practice draw samples from a broader distribution than the posterior. We propose a new approach to likelihood-free inference based on Bayesian conditional density estimation. Preliminary inferences based on limited simulation data are used to guide later simulations. In some cases, learning an accurate parametric representation of the entire true posterior distribution requires fewer model simulations than Monte Carlo ABC methods need to produce a single sample from an approximate posterior.

A new global optimization algorithm for functions of many continuous variables is presented, derived from the basic Simulated annealing method. Our main contribution lies in dealing with high-dimensionality minimization problems, which are often difficult to solve by all known minimization methods with or without gradient. In this article we take a special interest in the variables discretization issue. We also develop and implement several complementary stopping criteria. The original Metropolis iterative random search, which takes place in a Euclidean space Rn, is replaced by another similar exploration, performed within a succession of Euclidean spaces Rp, with p <<n. This Enhanced Simulated Annealing (ESA) algorithm was validated first on classical highly multimodal functions of 2 to 100 variables. We obtained significant reductions in the number of function evaluations compared to six other global optimization algorithms, selected according to previously published computational results for the same set of test functions. In most cases, ESA was able to closely approximate known global optima. The reduced ESA computational cost helped us to refine further the obtained global results, through the use of some local search. We have used this new minimizing procedure to solve complex circuit design problems, for which the objective function evaluation can be exceedingly costly.

The logistic loss is strictly convex and does not attain its infimum; consequently the solutions of logistic regression are in general off at infinity. This work provides a convergence analysis of stochastic and batch gradient descent for logistic regression. Firstly, under the assumption of separability, stochastic gradient descent minimizes the population risk at rate O(ln(t)/t) with high probability. Secondly, with or without separability, batch gradient descent minimizes the empirical risk at rate O(ln(t)/t). Furthermore, parameter convergence can be characterized along a unique pair of complementary subspaces defined by the problem instance: one subspace along which strong convexity induces parameters to converge at rate O(ln(t)/ √ t), and its orthogonal complement along which separability induces parameters to converge in direction at rate O(ln ln(t)/ ln(t)). 1 Overview Logistic regression is the task of finding a vector w ∈ R which approximately minimizes the population or empirical logistic risk, namely Rlog(w) := E [ `log(〈w,−yx〉) ] or R̂log(w) := 1 n n ∑ i=1 `log(〈w,−yixi〉), where `log(r) := ln ( 1 + exp(r) ) is the logistic loss. A traditional way to minimize Rlog or R̂log is to pick an arbitrary w0, and from there recursively construct stochastic gradient descent (SGD) or gradient descent (GD) iterates (wj)j≥0 via wj+1 := wj − ηjgj , where (ηj)j≥0 are step sizes, and gj := ` ′ log (〈 wj ,−yjxj 〉) (−yjxj) for SGD or gj := ∇R̂`log(wj) for GD. A common assumption is that the distribution on (x, y) is separable (Novikoff, 1962; Soudry et al., 2017). Under this assumption, the first result here is that SGD with constant step sizes minimizes the population risk Rlog at rate Õ(1/t) with high probability. Theorem 1.1 (Simplification of Theorem 2.1). Suppose there exists a unit vector ū with 〈ū, yx〉 ≥ γ, and |yx| ≤ 1 almost surely. Consider SGD iterates (wj)j≥0 as above, with w0 = 0 and ηj = 1. Then for any t ≥ 1, |wt| ≤ O(ln(t)), and with probability at least 1− δ, the average iterate ŵt := t−1 ∑ j<t wj satisfies Rlog(ŵt) ≤ O ( ln(t) + ln(t) ln(1/δ) t ) . This result improves upon prior work in the following ways. 1. Since `log exhibits an exponential tail and Lipschitz continuity, standard analyses give a Õ(1/ √ t) rate of SGD with step size ηj = 1/ √ j + 1, but only in expectation (Nesterov, 2004; Bubeck, 2015). With the O(ln(t)) bound on |wt| given by Theorem 1.1, Azuma-Hoeffding’s inequality gives a high-probability Õ(1/ √ t) rate. However, since standard analyses do not ensure a vanishing risk with constant step sizes, they do not give a Õ(1/t) rate. 1 ar X iv :1 80 3. 07 30 0v 2 [ cs .L G ] 1 J un 2 01 8 2. The only other prior high-probability Õ(1/t) rate is under strong convexity (e.g., (Rakhlin et al., 2012)): |wt − w̄| ≤ Õ(1/t) with high probability, where w̄ is the global optimum. However, `log is not strongly convex on R, and even in trivial cases the optimum of logistic regression may be off at infinity: suppose examples (xi) n i=1 lie on the positive real line, and labels (yi) n i=1 are all +1; then infw∈R R̂log(w) = 0, but R̂log(w) > 0 for every w ∈ R. Furthermore, not only do the guarantees here avoid strong convexity; the proofs are different. The proof of Theorem 1.1 is based on the perceptron proof, and might be of independent interest. Now consider batch gradient descent on empirical risk. In this case, a stronger result is possible: no assumption is made (separability is dropped), and the parameters may be precisely characterized. Theorem 1.2 (Simplification of Theorems 3.1, 3.2 and 3.6). Let examples ((xi, yi)) n i=1 be given with |xiyi| ≤ 1, along with a loss ` ∈ {`log, exp}, with corresponding empirical risk R̂ as above. Consider GD iterates (wj)j≥0 as above, with w0 = 0. 1. (Convergence in risk.) For any step sizes ηj ≤ 1 and any t ≥ 1, R̂(wt)− inf w∈Rd R̂(w) = O ( 1 t + ln(t) ∑ j<t ηj ) = { O(ln(t)/t) ηj = Ω(1), O(ln(t)/ √ t) ηj = Ω(1/ √ j + 1). 2. (Convergence in parameters.) There exists a unique subspace S, unique unit vector ū ∈ S⊥, and unique vector v̄ ∈ S which characterize convergence as follows. Fixing step size ηj := 1/ √ j + 1, letting ΠS denote orthogonal projection onto S, and defining optimal iterates w̄t := inf { R̂(w) : |w| ≤ |wt| } , if t = Ω(n ln(t)), then |ΠSwt| = Θ(1) and max { |ΠSwt − v̄|, |ΠSw̄t − v̄| } = O ( ln(t) √ t ) , |ΠS⊥wt| = Θ(ln(t)) and max ∣∣∣∣ wt |wt| − ū ∣∣∣∣2 , ∣∣∣∣ w̄t |w̄t| − ū ∣∣∣∣2 } = O ( ln ln t ln t ) . In particular, wt/|wt|→ ū and ΠSwt → v̄. To make sense of this, recall firstly the earlier discussion that the solutions are off at infinity in general. As will be made precise in Section 3.1, if examples are separable, then S = {0}, v̄ = 0, and ū is the unique maximum margin separator. The GD iterates not only keep driving R̂ to its infimum 0, but also closely track ū in direction, meaning wt/|wt|→ ū. Note that this is a strong mode of convergence: R̂(wt) = Õ(1/t) and |wt| = Θ(ln(t)) do not exclude the possibility that there is always a large angle between wt/|wt| and ū. The full part 2 of Theorem 1.2 gives a characterization of parameter convergence in the general setting: the parameter space can be split into a subspace S along which optimization is strongly convex, and its complement S⊥ along which the problem behaves as though separable. While there does exist recent prior work on parameter convergence of gradient descent for logistic regression (Soudry et al., 2017; Gunasekar et al., 2018; Nacson et al., 2018), it assumes separability. The analysis essentially for free gives convergence not only to (v̄, ū), but also to the optimal iterates w̄t. Consequently, logistic regression is implicitly regularized in a strong sense: it produces a sequence of iterates which closely track the outputs of a sequence of constraint optimization problems. This paper is organized as follows. SGD population risk convergence (Section 2). An online guarantee is given by a perceptron-style analysis, and transformed into a high-probability bound via a martingale Bernstein bound. Empirical samples structure (Section 3.1). The first step of the analysis is to derive the unique complementary subspaces (S, S⊥), along with unique optimal direction ū ∈ S⊥ and unique vector v̄ ∈ S. (S, S⊥) and ū are unique given the data ((xi, yi)) n i=1 and are not affected by the choice of ` or optimization method; v̄ depends on ` but not on the optimization method.

We present a framework of dynamic programming combinators that provides a high-level environment to describe the recursions typical of dynamic programming over sequence data in a style very similar to algebraic dynamic programming (ADP). Using a combination of type-level programming and stream fusion leads to a substantial increase in performance, without sacrificing much of the convenience and theoretical underpinnings of ADP.
 We draw examples from the field of computational biology, more specifically RNA secondary structure prediction, to demonstrate how to use these combinators and what differences exist between this library, ADP, and other approaches.
 The final version of the combinator library allows writing algorithms with performance close to hand-optimized C code.


Question: A comprehensive guide for implementing stochastic gradient MCMC algorithms.

This paper outlines a stochastic approximation technique for estimating the solution to a particular experimental equation. Consider a scenario where M(x), the expected outcome at a given level x, is a monotonic function whose exact form is unknown to the experimenter. The goal is to determine the value of x=0 that satisfies the equation M(x) = a, with a being a predetermined constant. The proposed methodology involves conducting a series of experiments at various levels, denoted as x1, x2,..., with the objective of converging these levels to the solution x=0 in a probabilistic sense. The method described herein ensures that the sequence of experimental levels will tend towards the desired solution with a high probability, facilitating an efficient and accurate estimation of the unknown value.

**Question**: A Complete Recipe for Stochastic Gradient MCMC

**Documents**: A Method for Stochastic Optimization
In many optimization problems, the objective function is not known in closed form but can only be evaluated through noisy observations. This is particularly prevalent in applications involving large datasets, where computing the full objective function is computationally prohibitive. In such scenarios, stochastic optimization methods become essential. We present a novel approach to stochastic optimization that can effectively handle the noise in the objective function and converge to the optimal solution with high probability. Our method iteratively updates the solution based on noisy observations, using a carefully designed learning rate that balances exploration and exploitation. We provide theoretical guarantees on the convergence of our method and demonstrate its efficacy through numerical experiments on various stochastic optimization problems.

**Similar Documents**: This paper introduces a cutting-edge technique for addressing optimization challenges in environments where the objective function is obscured by noise and can only be approximated through indirect, noisy measurements. Such situations are common in data-intensive applications, where the full calculation of the objective function is computationally infeasible due to the size of the datasets. Our proposed method is specifically tailored to navigate the uncertainties introduced by noise and to converge towards the optimal solution with a high degree of confidence. The approach involves a series of iterative adjustments to the solution, guided by noisy observations and underpinned by a learning rate strategy that strikes a balance between exploration of the solution space and exploitation of the current best estimate. We furnish rigorous theoretical evidence supporting the convergence properties of our method, complemented by empirical results from a range of stochastic optimization tasks that underscore its practical utility.

It is widely believed that in order to solve large scale global optimization problems an appropriate mixture of local approximation and global exploration is necessary. Local approximation, if first order information on the objective function is available, is efficiently performed by means of local optimization methods. Unfortunately, global exploration, in absence of some kind of global information on the problem, is a “blind” procedure, aimed at placing observations as evenly as possible in the search domain. Often this procedure reduces to uniform random sampling (like in Multistart algorithms, or in techniques based on clustering). In this paper we propose a new framework for global exploration which tries to guide random exploration towards the region of attraction of low-level local optima. The main idea originated by the use of smoothing techniques (based on gaussian convolutions): the possibility of applying a smoothing transformation not to the objective function but to the result of local searches seems to have never been explored yet. Although an exact smoothing of the results of local searches is impossible to implement, in this paper we propose a computational approximation scheme which has proven to be very efficient and (maybe more important) extremely robust in solving large scale global optimization problems with huge numbers of local optima. ∗Email: b.addis@ing.unifi.it Dip. Sistemi e Informatica Università di Firenze †Email: locatell@di.unito.it Dip. Informatica Università di Torino ‡Email: schoen@ing.unifi.it Dip. Sistemi e Informatica Università di Firenze

Gaussian processes allow for flexible specification of prior assumptions of unknown dynamics in state space models. We present a procedure for efficient Bayesian learning in Gaussian process state space models, where the representation is formed by projecting the problem onto a set of approximate eigenfunctions derived from the prior covariance structure. Learning under this family of models can be conducted using a carefully crafted particle MCMC algorithm. This scheme is computationally efficient and yet allows for a fully Bayesian treatment of the problem. Compared to conventional system identification tools or existing learning methods, we show competitive performance and reliable quantification of uncertainties in the model.

We describe an R package named huge which provides easy-to-use functions for estimating high dimensional undirected graphs from data. This package implements recent results in the literature, including Friedman et al. (2007), Liu et al. (2009, 2012) and Liu et al. (2010). Compared with the existing graph estimation package glasso, the huge package provides extra features: (1) instead of using Fortan, it is written in C, which makes the code more portable and easier to modify; (2) besides fitting Gaussian graphical models, it also provides functions for fitting high dimensional semiparametric Gaussian copula models; (3) more functions like data-dependent model selection, data generation and graph visualization; (4) a minor convergence problem of the graphical lasso algorithm is corrected; (5) the package allows the user to apply both lossless and lossy screening rules to scale up large-scale problems, making a tradeoff between computational and statistical efficiency.

In many applications of black-box optimization, one can evaluate multiple points simultaneously, e.g. when evaluating the performances of several different neural networks in a parallel computing environment. In this paper, we develop a novel batch Bayesian optimization algorithm — the parallel knowledge gradient method. By construction, this method provides the one-step Bayes optimal batch of points to sample. We provide an efficient strategy for computing this Bayes-optimal batch of points, and we demonstrate that the parallel knowledge gradient method finds global optima significantly faster than previous batch Bayesian optimization algorithms on both synthetic test functions and when tuning hyperparameters of practical machine learning algorithms, especially when function evaluations are noisy.

Surveying a suite of algorithms that offer a solution to managing large document archives.

We propose a simulation based algorithm for optimizing the average reward in a Markov Reward Process that depends on a set of parameters As a special case the method applies to Markov Decision Processes where optimization takes place within a parametrized set of policies The algorithm involves the simulation of a single sample path and can be implemented on line A convergence result with probability is provided This research was supported by contracts with Siemens AG Munich Germany and Alcatel Bell Belgium and by contract DMI with the National Science Foundation Introduction Markov Decision Processes and the associated dynamic programming DP methodology Ber a Put provide a general framework for posing and analyzing problems of se quential decision making under uncertainty DP methods rely on a suitably de ned value function that has to be computed for every state in the state space However many inter esting problems involve very large state spaces curse of dimensionality In addition DP assumes the availability of an exact model in the form of transition probabilities In many practical situations such a model is not available and one must resort to simulation or experimentation with an actual system For all of these reasons dynamic programming in its pure form may be inapplicable The e orts to overcome the aforementioned di culties involve two main ideas The use of simulation to estimate quantities of interest thus avoiding model based computations The use of parametric representations to overcome the curse of dimensionality Parametric representations and the associated algorithms can be broadly classi ed into three main categories a Parametrized value functions Instead of associating a value V i with each state i one uses a parametric form V i r where r is a vector of tunable parameters weights and V is a so called approximation architecture For example V i r could be the output of a multilayer perceptron with weights r when the input is i Other representations are possible e g involving polynomials linear combina tions of feature vectors state aggregation etc When the main ideas from DP are combined with such parametric representations one obtains methods that go un der the names of reinforcement learning or neuro dynamic programming see BT SB for textbook expositions as well as the references therein A key char acteristic is that policy optimization is carried out in an indirect fashion one tries to obtain a good approximation of the optimal value function of dynamic programming and uses it to construct policies that are close to optimal Such methods are reason ably well though not fully understood and there have been some notable practical successes see BT SB for an overview including the world class backgammon player by Tesauro Tes b Parametrized policies In an alternative approach which is the one considered in this paper the tuning of a parametrized value function is bypassed Instead one considers a class of policies described in terms of a parameter vector Simulation is employed to estimate the gradient of the performance metric with respect to and the policy is improved by updating in a gradient direction In some cases the re quired gradient can be estimated using IPA in nitesimal perturbation analysis see e g HC Gla CR and the references therein For general Markov processes and in the absence of special structure IPA is inapplicable but gradient estimation is still possible using likelihood ratio methods Gly Gly GG LEc GI c Actor critic methods A third approach which is a combination of the rst two includes parametrizations of the policy actor and of the value function critic BSA While such methods seem particularly promising theoretical understand ing has been limited to the impractical case of lookup representations one parameter per state KB This paper concentrates on methods based on policy parametrization and approx imate gradient improvement in the spirit of item b above While we are primarily interested in the case of Markov Decision Processes almost everything applies to the more general case of Markov Reward Processes that depend on a parameter vector and we proceed within this broader context We start with a formula for the gradient of the performance metric that has been presented in di erent forms and for various contexts in Gly CC FH JSJ TH CW We then suggest a method for estimating the terms that appear in that formula This leads to a simulation based method that updates the parameter vector at every regeneration time in an approximate gradient direction Furthermore we show how to construct an on line method that updates the parameter vector at each time step The resulting method has some conceptual similarities with those described in CR that reference assumes however the availability of an IPA estimator with certain guaranteed properties that are absent in our context and in JSJ which however does not contain convergence results The method that we propose only keeps in memory and updates K numbers where K is the dimension of Other than itself this includes a vector similar to the eligibility trace in Sutton s temporal di erence methods and as in JSJ an estimate of the average reward under the current value of If that estimate was accurate our method would be a standard stochastic gradient algorithm However as keeps changing is generally a biased estimate of the true average reward and the mathematical structure of our method is more complex than that of stochastic gradient algorithms For reasons that will become clearer later standard approaches e g martingale arguments or the ODE approach do not seem to su ce for establishing convergence and a more elaborate proof is necessary Our gradient estimator can also be derived or interpreted in terms of likelihood ratios Gly GG It takes the same form as the one presented in p of Gly but it is used di erently The development in Gly leads to a consistent estimator of the gradient assuming that a very large number of regenerative cycles are estimated while keeping the policy parameter at a xed value Presumably would be then updated after such a long simulation In contrast our method updates much more frequently and retains the desired convergence properties despite the fact that any single cycle results in a biased gradient estimate An alternative simulation based stochastic gradient method again based on a likeli hood ratio formula has been provided in Gly and uses the simulation of two regen erative cycles to construct an unbiased estimate of the gradient We note some of the di erences with the latter work First the methods in Gly involve a larger number of auxiliary quantities that are propagated in the course of a regenerative cycle Second our method admits a modi cation see Sections that can make it applicable even if the time until the next regeneration is excessive in which case likelihood ratio based methods su er from excessive variance Third our estimate  of the average reward is obtained as a weighted average of all past rewards not just over the last regenerative cycle In contrast an approach such as the one in Gly would construct an independent estimate of  during each regenerative cycle which should result in higher variance Finally our method brings forth and makes crucial use of the value di erential reward function of dy namic programming This is important because it paves the way for actor critic methods in which the variance associated with the estimates of the di erential rewards is poten tially reduced by means of learning value function approximation Indeed subsequent to the rst writing of this paper this latter approach has been pursued in KT SMS In summary the main contributions of this paper are as follows We introduce a new algorithm for updating the parameters of a Markov Reward Process on the basis of a single sample path The parameter updates can take place either during visits to a certain recurrent state or at every time step We also specialize the method to Markov Decision Processes with parametrically represented policies In this case the method does not require the transition probabilities to be known We establish that the gradient with respect to the parameter vector of the perfor mance metric converges to zero with probability which is the strongest possible result for gradient related stochastic approximation algorithms The method admits approximate variants with reduced variance such as the one described in Section or various types of actor critic methods The remainder of this paper is organized as follows In Section we introduce our framework and assumptions and state some background results including a formula for the gradient of the performance metric In Section we present an algorithm that per forms updates during visits to a certain recurrent state present our main convergence result and provide a heuristic argument Sections and deal with variants of the algo rithm that perform updates at every time step In Section we specialize our methods to the case of Markov Decision Processes that are optimized within a possibly restricted set of parametrically represented randomized policies We present some numerical results in Section and conclude in Section The lengthy proof of our main results is developed in the appendices Markov Reward Processes Depending on a Parameter In this section we present our general framework make a few assumptions and state some basic results that will be needed later We consider a discrete time nite state Markov chain fing with state space S f Ng whose transition probabilities depend on a parameter vector K and are denoted by pij P in j j in i Whenever the state is equal to i we receive a one stage reward that also depends on and is denoted by gi For every K let P be the stochastic matrix with entries pij Let P fP j Kg be the set of all such matrices and let P be its closure Note that every element of P is also a stochastic matrix and therefore de nes a Markov chain on the same s

We propose a computationally efficient limited memory Covariance Matrix Adaptation Evolution Strategy for large scale optimization, which we call the LM-CMA-ES. The LM-CMA-ES is a stochastic, derivative-free algorithm for numerical optimization of non-linear, non-convex optimization problems in continuous domain. Inspired by the limited memory BFGS method of Liu and Nocedal (1989), the LM-CMA-ES samples candidate solutions according to a covariance matrix reproduced from m direction vectors selected during the optimization process. The decomposition of the covariance matrix into Cholesky factors allows to reduce the time and memory complexity of the sampling to O(mn), where $n$ is the number of decision variables. When $n$ is large (e.g., n > 1000), even relatively small values of $m$ (e.g., m=20,30) are sufficient to efficiently solve fully non-separable problems and to reduce the overall run-time.

We study the generalization properties of stochastic gradient methods for learning with convex loss functions and linearly parameterized functions. We show that, in the absence of penalizations or constraints, the stability and approximation properties of the algorithm can be controlled by tuning either the step-size or the number of passes over the data. In this view, these parameters can be seen to control a form of implicit regularization. Numerical results complement the theoretical findings.

Stochastic Gradient Descent (SGD) has become popular for solving large scale supervised machine learning optimization problems such as SVM, due to their strong theoretical guarantees. While the closely related Dual Coordinate Ascent (DCA) method has been implemented in various software packages, it has so far lacked good convergence analysis. This paper presents a new analysis of Stochastic Dual Coordinate Ascent (SDCA) showing that this class of methods enjoy strong theoretical guarantees that are comparable or better than SGD. This analysis justifies the effectiveness of SDCA for practical applications.

Hierarchical Pitman-Yor Process priors are compelling methods for learning language models, outperforming point-estimate based methods. However, these models remain unpopular due to computational and statistical inference issues, such as memory and time usage, as well as poor mixing of sampler. In this work we propose a novel framework which represents the HPYP model compactly using compressed suffix trees. Then, we develop an efficient approximate inference scheme in this framework that has a much lower memory footprint compared to full HPYP and is fast in the inference time. The experimental results illustrate that our model can be built on significantly larger datasets compared to previous HPYP models, while being several orders of magnitudes smaller, fast for training and inference, and outperforming the perplexity of the state-of-the-art Modified Kneser-Ney countbased LM smoothing by up to 15%.


Question: A comprehensive guide for implementing stochastic gradient MCMC algorithms.

This paper introduces a novel methodology for Bayesian learning in the context of large-scale datasets, which is achieved through iterative learning from small mini-batches. The innovation lies in the strategic addition of noise to a standard stochastic gradient optimization algorithm. This noise calibration is critical as it ensures that as the stepsize is annealed, the iterates converge to samples drawn from the true posterior distribution. This integration of optimization and Bayesian posterior sampling not only streamlines the learning process but also inherently mitigates the risk of overfitting. A practical aspect of this method involves the development of a monitoring system for Monte Carlo estimates of posterior statistics. This system incorporates a "sampling threshold," which, once surpassed, triggers the collection of samples. The efficacy of this approach is demonstrated across three models: a Gaussian mixture model, logistic regression, and Independent Component Analysis (ICA) with natural gradients. This method not only enhances the efficiency of learning from big data but also ensures the accuracy of posterior sampling, making it a promising technique in the field of Bayesian learning.

Surveying a suite of algorithms that offer a solution to managing large document archives.

We describe DyNet, a toolkit for implementing neural network models based on dynamic declaration of network structure. In the static declaration strategy that is used in toolkits like Theano, CNTK, and TensorFlow, the user first defines a computation graph (a symbolic representation of the computation), and then examples are fed into an engine that executes this computation and computes its derivatives. In DyNet’s dynamic declaration strategy, computation graph construction is mostly transparent, being implicitly constructed by executing procedural code that computes the network outputs, and the user is free to use different network structures for each input. Dynamic declaration thus facilitates the implementation of more complicated network architectures, and DyNet is specifically designed to allow users to implement their models in a way that is idiomatic in their preferred programming language (C++ or Python). One challenge with dynamic declaration is that because the symbolic computation graph is defined anew for every training example, its construction must have low overhead. To achieve this, DyNet has an optimized C++ backend and lightweight graph representation. Experiments show that DyNet’s speeds are faster than or comparable with static declaration toolkits, and significantly faster than Chainer, another dynamic declaration toolkit. DyNet is released opensource under the Apache 2.0 license, and available at http://github.com/clab/dynet. Carnegie Mellon University, Pittsburgh, PA, USA Nara Institute of Science and Technology, Ikoma, Japan DeepMind, London, UK Bar Ilan University, Ramat Gan, Israel Allen Institute for Artificial Intelligence, Seattle, WA, USA University of Notre Dame, Notre Dame, IN, USA IBM T.J. Watson Research Center, Yorktown Heights, NY, USA University of Melbourne, Melbourne, Australia Johns Hopkins University, Baltimore, MD, USA Google, New York, NY, USA Google, Mountain View, CA, USA University of Washington, Seattle, USA Microsoft Research, Redmond, WA, USA University of Edinburgh, Edinburgh, UK 1 ar X iv :1 70 1. 03 98 0v 1 [ st at .M L ] 1 5 Ja n 20 17

Deep and recurrent neural networks (DNNs and RNNs respectively) are powerful models that were considered to be almost impossible to train using stochastic gradient descent with momentum. In this paper, we show that when stochastic gradient descent with momentum uses a well-designed random initialization and a particular type of slowly increasing schedule for the momentum parameter, it can train both DNNs and RNNs (on datasets with long-term dependencies) to levels of performance that were previously achievable only with Hessian-Free optimization. We find that both the initialization and the momentum are crucial since poorly initialized networks cannot be trained with momentum and well-initialized networks perform markedly worse when the momentum is absent or poorly tuned. Our success training these models suggests that previous attempts to train deep and recurrent neural networks from random initializations have likely failed due to poor initialization schemes. Furthermore, carefully tuned momentum methods su ce for dealing with the curvature issues in deep and recurrent network training objectives without the need for sophisticated second-order methods.

We propose a simulation based algorithm for optimizing the average reward in a Markov Reward Process that depends on a set of parameters As a special case the method applies to Markov Decision Processes where optimization takes place within a parametrized set of policies The algorithm involves the simulation of a single sample path and can be implemented on line A convergence result with probability is provided This research was supported by contracts with Siemens AG Munich Germany and Alcatel Bell Belgium and by contract DMI with the National Science Foundation Introduction Markov Decision Processes and the associated dynamic programming DP methodology Ber a Put provide a general framework for posing and analyzing problems of se quential decision making under uncertainty DP methods rely on a suitably de ned value function that has to be computed for every state in the state space However many inter esting problems involve very large state spaces curse of dimensionality In addition DP assumes the availability of an exact model in the form of transition probabilities In many practical situations such a model is not available and one must resort to simulation or experimentation with an actual system For all of these reasons dynamic programming in its pure form may be inapplicable The e orts to overcome the aforementioned di culties involve two main ideas The use of simulation to estimate quantities of interest thus avoiding model based computations The use of parametric representations to overcome the curse of dimensionality Parametric representations and the associated algorithms can be broadly classi ed into three main categories a Parametrized value functions Instead of associating a value V i with each state i one uses a parametric form V i r where r is a vector of tunable parameters weights and V is a so called approximation architecture For example V i r could be the output of a multilayer perceptron with weights r when the input is i Other representations are possible e g involving polynomials linear combina tions of feature vectors state aggregation etc When the main ideas from DP are combined with such parametric representations one obtains methods that go un der the names of reinforcement learning or neuro dynamic programming see BT SB for textbook expositions as well as the references therein A key char acteristic is that policy optimization is carried out in an indirect fashion one tries to obtain a good approximation of the optimal value function of dynamic programming and uses it to construct policies that are close to optimal Such methods are reason ably well though not fully understood and there have been some notable practical successes see BT SB for an overview including the world class backgammon player by Tesauro Tes b Parametrized policies In an alternative approach which is the one considered in this paper the tuning of a parametrized value function is bypassed Instead one considers a class of policies described in terms of a parameter vector Simulation is employed to estimate the gradient of the performance metric with respect to and the policy is improved by updating in a gradient direction In some cases the re quired gradient can be estimated using IPA in nitesimal perturbation analysis see e g HC Gla CR and the references therein For general Markov processes and in the absence of special structure IPA is inapplicable but gradient estimation is still possible using likelihood ratio methods Gly Gly GG LEc GI c Actor critic methods A third approach which is a combination of the rst two includes parametrizations of the policy actor and of the value function critic BSA While such methods seem particularly promising theoretical understand ing has been limited to the impractical case of lookup representations one parameter per state KB This paper concentrates on methods based on policy parametrization and approx imate gradient improvement in the spirit of item b above While we are primarily interested in the case of Markov Decision Processes almost everything applies to the more general case of Markov Reward Processes that depend on a parameter vector and we proceed within this broader context We start with a formula for the gradient of the performance metric that has been presented in di erent forms and for various contexts in Gly CC FH JSJ TH CW We then suggest a method for estimating the terms that appear in that formula This leads to a simulation based method that updates the parameter vector at every regeneration time in an approximate gradient direction Furthermore we show how to construct an on line method that updates the parameter vector at each time step The resulting method has some conceptual similarities with those described in CR that reference assumes however the availability of an IPA estimator with certain guaranteed properties that are absent in our context and in JSJ which however does not contain convergence results The method that we propose only keeps in memory and updates K numbers where K is the dimension of Other than itself this includes a vector similar to the eligibility trace in Sutton s temporal di erence methods and as in JSJ an estimate of the average reward under the current value of If that estimate was accurate our method would be a standard stochastic gradient algorithm However as keeps changing is generally a biased estimate of the true average reward and the mathematical structure of our method is more complex than that of stochastic gradient algorithms For reasons that will become clearer later standard approaches e g martingale arguments or the ODE approach do not seem to su ce for establishing convergence and a more elaborate proof is necessary Our gradient estimator can also be derived or interpreted in terms of likelihood ratios Gly GG It takes the same form as the one presented in p of Gly but it is used di erently The development in Gly leads to a consistent estimator of the gradient assuming that a very large number of regenerative cycles are estimated while keeping the policy parameter at a xed value Presumably would be then updated after such a long simulation In contrast our method updates much more frequently and retains the desired convergence properties despite the fact that any single cycle results in a biased gradient estimate An alternative simulation based stochastic gradient method again based on a likeli hood ratio formula has been provided in Gly and uses the simulation of two regen erative cycles to construct an unbiased estimate of the gradient We note some of the di erences with the latter work First the methods in Gly involve a larger number of auxiliary quantities that are propagated in the course of a regenerative cycle Second our method admits a modi cation see Sections that can make it applicable even if the time until the next regeneration is excessive in which case likelihood ratio based methods su er from excessive variance Third our estimate  of the average reward is obtained as a weighted average of all past rewards not just over the last regenerative cycle In contrast an approach such as the one in Gly would construct an independent estimate of  during each regenerative cycle which should result in higher variance Finally our method brings forth and makes crucial use of the value di erential reward function of dy namic programming This is important because it paves the way for actor critic methods in which the variance associated with the estimates of the di erential rewards is poten tially reduced by means of learning value function approximation Indeed subsequent to the rst writing of this paper this latter approach has been pursued in KT SMS In summary the main contributions of this paper are as follows We introduce a new algorithm for updating the parameters of a Markov Reward Process on the basis of a single sample path The parameter updates can take place either during visits to a certain recurrent state or at every time step We also specialize the method to Markov Decision Processes with parametrically represented policies In this case the method does not require the transition probabilities to be known We establish that the gradient with respect to the parameter vector of the perfor mance metric converges to zero with probability which is the strongest possible result for gradient related stochastic approximation algorithms The method admits approximate variants with reduced variance such as the one described in Section or various types of actor critic methods The remainder of this paper is organized as follows In Section we introduce our framework and assumptions and state some background results including a formula for the gradient of the performance metric In Section we present an algorithm that per forms updates during visits to a certain recurrent state present our main convergence result and provide a heuristic argument Sections and deal with variants of the algo rithm that perform updates at every time step In Section we specialize our methods to the case of Markov Decision Processes that are optimized within a possibly restricted set of parametrically represented randomized policies We present some numerical results in Section and conclude in Section The lengthy proof of our main results is developed in the appendices Markov Reward Processes Depending on a Parameter In this section we present our general framework make a few assumptions and state some basic results that will be needed later We consider a discrete time nite state Markov chain fing with state space S f Ng whose transition probabilities depend on a parameter vector K and are denoted by pij P in j j in i Whenever the state is equal to i we receive a one stage reward that also depends on and is denoted by gi For every K let P be the stochastic matrix with entries pij Let P fP j Kg be the set of all such matrices and let P be its closure Note that every element of P is also a stochastic matrix and therefore de nes a Markov chain on the same s

First order methods, which solely rely on gradient information, are commonly used in diverse machine learning (ML) and data analysis (DA) applications. This is attributed to the simplicity of their implementations, as well as low per-iteration computational/storage costs. However, they suffer from significant disadvantages; most notably, their performance degrades with increasing problem ill-conditioning. Furthermore, they often involve a large number of hyper-parameters, and are notoriously sensitive to parameters such as the step-size. By incorporating additional information from the Hessian, second-order methods, have been shown to be resilient to many such adversarial effects. However, these advantages of using curvature information come at the cost of higher per-iteration costs, which in “big data” regimes, can be computationally prohibitive. In this paper, we show that, contrary to conventional belief, second-order methods, when implemented appropriately, can be more efficient than first-order alternatives in many largescale ML/ DA applications. In particular, in convex settings, we consider variants of classical Newton’s method in which the Hessian and/or the gradient are randomly sub-sampled. We show that by effectively leveraging the power of GPUs, such randomized Newton-type algorithms can be significantly accelerated, and can easily outperform state of the art implementations of existing techniques in popular ML/ DA software packages such as TensorFlow. Additionally these randomized methods incur a small memory overhead compared to first-order methods. In particular, we show that for million-dimensional problems, our GPU accelerated sub-sampled Newton’s method achieves a higher test accuracy in milliseconds as compared with tens of seconds for first order alternatives.

We analyze stochastic algorithms for optimizing nonconvex, nonsmooth finite-sum problems, where the nonsmooth part is convex. Surprisingly, unlike the smooth case, our knowledge of this fundamental problem is very limited. For example, it is not known whether the proximal stochastic gradient method with constant minibatch converges to a stationary point. To tackle this issue, we develop fast stochastic algorithms that provably converge to a stationary point for constant minibatches. Furthermore, using a variant of these algorithms, we obtain provably faster convergence than batch proximal gradient descent. Our results are based on the recent variance reduction techniques for convex optimization but with a novel analysis for handling nonconvex and nonsmooth functions. We also prove global linear convergence rate for an interesting subclass of nonsmooth nonconvex functions, which subsumes several recent works.

Large multilayer neural networks trained with backpropagation have recently achieved state-ofthe-art results in a wide range of problems. However, using backprop for neural net learning still has some disadvantages, e.g., having to tune a large number of hyperparameters to the data, lack of calibrated probabilistic predictions, and a tendency to overfit the training data. In principle, the Bayesian approach to learning neural networks does not have these problems. However, existing Bayesian techniques lack scalability to large dataset and network sizes. In this work we present a novel scalable method for learning Bayesian neural networks, called probabilistic backpropagation (PBP). Similar to classical backpropagation, PBP works by computing a forward propagation of probabilities through the network and then doing a backward computation of gradients. A series of experiments on ten real-world datasets show that PBP is significantly faster than other techniques, while offering competitive predictive abilities. Our experiments also show that PBP provides accurate estimates of the posterior variance on the network weights.

In this paper we propose a method to use multiple generative models with latent variables for classi cation tasks. The standard approach to use generative models for classi cation is to train a separate model for each class. A novel data point is then classi ed by the model that attributes the highest probability. The algorithm we propose modi es the parameters of the models to improve the classi cation accuracy. Our approach is made computationally tractable by assuming that each of the models is deterministic, by which we mean that a data-point is associated to only a single latent state. The resulting algorithm is a variant of the support vector machine learning algorithm and in a limiting case the method is similar to the standard perceptron learning algorithm. We apply the method to two types of latent variable models. The rst has a discrete latent state space and the second, principal component analysis, has a continuous latent state space. We compare the e6ectiveness of both approaches on a handwritten digit recognition problem and on a satellite image recognition problem. c © 2003 Elsevier B.V. All rights reserved.

Many statistical models can be simulated forwards but have intractable likelihoods. Approximate Bayesian Computation (ABC) methods are used to infer properties of these models from data. Traditionally these methods approximate the posterior over parameters by conditioning on data being inside an -ball around the observed data, which is only correct in the limit →0. Monte Carlo methods can then draw samples from the approximate posterior to approximate predictions or error bars on parameters. These algorithms critically slow down as →0, and in practice draw samples from a broader distribution than the posterior. We propose a new approach to likelihood-free inference based on Bayesian conditional density estimation. Preliminary inferences based on limited simulation data are used to guide later simulations. In some cases, learning an accurate parametric representation of the entire true posterior distribution requires fewer model simulations than Monte Carlo ABC methods need to produce a single sample from an approximate posterior.

We establish theoretical results concerning all local optima of various regularized M-estimators, where both loss and penalty functions are allowed to be nonconvex. Our results show that as long as the loss function satisfies restricted strong convexity and the penalty function satisfies suitable regularity conditions, any local optimum of the composite objective function lies within statistical precision of the true parameter vector. Our theory covers a broad class of nonconvex objective functions, including corrected versions of the Lasso for errors-in-variables linear models; regression in generalized linear models using nonconvex regularizers such as SCAD and MCP; and graph and inverse covariance matrix estimation. On the optimization side, we show that a simple adaptation of composite gradient descent may be used to compute a global optimum up to the statistical precision εstat in log(1/εstat) iterations, which is the fastest possible rate of any first-order method. We provide a variety of simulations to illustrate the sharpness of our theoretical predictions. Disciplines Statistics and Probability This conference paper is available at ScholarlyCommons: http://repository.upenn.edu/statistics_papers/222 Regularized M -estimators with nonconvexity: Statistical and algorithmic theory for local optima Po-Ling Loh Department of Statistics University of California, Berkeley Berkeley, CA 94720 ploh@berkeley.edu Martin J. Wainwright Departments of Statistics and EECS University of California, Berkeley Berkeley, CA 94720 wainwrig@stat.berkeley.edu


Question: A comprehensive formula for implementing stochastic gradient Markov Chain Monte Carlo.

This paper introduces stochastic variational inference, an algorithm designed to scale the approximation of posterior distributions for a broad class of probabilistic models. We demonstrate the efficacy of this technique through its application to two probabilistic topic models: latent Dirichlet allocation and the hierarchical Dirichlet process topic model. To showcase the scalability of stochastic variational inference, we analyze extensive document collections, including 300,000 articles from Nature, 1.8 million articles from The New York Times, and 3.8 million articles from Wikipedia. Our results indicate that stochastic variational inference can effortlessly handle datasets of this magnitude, significantly outperforming traditional variational inference methods, which are limited to smaller data subsets. Additionally, we find that the Bayesian nonparametric topic model delivers superior performance compared to its parametric alternative. Stochastic variational inference thus enables the application of intricate Bayesian models to vast datasets, enhancing their practical utility in real-world scenarios.

The softmax representation of probabilities for categorical variables plays a prominent role in modern machine learning with numerous applications in areas such as large scale classification, neural language modeling and recommendation systems. However, softmax estimation is very expensive for large scale inference because of the high cost associated with computing the normalizing constant. Here, we introduce an efficient approximation to softmax probabilities which takes the form of a rigorous lower bound on the exact probability. This bound is expressed as a product over pairwise probabilities and it leads to scalable estimation based on stochastic optimization. It allows us to perform doubly stochastic estimation by subsampling both training instances and class labels. We show that the new bound has interesting theoretical properties and we demonstrate its use in classification problems.

We introduce a new class of upper bounds on the log partition function of a Markov random field (MRF). This quantity plays an important role in various contexts, including approximating marginal distributions, parameter estimation, combinatorial enumeration, statistical decision theory, and large-deviations bounds. Our derivation is based on concepts from convex duality and information geometry: in particular, it exploits mixtures of distributions in the exponential domain, and the Legendre mapping between exponential and mean parameters. In the special case of convex combinations of tree-structured distributions, we obtain a family of variational problems, similar to the Bethe variational problem, but distinguished by the following desirable properties: i) they are convex, and have a unique global optimum; and ii) the optimum gives an upper bound on the log partition function. This optimum is defined by stationary conditions very similar to those defining fixed points of the sum-product algorithm, or more generally, any local optimum of the Bethe variational problem. As with sum-product fixed points, the elements of the optimizing argument can be used as approximations to the marginals of the original model. The analysis extends naturally to convex combinations of hypertree-structured distributions, thereby establishing links to Kikuchi approximations and variants.

We propose a novel mixtures of Gaussian processes model in which the gating function is interconnected with a probabilistic logical model, in our case Markov logic networks. In this way, the resulting mixed graphical model, called Markov logic mixtures of Gaussian processes (MLxGP), solves joint Bayesian non-parametric regression and probabilistic relational inference tasks. In turn, MLxGP facilitates novel, interesting tasks such as regression based on logical constraints or drawing probabilistic logical conclusions about regression data, thus putting “machines reading regression data” in reach.

The predominant methodology in training deep learning advocates the use of stochastic gradient descent methods (SGDs). Despite its ease of implementation, SGDs are difficult to tune and parallelize. These problems make it challenging to develop, debug and scale up deep learning algorithms with SGDs. In this paper, we show that more sophisticated off-the-shelf optimization methods such as Limited memory BFGS (L-BFGS) and Conjugate gradient (CG) with line search can significantly simplify and speed up the process of pretraining deep algorithms. In our experiments, the difference between LBFGS/CG and SGDs are more pronounced if we consider algorithmic extensions (e.g., sparsity regularization) and hardware extensions (e.g., GPUs or computer clusters). Our experiments with distributed optimization support the use of L-BFGS with locally connected networks and convolutional neural networks. Using L-BFGS, our convolutional network model achieves 0.69% on the standard MNIST dataset. This is a state-of-theart result on MNIST among algorithms that do not use distortions or pretraining.

In this paper we present a new approach for constructing subgradient schemes for different types of nonsmooth problems with convex structure. Our methods are primaldual since they are always able to generate a feasible approximation to the optimum of an appropriately formulated dual problem. Besides other advantages, this useful feature provides the methods with a reliable stopping criterion. The proposed schemes differ from the classical approaches (divergent series methods, mirror descent methods) by presence of two control sequences. The first sequence is responsible for aggregating the support functions in the dual space, and the second one establishes a dynamically updated scale between the primal and dual spaces. This additional flexibility allows to guarantee a boundedness of the sequence of primal test points even in the case of unbounded feasible set. We present the variants of subgradient schemes for nonsmooth convex minimization, minimax problems, saddle point problems, variational inequalities, and stochastic optimization. In all situations our methods are proved to be optimal from the view point of worst-case black-box lower complexity bounds.

The basic theory of Markov chains has been known to mathematicians and engineers for close to 80 years, but it is only in the past decade that it has been applied explicitly to problems in speech processing. One of the major reasons why speech models, based on Markov chains, have not been developed until recently was the lack of a method for optimizing the parameters of the Markov model to match observed signal patterns. Such a method was proposed in the late 1960's and was immediately applied to speech processing in several research institutions. Continued refinements in the theory and implementation of Markov modelling techniques have greatly enhanced the method, leading to a wide range of applications of these models. It is the purpose of this tutorial paper to give an introduction to the theory of Markov models, and to illustrate how they have been applied to problems in speech recognition.

This paper considers additive factorial hidden Markov models, an extension to HMMs where the state factors into multiple independent chains, and the output is an additive function of all the hidden states. Although such models are very powerful, accurate inference is unfortunately difficult: exact inference is not computationally tractable, and existing approximate inference techniques are highly susceptible to local optima. In this paper we propose an alternative inference method for such models, which exploits their additive structure by 1) looking at the observed difference signal of the observation, 2) incorporating a “robust” mixture component that can account for unmodeled observations, and 3) constraining the posterior to allow at most one hidden state to change at a time. Combining these elements we develop a convex formulation of approximate inference that is computationally efficient, has no issues of local optima, and which performs much better than existing approaches in practice. The method is motivated by the problem of energy disaggregation, the task of taking a whole home electricity signal and decomposing it into its component appliances; applied to this task, our algorithm achieves state-of-the-art performance, and is able to separate many appliances almost perfectly using just the total aggregate signal.

We present a new method for solving stochastic differential equations based on Galerkin projections and extensions of Wiener’s polynomial chaos. Specifically, we represent the stochastic processes with an optimum trial basis from the Askey family of orthogonal polynomials that reduces the dimensionality of the system and leads to exponential convergence of the error. Several continuous and discrete processes are treated, and numerical examples show substantial speed-up compared to Monte Carlo simulations for low dimensional stochastic inputs.

Implementing E cient, Portable Computations for Machine Learning

For a complete copy of the book, please visit the MorganKaufmann website or Amazon.com.