Question: Accelerating training through disentangling variation modes in batch-normalized models.

This paper introduces Stochastic Depth, a novel training methodology designed to tackle the challenges associated with training extremely deep convolutional neural networks, which, despite their superior performance, can suffer from issues such as vanishing gradients, diminishing forward flow, and prolonged training times. The core idea behind Stochastic Depth is to train shorter networks while utilizing deeper architectures during testing. This seemingly contradictory approach is achieved by randomly selecting a subset of layers to be bypassed with the identity function for each mini-batch during training. This simple yet effective technique not only enhances the efficiency of training but also significantly improves test error performance across various datasets. In conjunction with residual networks, Stochastic Depth enables the training of networks with over 1200 layers, achieving notable reductions in test error, exemplified by a 4.91% error rate on the CIFAR-10 dataset. This methodology not only streamlines the training process but also showcases the potential for deeper networks to outperform shallower architectures in terms of accuracy.

Many deployed learned models are black boxes: given input, returns output. Internal information about the model, such as the architecture, optimisation procedure, or training data, is not disclosed explicitly as it might contain proprietary information or make the system more vulnerable. This work shows that such attributes of neural networks can be exposed from a sequence of queries. This has multiple implications. On the one hand, our work exposes the vulnerability of black-box neural networks to different types of attacks – we show that the revealed internal information helps generate more effective adversarial examples against the black box model. On the other hand, this technique can be used for better protection of private content from automatic recognition models using adversarial examples. Our paper suggests that it is actually hard to draw a line between white box and black box models.

Valuable training data is often owned by independent organizations and located in multiple data centers. Most deep learning approaches require to centralize the multi-datacenter data for performance purpose. In practice, however, it is often infeasible to transfer all data to a centralized data center due to not only bandwidth limitation but also the constraints of privacy regulations. Model averaging is a conventional choice for data parallelized training, but its ineffectiveness is claimed by previous studies as deep neural networks are often non-convex. In this paper, we argue that model averaging can be effective in the decentralized environment by using two strategies, namely, the cyclical learning rate and the increased number of epochs for local model training. With the two strategies, we show that model averaging can provide competitive performance in the decentralized mode compared to the data-centralized one. In a practical environment with multiple data centers, we conduct extensive experiments using state-of-the-art deep network architectures on different types of data. Results demonstrate the effectiveness and robustness of the proposed method.

Flow-based generative models, conceptually attractive due to tractability of both the exact log-likelihood computation and latent-variable inference, and efficiency of both training and sampling, has led to a number of impressive empirical successes and spawned many advanced variants and theoretical investigations. Despite their computational efficiency, the density estimation performance of flow-based generative models significantly falls behind those of state-of-the-art autoregressive models. In this work, we introduce masked convolutional generative flow (MACOW), a simple yet effective architecture of generative flow using masked convolution. By restricting the local connectivity in a small kernel, MACOW enjoys the properties of fast and stable training, and efficient sampling, while achieving significant improvements over Glow for density estimation on standard image benchmarks, considerably narrowing the gap to autoregressive models.

Modeling the distribution of natural images is a landmark problem in unsupervised learning. This task requires an image model that is at once expressive, tractable and scalable. We present a deep neural network that sequentially predicts the pixels in an image along the two spatial dimensions. Our method models the discrete probability of the raw pixel values and encodes the complete set of dependencies in the image. Architectural novelties include fast twodimensional recurrent layers and an effective use of residual connections in deep recurrent networks. We achieve log-likelihood scores on natural images that are considerably better than the previous state of the art. Our main results also provide benchmarks on the diverse ImageNet dataset. Samples generated from the model appear crisp, varied and globally coherent.

In this paper, we analyze the numerics of common algorithms for training Generative Adversarial Networks (GANs). Using the formalism of smooth two-player games we analyze the associated gradient vector field of GAN training objectives. Our findings suggest that the convergence of current algorithms suffers due to two factors: i) presence of eigenvalues of the Jacobian of the gradient vector field with zero real-part, and ii) eigenvalues with big imaginary part. Using these findings, we design a new algorithm that overcomes some of these limitations and has better convergence properties. Experimentally, we demonstrate its superiority on training common GAN architectures and show convergence on GAN architectures that are known to be notoriously hard to train.

The performance of neural network classifiers is determined by a number of hyperparameters, including learning rate, batch size, and depth. A number of attempts have been made to explore these parameters in the literature, and at times, to develop methods for optimizing them. However, exploration of parameter spaces has often been limited. In this note, I report the results of large scale experiments exploring these different parameters and their interactions. 1 Datasets and Libraries All experiments reported here were carried out using the Torch library [1] and CUDA (some of the experiments have been reproduced on a smaller scale with other libraries). The dataset for all the experiments is MNIST [3, 2]. Characters were deskewed prior to all experiments. Deskewing significantly reduces error rates in nearest neighbor classifiers. Skew corresponds to a simple one-parameter family of linear transformations in feature space and causes decision regions to become highly anisotropic. Without deskewing, differences in performance between different architectures might primarily reduce to their ability to “learn deskewing”. With deskewing, MNIST character classification become more of an instance of a typical classification problem. Prior results on classifying deskewed MNIST data both with neural networks and with other methods are shown in the table below. 2 Logistic vs Softmax Outputs Multi-Layer Perceptrons (MLPs) used for classification usually attempt to approximate posterior probabilities and use those as their discriminant function. Two common approaches to this are the use of least square regression with logistic output units trained with a least square error measure (“logistic outputs”) 1 ar X iv :1 50 8. 02 78 8v 1 [ cs .N E ] 1 2 A ug 2 01 5 Method Test Error Preprocessing Reference Reduced Set SVM deg 5 polynomial 1 deskewing LeCun et al. 1998 SVM deg 4 polynomial 1.1 deskewing LeCun et al. 1998 K-nearest-neighbors, L3 1.22 deskewing, noise removal, blurring, 2 pixel shift Kenneth Wilder, U. Chicago K-nearest-neighbors, L3 1.33 deskewing, noise removal, blurring, 1 pixel shift Kenneth Wilder, U. Chicago 2-layer NN, 300 HU 1.6 deskewing LeCun et al. 1998 Table 1: Other previously reported results on the MNIST database. and a softmax output layer (“softmax outputs”). In the limit of infinite amounts of training data, both approaches converge to true posterior probability estimates. Softmax output layers have the property that they are guaranteed to produce a normalized posterior probability distribution across all classes, while least square regression with logistic output units generates independent probability estimates for each class membership without any guarantees that these probabilities sum up to one. Softmax is often preferred, although there is no obvious theoretical reason why it should yield better discriminant functions or lower classification error for finite training sets. In OCR and speech recognition, some practitioners have observed that logistic outputs yield better posterior probability estimates and better results when combined with probabilistic language models. In addition, when the sum of the posterior probability estimates derived from logistic outputs differs significantly from unity, that is a strong indication that the input lies outside the training set and should be rejected. Figure 1 shows a scatterplot of test vs training error for a large number of MLPs with one hidden layer at different learning rates, different number of hidden units, and different batch sizes. Such scatterplots show what error rates are achievable by the different architectures, hyperparameter choices, initializations, and order of sample presentations. The lowest points in the vertical direction indicate the lowest test set error achievable by the architecture in this set of experiments. The scatterplot shows that logistic outputs achieve test set error rates of about 1.0% vs 1.1% for softmax outputs. At the same time, logistic outputs never achieve zero percent training set error, while softmax outputs frequently do. In order to ascertain that the difference in test set error between the two

The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target. However, because the targets change only once per epoch, Temporal Ensembling becomes unwieldy when learning large datasets. To overcome this problem, we propose Mean Teacher, a method that averages model weights instead of label predictions. As an additional benefit, Mean Teacher improves test accuracy and enables training with fewer labels than Temporal Ensembling. Without changing the network architecture, Mean Teacher achieves an error rate of 4.35% on SVHN with 250 labels, outperforming Temporal Ensembling trained with 1000 labels. We also show that a good network architecture is crucial to performance. Combining Mean Teacher and Residual Networks, we improve the state of the art on CIFAR-10 with 4000 labels from 10.55% to 6.28%, and on ImageNet 2012 with 10% of the labels from 35.24% to 9.11%.

Domain generalization (DG) is the challenging and topical problem of learning models that generalize to novel testing domain with different statistics than a set of known training domains. The simple approach of aggregating data from all source domains and training a single deep neural network end-to-end on all the data provides a surprisingly strong baseline that surpasses many prior published methods. In this paper we build on this strong baseline by designing an episodic training procedure that trains a single deep network in a way that exposes it to the domain shift that characterises a novel domain at runtime. Specifically, we decompose a deep network into feature extractor and classifier components, and then train each component by simulating it interacting with a partner who is badly tuned for the current domain. This makes both components more robust, ultimately leading to our networks producing state-of-the-art performance on three DG benchmarks. As a demonstration, we consider the pervasive workflow of using an ImageNet trained CNN as a fixed feature extractor for downstream recognition tasks. Using the Visual Decathlon benchmark, we demonstrate that our episodic-DG training improves the performance of such a general purpose feature extractor by explicitly training it for robustness to novel problems. This provides the largest-scale demonstration of heterogeneous DG to date.

In this paper we propose a new method for regularizing autoencoders by imposing an arbitrary prior on the latent representation of the autoencoder. Our method, named “adversarial autoencoder”, uses the recently proposed generative adversarial networks (GAN) in order to match the aggregated posterior of the hidden code vector of the autoencoder with an arbitrary prior. Matching the aggregated posterior to the prior ensures that there are no “holes” in the prior, and generating from any part of prior space results in meaningful samples. As a result, the decoder of the adversarial autoencoder learns a deep generative model that maps the imposed prior to the data distribution. We show how adversarial autoencoders can be used to disentangle style and content of images and achieve competitive generative performance on MNIST, Street View House Numbers and Toronto Face datasets.

Despite the success of generative adversarial networks (GANs) for image generation, the trade-off between visual quality and image diversity remains a significant issue. This paper achieves both aims simultaneously by improving the stability of training GANs. The key idea of the proposed approach is to implicitly regularize the discriminator using representative features. Focusing on the fact that standard GAN minimizes reverse Kullback-Leibler (KL) divergence, we transfer the representative feature, which is extracted from the data distribution using a pre-trained autoencoder (AE), to the discriminator of standard GANs. Because the AE learns to minimize forward KL divergence, our GAN training with representative features is influenced by both reverse and forward KL divergence. Consequently, the proposed approach is verified to improve visual quality and diversity of state of the art GANs using extensive evaluations.


Question: Acquiring structured finite-state models from unorganized observational data.

This paper presents the concept of Dynamic Movement Primitives (DMPs) as a critical framework for understanding motor control in both human and humanoid robotic systems. DMPs are viewed as the foundational building blocks that enable the generation of a vast array of movements, whether rhythmic or discrete, in a highly flexible and adaptable manner. These primitives are formalized as stable nonlinear attractor systems, which are particularly advantageous in robotics for their ability to quickly adjust to the dynamic and unpredictable environment. The versatility of DMPs in creating complex behaviors, such as locomotion or a tennis swing, while maintaining adaptability to perturbations, makes them a key component in autonomous robotics. Additionally, the DMP framework provides a structured approach that is also applicable to studies in computational neuroscience. A notable development in the DMP framework is the integration of well-established statistical learning methods, which has transformed DMPs from a more heuristic approach to a principled modeling technique. This enhancement has been substantiated through theoretical analyses, practical evaluations on humanoid robots, and empirical data from behavioral and brain imaging studies, all of which collectively support the DMP framework as a comprehensive model for motor control applicable to both robotics and biological systems.

An automata theory course can be taught in an interactive, hands-on manner using a computer. At Duke we have been using the software tool JFLAP to provide interaction and feedback in CPS 140, our automata theory course. JFLAP is a tool for designing and running nondeterministic versions of finite automata, pushdown automata, and Turing machines. Recently, we have enhanced JFLAP to allow one to study the proofs of several theorems that focus on conversions of languages, from one form to another, such as converting an NFA to a DFA and then to a minimum state DFA. In addition, our enhancements combined with other tools allow one to interactively study LL and LR parsing methods.

Standard algorithms for template-based information extraction (IE) require predefined template schemas, and often labeled data, to learn to extract their slot fillers (e.g., an embassy is the Target of a Bombing template). This paper describes an approach to template-based IE that removes this requirement and performs extraction without knowing the template structure in advance. Our algorithm instead learns the template structure automatically from raw text, inducing template schemas as sets of linked events (e.g., bombings include detonate, set off, and destroy events) associated with semantic roles. We also solve the standard IE task, using the induced syntactic patterns to extract role fillers from specific documents. We evaluate on the MUC-4 terrorism dataset and show that we induce template structure very similar to handcreated gold structure, and we extract role fillers with an F1 score of .40, approaching the performance of algorithms that require full knowledge of the templates.

Language learnabi l i ty has been inves t igated . This refers to the following s i tua t ion: A class of possible languages is specified, t oge the r wi th a method of present ing informat ion to the learner about an unknown language, which is to be chosen from the class. The quest ion is now asked, " I s the informat ion sufficient to determine which of the possible languages is the unknown language?" M a n y definitions of l ea rnab i l i ty are possible, bu t only the following is considered here: Time is quant ized and has a finite s t a r t ing time. At each t ime the learner receives a un i t of in format ion and is to make a guess as to the i den t i t y of the unknown language on the basis of the informat ion received so far. This process continues forever. The class of languages will be considered lea~nabIe with respect to the specified me thod of in format ion p resen ta t ion if there is an a lgor i thm t h a t the learner can use to make his guesses, the a lgor i thm hav ing the following p roper ty : Given any language of the class, there is some finite t ime af te r which the guesses will all be the same and they will be correct . In this pre l iminary invest igat ion, a language is t aken to be a set of s tr ings on some finite a lphabet . The a lphabe t is the same for all languages of tile class. Several var ia t ions of each of the following two basic methods of informat ion presenta t ion are inves t iga ted: A text for a language generates the s t r ings of the language in any order such t h a t every s t r ing of the language occurs a t least once. An informant for a language tells whe the r a s t r ing is in the language, and chooses the s t r ings in some order such t h a t every s t r ing occurs at least once. I t was found t h a t the class of context-sensi t ive languages is learnable from an informant, but that not even the class of regular languages is learnable from a text.

Recent approaches in classifying evolving data streams are based on supervised learning algorithms, which can be trained with labeled data only. Manual labeling of data is both costly and time consuming. Therefore, in a real streaming environment, where huge volumes of data appear at a high speed, labeled data may be very scarce. Thus, only a limited amount of training data may be available for building the classification models, leading to poorly trained classifiers. We apply a novel technique to overcome this problem by building a classification model from a training set having both unlabeled and a small amount of labeled instances. This model is built as micro-clusters using semi-supervised clustering technique and classification is performed with kappa-nearest neighbor algorithm. An ensemble of these models is used to classify the unlabeled data. Empirical evaluation on both synthetic data and real botnet traffic reveals that our approach, using only a small amount of labeled data for training, outperforms state-of-the-art stream classification algorithms that use twenty times more labeled data than our approach.

Echo State Networks (ESNs) have been shown to be effective for a number of tasks, including motor control, dynamic time series prediction, and memorizing musical sequences. However, their performance on natural language tasks has been largely unexplored until now. Simple Recurrent Networks (SRNs) have a long history in language modeling and show a striking similarity in architecture to ESNs. A comparison of SRNs and ESNs on a natural language task is therefore a natural choice for experimentation. Elman applies SRNs to a standard task in statistical NLP: predicting the next word in a corpus, given the previous words. Using a simple context-free grammar and an SRN with backpropagation through time (BPTT), Elman showed that the network was able to learn internal representations that were sensitive to linguistic processes that were useful for the prediction task. Here, using ESNs, we show that training such internal representations is unnecessary to achieve levels of performance comparable to SRNs. We also compare the processing capabilities of ESNs to bigrams and trigrams. Due to some unexpected regularities of Elman's grammar, these statistical techniques are capable of maintaining dependencies over greater distances than might be initially expected. However, we show that the memory of ESNs in this word-prediction task, although noisy, extends significantly beyond that of bigrams and trigrams, enabling ESNs to make good predictions of verb agreement at distances over which these methods operate at chance. Overall, our results indicate a surprising ability of ESNs to learn a grammar, suggesting that they form useful internal representations without learning them.

An approach to the online learning of Takagi-Sugeno (TS) type models is proposed in the paper. It is based on a novel learning algorithm that recursively updates TS model structure and parameters by combining supervised and unsupervised learning. The rule-base and parameters of the TS model continually evolve by adding new rules with more summarization power and by modifying existing rules and parameters. In this way, the rule-base structure is inherited and up-dated when new data become available. By applying this learning concept to the TS model we arrive at a new type adaptive model called the Evolving Takagi-Sugeno model (ETS). The adaptive nature of these evolving TS models in combination with the highly transparent and compact form of fuzzy rules makes them a promising candidate for online modeling and control of complex processes, competitive to neural networks. The approach has been tested on data from an air-conditioning installation serving a real building. The results illustrate the viability and efficiency of the approach. The proposed concept, however, has significantly wider implications in a number of fields, including adaptive nonlinear control, fault detection and diagnostics, performance analysis, forecasting, knowledge extraction, robotics, behavior modeling.

We presenta novel generati ve model for natural languagetree structuresin whichsemantic(lexical dependenc y) andsyntacticstructuresare scoredwith separatemodels.Thisfactorizationprovidesconceptual simplicity, straightforwardopportunitiesfor separatelyimproving the componentmodels,anda level of performancealreadycloseto thatof similar, non-factoredmodels.Most importantly, unlikeothermodernparsing models,thefactoredmodeladmitsanextremelyeffectiveA parsingalgorithm,which makesefficient,exactinferencefeasible.

There are several reasons why conventional speech recognition systems modeled on adult data fail to perform satisfactorily on children's speech input. For instance, children's vocal characteristics differ signi cantly from those of adults. In addition, their choices of vocabulary and sentence construction modalities usually do not conform to adult patterns. We describe comparative studies demonstrating the performance gain realized by adopting to children's acoustic and language model data to construct a children's speech recognition system.

We describe AutoClass, an approach to unsupervised classiication based upon the classical mixture model, supplemented by a Bayesian method for determining the optimal classes. We include a moderately detailed exposition of the mathematics behind the AutoClass system. We emphasize that no current unsupervised classiication system can produce maximally useful results when operated alone. It is the interaction between domain experts and the machine searching over the model space, that generates new knowledge. Both bring unique information and abilities to the database analysis task, and each enhances the others' eeectiveness. We illustrate this point with several applications of AutoClass to complex real world databases, and describe the resulting successes and failures. 6.1 Introduction This chapter is a summary of our experience in using an automatic classiication program (AutoClass) to extract useful information from databases. It also gives an outline of the principles that underlie automatic classiication in general, and AutoClass in particular. We are concerned with the problem of automatic discovery of classes in data (sometimes called clustering, or unsupervised learning), rather than the generation of class descriptions from labeled examples (called supervised learning). In some sense, automatic classiication aims at discovering the \natural" classes in the data. These classes reeect basic causal mechanisms that makes some cases look more like each other than the rest of the cases. The causal mechanisms may be as boring as sample biases in the data, or could reeect some major new discovery in the domain. Sometimes, these classes were well known to experts in the eld, but unknown to AutoClass, and other times

Spoken dialogue systems typically use predefined semantic slots to parse users' natural language inputs into unified semantic representations. To define the slots, domain experts and professional annotators are often involved, and the cost can be expensive. In this paper, we ask the following question: given a collection of unlabeled raw audios, can we use the frame semantics theory to automatically induce and fill the semantic slots in an unsupervised fashion? To do this, we propose the use of a state-of-the-art frame-semantic parser, and a spectral clustering based slot ranking model that adapts the generic output of the parser to the target semantic space. Empirical experiments on a real-world spoken dialogue dataset show that the automatically induced semantic slots are in line with the reference slots created by domain experts: we observe a mean averaged precision of 69.36% using ASR-transcribed data. Our slot filling evaluations also indicate the promising future of this proposed approach.


Question: Acquiring structured finite-state representations through unguided demonstrations.

This paper outlines a comprehensive strategy for enabling personal robots to exhibit adaptable and safe behaviors in human-centric environments, particularly focusing on the challenge of instantaneously adjusting their actions in response to unforeseen events. To achieve highly contact-reactive motions for tasks such as grasping and manipulation, our proposed framework integrates previous sensor experiences as a predictive model for future task executions. By associating typical movements with specific tasks, our system can leverage these past experiences to predict and adapt to upcoming situations. We utilize Dynamic Movement Primitives (DMPs), a type of dynamical system, to learn goal-directed actions from demonstrations. The dynamic properties of DMPs are enhanced by coupling them with both measured and predicted sensor data, creating a feedback mechanism that facilitates online adaptation of the movement strategy. This approach allows our system to generate a diverse array of potential movements that can effectively handle external disturbances and perception uncertainties, leading to robust and versatile behaviors. As a practical demonstration, we apply this method to a grasping task using the WAM robot arm, showcasing the system's ability to create truly adaptable and resilient actions in dynamic and uncertain environments.

The first needs a lot of data and suffers from covariate shift. It’s a supervised learning method when we fit the state-action pair mapping. The latter finds a cost function under which the expert is uniquely optimal. Compounding error is not a problem here, but IRL is super data inefficiend and requires RL in the inner loop. Since we only interested in the final policy, why do we learn the cost function? So, GAIL was born.

In recent years deep reinforcement learning (RL) systems have attained superhuman performance in a number of challenging task domains. However, a major limitation of such applications is their demand for massive amounts of training data. A critical present objective is thus to develop deep RL methods that can adapt rapidly to new tasks. In the present work we introduce a novel approach to this challenge, which we refer to as deep meta-reinforcement learning. Previous work has shown that recurrent networks can support meta-learning in a fully supervised context. We extend this approach to the RL setting. What emerges is a system that is trained using one RL algorithm, but whose recurrent dynamics implement a second, quite separate RL procedure. This second, learned RL algorithm can differ from the original one in arbitrary ways. Importantly, because it is learned, it is configured to exploit structure in the training domain. We unpack these points in a series of seven proof-of-concept experiments, each of which examines a key aspect of deep meta-RL. We consider prospects for extending and scaling up the approach, and also point out some potentially important implications for neuroscience.

We describe a method for identifying and performing functional analysis of structured regions that are embedded in natural language documents, such as tables or key-value lists. Such regions often encode information according to ad hoc schemas and avail themselves of visual cues in place of natural language grammar, presenting problems for standard information extraction algorithms. Unlike previous work in table extraction, which assumes a relatively noiseless two-dimensional layout, our aim is to accommodate a wide variety of naturally occurring structure types. Our approach has three main parts. First, we collect and annotate a a diverse sample of “naturally” occurring structures from several sources. Second, we use probabilistic text segmentation techniques, featurized by skip bigrams over spatial and token category cues, to automatically identify contiguous regions of structured text that share a common schema. Finally, we identify the records and fields within each structured region using a combination of distributional similarity and sequence alignment methods, guided by minimal supervision in the form of a single annotated record. We evaluate the last two components individually, and conclude with a discussion of further work.

Reinforcement learning holds the promise of enabling autonomous robots to learn large repertoires of behavioral skills with minimal human intervention. However, robotic applications of reinforcement learning often compromise the autonomy of the learning process in favor of achieving training times that are practical for real physical systems. This typically involves introducing hand-engineered policy representations and human-supplied demonstrations. Deep reinforcement learning alleviates this limitation by training general-purpose neural network policies, but applications of direct deep reinforcement learning algorithms have so far been restricted to simulated settings and relatively simple tasks, due to their apparent high sample complexity. In this paper, we demonstrate that a recent deep reinforcement learning algorithm based on off-policy training of deep Q-functions can scale to complex 3D manipulation tasks and can learn deep neural network policies efficiently enough to train on real physical robots. We demonstrate that the training times can be further reduced by parallelizing the algorithm across multiple robots which pool their policy updates asynchronously. Our experimental evaluation shows that our method can learn a variety of 3D manipulation skills in simulation and a complex door opening skill on real robots without any prior demonstrations or manually designed representations.

Linguistic structures exhibit a rich array of global phenomena, however commonly used Markov models are unable to adequately describe these phenomena due to their strong locality assumptions. We propose a novel hierarchical model for structured prediction over sequences and trees which exploits global context by conditioning each generation decision on an unbounded context of prior decisions. This builds on the success of Markov models but without imposing a fixed bound in order to better represent global phenomena. To facilitate learning of this large and unbounded model, we use a hierarchical PitmanYor process prior which provides a recursive form of smoothing. We propose prediction algorithms based on A* and Markov Chain Monte Carlo sampling. Empirical results demonstrate the potential of our model compared to baseline finite-context Markov models on part-of-speech tagging and syntactic parsing.

Humans and other animals often engage in activities for their own sakes rather than as steps toward solving practical problems. Psychologists call these intrinsically motivated behaviors. What we learn during intrinsically motivated behavior is essential for our development as competent autonomous entities able to efficiently solve a wide range of practical problems as they arise. In this paper we present initial results from a computational study of intrinsically motivated learning aimed at allowing artificial agents to construct and extend hierarchies of reusable skills that are needed for competent autonomy. At the core of the model are recent theoretical and algorithmic advances in computational reinforcement learning, specifically, new concepts related to skills and new learning algorithms for learning with skill hierarchies.

The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Until now neural networks have not been capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks that they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on a hand-written digit dataset and by learning several Atari 2600 games sequentially.

In this paper, we introduce Path Integral Networks (PI-Net), a recurrent network representation of the Path Integral optimal control algorithm. The network includes both system dynamics and cost models, used for optimal control based planning. PI-Net is fully differentiable, learning both dynamics and cost models end-to-end by back-propagation and stochastic gradient descent. Because of this, PI-Net can learn to plan. PI-Net has several advantages: it can generalize to unseen states thanks to planning, it can be applied to continuous control tasks, and it allows for a wide variety learning schemes, including imitation and reinforcement learning. Preliminary experiment results show that PI-Net, trained by imitation learning, can mimic control demonstrations for two simulated problems; a linear system and a pendulum swing-up problem. We also show that PI-Net is able to learn dynamics and cost models latent in the demonstrations.

Empirically, neural networks that attempt to learn programs from data have exhibited poor generalizability. Moreover, it has traditionally been difficult to reason about the behavior of these models beyond a certain level of input complexity. In order to address these issues, we propose augmenting neural architectures with a key abstraction: recursion. As an application, we implement recursion in the Neural Programmer-Interpreter framework on four tasks: grade-school addition, bubble sort, topological sort, and quicksort. We demonstrate superior generalizability and interpretability with small amounts of training data. Recursion divides the problem into smaller pieces and drastically reduces the domain of each neural network component, making it tractable to prove guarantees about the overall system’s behavior. Our experience suggests that in order for neural architectures to robustly learn program semantics, it is necessary to incorporate a concept like recursion.

This paper describes an architecture that combines the complementary strengths of probabilistic graphical models and declarative programming to enable robots to represent and reason with qualitative and quantitative descriptions of uncertainty and domain knowledge. An action language is used for the architecture’s low-level (LL) and high-level (HL) system descriptions, and the HL definition of recorded history is expanded to allow prioritized defaults. For any given objective, tentative plans created in the HL using commonsense reasoning are implemented in the LL using probabilistic algorithms, and the corresponding observations are added to the HL history. Tight coupling between the levels helps automate the selection of relevant variables and the generation of policies in the LL for each HL action, and supports reasoning with violation of defaults, noisy observations and unreliable actions in complex domains. The architecture is evaluated in simulation and on robots moving objects in indoor domains.


Question: Acquiring structured finite-state models through unguided demonstrations.

This paper explores the integration of task-level learning and planning for robotic systems engaged in object manipulation tasks within dynamic environments. Specifically, it addresses the challenge of preprogramming robots to execute intricate tasks, such as setting a dinner table, where the predetermined sequence of subtasks may become irrelevant due to environmental changes during execution. The proposed methodology focuses on learning the overarching goal of the task and employing a task planner to achieve this goal under varying initial environmental conditions. Recognizing that some tasks are governed by underlying constraints that must be adhered to, and that mere knowledge of the final goal is insufficient, two techniques for identifying these constraints are introduced. Firstly, the instructor can directly inform the system about the essential constraints. Alternatively, the robotic system can autonomously discern constraints through the aggregation and analysis of multiple observations. These identified constraints are then incorporated into the planning phase, ensuring that the execution of the task does not breach any constraints. The efficacy of this approach is demonstrated through real-world experiments on a robot performing pick-and-place tasks with a variety of objects of different sizes.

While the need for hierarchies within control systems is apparent, it is also clear to many researchers that such hierarchies should be learned. Learning both the structure and the component behaviors is a di cult task. The bene t of learning the hierarchical structures of behaviors is that the decomposition of the control structure into smaller transportable chunks allows previously learned knowledge to be applied to new but related tasks. Presented in this paper are improvements to Nested Q-learning (NQL) that allow more realistic learning of control hierarchies in reinforcement environments. Also presented is a simulation of a simple robot performing a series of related tasks that is used to compare both hierarchical and non-hierarchal learning techniques.

In this paper we discuss the recent evolution of spoken dialog systems in commercial deployments. Yet based on a simple finite state machine design paradigm, dialog systems reached today a higher level of complexity. The availability of massive amounts of data during deployment led to the development of continuous optimization strategy pushing the design and development of spoken dialog applications from an art to science. At the same time new methods for evaluating the subjective caller experience are available. Finally we describe the inevitable evolution for spoken dialog applications from speech only to multimodal interaction.

This paper describes discriminative language modeling for a large vocabulary speech recognition task. We contrast two parameter estimation methods: the perceptron algorithm, and a method based on maximizing the regularized conditional log-likelihood. The models are encoded as deterministic weighted finite state automata, and are applied by intersecting the automata with word-lattices that are the output from a baseline recognizer. The perceptron algorithm has the benefit of automatically selecting a relatively small feature set in just a couple of passes over the training data. We describe a method based on regularized likelihood that makes use of the feature set given by the perceptron algorithm, and initialization with the perceptron’s weights; this method gives an additional 0.5% reduction in word error rate (WER) over training with the perceptron alone. The final system achieves a 1.8% absolute reduction in WER for a baseline first-pass recognition system (from 39.2% to 37.4%), and a 0.9% absolute reduction in WER for a multi-pass recognition system (from 28.9% to 28.0%). 2006 Elsevier Ltd. All rights reserved.

Statistical supervised learning techniques have been successful for many natural language processing tasks, but they require labeled datasets, which can be expensive to obtain. On the other hand, unlabeled data (raw text) is often available "for free" in large quantities. Unlabeled data has shown promise in improving the performance of a number of tasks, e.g. word sense disambiguation, information extraction, and natural language parsing. In this thesis, we focus on two segmentation tasks, named-entity recognition and Chinese word segmentation. The goal of named-entity recognition is to detect and classify names of people, organizations, and locations in a sentence. The goal of Chinese word segmentation is to find the word boundaries in a sentence that has been written as a string of characters without spaces. Our approach is as follows: In a preprocessing step, we use raw text to cluster words and calculate mutual information statistics. The output of this step is then used as features in a supervised model, specifically a global linear model trained using the Perceptron algorithm. We also compare Markov and semi-Markov models on the two segmentation tasks. Our results show that features derived from unlabeled data substantially improves performance, both in terms of reducing the amount of labeled data needed to achieve a certain performance level and in terms of reducing the error using a fixed amount of labeled data. We find that sometimes semi-Markov models can also improve performance over Markov models. Thesis Supervisor: Michael Collins Title: Assistant Professor, CSAIL

The paper describes the authors’ experience in using knowledge representation and reasoning tools in the design of robots. The focus is on the systematic construction of models of the robot’s capabilities and its domain at different resolutions, and on establishing a clear relationship between the models at the different resolutions.

We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including: part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements.

Natural language interfaces for robot control aspire to find the best sequence of actions that reflect the behavior intended by the instruction. This is difficult because of the diversity of language, variety of environments, and heterogeneity of tasks. Previous work has demonstrated that probabilistic graphical models constructed from the parse structure of natural language can be used to identify motions that most closely resemble verb phrases. Such approaches however quickly succumb to computational bottlenecks imposed by construction and search the space of possible actions. Planning constraints, which define goal regions and separate the admissible and inadmissible states in an environment model, provide an interesting alternative to represent the meaning of verb phrases. In this paper we present a new model called the Distributed Correspondence Graph (DCG) to infer the most likely set of planning constraints from natural language instructions. A trajectory planner then uses these planning constraints to find a sequence of actions that resemble the instruction. Separating the problem of identifying the action encoded by the language into individual steps of planning constraint inference and motion planning enables us to avoid computational costs associated with generation and evaluation of many trajectories. We present experimental results from comparative experiments that demonstrate improvements in efficiency in natural language understanding without loss of accuracy.

We motivate and describe a new freely available human-human dialogue data set for interactive learning of visually grounded word meanings through ostensive definition by a tutor to a learner. The data has been collected using a novel, character-by-character variant of the DiET chat tool (Healey et al., 2003; Mills and Healey, submitted) with a novel task, where a Learner needs to learn invented visual attribute words (such as “burchak” for square) from a tutor. As such, the text-based interactions closely resemble face-to-face conversation and thus contain many of the linguistic phenomena encountered in natural, spontaneous dialogue. These include selfand other-correction, mid-sentence continuations, interruptions, overlaps, fillers, and hedges. We also present a generic n-gram framework for building user (i.e. tutor) simulations from this type of incremental data, which is freely available to researchers. We show that the simulations produce outputs that are similar to the original data (e.g. 78% turn match similarity). Finally, we train and evaluate a Reinforcement Learning dialogue control agent for learning visually grounded word meanings, trained from the BURCHAK corpus. The learned policy shows comparable performance to a rulebased system built previously.

Reinforcement learning holds the promise of enabling autonomous robots to learn large repertoires of behavioral skills with minimal human intervention. However, robotic applications of reinforcement learning often compromise the autonomy of the learning process in favor of achieving training times that are practical for real physical systems. This typically involves introducing hand-engineered policy representations and human-supplied demonstrations. Deep reinforcement learning alleviates this limitation by training general-purpose neural network policies, but applications of direct deep reinforcement learning algorithms have so far been restricted to simulated settings and relatively simple tasks, due to their apparent high sample complexity. In this paper, we demonstrate that a recent deep reinforcement learning algorithm based on offpolicy training of deep Q-functions can scale to complex 3D manipulation tasks and can learn deep neural network policies efficiently enough to train on real physical robots. We demonstrate that the training times can be further reduced by parallelizing the algorithm across multiple robots which pool their policy updates asynchronously. Our experimental evaluation shows that our method can learn a variety of 3D manipulation skills in simulation and a complex door opening skill on real robots without any prior demonstrations or manually designed representations.


Question: Acquiring structured finite-state models from unorganized observational data.

This paper explores the challenge of imitation learning in environments with extensive state spaces, where expert demonstrations are limited to a narrow subset of the potential states. Inverse Reinforcement Learning (IRL) serves as a powerful technique for extrapolating from these demonstrations, predicated on the hypothesis that the expert's actions are optimal within a Markov Decision Process (MDP) framework. Prior work in IRL often assumes the ability to compute (near)optimal policies for various reward functions, a condition that becomes increasingly untenable in scenarios with expansive or continuous state spaces. To address this limitation, we introduce a model-free IRL algorithm that minimizes the relative entropy between the empirical distribution of state-action trajectories under a uniform policy and those under the learned policy. This minimization is achieved through stochastic gradient descent. We contrast our novel method with established IRL algorithms that rely on approximate MDP models. Experimental outcomes from simulations involving car racing, gridworld, and ball-in-a-cup tasks substantiate the efficacy of our approach in acquiring effective policies from a restricted number of demonstrations. This advancement paves the way for more practical applications of IRL in complex, real-world settings.

This talk proposes a very simple “baseline architecture” for a learning agent that can handle stochastic, partially observable environments. The architecture uses reinforcement learning together with a method for representing temporal processes as graphical models. I will discuss methods for leaming the parameters and structure of such representations from sensory inputs, and for computing posterior probabilities. Some open problems remain before we can try out the complete agent; more arise when we consider scaling up. A second theme of the talk will be whether reinforcement learning can provide a good model of animal and human learning. To answer this question, we must do inverse reinforcement learning: given the observed behaviour, what reward signal, if any, is being optimized? This seems to be a very interesting problem for the COLT, UAI, and ML communities, and has been addressed in econometrics under the heading of structural estimation of Markov decision processes. 1 Learning in uncertain environments AI is about the construction of intelligent agents, i.e., systems that perceive and act effectively (according to some performance measure) in an environment. I have argued elsewhere Russell and Norvig (1995) that most AI research has focused on environments that are static, deterministic, discrete, and fully observable. What is to be done when, as in the real world, the environment is dynamic, stochastic, continuous, and partially observable? ‘This paper draws on a variety of research efforts supported by NSF @I-9634215), ONR (N00014-97-l-0941), and AR0 (DAAH04-96-1-0341). Permission to make digital or hard copies of all or p.art of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for prolit or commercial adwantage and that copies bear this notice and the full citation on the first page. To copy otherwise. to republish, to post on servers or to redistribute to lists, requires prior specific pemlission and/or a fee. COLT 98 Madison WI IJSA Copyright ACM 1998 1-5X1 13-057--0/9X/ 7...$5.00 In recent years, reinforcement learning (also called neurodynamic programming) has made rapid progress as an approachfor building agents automatically (Sutton, 1988; Kaelbling et al., 1996; Bertsekas & Tsitsiklis, 1996). The basic idea is that the performance measure is made available to the agent in the form of a rewardfunction specifying the reward for each state that the agent passes through. The performance measure is then the sum of the rewards obtained. For example, when a bumble bee forages, the reward function at each time step might be some combination of the distance flown (weighted negatively) and the nectar ingested. Reinforcement learning (RL) methods are essentially online algorithmd for solving Markovdecisionprocesses (MDPs). An MDP is defined by the reward function and a model, that is, the state transition probabilities conditioned on each possible action. RL algorithms can be model-based, where the agent learns a model, or model-free-e.g., Q-learning citeWatkins: 1989, which learns just a function Q(s, a) specifying the long-term value of taking action a in state s and acting optimally thereafter. Despite their successes, RL methods have been restricted largely tofully observable MDPs, in which the sensory input at each state is sufficient to identify the state. Obviously, in the real world, we must often deal with partially observable MDPs (POMDPs). Astrom (1965) proved that optimal decisions in POMDPs depend on the belief state b at each point in time, i.e., the posterior probability distribution over all possible actual states, given all evidence to date. The functions V and Q then become functions of b instead of s. Parr and Russell (1995) describes a very simple POMDP RL algorithm using an explicit representation of b as a vector of probabilities, and McCallum (1993) shows a way to approximate the belief state using recent percept sequences. Neither approach is likely to scale up to situations with large numbers of state variables and long-term temporal dependencies. What is needed is a way of representing the model compactly and updating the belief state efficiently given the model and each new observation. Dynamic Bayesian networks (Dean & Kanazawa, 1989) seem to have some of the required properties; in particular, they have significant advantages over other approaches such as Kalman filters and hidden Markov models. Our baseline architecture, shown in Figure 1, uses DBNs to represent and update the belief state as new sensor information arrives. Given a representation for b, the reward signal is used to learn a Q-function represented by some “black-box” function approximator such as a neural network. Provided we can handle hybrid (dis-

Narrative structure is an ubiquitous and intriguing phenomenon. By virtue of structure we recognize the presence of Villainy or Revenge in a story, even if that word is not actually present in the text. Narrative structure is an anvil for forging new artificial intelligence and machine learning techniques, and is a window into abstraction and conceptual learning as well as into culture and its influence on cognition. I advance our understanding of narrative structure by describing Analogical Story Merging (ASM), a new machine learning algorithm that can extract culturally-relevant plot patterns from sets of folktales. I demonstrate that ASM can learn a substantive portion of Vladimir Propp’s influential theory of the structure of folktale plots. The challenge was to take descriptions at one semantic level, namely, an event timeline as described in folktales, and abstract to the next higher level: structures such as Villainy, StuggleVictory, and Reward. ASM is based on Bayesian Model Merging, a technique for learning regular grammars. I demonstrate that, despite ASM’s large search space, a carefully-tuned prior allows the algorithm to converge, and furthermore it reproduces Propp’s categories with a chance-adjusted Rand index of 0.511 to 0.714. Three important categories are identified with F-measures above 0.8. The data are 15 Russian folktales, comprising 18,862 words, a subset of Propp’s original tales. This subset was annotated for 18 aspects of meaning by 12 annotators using the Story Workbench, a general text-annotation tool I developed for this work. Each aspect was doubly-annotated and adjudicated at inter-annotator F-measures that cluster around 0.7 to 0.8. It is the largest, most deeply-annotated narrative corpus assembled to date. The work has significance far beyond folktales. First, it points the way toward important applications in many domains, including information retrieval, persuasion and negotiation, natural language understanding and generation, and computational creativity. Second, abstraction from natural language semantics is a skill that underlies many cognitive tasks, and so this work provides insight into those processes. Finally, the work opens the door to a computational understanding of cultural influences on cognition and understanding cultural differences as captured in stories. Dissertation Supervisor: Patrick H. Winston Professor, Electrical Engineering and Computer Science Dissertation Committee: Whitman A. Richards Professor, Brain & Cognitive Sciences Peter Szolovits Professor, Electrical Engineering and Computer Science & Harvard-MIT Division of Health Sciences and Technology Joshua B. Tenenbaum Professor, Brain & Cognitive Sciences

Most human behaviors consist of multiple parts, steps, or subtasks. These structures guide our action planning and execution, but when we observe others, the latent structure of their actions is typically unobservable, and must be inferred in order to learn new skills by demonstration, or to assist others in completing their tasks. For example, an assistant who has learned the subgoal structure of a colleague’s task can more rapidly recognize and support their actions as they unfold. Here we model how humans infer subgoals from observations of complex action sequences using a nonparametric Bayesian model, which assumes that observed actions are generated by approximately rational planning over unknown subgoal sequences. We test this model with a behavioral experiment in which humans observed different series of goal-directed actions, and inferred both the number and composition of the subgoal sequences associated with each goal. The Bayesian model predicts human subgoal inferences with high accuracy, and significantly better than several alternative models and straightforward heuristics. Motivated by this result, we simulate how learning and inference of subgoals can improve performance in an artificial user assistance task. The Bayesian model learns the correct subgoals from fewer observations, and better assists users by more rapidly and accurately inferring the goal of their actions than alternative approaches.

A method is presented to learn object class models from unlab eled and unsegmented cluttered scenes for the purpose of visual object recognition. The variabili ty across a class of objects is modeled in a principled way, treating objects as flexible constellatio ns f rigid parts (features). Variability is represented by a joint probability density function (pdf) o n the shape of the constellation and the output of part detectors. Corresponding “constellation mo dels” can be learned in a completely unsupervised fashion. In a first stage, the learning method a utomatically identifies distinctive parts in the training set by applying a clustering algorithm to pat terns selected by an interest operator. It then learns the statistical shape model using expectation m aximization. Mixtures of constellation models can be defined and applied to “discover” object catego ri s in an unsupervised manner. The method achieves very good classification results on human fa ces, cars, leaves, handwritten letters, and cartoon characters.

Process mining includes the automated discovery of processes from event logs. Based on observed events (e.g., activities being executed or messages being exchanged) a process model is constructed. One of the essential problems in process mining is that one cannot assume to have seen all possible behavior. At best, one has seen a representative subset. Therefore, classical synthesis techniques are not suitable as they aim at finding a model that is able to exactly reproduce the log. Existing process mining techniques try to avoid such “overfitting” by generalizing the model to allow for more behavior. This generalization is often driven by the representation language and very crude assumptions about completeness. As a result, parts of the model are “overfitting” (allow only for what has actually been observed) while other parts may be “underfitting” (allow for much more behavior without strong support for it). None of the existing techniques enables the user to control the balance between “overfitting” and “underfitting”. To address this, we propose a two-step approach. First, using a configurable approach, a transition system is constructed. Then, using the “theory of regions”, the model is synthesized. The approach has been implemented in the context of ProM and overcomes many of the limitations of traditional approaches.

The increasing complexity of today’s testing scenarios for complex systems demands an integrated, open, and flexible approach to support the management of the overall test process. “Classical” model-based testing approaches, where a complete and precise formal specification serves as a reference for automatic test generation, are often impractical. Reasons are, on the one hand, the absence of a suitable formal specification. As complex systems are composed of several components, either hardware or software, often pre-built and third party, it is unrealistic to assume that a formal specification exists a priori. On the other hand, a sophisticated test execution environment is needed that can handle distributed test cases. This is because the test actions and observations can take place on different subsystems of the overall system. This thesis presents a novel approach to the integrated testing of complex systems. Our approach offers a coarse grained test environment, realized in terms of a component-based test design on top of a library of elementary but intuitively understandable test case fragments. The relations between the fragments are treated orthogonally, delivering a test design and execution environment enhanced by means of light-weight formal verification methods. In this way we are able to shift the test design issues from total experts of the system and the used test tools to experts of the system’s logic only. We illustrate the practical usability of our approach by means of industrial case studies in two different application domains: Computer Telephony Integrated solutions and Web-based applications. As an enhancement of our integrated test approach we provide an algorithm for generating approximate models for complex systems a posteriori. This is done by optimizing a standard machine learning algorithm according to domain-specific structural properties, i.e. properties like prefix-closeness, input-determinism, as well as independency and symmetries of events. The resulting models can never be exact, i.e. reflect the complete and correct behaviour of the considered system. Nevertheless they can be useful in practice, to represent the cumulative knowledge of the system in a consistent description.

Language acquisition in both natural and artificial language learning settings crucially depends on extracting information from ordered sequences. A shared sequence learning mechanism is thus assumed to underlie both natural and artificial language learning. A growing body of empirical evidence is consistent with this hypothesis. By means of artificial language learning experiments, we may therefore gain more insight in this shared mechanism. In this paper, we review empirical evidence from artificial language learning and computational modeling studies, as well as natural language data, and suggest that there are two key factors that help determine processing complexity in sequence learning, and thus in natural language processing. We propose that the specific ordering of non-adjacent dependencies (i.e. nested or crossed), as well as the number of non-adjacent dependencies to be resolved simultaneously (i.e. two or three) are important factors in gaining more insight into the boundaries of human sequence learning; and thus, also in natural language processing. The implications for theories of linguistic competence are discussed.

This paper proposes an improvement to the existing data-driven Neural Belief Tracking (NBT) framework for Dialogue State Tracking (DST). The existing NBT model uses a hand-crafted belief state update mechanism which involves an expensive manual retuning step whenever the model is deployed to a new dialogue domain. We show that this update mechanism can be learned jointly with the semantic decoding and context modelling parts of the NBT model, eliminating the last rule-based module from this DST framework. We propose two different statistical update mechanisms and show that dialogue dynamics can be modelled with a very small number of additional model parameters. In our DST evaluation over three languages, we show that this model achieves competitive performance and provides a robust framework for building resource-light DST models.

Statistical methods have long been the dominant approach in speech recognition and probabilistic modelling in ASR is now a mature technology. The use of statistical methods in other areas of spoken dialogue is however more recent and rather less mature. This paper reviews spoken dialogue systems from a statistical modelling perspective. The complete system is first presented as a partially observable Markov decision process. The various sub-components are then exposed by introducing appropriate intermediate variables. Samples of existing work are reviewed within this framework, including dialogue control and optimisation, semantic interpretation, goal detection, natural language generation and synthesis.

We propose Symbol-Refined Tree Substitution Grammars (SR-TSGs) for syntactic parsing. An SR-TSG is an extension of the conventional TSG model where each nonterminal symbol can be refined (subcategorized) to fit the training data. We aim to provide a unified model where TSG rules and symbol refinement are learned from training data in a fully automatic and consistent fashion. We present a novel probabilistic SR-TSG model based on the hierarchical Pitman-Yor Process to encode backoff smoothing from a fine-grained SR-TSG to simpler CFG rules, and develop an efficient training method based on Markov Chain Monte Carlo (MCMC) sampling. Our SR-TSG parser achieves an F1 score of 92.4% in the Wall Street Journal (WSJ) English Penn Treebank parsing task, which is a 7.7 point improvement over a conventional Bayesian TSG parser, and better than state-of-the-art discriminative reranking parsers.


Question: Acquiring structured finite-state models from unorganized observational data.

This paper explores the development of Linear Predictive Hidden Markov Models (LPHMMs) and their application in analyzing and processing the speech signal. Unlike traditional Hidden Markov Models (HMMs), LPHMMs integrate linear predictive coding (LPC) techniques to capture the temporal dynamics and spectral characteristics of speech more effectively. The LPHMM framework is designed to model the speech signal as a sequence of states, where each state is characterized by a set of LPC coefficients that describe the vocal tract's filtering properties at a particular instant. By combining the probabilistic framework of HMMs with the spectral accuracy of LPC, LPHMMs offer a more nuanced representation of speech, enabling improved performance in automatic speech recognition (ASR) and other speech processing tasks. The paper discusses the formulation of LPHMMs, detailing the process of estimating LPC coefficients and their integration into the HMM structure. It also presents experimental results comparing LPHMMs to conventional HMMs in various speech recognition scenarios, demonstrating the advantages of the LPHMM approach in terms of recognition accuracy and robustness to noise. Additionally, the paper examines the potential of LPHMMs in handling speaker variability and adapting to different acoustic environments, showcasing the versatility of this model in real-world speech applications.

**Question**: Understanding the Role of Attention Mechanisms in Neural Machine Translation

**Documents**: The role of attention in sequence-to-sequence learning

**Similar Documents**: This paper delves into the significance of attention mechanisms in sequence-to-sequence (Seq2Seq) learning, particularly within the context of neural machine translation (NMT). Attention mechanisms have revolutionized the way models process and generate sequences by enabling them to selectively focus on different parts of the input during the decoding phase. Unlike earlier Seq2Seq models that relied solely on a fixed context vector, attention allows the decoder to dynamically weigh the relevance of each input token when generating each output token. The paper discusses the architecture of attention-based NMT models, explaining how they compute attention weights and use them to produce context-aware translations. It also highlights the impact of attention on the model's ability to handle long sequences, manage syntactic and semantic complexities, and improve overall translation quality. Through extensive experiments and comparative analysis, the paper demonstrates the effectiveness of attention mechanisms in various NMT settings, from translating between different languages to adapting to domain-specific text. The paper further explores the interpretability of attention weights, illustrating how they can provide insights into the model's decision-making process and facilitate error analysis. Overall, the paper underscores

of “Any Domain Parsing: Automatic Domain Adaptation for Natural Language Parsing” by David McClosky , Ph.D., Brown University, May, 2010. Current efforts in syntactic parsing are largely data-driv en. These methods require labeled examples of syntactic structures to learn statistical patterns governing these structures. Labeled data typically requires expert annotators which makes it both time consuming and costly to p roduce. Furthermore, once training data has been created for one textual domain, portability to similar domains is limited. This domain-dependence has inspired a large body of work since syntactic parsing aims to capture syntactic patterns across an entire language rather than just a specific domain. The simplest approach to this task is to assume that the targe t domain is essentially the same as the source domain. No additional knowledge about the target domain is i cluded. A more realistic approach assumes that only raw text from the target domain is available. This a s umption lends itself well to semi-supervised learning methods since these utilize both labeled and unlab eled examples. This dissertation focuses on a family of semi-supervised me thods called self-training. Self-training creates semi-supervised learners from existing supervised learne rs with minimal effort. We first show results on self-training for constituency parsing within a single dom ain. While self-training has failed here in the past, we present a simple modification which allows it to succeed, p roducing state-of-the-art results for English constituency parsing. Next, we show how self-training is be neficial when parsing across domains and helps further when raw text is available from the target domain. On e of the remaining issues is that one must choose a training corpus appropriate for the target domain or perfo rmance may be severely impaired. Humans can do this in some situations, but this strategy becomes less prac tical as we approach larger data sets. We present a technique, Any Domain Parsing, which automatically detect s useful source domains and mixes them together to produce a customized parsing model. The resulting models perform almost as well as the best seen parsing models (oracle) for each target domain. As a result, we have a fully automatic syntactic constituency parser which can produce high-quality parses for all types of text, regardless of domain. Abstract of “Any Domain Parsing: Automatic Domain Adaptation for Natural Language Parsing” by David McClosky , Ph.D., Brown University, May, 2010.of “Any Domain Parsing: Automatic Domain Adaptation for Natural Language Parsing” by David McClosky , Ph.D., Brown University, May, 2010. Current efforts in syntactic parsing are largely data-driv en. These methods require labeled examples of syntactic structures to learn statistical patterns governing these structures. Labeled data typically requires expert annotators which makes it both time consuming and costly to p roduce. Furthermore, once training data has been created for one textual domain, portability to similar domains is limited. This domain-dependence has inspired a large body of work since syntactic parsing aims to capture syntactic patterns across an entire language rather than just a specific domain. The simplest approach to this task is to assume that the targe t domain is essentially the same as the source domain. No additional knowledge about the target domain is i cluded. A more realistic approach assumes that only raw text from the target domain is available. This a s umption lends itself well to semi-supervised learning methods since these utilize both labeled and unlab eled examples. This dissertation focuses on a family of semi-supervised me thods called self-training. Self-training creates semi-supervised learners from existing supervised learne rs with minimal effort. We first show results on self-training for constituency parsing within a single dom ain. While self-training has failed here in the past, we present a simple modification which allows it to succeed, p roducing state-of-the-art results for English constituency parsing. Next, we show how self-training is be neficial when parsing across domains and helps further when raw text is available from the target domain. On e of the remaining issues is that one must choose a training corpus appropriate for the target domain or perfo rmance may be severely impaired. Humans can do this in some situations, but this strategy becomes less prac tical as we approach larger data sets. We present a technique, Any Domain Parsing, which automatically detect s useful source domains and mixes them together to produce a customized parsing model. The resulting models perform almost as well as the best seen parsing models (oracle) for each target domain. As a result, we have a fully automatic syntactic constituency parser which can produce high-quality parses for all types of text, regardless of domain. Any Domain Parsing: Automatic Domain Adaptation for Natural Language Parsing by David McClosky B. S., University of Rochester, 2004 Sc. M., Brown University, 2006 A dissertation submitted in partial fulfillment of the requirements for the Degree of Doctor of Philosophy in the Department of Computer Science at Brown University. Providence, Rhode Island May, 2010 c © Copyright 2010 by David McClosky This dissertation by David McClosky is accepted in its prese nt form by the Department of Computer Science as satisfying the disser tation requirement for the degree of Doctor of Philosophy. Date Eugene Charniak, Director Recommended to the Graduate Council Date Mark Johnson, Reader Cognitive and Linguistic Sciences Date Dan Klein, Reader University of California at Berkeley Approved by the Graduate Council Date Sheila Bond Dean of the Graduate School

Most current speech recognition systems use hidden Markov models (HMMs) to deal with the temporal variability of speech and Gaussian mixture models to determine how well each state of each HMM fits a frame or a short window of frames of coefficients that represents the acoustic input. An alternative way to evaluate the fit is to use a feedforward neural network that takes several frames of coefficients as input and produces posterior probabilities over HMM states as output. Deep neural networks with many hidden layers, that are trained using new methods have been shown to outperform Gaussian mixture models on a variety of speech recognition benchmarks, sometimes by a large margin. This paper provides an overview of this progress and represents the shared views of four research groups who have had recent successes in using deep neural networks for acoustic modeling in speech recognition.

Unbounded, unordered, global-scale datasets are increasingly common in day-to-day business (e.g. Web logs, mobile usage statistics, and sensor networks). At the same time, consumers of these datasets have evolved sophisticated requirements, such as event-time ordering and windowing by features of the data themselves, in addition to an insatiable hunger for faster answers. Meanwhile, practicality dictates that one can never fully optimize along all dimensions of correctness, latency, and cost for these types of input. As a result, data processing practitioners are left with the quandary of how to reconcile the tensions between these seemingly competing propositions, often resulting in disparate implementations and systems. We propose that a fundamental shift of approach is necessary to deal with these evolved requirements in modern data processing. We as a field must stop trying to groom unbounded datasets into finite pools of information that eventually become complete, and instead live and breathe under the assumption that we will never know if or when we have seen all of our data, only that new data will arrive, old data may be retracted, and the only way to make this problem tractable is via principled abstractions that allow the practitioner the choice of appropriate tradeoffs along the axes of interest: correctness, latency, and cost. In this paper, we present one such approach, the Dataflow Model, along with a detailed examination of the semantics it enables, an overview of the core principles that guided its design, and a validation of the model itself via the real-world experiences that led to its development. We use the term “Dataflow Model” to describe the processing model of Google Cloud Dataflow [20], which is based upon technology from FlumeJava [12] and MillWheel [2]. This work is licensed under the Creative Commons AttributionNonCommercial-NoDerivs 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/. Obtain permission prior to any use beyond those covered by the license. Contact copyright holder by emailing info@vldb.org. Articles from this volume were invited to present their results at the 41st International Conference on Very Large Data Bases, August 31st September 4th 2015, Kohala Coast, Hawaii. Proceedings of the VLDB Endowment, Vol. 8, No. 12 Copyright 2015 VLDB Endowment 2150-8097/15/08.

Narrative structure is an ubiquitous and intriguing phenomenon. By virtue of structure we recognize the presence of Villainy or Revenge in a story, even if that word is not actually present in the text. Narrative structure is an anvil for forging new artificial intelligence and machine learning techniques, and is a window into abstraction and conceptual learning as well as into culture and its influence on cognition. I advance our understanding of narrative structure by describing Analogical Story Merging (ASM), a new machine learning algorithm that can extract culturally-relevant plot patterns from sets of folktales. I demonstrate that ASM can learn a substantive portion of Vladimir Propp’s influential theory of the structure of folktale plots. The challenge was to take descriptions at one semantic level, namely, an event timeline as described in folktales, and abstract to the next higher level: structures such as Villainy, StuggleVictory, and Reward. ASM is based on Bayesian Model Merging, a technique for learning regular grammars. I demonstrate that, despite ASM’s large search space, a carefully-tuned prior allows the algorithm to converge, and furthermore it reproduces Propp’s categories with a chance-adjusted Rand index of 0.511 to 0.714. Three important categories are identified with F-measures above 0.8. The data are 15 Russian folktales, comprising 18,862 words, a subset of Propp’s original tales. This subset was annotated for 18 aspects of meaning by 12 annotators using the Story Workbench, a general text-annotation tool I developed for this work. Each aspect was doubly-annotated and adjudicated at inter-annotator F-measures that cluster around 0.7 to 0.8. It is the largest, most deeply-annotated narrative corpus assembled to date. The work has significance far beyond folktales. First, it points the way toward important applications in many domains, including information retrieval, persuasion and negotiation, natural language understanding and generation, and computational creativity. Second, abstraction from natural language semantics is a skill that underlies many cognitive tasks, and so this work provides insight into those processes. Finally, the work opens the door to a computational understanding of cultural influences on cognition and understanding cultural differences as captured in stories. Dissertation Supervisor: Patrick H. Winston Professor, Electrical Engineering and Computer Science Dissertation Committee: Whitman A. Richards Professor, Brain & Cognitive Sciences Peter Szolovits Professor, Electrical Engineering and Computer Science & Harvard-MIT Division of Health Sciences and Technology Joshua B. Tenenbaum Professor, Brain & Cognitive Sciences

In this paper we present an overview of recent developments in discourse theory and parsing under the Linguistic Discourse Model (LDM) framework, a semantic theory of discourse structure. We give a novel approach to the problem of discourse segmentation based on discourse semantics and sketch a limited but robust approach to symbolic discourse parsing based on syntactic, semantic and lexical rules. To demonstrate the utility of the system in a real application, we briefly describe the architecture of the PALSUMM system, a symbolic summarization system being developed at FX Palo Alto Laboratory that uses discourse structures constructed using the theory outlined to summarize written English prose texts. 1

We describe a series o,f five statistical models o,f the translation process and give algorithms,for estimating the parameters o,f these models given a set o,f pairs o,f sentences that are translations o,f one another. We define a concept o,f word-by-word alignment between such pairs o,f sentences. For any given pair of such sentences each o,f our models assigns a probability to each of the possible word-by-word alignments. We give an algorithm for seeking the most probable o,f these alignments. Although the algorithm is suboptimal, the alignment thus obtained accounts well for the word-by-word relationships in the pair o,f sentences. We have a great deal o,f data in French and English from the proceedings o,f the Canadian Parliament. Accordingly, we have restricted our work to these two languages; but we,feel that because our algorithms have minimal linguistic content they would work well on other pairs o,f languages. We also ,feel, again because of the minimal linguistic content o,f our algorithms, that it is reasonable to argue that word-by-word alignments are inherent in any sufficiently large bilingual corpus.

Systems that extract structured information from natural language passages have been highly successful in specialized domains. The time is opportune for developing analogous applications for molecular biology and genomics. We present a system, GENIES, that extracts and structures information about cellular pathways from the biological literature in accordance with a knowledge model that we developed earlier. We implemented GENIES by modifying an existing medical natural language processing system, MedLEE, and performed a preliminary evaluation study. Our results demonstrate the value of the underlying techniques for the purpose of acquiring valuable knowledge from biological journals.

In many applications such as personal digital assistants, there is a constant need for new domains to increase the system’s coverage of user queries. A conventional approach is to learn a separate model every time a new domain is introduced. This approach is slow, inefficient, and a bottleneck for scaling to a large number of domains. In this paper, we introduce a framework that allows us to have a single model that can handle all domains: including unknown domains that may be created in the future as long as they are covered in the master schema. The key idea is to remove the need for distinguishing domains by explicitly predicting the schema of queries. Given permitted schema of a query, we perform constrained decoding on a lattice of slot sequences allowed under the schema. The proposed model achieves competitive and often superior performance over the conventional model trained separately per domain.

Language learnabi l i ty has been inves t igated . This refers to the following s i tua t ion: A class of possible languages is specified, t oge the r wi th a method of present ing informat ion to the learner about an unknown language, which is to be chosen from the class. The quest ion is now asked, " I s the informat ion sufficient to determine which of the possible languages is the unknown language?" M a n y definitions of l ea rnab i l i ty are possible, bu t only the following is considered here: Time is quant ized and has a finite s t a r t ing time. At each t ime the learner receives a un i t of in format ion and is to make a guess as to the i den t i t y of the unknown language on the basis of the informat ion received so far. This process continues forever. The class of languages will be considered lea~nabIe with respect to the specified me thod of in format ion p resen ta t ion if there is an a lgor i thm t h a t the learner can use to make his guesses, the a lgor i thm hav ing the following p roper ty : Given any language of the class, there is some finite t ime af te r which the guesses will all be the same and they will be correct . In this pre l iminary invest igat ion, a language is t aken to be a set of s tr ings on some finite a lphabet . The a lphabe t is the same for all languages of tile class. Several var ia t ions of each of the following two basic methods of informat ion presenta t ion are inves t iga ted: A text for a language generates the s t r ings of the language in any order such t h a t every s t r ing of the language occurs a t least once. An informant for a language tells whe the r a s t r ing is in the language, and chooses the s t r ings in some order such t h a t every s t r ing occurs at least once. I t was found t h a t the class of context-sensi t ive languages is learnable from an informant, but that not even the class of regular languages is learnable from a text.

The self-organizing map (SOM) represents an open set of input samples by a topologically organized, finite set of models. In this paper, a new version of the SOM is used for the clustering, organization, and visualization of a large database of symbol sequences (viz. protein sequences). This method combines two principles: the batch computing version of the SOM, and computation of the generalized median of symbol strings.