Question: Balancing aspects in the Neural Variational Inference dissertation.

This paper outlines a series of innovations in the training of deep variational autoencoders, significantly enhancing their capacity for unsupervised learning. Historically, variational autoencoders have been constrained by a shallow architecture, typically incorporating one or two layers of fully factorized stochastic latent variables, which has limited the complexity and expressiveness of the latent representation. Here, we detail three novel strategies to overcome these limitations, enabling the effective training of deep variational autoencoders with up to five stochastic layers. Firstly, we introduce a structure akin to the Ladder network to serve as the inference model, facilitating the extraction of more sophisticated and nuanced features. Secondly, we implement a warm-up phase during training to ensure that stochastic units remain active throughout the early stages, preventing premature convergence to suboptimal solutions. Lastly, we incorporate batch normalization to stabilize and accelerate the learning process. Through these advancements, we achieve state-of-the-art performance in terms of log-likelihood on a variety of benchmark datasets, demonstrating the enhanced capabilities of deep variational autoencoders in generative modeling.

In this work we investigate different avenues of improving the Neural Algorithm of Artistic Style [7]. While showing great results when transferring homogeneous and repetitive patterns, the original style representation often fails to capture more complex properties, like having separate styles of foreground and background. This leads to visual artifacts and undesirable textures appearing in unexpected regions when performing style transfer. We tackle this issue with a variety of approaches, mostly by modifying the style representation in order for it to capture more information and impose a tighter constraint on the style transfer result. In our experiments, we subjectively evaluate our best method as producing from barely noticeable to significant improvements in the quality of style transfer.

Neural machine translation, a recently proposed approach to machine translation based purely on neural networks, has shown promising results compared to the existing approaches such as phrasebased statistical machine translation. Despite its recent success, neural machine translation has its limitation in handling a larger vocabulary, as training complexity as well as decoding complexity increase proportionally to the number of target words. In this paper, we propose a method based on importance sampling that allows us to use a very large target vocabulary without increasing training complexity. We show that decoding can be efficiently done even with the model having a very large target vocabulary by selecting only a small subset of the whole target vocabulary. The models trained by the proposed approach are empirically found to match, and in some cases outperform, the baseline models with a small vocabulary as well as the LSTM-based neural machine translation models. Furthermore, when we use an ensemble of a few models with very large target vocabularies, we achieve performance comparable to the state of the art (measured by BLEU) on both the English→German and English→French translation tasks of WMT’14.

We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1 2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.

In this paper we propose a new framework for learning from large scale datasets based on iterative learning from small mini-batches. By adding the right amount of noise to a standard stochastic gradient optimization algorithm we show that the iterates will converge to samples from the true posterior distribution as we anneal the stepsize. This seamless transition between optimization and Bayesian posterior sampling provides an inbuilt protection against overfitting. We also propose a practical method for Monte Carlo estimates of posterior statistics which monitors a “sampling threshold” and collects samples after it has been surpassed. We apply the method to three models: a mixture of Gaussians, logistic regression and ICA with natural gradients.

Software frameworks for neural networks play key roles in the development and application of deep learning methods. However, as new types of deep learning models are developed, existing frameworks designed for convolutional neural networks are becoming less useful. In this paper, we introduce Chainer, a Pythonbased, standalone open source framework for deep learning models. Chainer provides a flexible, intuitive, and high performance means of implementing a full range of deep learning models, including state-of-the-art models such as recurrent neural networks and variational autoencoders.

The Differentiable Neural Computer (DNC) can learn algorithmic and question answering tasks. An analysis of its internal activation patterns reveals three problems: Most importantly, the lack of key-value separation makes the address distribution resulting from content-based look-up noisy and flat, since the value influences the score calculation, although only the key should. Second, DNC’s deallocation of memory results in aliasing, which is a problem for content-based look-up. Thirdly, chaining memory reads with the temporal linkage matrix exponentially degrades the quality of the address distribution. Our proposed fixes of these problems yield improved performance on arithmetic tasks, and also improve the mean error rate on the bAbI question answering dataset by 43%.

We show how the success of deep learning could depend not only on mathematics but also on physics: although well-known mathematical theorems guarantee that neural networks can approximate arbitrary functions well, the class of functions of practical interest can frequently be approximated through “cheap learning” with exponentially fewer parameters than generic ones. We explore how properties frequently encountered in physics such as symmetry, locality, compositionality, and polynomial log-probability translate into exceptionally simple neural networks. We further argue that when the statistical process generating the data is of a certain hierarchical form prevalent in physics and machine-learning, a deep neural network can be more efficient than a shallow one. We formalize these claims using information theory and discuss the relation to the renormalization group. We prove various “no-flattening theorems” showing when efficient linear deep networks cannot be accurately approximated by shallow ones without efficiency loss; for example, we show that n variables cannot be multiplied using fewer than 2 neurons in a single hidden layer.

We investigate neural circuits in the exacting setting that (i) the acquisition of a piece of knowledge can occur from a single interaction, (ii) the result of each such interaction is a rapidly evaluatable subcircuit, (iii) hundreds of thousands of such subcircuits can be acquired in sequence without substantially degrading the earlier ones, and (iv) recall can be in the form of a rapid evaluation of a composition of subcircuits that have been so acquired at arbitrary different earlier times.We develop a complexity theory, in terms of asymptotically matching upper and lower bounds, on the capacity of a neural network for executing, in this setting, the following action, which we call {\it association}: Each action sets up a subcircuit so that the excitation of a chosen set of neurons A will in future cause the excitation of another chosen set B.% As model of computation we consider the neuroidal model, a fully distributed model in which the quantitative resources n, the neuron numbers, d, the number of other neurons each neuron is connected to, and k, the inverse of the maximum synaptic strength, are all accounted for.A succession of experiences, possibly over a lifetime, results in the realization of a complex set of subcircuits. The composability requirement constrains the model to ensure that, for each association as realized by a subcircuit, the excitation in the triggering set of neurons A is quantitatively similar to that in the triggered set B, and also that the unintended excitation in the rest of the system is negligible. These requirements ensure that chains of associations can be triggeredWe first analyze what we call the Basic Mechanism, which uses only direct connections between neurons in the triggering set A and the target set B. We consider random networks of n neurons with expected number d of connections to and from each. We show that in the composable context capacity growth is limited by d^2, a severe limitation if the network is sparse, as it is in cortex. We go on to study the Expansive Mechanism, that additionally uses intermediate relay neurons which have high synaptic weights. For this mechanism we show that the capacity can grow as dn, to within logarithmic factors. From these two results it follows that in the composable regime, for the realistic cortical estimate of d=n^{\frac{1}{2}, superlinear capacity of order n^{\frac{3}{2}} in terms of the neuron numbers can be realized by the Expansive Mechanism, instead of the linear order n to which the Basic Mechanism is limited. More generally, for both mechanisms, we establish matching upper and lower bounds on capacity in terms of the parameters n, d, and the inverse maximum synaptic strength k.The results as stated above assume that in a set of associations, a target B can be triggered by at most one set A. It can be shown that the capacities are similar if the number m of As that can trigger a B is greater than one but small, but become severely constrained if m exceeds a certain threshold.

This paper introduces Non-Autonomous Input-Output Stable Network (NAIS-Net), a very deep architecture where each stacked processing block is derived from a time-invariant non-autonomous dynamical system. Non-autonomy is implemented by skip connections from the block input to each of the unrolled processing stages and allows stability to be enforced so that blocks can be unrolled adaptively to a pattern-dependent processing depth. NAIS-Net induces non-trivial, Lipschitz input-output maps, even for an infinite unroll length. We prove that the network is globally asymptotically stable so that for every initial condition there is exactly one input-dependent equilibrium assuming tanh units, and multiple stable equilibria for ReL units. An efficient implementation that enforces the stability under derived conditions for both fully-connected and convolutional layers is also presented. Experimental results show how NAIS-Net exhibits stability in practice, yielding a significant reduction in generalization gap compared to ResNets.

As book review editor of the IEEE Transactions on Neural Networks, Mohamad Hassoun has had the opportunity to assess the multitude of books on artificial neural networks that have Historically the data pairs and logical, fashion transition while associative memories commonly? This organization of such as a, clear and algorithms search simulated annealing learning algorithm. After the utility of how to, discover form. More than three different levels of the cost function will. Dean pomerleau in reinforcement and they, discovered two chapters on. This view the application for handwriting recognition this approach by arrows originating.


Question: Balancing considerations in neural variational inference dissertation.

This paper investigates the effectiveness of wide residual networks (WRNs) in addressing the limitations of deep residual networks, particularly in terms of feature reuse and training efficiency. Deep residual networks have been successful in enhancing performance by increasing depth, but each incremental gain in accuracy has required a significant increase in the number of layers, leading to diminished feature reuse and slower training times. To mitigate these issues, the authors conduct an in-depth analysis of the ResNet block architecture and propose a novel approach: increasing the width of the network while decreasing its depth. This results in wide residual networks, which are shown to outperform their thin and very deep counterparts in terms of both accuracy and efficiency. Remarkably, a simple 16-layer WRN surpasses the performance of much deeper networks, including those with thousands of layers, setting new benchmarks on datasets such as CIFAR, SVHN, COCO, and achieving substantial improvements on ImageNet. The superior performance of WRNs demonstrates the benefits of prioritizing width over depth in network architecture, offering a promising alternative for optimizing deep learning models. The authors' code and models are accessible at https://github.com/szagoruyko/wide-residual-networks, enabling further research and experimentation in this area.

Graphical models bring together graph theory and probability theory in a powerful formalism for multivariate statistical modeling. In statistical signal processing— as well as in related fields such as communication theory, control theory and bioinformatics—statistical models have long been formulated in terms of graphs, and algorithms for computing basic statistical quantities such as likelihoods and marginal probabilities have often been expressed in terms of recursions operating on these graphs. Examples include hidden Markov models, Markov random fields, the forward-backward algorithm and Kalman filtering [ Rabiner and Juang (1993); Pearl (1988); Kailath et al. (2000)]. These ideas can be understood, unified and generalized within the formalism of graphical models. Indeed, graphical models provide a natural framework for formulating variations on these classical architectures, and for exploring entirely new families of statistical models. The recursive algorithms cited above are all instances of a general recursive algorithm known as the junction tree algorithm [ Lauritzen and Spiegelhalter, 1988]. The junction tree algorithm takes advantage of factorization properties of the joint probability distribution that are encoded by the pattern of missing edges in a graphical model. For suitably sparse graphs, the junction tree algorithm provides a systematic and practical solution to the general problem of computing likelihoods and other statistical quantities associated with a graphical model. Unfortunately, many graphical models of practical interest are not " suitably sparse, " so that the junction tree algorithm no longer provides a viable computational solution to the problem of computing marginal probabilities and other expectations. One popular source of methods for attempting to cope with such cases is the Markov chain Monte Carlo (MCMC) framework, and indeed there is a significant literature on

In machine learning, there is a fundamental trade-off between ease of optimization and expressive power. Neural Networks, in particular, have enormous expressive power and yet are notoriously challenging to train. The nature of that optimization challenge changes over the course of learning. Traditionally in deep learning, one makes a static trade-off between the needs of early and late optimization. In this paper, we investigate a novel framework, GradNets, for dynamically adapting architectures during training to get the benefits of both. For example, we can gradually transition from linear to non-linear networks, deterministic to stochastic computation, shallow to deep architectures, or even simple downsampling to fully differentiable attention mechanisms. Benefits include increased accuracy, easier convergence with more complex architectures, solutions to test-time execution of batch normalization, and the ability to train networks of up to 200 layers.

We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1 2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.

The goal of this paper is to demonstrate a method for tensorizing neural networks based upon an efficient way of approximating scale invariant quantum states, the Multi-scale Entanglement Renormalization Ansatz (MERA). We employ MERA as a replacement for linear layers in a neural network and test this implementation on the CIFAR-10 dataset. The proposed method outperforms factorization using tensor trains, providing greater compression for the same level of accuracy and greater accuracy for the same level of compression. We demonstrate MERAlayers with 3900 times fewer parameters and a reduction in accuracy of less than 1% compared to the equivalent fully connected layers.

We consider the problem of density estimation on Riemannian manifolds. Density estimation on manifolds has many applications in fluid-mechanics, optics and plasma physics and it appears often when dealing with angular variables (such as used in protein folding, robot limbs, gene-expression) and in general directional statistics. In spite of the multitude of algorithms available for density estimation in the Euclidean spaces R that scale to large n (e.g. normalizing flows, kernel methods and variational approximations), most of these methods are not immediately suitable for density estimation in more general Riemannian manifolds. We revisit techniques related to homeomorphisms from differential geometry for projecting densities to sub-manifolds and use it to generalize the idea of normalizing flows to more general Riemannian manifolds. The resulting algorithm is scalable, simple to implement and suitable for use with automatic differentiation. We demonstrate concrete examples of this method on the n-sphere S. In recent years, there has been much interest in applying variational inference techniques to learning large scale probabilistic models in various domains, such as images and text [1, 2, 3, 4, 5, 6]. One of the main issues in variational inference is finding the best approximation to an intractable posterior distribution of interest by searching through a class of known probability distributions. The class of approximations used is often limited, e.g., mean-field approximations, implying that no solution is ever able to resemble the true posterior distribution. This is a widely raised objection to variational methods, in that unlike MCMC, the true posterior distribution may not be recovered even in the asymptotic regime. To address this problem, recent work on Normalizing Flows [7], Inverse Autoregressive Flows [8], and others [9, 10] (referred collectively as normalizing flows), focused on developing scalable methods of constructing arbitrarily complex and flexible approximate posteriors from simple distributions using transformations parameterized by neural networks, which gives these models universal approximation capability in the asymptotic regime. In all of these works, the distributions of interest are restricted to be defined over high dimensional Euclidean spaces. There are many other distributions defined over special homeomorphisms of Euclidean spaces that are of interest in statistics, such as Beta and Dirichlet (n-Simplex); Norm-Truncated Gaussian (n-Ball); Wrapped Cauchy and Von-Misses Fisher (n-Sphere), which find little applicability in variational inference with large scale probabilistic models due to the limitations related to density complexity and gradient computation [11, 12, 13, 14]. Many such distributions are unimodal and generating complicated distributions from them would require creating mixture densities or using auxiliary random variables. Mixture methods require further knowledge or tuning, e.g. number of mixture components necessary, and a heavy computational burden on the gradient computation in general, e.g. with quantile functions [15]. Further, mode complexity increases only linearly with mixtures as opposed to exponential increase with normalizing flows. Conditioning on auxiliary variables [16] on the other hand constrains the use of the created distribution, due to the need for integrating out the auxiliary factors in certain scenarios. In all of these methods, computation of low-variance gradients is difficult due to the fact that simulation of random variables cannot be in general reparameterized (e.g. rejection sampling [17]). In this work, we present methods that generalizes previous work on improving variational inference in R using normalizing flows to Riemannian manifolds of interest such as spheres S, tori T and their product topologies with R, like infinite cylinders. ar X iv :1 61 1. 02 30 4v 1 [ st at .M L ] 7 N ov 2 01 6 Figure 1: Left: Construction of a complex density on S by first projecting the manifold to R, transforming the density and projecting it back to S. Right: Illustration of transformed (S → R) densities corresponding to an uniform density on the sphere. Blue: empirical density (obtained by Monte Carlo); Red: Analytical density from equation (4); Green: Density computed ignoring the intrinsic dimensionality of S. These special manifolds M ⊂ R are homeomorphic to the Euclidean space R where n corresponds to the dimensionality of the tangent space of M at each point. A homeomorphism is a continuous function between topological spaces with a continuous inverse (bijective and bicontinuous). It maps point in one space to the other in a unique and continuous manner. An example manifold is the unit 2-sphere, the surface of a unit ball, which is embedded in R and homeomorphic to R (see Figure 1). In normalizing flows, the main result of differential geometry that is used for computing the density updates is given by, d~x = |det Jφ| d~u and represents the relationship between differentials (infinitesimal volumes) between two equidimensional Euclidean spaces using the Jacobian of the function φ : R → R that transforms one space to the other. This result only applies to transforms that preserve the dimensionality. However, transforms that map an embedded manifold to its intrinsic Euclidean space, do not preserve the dimensionality of the points and the result above become obsolete. Jacobian of such transforms φ : R → R with m > n are rectangular and an infinitesimal cube on R maps to an infinitesimal parallelepiped on the manifold. The relation between these volumes is given by d~x = √ det G d~u, where G = J φ Jφ is the metric induced by the embedding φ on the tangent space TxM, [18, 19, 20]. The correct formula for computing the density over M now becomes : ∫

It is possible to combine multiple latent-variable models of the same data by multiplying their probability distributions together and then renormalizing. This way of combining individual expert models makes it hard to generate samples from the combined model but easy to infer the values of the latent variables of each expert, because the combination rule ensures that the latent variables of different experts are conditionally independent when given the data. A product of experts (PoE) is therefore an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary. Training a PoE by maximizing the likelihood of the data is difficult because it is hard even to approximate the derivatives of the renormalization term in the combination rule. Fortunately, a PoE can be trained using a different objective function called contrastive divergence whose derivatives with regard to the parameters can be approximated accurately and efficiently. Examples are presented of contrastive divergence learning using several types of expert on several types of data.

We study probabilistic models of natural images and extend the autoregressive family of PixelCNN models by incorporating latent variables. Subsequently, we describe two new generative image models that exploit different image transformations as latent variables: a quantized grayscale view of the image or a multi-resolution image pyramid. The proposed models tackle two known shortcomings of existing PixelCNN models: 1) their tendency to focus on low-level image details, while largely ignoring high-level image information, such as object shapes, and 2) their computationally costly procedure for image sampling. We experimentally demonstrate benefits of our LatentPixelCNN models, in particular showing that they produce much more realistically looking image samples than previous stateof-the-art probabilistic models.

We show how the success of deep learning could depend not only on mathematics but also on physics: although well-known mathematical theorems guarantee that neural networks can approximate arbitrary functions well, the class of functions of practical interest can frequently be approximated through “cheap learning” with exponentially fewer parameters than generic ones. We explore how properties frequently encountered in physics such as symmetry, locality, compositionality, and polynomial log-probability translate into exceptionally simple neural networks. We further argue that when the statistical process generating the data is of a certain hierarchical form prevalent in physics and machine-learning, a deep neural network can be more efficient than a shallow one. We formalize these claims using information theory and discuss the relation to the renormalization group. We prove various “no-flattening theorems” showing when efficient linear deep networks cannot be accurately approximated by shallow ones without efficiency loss; for example, we show that n variables cannot be multiplied using fewer than 2 neurons in a single hidden layer.

The main aim of this dissertation is to study the topic of wavelet neural networks and see how they are useful for dynamical systems applications such as predicting chaotic time series and nonlinear noise reduction. To do this, the theory of wavelets has been studied in the first chapter, with the emphasis being on discrete wavelets. The theory of neural networks and its current applications in the modelling of dynamical systems has been shown in the second chapter. This provides sufficient background theory to be able to model and study wavelet neural networks. In the final chapter a wavelet neural network is implemented and shown to accurately estimate the dynamics of a chaotic system, enabling prediction and enhancing methods already available in nonlinear noise reduction.

In this paper we propose a new method for regularizing autoencoders by imposing an arbitrary prior on the latent representation of the autoencoder. Our method, named “adversarial autoencoder”, uses the recently proposed generative adversarial networks (GAN) in order to match the aggregated posterior of the hidden code vector of the autoencoder with an arbitrary prior. Matching the aggregated posterior to the prior ensures that there are no “holes” in the prior, and generating from any part of prior space results in meaningful samples. As a result, the decoder of the adversarial autoencoder learns a deep generative model that maps the imposed prior to the data distribution. We show how adversarial autoencoders can be used to disentangle style and content of images and achieve competitive generative performance on MNIST, Street View House Numbers and Toronto Face datasets.


Question: Balancing act: Exploring trade-offs in neural variational inference for thesis work.

This paper introduces an innovative class of subgradient methods designed to dynamically adapt to the evolving geometry of data encountered during earlier iterations, thereby enhancing gradient-based learning through more informed steps. Inspired by recent developments in stochastic optimization and online learning, our methods utilize proximal functions to govern the gradient steps. The adaptive modification of the proximal function is a key feature of our approach, greatly simplifying the selection of a learning rate and ensuring regret bounds that match those of the optimally chosen proximal function in hindsight. We provide detailed algorithms for adaptive subgradient methods tailored to empirical risk minimization problems, focusing on widely used regularization functions and domain constraints. Through both theoretical analysis and empirical testing, we demonstrate that our adaptive subgradient methods outperform conventional, non-adaptive subgradient algorithms, effectively identifying highly predictive but infrequently observed features akin to finding needles in haystacks.

The Adaptive Subgradient Methods for Online Learning and Stochastic Optimization paper presents a novel framework for gradient-based learning that dynamically adjusts to the characteristics of data observed during earlier iterations. This adaptive approach allows for more refined and informative learning steps, particularly in scenarios where very predictive but rarely encountered features are present. The methods draw upon recent advancements in stochastic optimization and online learning, leveraging proximal functions to control the gradient steps. A significant contribution is the introduction of an adaptive mechanism for modifying the proximal function, which eases the challenge of setting a learning rate and guarantees regret bounds equivalent to those achievable with the best hindsight-selected proximal function. The paper offers several efficient algorithms for empirical risk minimization tasks, specifically addressing common regularization functions and domain constraints. Through both theoretical analysis and practical experimentation, it is shown that adaptive subgradient methods significantly outperform their non-adaptive counterparts, showcasing their superiority in addressing empirical risk minimization problems.

Memory-augmented recurrent neural networks (M-RNNs) have demonstrated empirically that they are very attractive for many applications, but a good theoretical understanding of their behaviors is unclear yet. In this paper, three analytical indicators named duration, addressability, and capacity of general forms of the additional memory in M-RNNs are formalized. The analysis results of the interactions among these indicators reveal that it is hard for an M-RNN to simultaneously provide good performance on more than two out of three of indicators. Meanwhile, the duration, addressability, and capacity are applied to analyze and compare two M-RNNs: long short term memory and neural turing machine for different cases. The comparison results show that none of the models has better performance on one indicator than the other model all the time. Moreover, it is found that separating memory system into sub-memories can bring the increasing duration and addressability and the decreasing capacity for the whole memory system.

Regularized training of an autoencoder typically results in hidden unit biases that take on large negative values. We show that negative biases are a natural result of using a hidden layer whose responsibility is to both represent the input data and act as a selection mechanism that ensures sparsity of the representation. We then show that negative biases impede the learning of data distributions whose intrinsic dimensionality is high. We also propose a new activation function that decouples the two roles of the hidden layer and that allows us to learn representations on data with very high intrinsic dimensionality, where standard autoencoders typically fail. Since the decoupled activation function acts like an implicit regularizer, the model can be trained by minimizing the reconstruction error of training data, without requiring any additional regularization.

Modern convolutional networks, incorporating rectifiers and max-pooling, are neither smooth nor convex; standard guarantees therefore do not apply. Nevertheless, methods from convex optimization such as gradient descent and Adam are widely used as building blocks for deep learning algorithms. This paper provides the first convergence guarantee applicable to modern convnets, which furthermore matches a lower bound for convex nonsmooth functions. The key technical tool is the neural Taylor approximation – a straightforward application of Taylor expansions to neural networks – and the associated Taylor loss. Experiments on a range of optimizers, layers, and tasks provide evidence that the analysis accurately captures the dynamics of neural optimization. The second half of the paper applies the Taylor approximation to isolate the main difficulty in training rectifier nets – that gradients are shattered – and investigates the hypothesis that, by exploring the space of activation configurations more thoroughly, adaptive optimizers such as RMSProp and Adam are able to converge to better solutions.

This tutorial is a worked-out version of a 5-hour course originally held at AIS in September/October 2002. It has two distinct components. First, it contains a mathematically-oriented crash course on traditional training methods for recurrent neural networks, covering back-propagation through time (BPTT), real-time recurrent learning (RTRL), and extended Kalman filtering approaches (EKF). This material is covered in Sections 2 – 5. The remaining sections 1 and 6 – 9 are much more gentle, more detailed, and illustrated with simple examples. They are intended to be useful as a stand-alone tutorial for the echo state network (ESN) approach to recurrent neural network training.

Evolution strategies (ES) are a family of black-box optimization algorithms able to train deep neural networks roughly as well as Q-learning and policy gradient methods on challenging deep reinforcement learning (RL) problems, but are much faster (e.g. hours vs. days) because they parallelize better. However, many RL problems require directed exploration because they have reward functions that are sparse or deceptive (i.e. contain local optima), and it is unknown how to encourage such exploration with ES. Here we show that algorithms that have been invented to promote directed exploration in small-scale evolved neural networks via populations of exploring agents, specifically novelty search (NS) and quality diversity (QD) algorithms, can be hybridized with ES to improve its performance on sparse or deceptive deep RL tasks, while retaining scalability. Our experiments confirm that the resultant new algorithms, NS-ES and two QD algorithms, NSR-ES and NSRA-ES, avoid local optima encountered by ES to achieve higher performance on Atari and simulated robots learning to walk around a deceptive trap. This paper thus introduces a family of fast, scalable algorithms for reinforcement learning that are capable of directed exploration. It also adds this new family of exploration algorithms to the RL toolbox and raises the interesting possibility that analogous algorithms with multiple simultaneous paths of exploration might also combine well with existing RL algorithms outside ES.

Unlike traditional approaches that focus on the quantization at the network level, in this work we propose to minimize the quantization effect at the tensor level. We analyze the trade-off between quantization noise and clipping distortion in low precision networks. We identify the statistics of various tensors, and derive exact expressions for the mean-square-error degradation due to clipping. By optimizing these expressions, we show marked improvements over standard quantization schemes that normally avoid clipping. For example, just by choosing the accurate clipping values, more than 40% accuracy improvement is obtained for the quantization of VGG16-BN to 4-bits of precision. Our results have many applications for the quantization of neural networks at both training and inference time. One immediate application is for a rapid deployment of neural networks to low-precision accelerators without time-consuming fine tuning or the availability of the full datasets.

We propose a quantum generalisation of a classical neural network. The classical neurons are firstly rendered reversible by adding ancillary bits. Then they are generalised to being quantum reversible, i.e. unitary. (The classical networks we generalise are called feedforward, and have step-function activation functions.) The quantum network can be trained efficiently using gradient descent on a cost function to perform quantum generalisations of classical tasks. We demonstrate numerically that it can: (i) compress quantum states onto a minimal number of qubits, creating a quantum autoencoder, and (ii) discover quantum communication protocols such as teleportation. Our general recipe is theoretical and implementation-independent. The quantum neuron module can naturally be implemented photonically.

Although image compression has been actively studied for decades, there has been relatively little research on learning to compress images with modern neural networks. Standard approaches, such as those employing patch-based autoencoders, have shown a great deal of promise but cannot compete with popular image codecs because they fail to address three questions: 1) how to effectively binarize activations: in the absence of binarization, a bottleneck layer alone tends not to lead to efficient compression; 2) how to achieve variable-rate encoding: a standard autoencoder generates a fixed-length code for each fixed-resolution input patch, resulting in the same cost for lowand high-entropy patches, and requiring the network to be completely retrained to achieve different compression rates; and 3) how to avoid block artifacts: patch-based approaches are prone to block discontinuities. We propose a general framework for variable-rate image compression and a novel architecture based on convolutional and deconvolutional recurrent networks, including LSTMs, that address these issues and report promising results compared to existing baseline codecs. We evaluate the proposed methods on a large-scale benchmark consisting of tiny images (32× 32), which proves to be very challenging for all the methods.

Unsupervised neural networks, such as restricted Boltzmann machines (RBMs) and deep belief networks (DBNs), are powerful tools for feature selection and pattern recognition tasks. We demonstrate that overfitting occurs in such models just as in deep feedforward neural networks, and discuss possible regularization methods to reduce overfitting. We also propose a “partial” approach to improve the efficiency of Dropout/DropConnect in this scenario, and discuss the theoretical justification of these methods from model convergence and likelihood bounds. Finally, we compare the performance of these methods based on their likelihood and classification error rates for various pattern recognition data sets.

Synthetic likelihood is an attractive approach to likelihood-free inference when an approximately Gaussian summary statistic for the data, informative for inference about the parameters, is available. The synthetic likelihood method derives an approximate likelihood function from a plug-in normal density estimate for the summary statistic, with plug-in mean and covariance matrix obtained by Monte Carlo simulation from the model. In this article, we develop alternatives to Markov chain Monte Carlo implementations of Bayesian synthetic likelihoods with reduced computational overheads. Our approach uses stochastic gradient variational inference methods for posterior approximation in the synthetic likelihood context, employing unbiased estimates of the log likelihood. We compare the new method with a related likelihood free variational inference technique in the literature, while at the same time improving the implementation of that approach in a number of ways. These new algorithms are feasible to implement in situations which are challenging for conventional approximate Bayesian computation (ABC) methods, in terms of the dimensionality of the parameter and summary statistic.


Question: Balancing considerations in neural variational inference dissertation.

This paper provides a comprehensive overview of deep learning in neural networks, highlighting the remarkable achievements of deep artificial neural networks, including recurrent networks, in the fields of pattern recognition and machine learning. Deep learners are differentiated from shallow learners by the depth of their credit assignment paths, which represent the causal links between actions and their effects, potentially allowing for learning across multiple levels of abstraction. The survey delves into the evolution of deep supervised learning, recounting the history of backpropagation—a pivotal algorithm in neural network training. Additionally, it explores unsupervised learning techniques, which enable networks to learn without explicit guidance, as well as reinforcement learning and evolutionary computation, which focus on learning through interaction and natural selection principles, respectively. The paper also discusses indirect search strategies for finding compact programs that encode complex and expansive deep networks, an approach that can significantly reduce the computational resources required for training large-scale neural networks. This overview not only encapsulates the advancements in deep learning up to the turn of the millennium but also reflects on the foundational research that has shaped the landscape of modern machine learning.

An unsupervised learning algorithm for a multilayer network of stochastic neurons is described. Bottom-up "recognition" connections convert the input into representations in successive hidden layers, and top-down "generative" connections reconstruct the representation in one layer from the representation in the layer above. In the "wake" phase, neurons are driven by recognition connections, and generative connections are adapted to increase the probability that they would reconstruct the correct activity vector in the layer below. In the "sleep" phase, neurons are driven by generative connections, and recognition connections are adapted to increase the probability that they would produce the correct activity vector in the layer above.

Gaussian processes (GPs) are a good choice for function approximation as they are flexible, robust to overfitting, and provide well-calibrated predictive uncertainty. Deep Gaussian processes (DGPs) are multi-layer generalizations of GPs, but inference in these models has proved challenging. Existing approaches to inference in DGP models assume approximate posteriors that force independence between the layers, and do not work well in practice. We present a doubly stochastic variational inference algorithm that does not force independence between layers. With our method of inference we demonstrate that a DGP model can be used effectively on data ranging in size from hundreds to a billion points. We provide strong empirical evidence that our inference scheme for DGPs works well in practice in both classification and regression.

This tutorial is a worked-out version of a 5-hour course originally held at AIS in September/October 2002. It has two distinct components. First, it contains a mathematically-oriented crash course on traditional training methods for recurrent neural networks, covering back-propagation through time (BPTT), real-time recurrent learning (RTRL), and extended Kalman filtering approaches (EKF). This material is covered in Sections 2 – 5. The remaining sections 1 and 6 – 9 are much more gentle, more detailed, and illustrated with simple examples. They are intended to be useful as a stand-alone tutorial for the echo state network (ESN) approach to recurrent neural network training.

Learning an interpretable factorised representation of the independent data generative factors of the world without supervision is an important precursor for the development of artificial intelligence that is able to learn and reason in the same way that humans do. We introduce β-VAE, a new state-of-the-art framework for automated discovery of interpretable factorised latent representations from raw image data in a completely unsupervised manner. Our approach is a modification of the variational autoencoder (VAE) framework. We introduce an adjustable hyperparameter β that balances latent channel capacity and independence constraints with reconstruction accuracy. We demonstrate that β-VAE with appropriately tuned β > 1 qualitatively outperforms VAE (β = 1), as well as state of the art unsupervised (InfoGAN) and semi-supervised (DC-IGN) approaches to disentangled factor learning on a variety of datasets (celebA, faces and chairs). Furthermore, we devise a protocol to quantitatively compare the degree of disentanglement learnt by different models, and show that our approach also significantly outperforms all baselines quantitatively. Unlike InfoGAN, β-VAE is stable to train, makes few assumptions about the data and relies on tuning a single hyperparameter β, which can be directly optimised through a hyperparameter search using weakly labelled data or through heuristic visual inspection for purely unsupervised data.

We introduce a new regression framework, Gaussian process regression networks (GPRN), which combines the structural properties of Bayesian neural networks with the nonparametric flexibility of Gaussian processes. GPRN accommodates input (predictor) dependent signal and noise correlations between multiple output (response) variables, input dependent length-scales and amplitudes, and heavy-tailed predictive distributions. We derive both elliptical slice sampling and variational Bayes inference procedures for GPRN. We apply GPRN as a multiple output regression and multivariate volatility model, demonstrating substantially improved performance over eight popular multiple output (multi-task) Gaussian process models and three multivariate volatility models on real datasets, including a 1000 dimensional gene expression dataset.

Information bottleneck (IB) is a method for extracting information from one random variable X that is relevant for predicting another random variable Y . To do so, IB identifies an intermediate “bottleneck” variable T that has low mutual information I(X;T ) and high mutual information I(Y ;T ). The IB curve characterizes the set of bottleneck variables that achieve maximal I(Y ;T ) for a given I(X;T ), and is typically explored by optimizing the IB Lagrangian, I(Y ;T )− βI(X;T ). Recently, there has been interest in applying IB to supervised learning, particularly for classification problems that use neural networks. In most classification problems, the output class Y is a deterministic function of the input X , which we refer to as “deterministic supervised learning”. We demonstrate three pathologies that arise when IB is used in any scenario where Y is a deterministic function of X: (1) the IB curve cannot be recovered by optimizing the IB Lagrangian for different values of β; (2) there are “uninteresting” solutions at all points of the IB curve; and (3) for classifiers that achieve low error rates, the activity of different hidden layers will not exhibit a strict trade-off between compression and prediction, contrary to a recent proposal. To address problem (1), we propose a functional that, unlike the IB Lagrangian, can recover the IB curve in all cases. We finish by demonstrating these issues on the MNIST dataset.

We discsus Hinton’s (1989) relative payoff procedure (RPP), a static reinforcement learning algorithm whose foundation is not stochastic gradient ascent. We show circumstances under which applying the RPP is guaranteed to increase the mean return, even though it can make large changes in the values of the parameters. The proof is based on a mapping between the RPP and a form of the expectation-maximisation procedure of Dempster, Laird & Rubin (1976).

In predictive modeling applications, it is often of interest to determine the relative contribution of subsets of features in explaining the variability of an outcome. It is useful to consider this variable importance as a function of the unknown, underlying data-generating mechanism rather than the specific predictive algorithm used to fit the data. In this paper, we connect these ideas in nonparametric variable importance to machine learning, and provide a method for efficient estimation of variable importance when building a predictive model using a neural network. We show how a single augmented neural network with multi-task learning simultaneously estimates the importance of many feature subsets, improving on previous procedures for estimating importance. We demonstrate on simulated data that our method is both accurate and computationally efficient, and apply our method to both a study of heart disease and for predicting mortality in ICU patients.

We introduce dropout compaction, a novel method for training feed-forward neural networks which realizes the performance gains of training a large model with dropout regularization, yet extracts a compact neural network for run-time efficiency. In the proposed method, we introduce a sparsity-inducing prior on the per unit dropout retention probability so that the optimizer can effectively prune hidden units during training. By changing the prior hyperparameters, we can control the size of the resulting network. We performed a systematic comparison of dropout compaction and competing methods on several realworld speech recognition tasks and found that dropout compaction achieved comparable accuracy with fewer than 50% of the hidden units, translating to a 2.5x speedup in run-time.

Recent work has shown how denoising and contractive autoencoders implicitly capture the structure of the data-generating density, in the case where the corruption noise is Gaussian, the reconstruction error is the squared error, and the data is continuous-valued. This has led to various proposals for sampling from this implicitly learned density function, using Langevin and Metropolis-Hastings MCMC. However, it remained unclear how to connect the training procedure of regularized auto-encoders to the implicit estimation of the underlying datagenerating distribution when the data are discrete, or using other forms of corruption process and reconstruction errors. Another issue is the mathematical justification which is only valid in the limit of small corruption noise. We propose here a different attack on the problem, which deals with all these issues: arbitrary (but noisy enough) corruption, arbitrary reconstruction loss (seen as a log-likelihood), handling both discrete and continuous-valued variables, and removing the bias due to non-infinitesimal corruption noise (or non-infinitesimal contractive penalty).


Question: Language-based multi-sensory interfaces for transferring control in self-driving vehicles.

This investigation delves into the impact of verbal explanations on driver comprehension, trust, and safety when operating semi-autonomous vehicles. Through a driving simulation featuring an auto-braking system, we examined how different message contents, accompanying the car's autonomous maneuvers, influence driver behavior and safety. Messages that solely conveyed "how" information, detailing the action ("The car is braking"), were found to degrade driving performance. Conversely, messages that included "why" information, elucidating the rationale behind the action ("Obstacle ahead"), were favored by drivers and correlated with improved performance. The provision of both "how and why" information led to the highest safety standards but also triggered negative emotions in drivers. Our findings indicate that automotive manufacturers must not only focus on the functionality of autonomous actions but also on the effective communication of these actions to drivers, to enhance overall safety.

**Question**: The role of machine learning in enhancing cybersecurity measures

**Documents**: Deep learning for network intrusion detection: A survey
Intrusion detection systems (IDS) are an important part of network security. Recently, deep learning (DL) models have been proposed as a way to improve the accuracy of IDS. This paper surveys the recent literature on DL-based IDS and provides a comprehensive review of the different DL architectures used in IDS, their performance, and their limitations. The survey also discusses the challenges in using DL for IDS and proposes future research directions. The results of the survey show that DL models can significantly improve the accuracy of IDS, but also highlight the need for larger and more diverse datasets, as well as more interpretability and explainability of the DL models.

**Similar Documents**: This comprehensive review explores the integration of deep learning (DL) techniques into network intrusion detection systems (IDS) to bolster cybersecurity. The paper critically examines the recent advancements in DL-based IDS, presenting an in-depth analysis of various DL architectures employed, their effectiveness, and inherent constraints. It highlights the benefits of DL models in enhancing the precision of IDS while acknowledging the challenges they present, such as the requirement for extensive and diverse datasets, and the need for improved interpretability and transparency of DL algorithms. The findings underscore the significant enhancement in IDS accuracy through DL models, yet they also underscore the necessity for ongoing research to address the existing limitations and to push the boundaries of what is achievable in cybersecurity with DL.

This paper presents the concept for a test setup to prototype control algorithms for a multi-finger gripper. The human-robot interface has to provide enough degrees of freedom (DOF) to intuitively control advanced gripper and to accomplish this task a simple sensor glove equipped with flex and force sensors has been prepared for this project. The software architecture has to support both real hardware and simulation, as well as flexible communication standards, therefore, ROS architecture was employed in this matter. Paper presents some preliminary results for using sensor glove and simulated model of the three finger gripper.

A robust methodology for detecting and evaluating driver distraction induced by in-vehicle information system using artificial neural network and fuzzy logic is introduced in this paper. An artificial neural network is used to predict driver's performance on a specific road segment. The predicted performance-based measures are compared to the driving with secondary task accomplishment. Fuzzy logic is applied to fuse the variables into a single output, which constitutes a level of driver distraction in percentage. The technique was tested on a vehicle simulator by ten drivers that exploited in-vehicle information system as a secondary activity. The driver-in-the-loop experiment outcomes are discussed.

Advances in interactive media technologies have reached the stage where users need no longer to act as passive observers. As the technology evolves further we will be able to engage in computer based synthetic performances that mimic events in the real or natural world. Performances in this context refers to plays, films, computer based simulations, engineering mock-ups, etc. At a more fundamental level, we are dealing with complex interactions of the human sensory and perceptual systems with a stimulus environment (covering visual, auditory and other components). Authors or rather directors of these new media environments will exploit computer generated perceptual illusions to convince the person (participant) who is interacting with the system that they are present in the environment. Taken to the limit the participant would not be able to distinguish between being in a real or natural environment compared with being in a purely synthetic environment. Unfortunately, the performance of current technology falls considerably short of even getting even close to providing a faithful perceptual illusion of the real world. Eventually, VR authoring tools will become more sophisticated and it will be possible to produce realistic virtual environments that match more closely with the real world. Some VR practitioners have attempted to collect data that they purport to represent the degree of presence a user experiences. Unfortunately, we do not yet have an agreed definition of presence and seem undecided whether it is a meaningful metric in its own right. This paper will examine the dangers of attempting to measure the 'presence' of a VR system as a one-dimensional parameter. The question of whether presence is a valid measure of a VR system will also be addressed. An important differentiating characteristic of VR systems compared with other human-computer interfaces are their ability to create a sense of 'being-in' the computer generated environment. Other forms of media such as film and TV are also known to induce a sense of 'being-in' the environment. Some VR practitioners have tended to use the term presence to describe this effect This means that people who are engaged in the virtual environment feel as though they are actually part of the virtual environment. In order to be perceptually present in a virtual environment it is first important to understand what it means to be present in the real-world. A good example of real world experience is that of a roller coaster ride. The sensory …

We present the design and evaluation of earPod: an eyes-free menu technique using touch input and reactive auditory feedback. Studies comparing earPod with an iPod-like visual menu technique on reasonably-sized static menus indicate that they are comparable in accuracy. In terms of efficiency (speed), earPod is initially slower, but outperforms the visual technique within 30 minutes of practice. Our results indicate that earPod is potentially a reasonable eyes-free menu technique for general use, and is a particularly exciting technique for use in mobile device interfaces.

Various interaction techniques have been developed in the field of virtual and augmented reality. Whereas techniques for object selection, manipulation, travel, and wayfinding have already been covered in existing taxonomies in some detail, application control techniques have not yet been sufficiently considered. However, they are needed by almost every mixed reality application, e.g. for choosing from alternative objects or options. For this purpose a great variety of distinct three-dimensional (3D) menu selection techniques is available. This paper surveys existing 3D menus from the corpus of literature and classifies them according to various criteria. The taxonomy introduced here assists developers of interactive 3D applications to better evaluate their options when choosing, optimizing, and implementing a 3D menu technique. Since the taxonomy spans the design space for 3D menu solutions, it also aids researchers in identifying opportunities to improve or create novel virtual menu techniques. r 2006 Elsevier Ltd. All rights reserved.

What machine learning (ML) technique is used for system intelligence implementation in ADAS (advanced driving assistance system)? This paper tries to answer this question. This paper analyzes ADAS and ML independently and then relate which ML technique is applicable to what ADAS component and why. The paper gives a good grasp of the current state-of-the-art. Sample works in supervised, unsupervised, deep and reinforcement learnings are presented; their strengths and rooms for improvements are also discussed. This forms part of the basics in understanding autonomous vehicle. This work is a contribution to the ongoing research in ML aimed at reducing road traffic accidents and fatalities, and the invocation of safe driving.

As an envisaged future of transportation, self-driving cars are being discussed from various perspectives, including social, economical, engineering, computer science, design, and ethics. On the one hand, self-driving cars present new engineering problems that are being gradually successfully solved. On the other hand, social and ethical problems are typically being presented in the form of an idealized unsolvable decision-making problem, the so-called trolley problem, which is grossly misleading. We argue that an applied engineering ethical approach for the development of new technology is what is needed; the approach should be applied, meaning that it should focus on the analysis of complex real-world engineering problems. Software plays a crucial role for the control of self-driving cars; therefore, software engineering solutions should seriously handle ethical and social considerations. In this paper we take a closer look at the regulative instruments, standards, design, and implementations of components, systems, and services and we present practical social and ethical challenges that have to be met, as well as novel expectations for software engineering.

125 years after Bertha Benz completed the first overland journey in automotive history, the Mercedes Benz S-Class S 500 INTELLIGENT DRIVE followed the same route from Mannheim to Pforzheim, Germany, in fully autonomous manner. The autonomous vehicle was equipped with close-to-production sensor hardware and relied solely on vision and radar sensors in combination with accurate digital maps to obtain a comprehensive understanding of complex traffic situations. The historic Bertha Benz Memorial Route is particularly challenging for autonomous driving. The course taken by the autonomous vehicle had a length of 103 km and covered rural roads, 23 small villages and major cities (e.g. downtown Mannheim and Heidelberg). The route posed a large variety of difficult traffic scenarios including intersections with and without traffic lights, roundabouts, and narrow passages with oncoming traffic. This paper gives an overview of the autonomous vehicle and presents details on vision and radar-based perception, digital road maps and video-based self-localization, as well as motion planning in complex urban scenarios.

Engineering automatic speech recognition (ASR) for speech to speech (S2S) translation systems, especially targeting languages and domains that do not have readily available spoken language resources, is immensely challenging due to a number of reasons. In addition to contending with the conventional data-hungry speech acoustic and language modeling needs, these designs have to accommodate varying requirements imposed by the domain needs and characteristics, target device and usage modality (such as phrase-based, or spontaneous free form interactions, with or without visual feedback) and huge spoken language variability arising due to socio-linguistic and cultural differences of the users. This paper, using case studies of creating speech translation systems between English and languages such as Pashto and Farsi, describes some of the practical issues and the solutions that were developed for multilingual ASR development. These include novel acoustic and language modeling strategies such as language adaptive recognition, active-learning based language modeling, class-based language models that can better exploit resource poor language data, efficient search strategies, including N-best and confidence generation to aid multiple hypotheses translation, use of dialog information and clever interface choices to facilitate ASR, and audio interface design for meeting both usability and robustness requirements

Clumsy intermediary devices constrain our interaction with computers and their applications. Glove-based input devices let us apply our manual dexterity to the task. We provide a basis for understanding the field by describing key hand-tracking technologies and applications using glove-based input. The bulk of development in glove-based input has taken place very recently, and not all of it is easily accessible in the literature. We present a cross-section of the field to date. Hand-tracking devices may use the following technologies: position tracking, optical tracking, marker systems, silhouette analysis, magnetic tracking or acoustic tracking. Actual glove technologies on the market include: Sayre glove, MIT LED glove, Digital Data Entry Glove, DataGlove, Dexterous HandMaster, Power Glove, CyberGlove and Space Glove. Various applications of glove technologies include projects into the pursuit of natural interfaces, systems for understanding signed languages, teleoperation and robotic control, computer-based puppetry, and musical performance.<<ETX>>


Question: Language-based multimodal presentations for transferring control in autonomous vehicles during handover scenarios.

An International Survey on Attitudes Toward Autonomous Vehicles: Insights from 5000 Respondents
This research explored public sentiment regarding the adoption, apprehensions, and purchasing intentions for partially, highly, and fully automated vehicles through an extensive online questionnaire. Gathering 5000 responses from 109 countries, including detailed analysis from 40 countries with a minimum of 25 respondents, the study aimed to identify cross-national variations and correlations with demographic factors, such as age, gender, and personality traits, measured using a concise version of the Big Five Inventory. Findings revealed that, on average, participants found manual driving to be the most pleasurable form of driving. Responses were varied, with 22% indicating no willingness to pay extra for fully automated driving systems, while 5% expressed a readiness to pay over $30,000, and 33% anticipated high enjoyment from fully automated driving. A majority, 69% of respondents, forecasted that fully automated driving would achieve a 50% market share by 2050. Among the primary concerns raised were software hacking/misuse, legal uncertainties, and safety issues. Notably, individuals with higher neuroticism scores showed slight discomfort with data transmission, whereas those scoring higher on agreeableness were somewhat more at ease with it. Respondents from countries with more advanced development, characterized by lower accident rates, higher education levels, and income, expressed less comfort with their vehicles transmitting data, with cross-national correlations ranging from q = 0.80 to q = 0.90. The study's outcomes highlight key areas of enthusiasm and apprehension among the global populace, offering valuable insights for vehicle manufacturers and other stakeholders in the autonomous vehicle industry.

extraordinarily difficult and poses challenges for both integration and grounding. Here, I propose a unifying idea that meets both challenges: language games. A language game is a sequence of verbal interactions between two agents situated in a specific environment. Language games both integrate the various activities required for dialogue and ground unknown words or phrases in a specific context, which helps constrain possible meanings. Over the past five years, I have been working with a team to develop and test language games on progressively more sophisticated systems, from relatively simple camera-based systems to humanoid robots. The results of our work show that language games are a useful way to both understand and design human–robot interaction.

When animation of a humanoid figure is to be generated at run-time, instead of by replaying precomposed motion clips, some method is required of specifying the avatar’s movements in a form from which the required motion data can be automatically generated. This form must be of a more abstract nature than raw motion data: ideally, it should be independent of the particular avatar’s proportions, and both writable by hand and suitable for automatic generation from higher-level descriptions of the required actions. We describe here the development and implementation of such a scripting language for the particular area of sign languages of the deaf, called SiGML (Signing Gesture Markup Language), based on the existing HamNoSys notation for sign languages. We conclude by suggesting how this work may be extended to more general animation for interactive virtual reality applications.

The recent DARPA-sponsored automated vehicle "Challenges" have generated strong interest in both the research community and the general public, raising consciousness about the possibilities for vehicle automation. Driverless vehicles make good subjects for the visually-oriented media, and they pose enough interesting research challenges to occupy generations of graduate students. However, automated vehicles also have the potential to help solve a variety of real-world problems. Engineers need to think carefully about which of those problems we are actually solving.A well-engineered system should be designed to satisfy specific needs, and those needs should be reflected in the definition of system requirements. Alternative technological approaches can then be evaluated and traded off based on their ability to meet the requirements. The article describes the rather different needs of the public road transportation system and the military transportation system, and then shows how those needs influence the requirements for automated vehicle systems. These requirements point toward significantly different technical approaches, but it is possible to find some limited areas of technical commonality.

Current outdoor localization techniques fail to provide the required accuracy for estimating the car's lane. In this paper, we present LaneQuest: a system that leverages the ubiquitous and low-energy inertial sensors available in commodity smart-phones to provide an accurate estimate of the car's current lane. LaneQuest leverages hints from the phone sensors about the surrounding environment to detect the car's lane. For example, a car making a right turn most probably will be in the right-most lane, a car passing by a pothole will be in a specific lane, and the car's angular velocity when driving through a curve reflects its lane. Our investigation shows that there are amble opportunities in the environment, i.e. lane “anchors”, that provide cues about the car's lane. To handle the ambiguous location, sensors noise, and fuzzy lane anchors; LaneQuest employs a novel probabilistic lane estimation algorithm. Furthermore, it uses an unsupervised crowd-sourcing approach to learn the position and lane-span distribution of the different lane-level anchors. Our evaluation results from implementation on different android devices and 260Km driving traces by 13 drivers in different cities shows that LaneQuest can detect the different lane-level anchors with an average precision and recall of more than 90%. This leads to an accurate detection of the exact car's lane position 80% of the time, increasing to 89% of the time to within one lane. This comes with a low-energy footprint, allowing LaneQuest to be implemented on the energy-constrained mobile devices.

Modern vehicles are evolving with more electronic components than ever before (In this paper, “vehicle” means “automotive vehicle.” It is also equal to “car.”) One notable example is graphical processing unit (GPU), which is a key component to implement a digital cluster. To implement the digital cluster that displays all the meters (such as speed and fuel gauge) together with infotainment services (such as navigator and browser), the GPU needs to be virtualized; however, GPU virtualization for the digital cluster has not been addressed yet. This paper presents a Virtualized Automotive DIsplay (VADI) system to virtualize a GPU and its attached display device. VADI manages two execution domains: one for the automotive control software and the other for the in-vehicle infotainment (IVI) software. Through GPU virtualization, VADI provides GPU rendering to both execution domains, and it simultaneously displays their images on a digital cluster. In addition, VADI isolates GPU from the IVI software in order to protect it from potential failures of the IVI software. We implement VADI with Vivante GC2000 GPU and perform experiments to ensure requirements of International Standard Organization (ISO) safety standards. The results show that VADI guarantees 30 frames per second (fps), which is the minimum frame rate for digital cluster mandated by ISO safety standards even with the failure of the IVI software. It also achieves 60 fps in a synthetic workload.

OBJECTIVE
This study examined the effectiveness of rear-end collision warnings presented in different sensory modalities as a function of warning timing in a driving simulator.

BACKGROUND
The proliferation of in-vehicle information and entertainment systems threatens driver attention and may increase the risk of rear-end collisions. Collision warning systems have been shown to improve inattentive and/or distracted driver response time (RT) in rear-end collision situations. However, most previous rear-end collision warning research has not directly compared auditory, visual, and tactile warnings.

METHOD
Sixteen participants in a fixed-base driving simulator experienced four warning conditions: no warning, visual, auditory, and tactile. The warnings activated when the time-to-collision (TTC) reached a critical threshold of 3.0 or 5.0 s. Driver RT was captured from a warning below critical threshold to brake initiation.

RESULTS
Drivers with a tactile warning had the shortest mean RT. Drivers with a tactile warning had significantly shorter RT than drivers without a warning and had a significant advantage over drivers with visual warnings.

CONCLUSION
Tactile warnings show promise as effective rear-end collision warnings.

APPLICATION
The results of this study can be applied to the future design and evaluation of automotive warnings designed to reduce rear-end collisions.

Driverless cars and driver-assisted vehicles are becoming prevalent in today's transportation systems. Companies like TESLA, Audi, and Google are among the leaders in this new era of the automotive industry. Modern vehicles, equipped with sophisticated navigation devices, complex driver-assisted systems, and a whole lot of safety features, bring broader impacts to our quality of life, economic development, and environmental sustainability. Instant safety messages such as pre-collision warnings, blind-spot detection, pedestrian and object awareness significantly improve the safety for drivers, passengers, and pedestrians. As a result, vehicles would be able to travel closely yet safely together, forming a platoon, thus resulting in a reduction of traffic congestion and fuel consumption. Driverless cars also have non-safety-related applications: which are used to facilitate traffic management and infotainment dissemination for drivers and passengers. Vehicular Ad hoc Network (VANET), the wireless communication technology enabling driverless cars, features not only dynamic topology but also high mobility. The vehicle nodes move rapidly on streets and highways. Their movements also depend on but not limited to road traffic, speed limits, and behavior of nearby vehicles. With massive amount of messages exchanged among driverless cars that command the cars' movements at high speeds and in close distances, any malicious alternation could lead to disastrous accidents. Message authentication to ensure data integrity is paramount to attack preparation. This paper presents a novel message authentication scheme that protects cars from bogus messages and makes VANET resilient to Denial-of-Service (DoS) attacks. The work includes a simulation framework that integrates vehicle and data traffic models to validate the effectiveness of the proposed message authentication scheme.

What machine learning (ML) technique is used for system intelligence implementation in ADAS (advanced driving assistance system)? This paper tries to answer this question. This paper analyzes ADAS and ML independently and then relate which ML technique is applicable to what ADAS component and why. The paper gives a good grasp of the current state-of-the-art. Sample works in supervised, unsupervised, deep and reinforcement learnings are presented; their strengths and rooms for improvements are also discussed. This forms part of the basics in understanding autonomous vehicle. This work is a contribution to the ongoing research in ML aimed at reducing road traffic accidents and fatalities, and the invocation of safe driving.

This paper describes a hybrid model and a model predictive control (MPC) strategy for solving a traction control problem. The problem is tackled in a systematic way from modeling to control synthesis and implementation. The model is described first in the Hybrid Systems Description Language to obtain a mixed-logical dynamical (MLD) hybrid model of the open-loop system. For the resulting MLD model, we design a receding horizon finite-time optimal controller. The resulting optimal controller is converted to its equivalent piecewise affine form by employing multiparametric programming techniques, and finally experimentally tested on a car prototype. Experiments show that good and robust performance is achieved in a limited development time by avoiding the design of ad hoc supervisory and logical constructs usually required by controllers developed according to standard techniques.

This paper presents the results of the wearable computing prototype supporting trainingand qualification activities at the SKODA production facilities in Czech Republic. The emerged prototype is based upon the first of the 2 main “Variant Production Showcases” (training and assembly-line) which are to be implemented in the WearIT@work project (EC IP 004216). As an introduction, the authors of this paper investigate current training processes at Skoda, and derive the potential benefits and risks of applying wearable computing technology. Accordingly, the approach of creating the wearable prototypes, via usability experiments at the Skoda production site, is explained in detail. As a preliminary result, the first functional prototypes, including a task recognition prototype, based upon the components of the European Wearable Computing Platform, are described. The paper is rounded up by providing a short outlook regarding the second envisaged test case, which is focussed upon selected assembly line operations of blue collar workers.


Question: Language-based multimodal presentations for transferring control in autonomous vehicles during handover procedures.

This paper critically examines the impact of adaptive cruise control (ACC) and highly automated driving (HAD) systems on driver workload and situation awareness, synthesizing empirical evidence from recent studies. Researchers from the Department of BioMechanical Engineering at Delft University of Technology, the Centre for Transport Studies at the University of Twente, TNO Human Factors, and the Transportation Research Group at the University of Southampton have collaborated to review the available data. The investigation reveals that ACC and HAD can significantly influence both workload and situation awareness, with mixed outcomes depending on the driving scenario and the level of automation. While ACC tends to reduce workload, it may also lead to decreased situation awareness, as drivers become overly reliant on the technology. In contrast, HAD systems, which take over more driving tasks, can maintain or even enhance situation awareness by providing clear and timely information to the driver. The review underscores the importance of designing HAD systems that not only manage driving tasks but also actively engage the driver, thereby preserving their situational awareness and readiness to intervene when necessary. This research highlights the need for further investigation into the optimal interaction between drivers and automated driving systems to ensure safety and efficiency on the road.

Recent progress in autonomous and semiautonomous driving has been made possible in part through an assortment of sensors that provide the intelligent agent with an enhanced perception of its surroundings. It has been clear for quite some while now that for intelligent vehicles to function effectively in all situations and conditions, a fusion of different sensor technologies is essential. Consequently, the availability of synchronized multi-sensory data streams are necessary to promote the development of fusion based algorithms for low, mid and high level semantic tasks. In this paper, we provide a comprehensive description of our heavily sensorized, panoramic testbed capable of providing high quality data from a slew of synchronized and calibrated sensors such as cameras, LIDARs, radars, and the IMU/GPS. The vehicle has recorded over 100 hours of real world data for a very diverse set of weather, traffic and daylight conditions. All captured data is accurately calibrated and synchronized using timestamps, and stored safely in high performance servers mounted inside the vehicle itself. Details on the testbed instrumentation, sensor layout, sensor outputs, calibration and synchronization are described in this paper.

Software testing is all too often simply a bug hunt rather than a wellconsidered exercise in ensuring quality. A more methodical approach than a simple cycle of system-level test-fail-patch-test will be required to deploy safe autonomous vehicles at scale. The ISO 26262 development V process sets up a framework that ties each type of testing to a corresponding design or requirement document, but presents challenges when adapted to deal with the sorts of novel testing problems that face autonomous vehicles. This paper identifies five major challenge areas in testing according to the V model for autonomous vehicles: driver out of the loop, complex requirements, non-deterministic algorithms, inductive learning algorithms, and failoperational systems. General solution approaches that seem promising across these different challenge areas include: phased deployment using successively relaxed operational scenarios, use of a monitor/actuator pair architecture to separate the most complex autonomy functions from simpler safety functions, and fault injection as a way to perform more efficient edge case testing. While significant challenges remain in safety-certifying the type of algorithms that provide high-level autonomy themselves, it seems within reach to instead architect the system and its accompanying design process to be able to employ existing software safety approaches.

This paper presents vehicle control system for automatic valet parking with infrastructure sensors. First, we describe the automatic valet parking service system. In the service, vehicle moves autonomously based on sensing data generated by infrastructure sensors. Second, we implement vehicle control system for automatic valet parking. We design hardware and software components focusing on minimizing the add-on devices and maximizing data integration. Finally, we do a feasibility test for verification of the system.

We present the SimSensei system, a fully automatic virtual agent that conducts interviews to assess indicators of psychological distress. We emphasize on the perception part of the system, a multimodal framework which captures and analyzes user state for both behavioral understanding and interactional purposes.

A robot perceives space in order to navigate, map, search and investigate its surroundings. One application area where humans interact with robots is search and rescue. The robot may have unique capabilities such as seeing outside the visible spectrum and being able to navigate in tight and hazardous spaces. In such an operation, the human may also be an effective search agent. Thus, it would be beneficial if the robot’s spatial perception was conveyed to the human via a secondary peripheral modality. We have worked on developing a visual to tactile substitution.device for people who are visually impaired or blind. We propose to use this similar technique to the remote robot in order that it convey its spatial perception to the human operator or co-worker.

Advanced Driver Assistance Systems (ADAS) have made driving safer over the last decade. They prepare vehicles for unsafe road conditions and alert drivers if they perform a dangerous maneuver. However, many accidents are unavoidable because by the time drivers are alerted, it is already too late. Anticipating maneuvers beforehand can alert drivers before they perform the maneuver and also give ADAS more time to avoid or prepare for the danger. In this work we anticipate driving maneuvers a few seconds before they occur. For this purpose we equip a car with cameras and a computing device to capture the driving context from both inside and outside of the car. We propose an Autoregressive Input-Output HMM to model the contextual information alongwith the maneuvers. We evaluate our approach on a diverse data set with 1180 miles of natural freeway and city driving and show that we can anticipate maneuvers 3.5 seconds before they occur with over 80% F1-score in real-time.

The automation of the overtaking maneuver is considered to be one of the toughest challenges in the development of autonomous vehicles. This operation involves two vehicles (the overtaking and the overtaken) cooperatively driving, as well as the surveillance of any other vehicles that are involved in the maneuver. This operation consists of two lane changes-one from the right to the left lane of the road, and the other is to return to the right lane after passing. Lane-change maneuvers have been used to move into or out of a circulation lane or platoon; however, overtaking operations have not received much coverage in the literature. In this paper, we present an overtaking system for autonomous vehicles equipped with path-tracking and lane-change capabilities. The system uses fuzzy controllers that mimic human behavior and reactions during overtaking maneuvers. The system is based on the information that is supplied by a high-precision Global Positioning System and a wireless network environment. It is able to drive an automated vehicle and overtake a second vehicle that is driving in the same lane of the road.

Spoken dialogue systems have received increased interest because they are potentially much more natural and powerful methods of communicating with machines than are current graphics-based interfaces. Wired for Speech presents basic research in the psychological and sociological aspects of voice synthesis and recognition. Its major lesson is that people attribute human characteristics to spoken dialogue systems for reasons related to human evolution. But although it contains interesting basic research, the book is mainly aimed at giving technological or marketing advice to those seeking to use voice interfaces when creating commercial applications. The book is oriented around a series of simple experiments designed to show just how pervasive psychological and social influences can be on the opinions and behaviors of people confronted with voice interfaces. Each chapter describes a basic research hypothesis, introduces an experiment to test it, and discusses its implications for designing voice interfaces: gender, personality, accent, ethnicity, emotion, number of distinct voices, use of " I " by the system, voices in concert with faces, mixed synthetic and recorded voices, context, and the effects of errors in human–computer cooperation. Although Wired for Speech is very accessible , especially to the non-scientist, it is written with an unusual bibliography style for an academic book: All references and details are given in a notes section at the end of the book, making up one third of the content. This narrative exposition style will probably not satisfy either type of reader: Scientists will be frustrated at the imprecision in argumen-tation, lack of detail in the book itself, and continually having to refer to the notes. The lack of detail also prevents the book from serving as a reference work. Meanwhile those needing advice when implementing voice interfaces will be puzzled at references to Chomsky, Grice, and Zemlin. Thus the book seems to suffer from not knowing its audience well, which is odd as this is precisely the lesson that the book tries to impart. Another complaint from the scientific point of view is the obsession this particular book has with casting every experimental result as advice for the business and marketing side of voice interfaces, typically concentrating on Web-based e-marketing examples such as buying books on line. Most of the experiments have sample sizes of between 40 and 50, but the authors seem ready to invite multi-billion-dollar businesses to immediately base their deployed systems on these results. Finally, and fundamentally, this …

Introduction of automated driving increases complexity of automotive systems. As a result, architecture design becomes a major concern for ensuring non-functional requirements such as safety, and modifiability. In the ISO 26262 standard, architecture patterns are recommended for system development. However, the existing architecture patterns may not be able to answer requirements of automated driving completely. When applying these patterns in the automated driving context, modification and analysis of these patterns are needed. In this paper, we present a novel architecture pattern for safety critical automated driving functions. In addition, we propose a generic approach to compare our pattern with a number of existing ones. The comparison results can be used as a basis for project specific architectural decisions. Our Safety Channel pattern is validated by its implementation for a real-life truck platooning application.

This paper presents a set of pinch glove-based user interface tools for an outdoor wearable augmented reality computer system. The main form of user interaction is the use of hand and head gestures. We have developed a set of augmented reality information presentation techniques. To support direct manipulation, the following three selection techniques have been implemented: two-handed framing, line of sight and laser beam. A new glove-based text entry mechanism has been developed to support symbolic manipulation. A scenario for a military logistics task is described to illustrate the functionality of this form of interaction.