Question: Deep versus shallow architecture in multilayer neural networks.

Multilayer feedforward networks exhibit universal approximation capabilities

Multilayer feedforward networks, characterized by their layered structure and absence of feedback connections, have been widely studied for their powerful function approximation properties. These networks, with a sufficient number of hidden layers and units, have been proven to be universal approximators. This means they can approximate any continuous function to any desired degree of accuracy, given a sufficient number of parameters. The universal approximation theorem, a cornerstone of neural network theory, demonstrates that even a single hidden layer feedforward network can represent a vast range of functions. However, the depth and complexity of the network can significantly impact its performance and learning efficiency. Deeper networks, with multiple hidden layers, have the potential to learn more complex representations and solve intricate problems that are not easily handled by shallow architectures. The architecture of multilayer feedforward networks, therefore, plays a crucial role in their ability to model and generalize from data, making them a versatile tool in various machine learning applications.

Classifying neural signals is a crucial step in the brain-computer interface (BCI). Although Deep Neural Network (DNN) has been shown to be surprisingly good at classification, DNN suffers from long training time and catastrophic forgetting. Catastrophic forgetting refers to a phenomenon in which a DNN tends to forget previously learned task when it learns a new task. Here we argue that the solution to this problem may be found in the human brain, specifically, by combining functions of the two regions: the striatum and the hippocampus, which is pivotal for reinforcement learning and memory recall relevant to the current context, respectively. The mechanism of these brain regions provides insights into resolving catastrophic forgetting and long training time of DNNs. Referring to the hippocampus-striatum network we discuss design principles of combining different types of DNNs for building a new BCI architecture, called “Meta BCI”.

-We give conditions ensuring that multilayer jeedJorward networks with as Jew as a single hidden layer and an appropriately smooth hidden layer activation fimction are capable o f arbitrarily accurate approximation to an arbitrao' function and its derivatives. In fact, these networks can approximate functions that are not dtifferentiable in the classical sense, but possess only a generalized derivative, as is the case jor certain piecewise dtlf[erentiable Junctions. The conditions imposed on the hidden layer activation function are relatively mild: the conditions imposed on the domain o f the fimction to be approximated have practical intplications. Our approximation results provide a previously missing theoretical justification jor the use of multilayer /~'ed/brward networks in applications requiring simultaneous approximation o f a function and its derivatives. Keywords--Approximation, Derivatives, Sobolev space, Feedforward networks. 1. I N T R O D U C T I O N The capability of sufficiently complex multilayer feedforward networks to approximate an unknown mapping./~. R' ----~ R arbitrarily well has been recently investigated by Cybenko (1989), Funahashi (1989), Hecht-Nielsen (1989), Hornik, Stinchcombe, and White (1989) (HSW) (all for sigmoid hidden layer activation functions) and Stinchcombe and White (1989) (SW) (non-sigmoid hidden layer activation functions). In applications, it may be desirable to approximate not only the unknown mapping, but also its unknown derivatives. This is the case in Jordan's (1989) recent investigation of robot learning of smooth movement . Jordan states: The Jacobian matrix 8z/Ox . . . is the matrix that relates small changes in the controller output to small changes in the task space results and cannot be assumed to be available a priori, or provided by the environment. However, all of the derivatives in the matrix are forward derivatives. They are easily obtained by differentiation if a forward model is available. The forward model itself must be learned, but this can be achieved directly by system idenAcknowledgements: We arc indebted to Angelo Melino for pressing us on the issue addressed here and to the referees for numerous helpful suggestions. White's participation was supported by NSF Grant SES-8806990. Requests for reprints should be sent to Halbert White, Department of Economics, D-008, University of California, San Diego. La Jolla, CA 92093. tification. Once the model is accurate over a particular domain, its derivatives provide a learning operator that allows the system to convert errors in task space into errors in articulatory space and thereby change the controller. Thus, learning an adequate approximation to the Jacobian matrix of an unknown mapping is a key component of Jordan 's approach to robot learning of smooth movement . Despite the success of Jordan 's experiments, there is no existing theoretical guarantee that multilayer feedforward networks generally have the capability to approximate an unknown mapping and its derivatives simultaneously. For example, a network with hard limiting hidden layer activations approximates unknown mappings with a piecewise-constant function, the first derivatives of which exist and are zero almost everywhere. Obviously, the derivatives of such a network output function cannot approximate the derivatives of an arbitrary function. Intuition suggests that networks having smooth hidden layer activation functions ought to have output function derivatives that will approximate the derivatives of an unknown mapping. However , the justification for this intuition is not obvious. Consider the class of single hidden layer feedforward networks having network output functions belonging to the set 2. "(G) -~ {g : x"-~ R[ g(x) : /:,G(_i';,,):

A neural network architecture for the learning of recognition categories is derived. Real-time network dynamics are completely characterized through mathematical analysis and computer simulations. The architecture self-organizes and self-stabilizes its recognition codes in response to arbitrary orderings of arbitrarily many and arbitrarily complex binary input patterns. Top-down attentional and matching mechanisms are critical in self-stabilizing the code learning process. The architecture embodies a parallel search scheme which updates itself adaptively as the learning process unfolds. After learning self-stabilizes, the search process is automatically disengaged. Thereafter input patterns directly access their recognition codes without any search. Thus recognition time does not grow as a function of code complexity. A novel input pattern can directly access a category if it shares invariant properties with the set of familiar exemplars of that category. These invariant properties emerge in the form of learned critical feature patterns, or prototypes. The architecture possesses a context-sensitive self-scaling property which enables its emergent critical feature patterns to form. They detect and remember statistically predictive configurations of featural elements which are derived ~ ..from the set of all input patterns that are ever experienced. Four types of attentional

Recently, there have been increasing demands to construct compact deep architectures to remove unnecessary redundancy and to improve the inference speed. While many recent works focus on reducing the redundancy by eliminating unneeded weight parameters, it is not possible to apply a single deep network for multiple devices with different resources. When a new device or circumstantial condition requires a new deep architecture, it is necessary to construct and train a new network from scratch. In this work, we propose a novel deep learning framework, called a nested sparse network, which exploits an n-in-1-type nested structure in a neural network. A nested sparse network consists of multiple levels of networks with a different sparsity ratio associated with each level, and higher level networks share parameters with lower level networks to enable stable nested learning. The proposed framework realizes a resource-aware versatile architecture as the same network can meet diverse resource requirements, i.e., anytime property. Moreover, the proposed nested network can learn different forms of knowledge in its internal networks at different levels, enabling multiple tasks using a single network, such as coarse-to-fine hierarchical classification. In order to train the proposed nested network, we propose efficient weight connection learning and channel and layer scheduling strategies. We evaluate our network in multiple tasks, including adaptive deep compression, knowledge distillation, and learning class hierarchy, and demonstrate that nested sparse networks perform competitively, but more efficiently, compared to existing methods.

Deep neural networks (DNNs) are widely used in machine translation (MT). This article gives an overview of DNN applications in various aspects of MT.

In this work, we establish the relation between optimal control and training deep Convolution Neural Networks (CNNs). We show that the forward propagation in CNNs can be interpreted as a time-dependent nonlinear differential equation and learning can be seen as controlling the parameters of the differential equation such that the network approximates the data-label relation for given training data. Using this continuous interpretation, we derive two new methods to scale CNNs with respect to two different dimensions. The first class of multiscale methods connects low-resolution and high-resolution data using prolongation and restriction of CNN parameters inspired by algebraic multigrid techniques. We demonstrate that our method enables classifying highresolution images using CNNs trained with low-resolution images and vice versa and warm-starting the learning process. The second class of multiscale methods connects shallow and deep networks and leads to new training strategies that gradually increase the depths of the CNN while re-using parameters for initializations.

We present the soft exponential activation function for artificial neural networks that continuously interpolates between logarithmic, linear, and exponential functions. This activation function is simple, differentiable, and parameterized so that it can be trained as the rest of the network is trained. We hypothesize that soft exponential has the potential to improve neural network learning, as it can exactly calculate many natural operations that typical neural networks can only approximate, including addition, multiplication, inner product, distance, and sinusoids.

Learning to store information over extended time intervals by recurrent backpropagation takes a very long time, mostly because of insufficient, decaying error backflow. We briefly review Hochreiter's (1991) analysis of this problem, then address it by introducing a novel, efficient, gradient based method called long short-term memory (LSTM). Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units. Multiplicative gate units learn to open and close access to the constant error flow. LSTM is local in space and time; its computational complexity per time step and weight is O. 1. Our experiments with artificial data involve local, distributed, real-valued, and noisy pattern representations. In comparisons with real-time recurrent learning, back propagation through time, recurrent cascade correlation, Elman nets, and neural sequence chunking, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex, artificial long-time-lag tasks that have never been solved by previous recurrent network algorithms.

We extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes. The combined system is analogous to a Turing Machine or Von Neumann architecture but is differentiable end-toend, allowing it to be efficiently trained with gradient descent. Preliminary results demonstrate that Neural Turing Machines can infer simple algorithms such as copying, sorting, and associative recall from input and output examples.

Deep and recurrent neural networks (DNNs and RNNs respectively) are powerful models that were considered to be almost impossible to train using stochastic gradient descent with momentum. In this paper, we show that when stochastic gradient descent with momentum uses a well-designed random initialization and a particular type of slowly increasing schedule for the momentum parameter, it can train both DNNs and RNNs (on datasets with long-term dependencies) to levels of performance that were previously achievable only with Hessian-Free optimization. We find that both the initialization and the momentum are crucial since poorly initialized networks cannot be trained with momentum and well-initialized networks perform markedly worse when the momentum is absent or poorly tuned. Our success training these models suggests that previous attempts to train deep and recurrent neural networks from random initializations have likely failed due to poor initialization schemes. Furthermore, carefully tuned momentum methods su ce for dealing with the curvature issues in deep and recurrent network training objectives without the need for sophisticated second-order methods.


Question: Deep versus shallow architecture in multilayer neural networks.

A neural network architecture capable of achieving arbitrary precision in function approximation is demonstrated
The authors illustrate that a specific configuration of a feedforward neural network, consisting of multiple inputs, a single output, and a single hidden layer, can achieve exceptional approximation capabilities. This network, with its fixed and predetermined connections from the input to the hidden layer, utilizes a monotonic activation function at the hidden layer nodes and a linear output. Interestingly, this network can be viewed as a variant of a Fourier network, inheriting the approximation properties of Fourier series. This implies that, with a sufficient number of hidden units, the network can approximate any square integrable function to any desired level of accuracy, effectively eliminating any avoidable errors in its predictions. This finding underscores the potential of such networks in achieving high-precision function approximation tasks.

-A neural network learning procedure has been applied to the classification ~/sonar returns [kom two undersea targets, a metal cylinder and a similarly shaped rock. Networks with an intermediate layer ~/ hidden processing units achieved a classification accuracy as high as 100% on a training set of l04 returns. These net~orks correctly classified up to 90.4% of 104 test returns not contained in the training set. This perfi~rmance was better than that of a nearest neighbor classifier, which was 82.7%. and was close to that of an optimal Bayes classifie~ Specific signal features extracted by hidden units in a trained network were identified and related to coding schemes in the pattern of connection strengths between the input and the hidden units. Network perlbrmance and class[/~cation strategy was comparable to that of trained human listeners. Keywords--Learning algorithms, Hidden units. Multilayered neural network, Sonar, Signal processing.

We present an application of back-propagation networks to handwritten digit recognition. Minimal preprocessing of the data was required, but architecture of the network was highly constrained and specifically designed for the task. The input of the network consists of normalized images of isolated digits. The method has 1 % error rate and about a 9% reject rate on zipcode digits provided by the U.S. Postal Service.

Deep Neural nets (NNs) with millions of parameters are at the heart of many stateof-the-art computer vision systems today. However, recent works have shown that much smaller models can achieve similar levels of performance. In this work, we address the problem of pruning parameters in a trained NN model. Instead of removing individual weights one at a time as done in previous works, we remove one neuron at a time. We show how similar neurons are redundant, and propose a systematic way to remove them. Our experiments in pruning the densely connected layers show that we can remove upto 85% of the total parameters in an MNIST-trained network, and about 35% for AlexNet without significantly affecting performance. Our method can be applied on top of most networks with a fully connected layer to give a smaller network.

Deep neural networks (DNNs) have achieved unprecedented success in a wide array of tasks. However, the performance of these systems depends directly on their hyper-parameters which often must be selected by an expert. Optimizing the hyper-parameters remains a substantial obstacle in designing DNNs in practice. In this work, we propose to select them using particle swarm optimization (PSO). Such biologically-inspired approaches have not been extensively exploited for this task. We demonstrate that PSO efficiently explores the solution space, allowing DNNs of a minimal topology to obtain competitive classification performance over the MNIST dataset. We showed that very small DNNs optimized by PSO retrieve promising classification accuracy for CIFAR-10. Also, PSO improves the performance of existing architectures. Extensive experimental study, backed-up with the statistical tests, revealed that PSO is an effective technique for automating hyper-parameter selection and efficiently exploits computational resources.

In the deep neural network (DNN), the hidden layers can be considered as increasingly complex feature transformations and the final softmax layer as a log-linear classifier making use of the most abstract features computed in the hidden layers. While the loglinear classifier should be different for different languages, the feature transformations can be shared across languages. In this paper we propose a shared-hidden-layer multilingual DNN (SHL-MDNN), in which the hidden layers are made common across many languages while the softmax layers are made language dependent. We demonstrate that the SHL-MDNN can reduce errors by 3-5%, relatively, for all the languages decodable with the SHL-MDNN, over the monolingual DNNs trained using only the language specific data. Further, we show that the learned hidden layers sharing across languages can be transferred to improve recognition accuracy of new languages, with relative error reductions ranging from 6% to 28% against DNNs trained without exploiting the transferred hidden layers. It is particularly interesting that the error reduction can be achieved for the target language that is in different families of the languages used to learn the hidden layers.

The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains. In this work, we propose a new activation function, named Swish, which is simply f(x) = x · sigmoid(x). Our experiments show that Swish tends to work better than ReLU on deeper models across a number of challenging datasets. For example, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9% for Mobile NASNetA and 0.6% for Inception-ResNet-v2. The simplicity of Swish and its similarity to ReLU make it easy for practitioners to replace ReLUs with Swish units in any neural network.

Deep neural networks (DNNs) are powerful types of artificial neural networks (ANNs) that use several hidden layers. They have recently gained considerable attention in the speech transcription and image recognition community (Krizhevsky et al., 2012) for their superior predictive properties including robustness to overfitting. However their application to financial market prediction has not been previously researched, partly because of their computational complexity. This paper describes the application of DNNs to predicting financial market movement directions. A critical step in the viability of the approach in practice is the ability to effectively deploy the algorithm on general purpose high performance computing infrastructure. Using an Intel Xeon Phi co-processor with 61 cores, we describe the process for efficient implementation of the batched stochastic gradient descent algorithm and demonstrate a 11.4x speedup on the Intel Xeon Phi over a serial implementation on the Intel Xeon.

Three Bayesian ideas are presented for supervised adaptive classifiers. First, it is argued that the output of a classifier should be obtained by marginalizing over the posterior distribution of the parameters; a simple approximation to this integral is proposed and demonstrated. This involves a "moderation" of the most probable classifier's outputs, and yields improved performance. Second, it is demonstrated that the Bayesian framework for model comparison described for regression models in MacKay (1992a,b) can also be applied to classification problems. This framework successfully chooses the magnitude of weight decay terms, and ranks solutions found using different numbers of hidden units. Third, an information-based data selection criterion is derived and demonstrated within this framework.

The game of Go is more challenging than other board games, due to the difficulty of constructing a position or move evaluation function. In this paper we investigate whether deep convolutional networks can be used to directly represent and learn this knowledge. We train a large 12-layer convolutional neural network by supervised learning from a database of human professional games. The network correctly predicts the expert move in 55% of positions, equalling the accuracy of a 6 dan human player. When the trained convolutional network was used directly to play games of Go, without any search, it beat the traditional-search program GnuGo in 97% of games, and matched the performance of a state-of-the-art Monte-Carlo tree search that simulates two million positions per move.

Card games are interesting for many reasons besides their connection with gambling. Bridge is being a game of imperfect information, it is a well defined, decision making game. The estimation of the number of tricks to be taken by one pair of bridge players is called Double Dummy Bridge Problem (DDBP). Artificial Neural Networks are Non – Linear mapping structures based on the function of the human brain. Feed Forward Neural Network is used to solve the DDBP in contract bridge. The learning methodology, supervised learning was used in Back – Propagation Network (BPN) for training and testing the bridge sample deal. In our study we compared back – Propagation algorithm and obtained that Resilient Back – Propagation algorithms by using Hyperbolic Tangent function and Resilient Back – Propagation algorithm produced better result than the other. Among various neural network architectures, in this study we used four network architectures viz., 26x4, 52, 104 and 52x4 for solving DDBP in contract bridge.


Question: Deep versus shallow architecture in multilayer neural networks.

This paper presents a neural network architecture designed for learning and recognizing complex patterns in real-time. The architecture, characterized through rigorous mathematical analysis and simulations, is capable of self-organizing and self-stabilizing its recognition codes in response to a wide range of binary input patterns, regardless of their complexity or the order in which they are presented. Attentional and matching mechanisms are crucial in the stabilization of the code learning process, ensuring the network's robustness and adaptability. The architecture features a parallel search mechanism that dynamically updates itself as learning progresses, automatically disengaging once self-stabilization is achieved. This results in direct access to recognition codes without the need for search, keeping recognition time independent of code complexity. Novel input patterns can directly access a category if they share invariant properties with familiar exemplars, which are represented by learned critical feature patterns or prototypes. These prototypes emerge as the network learns to detect and remember statistically predictive configurations of features derived from all experienced input patterns. The architecture's context-sensitive self-scaling property facilitates the formation of these critical feature patterns, enhancing its ability to handle an expanding set of input patterns. Four distinct attentional mechanisms are integral to this process, each contributing to the architecture's overall performance and adaptability.

While natural languages are compositional, how state-of-the-art neural models achieve compositionality is still unclear. We propose a deep network, which not only achieves competitive accuracy for text classification, but also exhibits compositional behavior. That is, while creating hierarchical representations of a piece of text, such as a sentence, the lower layers of the network distribute their layer-specific attention weights to individual words. In contrast, the higher layers compose meaningful phrases and clauses, whose lengths increase as the networks get deeper until fully composing the sentence.

We introduce DropConnect, a generalization of Dropout (Hinton et al., 2012), for regularizing large fully-connected layers within neural networks. When training with Dropout, a randomly selected subset of activations are set to zero within each layer. DropConnect instead sets a randomly selected subset of weights within the network to zero. Each unit thus receives input from a random subset of units in the previous layer. We derive a bound on the generalization performance of both Dropout and DropConnect. We then evaluate DropConnect on a range of datasets, comparing to Dropout, and show state-of-the-art results on several image recognition benchmarks by aggregating multiple DropConnect-trained models.

This chapter critically reviews some of the important methods being used for building quantitative structure-activity relationship (QSAR) models using the artificial neural networks (ANNs). It attends predominantly to the use of multilayer ANNs in the regression analysis of structure-activity data. The highlighted topics cover the approximating ability of ANNs, the interpretability of the resulting models, the issues of generalization and memorization, the problems of overfitting and overtraining, the learning dynamics, regularization, and the use of neural network ensembles. The next part of the chapter focuses attention on the use of descriptors. It reviews different descriptor selection and preprocessing techniques; considers the use of the substituent, substructural, and superstructural descriptors in building common QSAR models; the use of molecular field descriptors in three-dimensional QSAR studies; along with the prospects of "direct" graph-based QSAR analysis. The chapter starts with a short historical survey of the main milestones in this area.

We address the question of when a network can be expected to generalize from m random training examples chosen from some arbitrary probability distribution, assuming that future test examples are drawn from the same distribution. Among our results are the following bounds on appropriate sample vs. network size. Assume 0 < ∊ 1/8. We show that if m O(W/∊ log N/∊) random examples can be loaded on a feedforward network of linear threshold functions with N nodes and W weights, so that at least a fraction 1 ∊/2 of the examples are correctly classified, then one has confidence approaching certainty that the network will correctly classify a fraction 1 ∊ of future test examples drawn from the same distribution. Conversely, for fully-connected feedforward nets with one hidden layer, any learning algorithm using fewer than (W/∊) random training examples will, for some distributions of examples consistent with an appropriate weight choice, fail at least some fixed fraction of the time to find a weight choice that will correctly classify more than a 1 ∊ fraction of the future test examples.

Recently, there have been increasing demands to construct compact deep architectures to remove unnecessary redundancy and to improve the inference speed. While many recent works focus on reducing the redundancy by eliminating unneeded weight parameters, it is not possible to apply a single deep network for multiple devices with different resources. When a new device or circumstantial condition requires a new deep architecture, it is necessary to construct and train a new network from scratch. In this work, we propose a novel deep learning framework, called a nested sparse network, which exploits an n-in-1-type nested structure in a neural network. A nested sparse network consists of multiple levels of networks with a different sparsity ratio associated with each level, and higher level networks share parameters with lower level networks to enable stable nested learning. The proposed framework realizes a resource-aware versatile architecture as the same network can meet diverse resource requirements, i.e., anytime property. Moreover, the proposed nested network can learn different forms of knowledge in its internal networks at different levels, enabling multiple tasks using a single network, such as coarse-to-fine hierarchical classification. In order to train the proposed nested network, we propose efficient weight connection learning and channel and layer scheduling strategies. We evaluate our network in multiple tasks, including adaptive deep compression, knowledge distillation, and learning class hierarchy, and demonstrate that nested sparse networks perform competitively, but more efficiently, compared to existing methods.

Currently, the neural network architecture design is mostly guided by the indirect metric of computation complexity, i.e., FLOPs. However, the direct metric, e.g., speed, also depends on the other factors such as memory access cost and platform characterics. Thus, this work proposes to evaluate the direct metric on the target platform, beyond only considering FLOPs. Based on a series of controlled experiments, this work derives several practical guidelines for efficient network design. Accordingly, a new architecture is presented, called ShuffleNet V2. Comprehensive ablation experiments verify that our model is the stateof-the-art in terms of speed and accuracy tradeoff.

In this work, we establish the relation between optimal control and training deep Convolution Neural Networks (CNNs). We show that the forward propagation in CNNs can be interpreted as a time-dependent nonlinear differential equation and learning can be seen as controlling the parameters of the differential equation such that the network approximates the data-label relation for given training data. Using this continuous interpretation, we derive two new methods to scale CNNs with respect to two different dimensions. The first class of multiscale methods connects low-resolution and high-resolution data using prolongation and restriction of CNN parameters inspired by algebraic multigrid techniques. We demonstrate that our method enables classifying highresolution images using CNNs trained with low-resolution images and vice versa and warm-starting the learning process. The second class of multiscale methods connects shallow and deep networks and leads to new training strategies that gradually increase the depths of the CNN while re-using parameters for initializations.

In this work, we investigate the effect of convolutional network depth, receptive field size, dropout layers, rectified activation unit type and dataset noise on its accuracy in Tiny-ImageNet Challenge settings. In order to make a thorough evaluation of the cause of the peformance improvement, we start with a basic 5 layer model with 5×5 convolutional receptive fields. We keep increasing network depth or reducing receptive field size, and continue applying modern techniques, such as PReLu and dropout, to the model. Our model achieves excellent performance even compared to state-of-the-art results, with 0.444 final error rate on the test set.

To better understand deep architectures and unsupervised learning, uncluttered by hardware details, we develop a general autoencoder framework for the comparative study of autoencoders, including Boolean autoencoders. We derive several results regarding autoencoders and autoencoder learning, including results on learning complexity, vertical and horizontal composition, and fundamental connections between critical points and clustering. Possible implications for the theory of deep architectures are discussed.

The increasing use of deep neural networks for safety-critical applications, such as autonomous driving and flight control, raises concerns about their safety and reliability. Formal verification can address these concerns by guaranteeing that a deep learning system operates as intended, but the state-of-the-art is limited to small systems. In this work-in-progress report we give an overview of our work on mitigating this difficulty, by pursuing two complementary directions: devising scalable verification techniques, and identifying design choices that result in deep learning systems that are more amenable to verification. ACM Reference Format: Lindsey Kuper, Guy Katz, Justin Gottschlich, Kyle Julian, Clark Barrett, and Mykel J. Kochenderfer. 2018. Toward Scalable Verification for SafetyCritical Deep Networks. In Proceedings of SysML Conference (SysML). ACM, New York, NY, USA, 3 pages.


Question: Deep versus shallow architecture in multilayer neural networks.

This paper extends the findings of K.M. Hornik, M. Stinchcombe, and H. White from the University of California, San Diego, Department of Economics (Discussion Paper, June 1988; to be published in Neural Networks), which established that multilayer feedforward networks, even with a single hidden layer and no output layer squashing, can function as universal approximators. These networks, equipped with arbitrary sigmoid activation functions in the hidden layer, can approximate any mapping with remarkable accuracy, given a sufficient number of hidden units. The current research, however, relaxes the requirement for a sigmoid activation function in the hidden layer. Instead, the activation function can be a significantly more generalized nonlinear function, thereby broadening the applicability of multilayer feedforward networks. The universal approximation capability of these networks is primarily attributed to the presence of intermediate layers with a large number of parallel processors, rather than the specific characteristics of the intermediate-layer activation function. Notably, the use of sigmoid activation functions is not a prerequisite for achieving universal approximation.

Very deep convolutional networks with hundreds of layers have led to significant reductions in error on competitive benchmarks. Although the unmatched expressiveness of the many layers can be highly desirable at test time, training very deep networks comes with its own set of challenges. The gradients can vanish, the forward flow often diminishes, and the training time can be painfully slow. To address these problems, we propose stochastic depth, a training procedure that enables the seemingly contradictory setup to train short networks and use deep networks at test time. We start with very deep networks but during training, for each mini-batch, randomly drop a subset of layers and bypass them with the identity function. This simple approach complements the recent success of residual networks. It reduces training time substantially and improves the test error significantly on almost all data sets that we used for evaluation. With stochastic depth we can increase the depth of residual networks even beyond 1200 layers and still yield meaningful improvements in test error (4.91% on CIFAR-10).

The driving force behind deep networks is their ability to compactly represent rich classes of functions. The primary notion for formally reasoning about this phenomenon is expressive efficiency, which refers to a situation where one network must grow unfeasibly large in order to realize (or approximate) functions of another. To date, expressive efficiency analyses focused on the architectural feature of depth, showing that deep networks are representationally superior to shallow ones. In this paper we study the expressive efficiency brought forth by connectivity, motivated by the observation that modern networks interconnect their layers in elaborate ways. We focus on dilated convolutional networks, a family of deep models delivering state of the art performance in sequence processing tasks. By introducing and analyzing the concept of mixed tensor decompositions, we prove that interconnecting dilated convolutional networks can lead to expressive efficiency. In particular, we show that even a single connection between intermediate layers can already lead to an almost quadratic gap, which in large-scale settings typically makes the difference between a model that is practical and one that is not. Empirical evaluation demonstrates how the expressive efficiency of connectivity, similarly to that of depth, translates into gains in accuracy. This leads us to believe that expressive efficiency may serve a key role in the development of new tools for deep network design.

This is the second paper in a series of three that explores the emergence of several prominent features of the functional architecture of visual cortex, in a "modular self-adaptive network" containing several layers of cells with parallel feedforward connections whose strengths develop according to a Hebb-type correlation-rewarding rule. In the present paper I show that orientation-selective cells, similar to the "simple" cortical cells of Hubel and Wiesel [Hubel, D. H. & Wiesel, T. N. (1962) J. Physiol. 160, 106-154], emerge in such a network. No orientation preference is specified to the system at any stage, the orientation-selective cell layer emerges even in the absence of environmental input to the system, and none of the basic developmental rules is specific to visual processing.

The predominant methodology in training deep learning advocates the use of stochastic gradient descent methods (SGDs). Despite its ease of implementation, SGDs are difficult to tune and parallelize. These problems make it challenging to develop, debug and scale up deep learning algorithms with SGDs. In this paper, we show that more sophisticated off-the-shelf optimization methods such as Limited memory BFGS (L-BFGS) and Conjugate gradient (CG) with line search can significantly simplify and speed up the process of pretraining deep algorithms. In our experiments, the difference between LBFGS/CG and SGDs are more pronounced if we consider algorithmic extensions (e.g., sparsity regularization) and hardware extensions (e.g., GPUs or computer clusters). Our experiments with distributed optimization support the use of L-BFGS with locally connected networks and convolutional neural networks. Using L-BFGS, our convolutional network model achieves 0.69% on the standard MNIST dataset. This is a state-of-theart result on MNIST among algorithms that do not use distortions or pretraining.

Deep and recurrent neural networks (DNNs and RNNs respectively) are powerful models that were considered to be almost impossible to train using stochastic gradient descent with momentum. In this paper, we show that when stochastic gradient descent with momentum uses a well-designed random initialization and a particular type of slowly increasing schedule for the momentum parameter, it can train both DNNs and RNNs (on datasets with long-term dependencies) to levels of performance that were previously achievable only with Hessian-Free optimization. We find that both the initialization and the momentum are crucial since poorly initialized networks cannot be trained with momentum and well-initialized networks perform markedly worse when the momentum is absent or poorly tuned. Our success training these models suggests that previous attempts to train deep and recurrent neural networks from random initializations have likely failed due to poor initialization schemes. Furthermore, carefully tuned momentum methods su ce for dealing with the curvature issues in deep and recurrent network training objectives without the need for sophisticated second-order methods.

We address the question of when a network can be expected to generalize from m random training examples chosen from some arbitrary probability distribution, assuming that future test examples are drawn from the same distribution. Among our results are the following bounds on appropriate sample vs. network size. Assume 0 < ∊ 1/8. We show that if m O(W/∊ log N/∊) random examples can be loaded on a feedforward network of linear threshold functions with N nodes and W weights, so that at least a fraction 1 ∊/2 of the examples are correctly classified, then one has confidence approaching certainty that the network will correctly classify a fraction 1 ∊ of future test examples drawn from the same distribution. Conversely, for fully-connected feedforward nets with one hidden layer, any learning algorithm using fewer than (W/∊) random training examples will, for some distributions of examples consistent with an appropriate weight choice, fail at least some fixed fraction of the time to find a weight choice that will correctly classify more than a 1 ∊ fraction of the future test examples.

Deep neural networks (DNNs) are powerful types of artificial neural networks (ANNs) that use several hidden layers. They have recently gained considerable attention in the speech transcription and image recognition community (Krizhevsky et al., 2012) for their superior predictive properties including robustness to overfitting. However their application to financial market prediction has not been previously researched, partly because of their computational complexity. This paper describes the application of DNNs to predicting financial market movement directions. A critical step in the viability of the approach in practice is the ability to effectively deploy the algorithm on general purpose high performance computing infrastructure. Using an Intel Xeon Phi co-processor with 61 cores, we describe the process for efficient implementation of the batched stochastic gradient descent algorithm and demonstrate a 11.4x speedup on the Intel Xeon Phi over a serial implementation on the Intel Xeon.

Pattern recognition is one of the traditional uses of neural networks. When trained with gradient-based learning methods, these networks can learn the classification of input data by example. An introduction to classifiers and gradient-based learning is given. It is shown how several perceptrons can be combined and trained gradient-based. Furthermore, an overview of convolutional neural networks, as well as a real-world example, are discussed.

We revisit the idea of using deep neural networks for one-shot decoding of random and structured codes, such as polar codes. Although it is possible to achieve maximum a posteriori (MAP) bit error rate (BER) performance for both code families and for short codeword lengths, we observe that (i) structured codes are easier to learn and (ii) the neural network is able to generalize to codewords that it has never seen during training for structured, but not for random codes. These results provide some evidence that neural networks can learn a form of decoding algorithm, rather than only a simple classifier. We introduce the metric normalized validation error (NVE) in order to further investigate the potential and limitations of deep learning-based decoding with respect to performance and complexity.

Inspired by the information theoretic idea of minimum description length, we add a term to the back propagation cost function that penalizes network complexity. We give the details of the procedure, called weight-elimination, describe its dynamics, and clarify the meaning of the parameters involved. From a Bayesian perspective, the complexity term can be usefully interpreted as an assumption about prior distribution of the weights. We use this procedure to predict the sunspot time series and the notoriously noisy series of currency exchange rates.


Question: A reversed Yarbus procedure: forecasting the observer's task from eye movement patterns.

This research delves into the characteristics of visual representations that guide ongoing performance in sensorimotor tasks, focusing on how visual information from various gaze positions is integrated. To investigate this, we employed a method involving task-relevant display modifications during saccadic eye movements. Participants were tasked with replicating a pattern of colored blocks on a computer screen, utilizing a mouse to move the blocks. Eye movements were tracked using a dual-Purkinje eye tracker, while the color of the blocks in the pattern was altered at various stages of the task. When the saccade target changed color during the eye movement, fixation durations on the model pattern increased, with the magnitude of increase dependent on the task stage when the change occurred. This finding implies that fixations on identical visual stimuli serve distinct functions at different task points. The results also suggest that the visual information retained between successive fixations is contingent on immediate task requirements, aligning with earlier propositions that visual representations are both limited and task-specific. Changes to blocks beyond the saccade target induced more pronounced increases in fixation duration, indicating that a holistic aspect of the pattern is preserved across fixations. Fixation durations revealed display change effects not evident in perceptual reports, highlighting the importance of distinguishing between processes operating at various levels of description and time scales. Our subjective experience of the environment may encapsulate events over a longer timescale than those governing the intricate workings of the perceptual-motor apparatus. This study contributes to our understanding of how task constraints shape visual working memory in sensorimotor tasks.

Object recognition is a key function in both human and machine vision. While brain decoding of seen and imagined objects has been achieved, the prediction is limited to training examples. We present a decoding approach for arbitrary objects using the machine vision principle that an object category is represented by a set of features rendered invariant through hierarchical processing. We show that visual features, including those derived from a deep convolutional neural network, can be predicted from fMRI patterns, and that greater accuracy is achieved for low-/high-level features with lower-/higher-level visual areas, respectively. Predicted features are used to identify seen/imagined object categories (extending beyond decoder training) from a set of computed features for numerous object images. Furthermore, decoding of imagined objects reveals progressive recruitment of higher-to-lower visual representations. Our results demonstrate a homology between human and machine vision and its utility for brain-based information retrieval.

The purpose of this study is to establish a relationship between the skin lines on the upper third of the face in cadavers, which represent the muscle activity in life and the skin lines achieved by voluntary contraction of the forehead, glabellar, and orbital muscles in patients. Anatomical dissection of fresh cadavers was performed in 20 fresh cadavers, 11 females and 9 males, with ages ranging from 53 to 77 years. Subcutaneous dissection identified the muscle shape and the continuity of the fibers of the eyebrow elevator and depress muscles. Subgaleal dissection identified the cutaneous insertions of the muscles. They were correlated with skin lines on the upper third of the face of the cadavers that represent the muscle activity in life. Voluntary contraction was performed by 20 voluntary patients, 13 females and 7 males, with ages ranging from 35 to 62 years. Distinct patterns of skin lines on the forehead, glabellar and orbital areas, and eyebrow displacement were identified. The frontalis exhibited four anatomical shapes with four different patterns of horizontal parallel lines on the forehead skin. The corrugator supercilii showed three shapes of muscles creating six patterns of vertical glabellar lines, three symmetrical and three asymmetrical. The orbicularis oculi and procerus had single patterns. The skin lines exhibited in voluntary contraction of the upper third of the face in patients showed the same patterns of the skin lines achieved in cadavers. Skin lines in cadavers, which are the expression of the muscle activity in life, were similar to those achieved in the voluntary contraction of patients, allowing us to assert that the muscle patterns of patients were similar to those identified in cadavers. This journal requires that authors assign a level of evidence to each submission to which Evidence-Based Medicine rankings are applicable. This excludes Review Articles, Book Reviews, and manuscripts that concern Basic Science, Animal Studies, Cadaver Studies, and Experimental Studies. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors http://www.springer.com/00266 .

A fruitful quantitative approach to understanding how the brain makes decisions has been to look at the time needed to make a decision, and how it is affected by factors such as the supply of information, or an individual's expectations. This approach has led to a model of decision-making, consistent with recent neurophysiological data, that explains the observed variability of reaction times and correctly predicts the effects of altered expectations. Can it also predict what happens when the urgency of making the response changes? We asked subjects to make eye movements to low-visibility targets either as fast or as accurately as possible, and found that the model does indeed predict the timing of their responses: the degree of urgency seems to influence the criterion level at which a decision signal triggers a response.

Existing research suggests that individual personality differences are correlated with a user's speed and accuracy in solving problems with different types of complex visualization systems. We extend this research by isolating factors in personality traits as well as in the visualizations that could have contributed to the observed correlation. We focus on a personality trait known as "locus of control” (LOC), which represents a person's tendency to see themselves as controlled by or in control of external events. To isolate variables of the visualization design, we control extraneous factors such as color, interaction, and labeling. We conduct a user study with four visualizations that gradually shift from a list metaphor to a containment metaphor and compare the participants' speed, accuracy, and preference with their locus of control and other personality factors. Our findings demonstrate that there is indeed a correlation between the two: participants with an internal locus of control perform more poorly with visualizations that employ a containment metaphor, while those with an external locus of control perform well with such visualizations. These results provide evidence for the externalization theory of visualization. Finally, we propose applications of these findings to adaptive visual analytics and visualization evaluation.

Image saliency detection is very useful in many computer vision tasks while it still remains a challenging problem. In this paper, we propose a new computational saliency detection model which is implemented with a coarse to fine strategy under the Bayesian framework. First, saliency points are applied to get a coarse location of the saliency region. And then, based on the rough region, we compute a prior map for the Bayesian model to achieve the final saliency map. Experimental results on a public available dataset show the effectiveness of the proposed prior map and the strength of our saliency map compared with several previous method.

This paper presents a method for the detection and recognition of social interactions in a day-long first-person video of u social event, like a trip to an amusement park. The location and orientation of faces are estimated and used to compute the line of sight for each face. The context provided by all the faces in a frame is used to convert the lines of sight into locations in space to which individuals attend. Further, individuals are assigned roles based on their patterns of attention. The rotes and locations of individuals are analyzed over time to detect and recognize the types of social interactions. In addition to patterns of face locations and attention, the head movements of the first-person can provide additional useful cues as to their attentional focus. We demonstrate encouraging results on detection and recognition of social interactions in first-person videos captured from multiple days of experience in amusement parks.

We review some applications of human-computer interaction that alleviate the complexity of visual recognition by partitioning it into human and machine tasks to exploit the differences between human and machine capabilities. Human involvement offers advantages, both in the design of automated pattern classification systems, and at the operational level of some image retrieval and classification tasks. Recent development of interactive systems has benefited from the convergence of computer vision and psychophysics in formulating visual tasks as computational processes. Computer-aided classifier design and exploratory data analysis are already well established in pattern recognition and machine learning, but interfaces and functionality are improving. On the operational side, earlier recognition systems made use of human talent only in preprocessing and in coping with rejects. Most current content-based image retrieval systems make use of relevance feedback without direct image interaction. In contrast, some visual object classification systems can exploit such interaction. They require, however, a domain-specific visible model that makes sense to both human and computer.

People routinely make various trait judgments from facial appearance, and such judgments affect important social outcomes. These judgments are highly correlated with each other, reflecting the fact that valence evaluation permeates trait judgments from faces. Trustworthiness judgments best approximate this evaluation, consistent with evidence about the involvement of the amygdala in the implicit evaluation of face trustworthiness. Based on computer modeling and behavioral experiments, I argue that face evaluation is an extension of functionally adaptive systems for understanding the communicative meaning of emotional expressions. Specifically, in the absence of diagnostic emotional cues, trustworthiness judgments are an attempt to infer behavioral intentions signaling approach/avoidance behaviors. Correspondingly, these judgments are derived from facial features that resemble emotional expressions signaling such behaviors: happiness and anger for the positive and negative ends of the trustworthiness continuum, respectively. The emotion overgeneralization hypothesis can explain highly efficient but not necessarily accurate trait judgments from faces, a pattern that appears puzzling from an evolutionary point of view and also generates novel predictions about brain responses to faces. Specifically, this hypothesis predicts a nonlinear response in the amygdala to face trustworthiness, confirmed in functional magnetic resonance imaging (fMRI) studies, and dissociations between processing of facial identity and face evaluation, confirmed in studies with developmental prosopagnosics. I conclude with some methodological implications for the study of face evaluation, focusing on the advantages of formally modeling representation of faces on social dimensions.

What are the computational tasks that an executive controller for visual attention must solve? This question is posed in the context of the Selective Tuning model of attention. The range of required computations go beyond top-down bias signals or region-of-interest determinations, and must deal with overt and covert fixations, process timing and synchronization, information routing, memory, matching control to task, spatial localization, priming, and coordination of bottom-up with top-down information. During task execution, results must be monitored to ensure the expected results. This description includes the kinds of elements that are common in the control of any kind of complex machine or system. We seek a mechanistic integration of the above, in other words, algorithms that accomplish control. Such algorithms operate on representations, transforming a representation of one kind into another, which then forms the input to yet another algorithm. Cognitive Programs (CPs) are hypothesized to capture exactly such representational transformations via stepwise sequences of operations. CPs, an updated and modernized offspring of Ullman's Visual Routines, impose an algorithmic structure to the set of attentional functions and play a role in the overall shaping of attentional modulation of the visual system so that it provides its best performance. This requires that we consider the visual system as a dynamic, yet general-purpose processor tuned to the task and input of the moment. This differs dramatically from the almost universal cognitive and computational views, which regard vision as a passively observing module to which simple questions about percepts can be posed, regardless of task. Differing from Visual Routines, CPs explicitly involve the critical elements of Visual Task Executive (vTE), Visual Attention Executive (vAE), and Visual Working Memory (vWM). Cognitive Programs provide the software that directs the actions of the Selective Tuning model of visual attention.

This paper introduces the development of a practical brain-computer interface at Tsinghua University. The system uses frequency-coded steady-state visual evoked potentials to determine the gaze direction of the user. To ensure more universal applicability of the system, approaches for reducing user variation on system performance have been proposed. The information transfer rate (ITR) has been evaluated both in the laboratory and at the Rehabilitation Center of China, respectively. The system has been proved to be applicable to >90% of people with a high ITR in living environments.


Question: A reverse Yarbus procedure: forecasting the observer's task from eye movement patterns.

This study explores the pivotal role of visual attention in the execution of saccadic eye movements through a series of experiments. The first experiment tasked participants with performing a saccade to a designated spot while also identifying a visual target displayed just before the eye movement. The findings revealed that target detection was most accurate when the target's location matched the saccade's destination, indicating that visual attention is harnessed in the planning or execution of saccadic eye movements. In the second experiment, participants were instructed to focus their attention on a specific location and then to make a saccade to either the same location or an alternative one. Results showed that target detection was consistently better at the saccade location, irrespective of where attention was directed. This evidence underscores the inability to decouple eye movement from attention, suggesting that visuospatial attention plays a critical role in the orchestration of voluntary saccadic eye movements.

A large number of road accidents in the world occur due to driver’s not taking proper breaks while driving long distances. This System is used to detect fatigue behaviour of the driver well and before any accident occurs. In this system driver’s facial appearance is extracted via high speed camera which is placed in car in front of driver. This camera will capture the frames at every instant of time. Each of this frame will be processed using image processing algorithm. The mouth and eye region is separated from each of the frames and analyzed for concluding whether the driver is dozy or not. The system draws conclusion if the eyes are found closed for five consecutive frame. If doziness is detected, a signal alert is triggered. In case accidents occurs, message is sent to the desired number.

In this paper we first point out a fatal drawback that the widely used Granger causality (GC) needs to estimate the autoregressive model, which is equivalent to taking a series of backward recursive operations which are infeasible in many irreversible chemical reaction models. Thus, new causality (NC) proposed by Hu et al. (2011) is theoretically shown to be more sensitive to reveal true causality than GC. We then apply GC and NC to motor imagery (MI) which is an important mental process in cognitive neuroscience and psychology and has received growing attention for a long time. We study causality flow during MI using scalp electroencephalograms from nine subjects in Brain-computer interface competition IV held in 2008. We are interested in three regions: Cz (central area of the cerebral cortex), C3 (left area of the cerebral cortex), and C4 (right area of the cerebral cortex) which are considered to be optimal locations for recognizing MI states in the literature. Our results show that: 1) there is strong directional connectivity from Cz to C3/C4 during left- and right-hand MIs based on GC and NC; 2) during left-hand MI, there is directional connectivity from C4 to C3 based on GC and NC; 3) during right-hand MI, there is strong directional connectivity from C3 to C4 which is much clearly revealed by NC than by GC, i.e., NC largely improves the classification rate; and 4) NC is demonstrated to be much more sensitive to reveal causal influence between different brain regions than GC.

This paper contributes to the nascent body of literature on pointing performance in Virtual Environments (VEs), comparing gaze- and hand-based pointing. Contrary to previous findings, preliminary results indicate that gaze-based pointing is slower than hand-based pointing for distant objects.

This paper is conceived as a guide that will describe the very well known Spatial Orienting paradigm, used to explore attentional processes in healthy individuals as well as in people suffering from psychiatric disorders and brain-damaged patients. The paradigm was developed in the late 1970s, and since then, it has been used in thousands of attentional studies. In this review, we attempt to describe, the paradigm for the naïf reader, and explain in detail when is it used, which variables are usually manipulated, how to interpret its results, and how can it be adapted to different populations and methodologies. The main goal of this review is to provide a practical guide to researchers who have never used the paradigm that will help them design their experiments, as a function of their theoretical and experimental needs. We also focus on how to adapt the paradigm to different technologies (such as event-related potentials, functional resonance imaging, or transcranial magnetic stimulation), and to different populations by presenting an example of its use in brain-damaged patients.

What if we could effectively read the mind and transfer human visual capabilities to computer vision methods? In this paper, we aim at addressing this question by developing the first visual object classifier driven by human brain signals. In particular, we employ EEG data evoked by visual object stimuli combined with Recurrent Neural Networks (RNN) to learn a discriminative brain activity manifold of visual categories in a reading the mind effort. Afterward, we transfer the learned capabilities to machines by training a Convolutional Neural Network (CNN)–based regressor to project images onto the learned manifold, thus allowing machines to employ human brain–based features for automated visual classification. We use a 128-channel EEG with active electrodes to record brain activity of several subjects while looking at images of 40 ImageNet object classes. The proposed RNN-based approach for discriminating object classes using brain signals reaches an average accuracy of about 83%, which greatly outperforms existing methods attempting to learn EEG visual object representations. As for automated object categorization, our human brain–driven approach obtains competitive performance, comparable to those achieved by powerful CNN models and it is also able to generalize over different visual datasets.

This paper introduces the development of a practical brain-computer interface at Tsinghua University. The system uses frequency-coded steady-state visual evoked potentials to determine the gaze direction of the user. To ensure more universal applicability of the system, approaches for reducing user variation on system performance have been proposed. The information transfer rate (ITR) has been evaluated both in the laboratory and at the Rehabilitation Center of China, respectively. The system has been proved to be applicable to >90% of people with a high ITR in living environments.

This study investigated facial expression recognition in peripheral relative to central vision, and the factors accounting for the recognition advantage of some expressions in the visual periphery. Whole faces or only the eyes or the mouth regions were presented for 150 ms, either at fixation or extrafoveally (2.5° or 6°), followed by a backward mask and a probe word. Results indicated that (a) all the basic expressions were recognized above chance level, although performance in peripheral vision was less impaired for happy than for non-happy expressions, (b) the happy face advantage remained when only the mouth region was presented, and (c) the smiling mouth was the most visually salient and most distinctive facial feature of all expressions. This suggests that the saliency and the diagnostic value of the smile account for the advantage in happy face recognition in peripheral vision. Because of saliency, the smiling mouth accrues sensory gain and becomes resistant to visual degradation due to stimulus eccentricity, thus remaining accessible extrafoveally. Because of diagnostic value, the smile provides a distinctive single cue of facial happiness, thus bypassing integration of face parts and reducing susceptibility to breakdown of configural processing in peripheral vision.

The face is our primary source of visual information for identifying people and reading their emotional and mental states. With the exception of prosopagnosics (who are unable to recognize faces) and those suffering from such disorders of social cognition as autism, people are extremely adept at these two tasks. However, our cognitive powers in this regard come at the price of reading too much into the human face. The face is often treated as a window into a person's true nature. Given the agreement in social perception of faces, this paper discusses that it should be possible to model this perception.

For a learning platform to be adapti ve, it needs to monitor the user in r ealtime. At the ScienceCampus Tübingen, we conduct empirical research on learning strategies, in particular with regard to system adaptivity. Our experimental Webbased adaptiv e multimedia learning environment is able to communicate with an eyetracking system in real-time, and l inks gaze data to browser-based learning content. By processing simple evaluation scripts in the browser, we can control the learning process by adapting the learning content in response to predefined conditions and user inputs. The experimental system serves as a basis for a series of psyc hologi al studies in the learning context. We have already conducted some studies with our experimental learning platform, but its devel opment is still ongoing. In particular, we are adding new feature s required for future studies. In this paper, we desc ribe the actual state of our experimental learning environment, lessons learned from its a pplication in the studies, and our plans for future devel opment.

The advancement in technology indicates that there is an opportunity to enhance Human-Computer Interaction by way of affective state recognition. Affective state recognition is typically based on passive stimuli such as watching video clips, which does not reflect genuine interaction. This paper presents a study on affective state recognition using active stimuli, i.e. facial expressions of users when they attempt computerised tasks, particularly across typical usage of computer systems. A data collection experiment is presented for acquiring data from normal users whilst they interact with software, attempting to complete a set of predefined tasks. In addition, a hierarchical machine learning approach is presented for facial expression-based affective state recognition, which employs an Euclidean distance-based feature representation, conjointly with a customised encoding for users’ self-reported affective states. Consequently, the aim is to find the potential relationship between the facial expressions, as defined by Paul Ekman, and the self-reported emotional states specified by users using Russells Circumplex model, in relation to the actual feelings and affective states. The main findings of this study suggest that facial expressions cannot precisely reveal the actual feelings of users whilst interacting with common computerised tasks. Moreover, during active interaction tasks more variation occurs within the facial expressions of participants than occurs within passive interaction.


Question: A reverse Yarbus procedure: forecasting the observer's assignment from eye movement designs.

This paper introduces a novel model for selective visual attention that focuses on the initial stages of object recognition in complex scenes. Drawing from the coherence theory of visual cognition, our model emphasizes the role of proto-objects, which are transient, unconfirmed units of visual information that precede full object recognition. These proto-objects can be selectively attended to and then validated as genuine objects through further processing. Our biologically inspired model illustrates how proto-objects can be formed and attended to within natural scenes. We show that this model can enhance a pre-existing object recognition system in the cortex, allowing it to progress from identifying isolated objects to sequentially recognizing all objects within a more intricate scene. This advancement represents a significant step towards understanding the mechanisms underlying visual attention and object recognition in complex environments.

Three experiments investigated the roles of intrinsic directions of a scene and observer's viewing direction in recognizing the scene. Participants learned the locations of seven objects along an intrinsic direction that was different from their viewing direction and then recognized spatial arrangements of three or six of these objects from different viewpoints. The results showed that triplets with two objects along the intrinsic direction (intrinsic triplets) were easier to recognize than triplets with two objects along the study viewing direction (non-intrinsic triplets), even when the intrinsic triplets were presented at a novel test viewpoint and the non-intrinsic triplets were presented at the familiar test viewpoint. The results also showed that configurations with the same three or six objects were easier to recognize at the familiar test viewpoint than other viewpoints. These results support and develop the model of spatial memory and navigation proposed by Mou, McNamara, Valiquette, and Rump [Mou, W., McNamara, T. P., Valiquiette C. M., & Rump, B. (2004). Allocentric and egocentric updating of spatial memories. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30, 142-157].

Parallel coordinates is a visualization technique that provides an unbiased representation of high-dimensional data. The parallel configuration of axes treats data dimensions uniformly and is well suited for exploratory visualization. However, first-time users of parallel coordinate visualizations can find the representation confusing and difficult to understand.We used eye tracking to study how parallel coordinate visualizations are perceived, and compared the results to the optimal visual scan path required to complete the tasks. The results indicate that even first-time users quickly learn how to use parallel coordinate visualizations, pay attention to the correct task-specific areas in the visualization, and become rapidly proficient with it.

One of the ultimate goals of neuroscience is decoding someone’s intentions directly from his/her brain activities. In this thesis, we aim at pursuing this goal in different scenarios. Firstly, we show the possibility of creating a user-centric music/movie recommender system by employing neurophysiological signals. Regarding this, we employed a brain decoding paradigm in order to classify the features extracted from brain signals of participants watching movie/music video clips, into our target classes (two broad music genres and four broad movie genres). Our results provide a preliminary experimental evidence towards user-centric music/movie content retrieval by exploiting brain signals. Secondly, we addressed one of the main issue of the applications of brain decoding algorithms. Generally, the performance of such algorithms suffers from the constraint of having few and noisy samples, which is the case in most of the neuroimaging datasets. In order to overcome this limitation, we employed an adaptation paradigm in order to transfer knowledge from another domain (e.g. large-scale image domain) to the brain domain. We experimentally show that such adaptation procedure leads to improved results. We performed such adaptation pipeline on different tasks (i.e. object recognition and genre classification) using different neuroimaging modalities (i.e. fMRI, EEG, and MEG). Thirdly, we aimed at one of the fundamental goals in brain decoding which is reconstructing the external stimuli using only the brain features. Under this scenario, we show the possibility of regressing the stimuli spectrogram using time-frequency analysis of the brain signals. Finally, we conclude the thesis by summarizing our contributions and discussing the future directions and applications of our research.

It is well established that heart rate variability (HRV) plays an important role in social communication. Polyvagal theory suggests that HRV may provide a sensitive marker of one's ability to respond and recognize social cues. The aim of the present study was to directly test this hypothesis. Resting-state HRV was collected and performance on the Reading the Mind in the Eyes Test was assessed in 65 volunteers. HRV was positively associated with performance on this emotion recognition task confirming our hypothesis and these findings were retained after controlling for a variety of confounding variables known to influence HRV - sex, BMI, smoking habits, physical activity levels, depression, anxiety, and stress. Our data suggests that increased HRV may provide a novel marker of one's ability to recognize emotions in humans. Implications for understanding the biological basis of emotion recognition, and social impairment in humans are discussed.

Recent advances in human neuroimaging have shown that it is possible to accurately decode a person's conscious experience based only on non-invasive measurements of their brain activity. Such 'brain reading' has mostly been studied in the domain of visual perception, where it helps reveal the way in which individual experiences are encoded in the human brain. The same approach can also be extended to other types of mental state, such as covert attitudes and lie detection. Such applications raise important ethical issues concerning the privacy of personal thought.

Introduction Vision, due to its significance in surviving and socializing, is one of the most important and  extensively studied sensory functions in the human brain. In order to fully understand visual  information processing, or more specifically, visual recognition, David Marr proposed the  Tri-level Hypothesis [29], in that three levels of the system should be studied: the computational  goal of the system, the internal representation or the algorithm the system uses to achieve the  goal, and the neural substrates that implement the system. It is well-known that visual  recognition in the human brain is implemented by the ventral visual pathway [32], which  receives visual information from the retina and goes through a layered structure including V1  (also known as the primary visual cortex), V2, V4, before reaching the inferior temporal cortex  (IT). The topographic mapping between the retina and the human visual cortex follows a  log-polar transformation, in which the Cartesian coordinates of the retina are transformed to  polar coordinates (polar angle and eccentricity) in the human visual cortex. From V1 to V4, each  of the visual areas contains a complete polar angle and eccentricity map, and have a increasing  receptive field (RF) size and respond to increasing complex visual features [12,22]. In higher  level visual areas such as the ventral temporal cortex (VTC), studies using functional magnetic  resonance imaging (fMRI) and diffusion tensor imaging (DTI) have found there exists visual  category-selective regions, such as the Fusiform Face Area (FFA, [26]) for face processing, the  Parahippocampal Place Area (PPA, [13]) for scene perception, and the Lateral Occipital  Complex (LOC, [19]) for general object recognition. Grill-Spector and Malach [20] proposed that  layer-based hierarchical processing, together with the functional specialization, are the two  organizing principles of the human visual cortex. More recent studies have shown that the  central-biased face recognition pathway (FFA) and the peripheral-biased scene recognition  pathway (PPA) are both functionally and anatomically segregated by mid-fusiform sulcus (MFS),  which enables parallel fast processing of visual recognition tasks in the ventral pathway [18, 43,  21, 44].

Voluntarily directing visual attention to a cued position in space leads to improved processing of forthcoming visual stimuli at the attended position, and attenuated processing of competing stimuli elsewhere, due to anticipatory tuning of visual cortex activity. In EEG, recent evidence points to a determining role of modulations of posterior alpha-band activity (8-14 Hz) in such anticipatory facilitation (alpha-power decreases) versus inhibition (alpha-power increases). Yet, while such alpha-modulations are a common finding, the direction of modulation varies to a great extent across studies implying dependence on task demands. Here, we reveal opposite modulation of posterior alpha-power with early/initiation versus later/sustained processes of anticipatory attention orienting. Marked alpha-decreases were observed during shifting of attention (initial 700 ms) over occipito-parietal areas processing to-be-attended visual space, while alpha-increases dominated in the subsequent maintenance phase (>700 ms) over occipito-parietal cortex tuned to unattended positions. Notably, the presence of alpha-modulation strongly depended on individual resting alpha-power. Overall, this provides further support to an active facilitative versus inhibitory role of alpha-power decreases and increases and suggests that these attention-related changes are differentially deployed during anticipatory attention orienting to prepare versus maintain the cortex for optimal target processing.

We present a method for creating black-and-white illustrations from photographs of human faces. In addition an interactive technique is demonstrated for deforming these black-and-white facial illustrations to create caricatures which highlight and exaggerate representative facial features. We evaluate the effectiveness of the resulting images through psychophysical studies to assess accuracy and speed in both recognition and learning tasks. These studies show that the facial illustrations and caricatures generated using our techniques are as effective as photographs in recognition tasks. For the learning task we find that illustrations are learned two times faster than photographs and caricatures are learned one and a half times faster than photographs. Because our techniques produce images that are effective at communicating complex information, they are useful in a number of potential applications, ranging from entertainment and education to low bandwidth telecommunications and psychology research.

In this work we describe experiments with eigenfaces for recognition and interactive search in a large-scale face database. Accurate visual recognition is demonstrated using a database of O(10) faces. The problem of recognition under general viewing orientation is also examined. A view-based multiple-observer eigenspace technique is proposed for use in face recognition under variable pose. In addition, a modular eigenspace description technique is used which incorporates salient features such as the eyes, nose and mouth, in an eigenfeature layer. This modular representation yields higher recognition rates as well as a more robust framework for face recognition. An automatic feature extraction technique using feature eigentemplates is also demonstrated.

This work evaluates the feasibility of a motor imagery-based optical brain-computer interface (BCI) for humanoid robot control. The functional near-infrared spectroscopy (fNIRS) based BCI-robot system developed in this study operates through a high-level control mechanism where user specifies a target action through the BCI and the robot performs the set of micro operations necessary to fulfill the identified goal. For the evaluation of the system, four motor imagery tasks (left hand, right hand, left foot, and right foot) were mapped to operational commands (turn left, turn right, walk forward, walk backward) that were sent to the robot in real time to direct the robot navigating a small room. An ecologically valid offline analysis with minimal preprocessing shows that seven subjects could achieve an average accuracy of 32.5 %. This was increased to 43.6 % just by including calibration data from the same day of the robot control using the same cap setup, indicating that day-of calibration following the initial training may be important for BCI control.