Question: Using HMM and the Face Edge Length Model for facial recognition under conditions of partial occlusion.

This paper introduces an innovative technique for face recognition in video sequences, particularly effective under partial occlusion. The method models human faces using a low-dimensional appearance manifold in the high-dimensional image space. This manifold is decomposed into subsets, referred to as pose manifolds, with each approximated as an affine plane. The construction of this representation involves sampling exemplars from video sequences, clustering these samples using the K-means algorithm, and then representing each cluster as a plane derived from principal component analysis (PCA). The transitions between pose manifolds, which encode the probability of moving from one image to another, are learned from training video sequences. Face recognition in test sequences is achieved through a maximum a posteriori formulation, combining the likelihood of the input image belonging to a specific pose manifold with the transition probability to this manifold from the preceding frame. To address partial occlusion, a weight mask is incorporated into the recognition process. Comprehensive experimental results confirm the superiority of the proposed algorithm over existing frame-based face recognition methods that use temporal voting schemes, especially when dealing with partial occlusions.

A facial recognition system is a computer application for automatically identifying or verifying a person from a digital image or a video frame from a video source. One of the way is to do this is by comparing selected facial features from the image and a facial database. It is typically used in security systems and can be compared to other biometrics such as fingerprint or eye iris recognition systems. In this paper we focus on 3-D facial recognition system and biometric facial recognition system. We do critics on facial recognition system giving effectiveness and weaknesses. This paper also introduces scope of recognition system in India. Face recognition has received substantial attention from researchers in biometrics, pattern recognition field and computer vision communities. Face recognition can be applied in Security measure at Air ports, Passport verification, Criminals list verification in police department, Visa processing, Verification of Electoral identification and Card Security measure at ATM’s. Principal Component Analysis (PCA) is a technique among the most common feature extraction techniques used in Face Recognition. In this paper, a face recognition system for personal identification and verification using Principal Component Analysis with different distance classifiers is proposed. The test results in the ORL face database produces interesting results from the point of view of recognition success, rate, and robustness of the face recognition algorithm. Different classifiers were used to match the image of a person to a class (a subject) obtained from the training data. These classifiers are: the City-Block Distance Classifier, the Euclidian distance classifier, the Squared Euclidian Distance Classifier, and the Squared Chebyshev distance Classifier.

In this paper, we present a patch-based regression framework for addressing the human age and head pose estimation problems. Firstly, each image is encoded as an ensemble of orderless coordinate patches, the global distribution of which is described by Gaussian mixture models (GMM), and then each image is further expressed as a specific distribution model by Maximum a Posteriori adaptation from the global GMM. Then the patch-kernel is designed for characterizing the Kullback-Leibler divergence between the derived models for any two images, and its discriminating power is further enhanced by a weak learning process, called inter-modality similarity synchronization. Finally, kernel regression is employed for ultimate human age or head pose estimation. These three stages are complementary to each other, and jointly minimize the regression error. The effectiveness of this regression framework is validated by three experiments: 1) on the YAMAHA aging database, our solution brings a more than 50% reduction in age estimation error compared with the best reported results; 2) on the FG-NET aging database, our solution based on raw image features performs even better than the state-of-the-art algorithms which require fine face alignment for extracting warped appearance features; and 3) on the CHIL head pose database, our solution significantly outperforms the best one reported in the CLEAR07 evaluation.

Keywords: Eigenface, face Recognition, k-means clustering, principle component analysis (PCA)

Traditional photometric stereo algorithms employ a Lambertian reflectance model with a varying albedo field and involve the appearance of only one object. In this paper, we generalize photometric stereo algorithms to handle all appearances of all objects in a class, in particular the human face class, by making use of the linear Lambertian property. A linear Lambertian object is one which is linearly spanned by a set of basis objects and has a Lambertian surface. The linear property leads to a rank constraint and, consequently, a factorization of an observation matrix that consists of exemplar images of different objects (e.g., faces of different subjects) under different, unknown illuminations. Integrability and symmetry constraints are used to fully recover the subspace bases using a novel linearized algorithm that takes the varying albedo field into account. The effectiveness of the linear Lambertian property is further investigated by using it for the problem of illumination-invariant face recognition using just one image. Attached shadows are incorporated in the model by a careful treatment of the inherent nonlinearity in Lambert's law. This enables us to extend our algorithm to perform face recognition in the presence of multiple illumination sources. Experimental results using standard data sets are presented

Between October 2000 and December 2000 we collected a database of over 40,000 facial images of 68 people. Using the CMU 3D Room we imaged each person across 13 different poses, under 43 different illumination conditions, and with 4 different expressions. We call this database the CMU Pose, Illumination, and Expression (PIE) database. In this paper we describe the imaging hardware, the collection procedure, the organization of the database, several potential uses of the database, and how to obtain the database.

Person-independent and pose-invariant estimation of eye-gaze is important for situation analysis and for automated video annotation. We propose a fast cascade regression based method that first estimates the location of a dense set of markers and their visibility, then reconstructs face shape by fitting a part-based 3D model. Next, the reconstructed 3D shape is used to estimate a canonical view of the eyes for 3D gaze estimation. The model operates in a feature space that naturally encodes local ordinal properties of pixel intensities leading to photometric invariant estimation of gaze. To evaluate the algorithm in comparison with alternative approaches, three publicly-available databases were used, Boston University Head Tracking, Multi-View Gaze and CAVE Gaze datasets. Precision for head pose and gaze averaged 4 degrees or less for pitch, yaw, and roll. The algorithm outperformed alternative methods in both datasets.

Facial composites are widely used by law enforcement agencies to assist in the identification and apprehension of suspects involved in criminal activities. These composites, generated from witness descriptions, are posted in public places and media with the hope that some viewers will provide tips about the identity of the suspect. This method of identifying suspects is slow, tedious, and may not lead to the timely apprehension of a suspect. Hence, there is a need for a method that can automatically and efficiently match facial composites to large police mugshot databases. Because of this requirement, facial composite recognition is an important topic for biometrics researchers. While substantial progress has been made in nonforensic facial composite (or viewed composite) recognition over the past decade, very little work has been done using operational composites relevant to law enforcement agencies. Furthermore, no facial composite to mugshot matching systems have been documented that are readily deployable as standalone software. Thus, the contributions of this paper include: 1) an exploration of composite recognition use cases involving multiple forms of facial composites; 2) the FaceSketchID System, a scalable, and operationally deployable software system that achieves state-of-the-art matching accuracy on facial composites using two algorithms (holistic and component based); and 3) a study of the effects of training data on algorithm performance. We present experimental results using a large mugshot gallery that is representative of a law enforcement agency's mugshot database. All results are compared against three state-of-the-art commercial-off-the-shelf face recognition systems.

We present a distributed representation of pose and appearance of people called the “poselet activation vector”. First we show that this representation can be used to estimate the pose of people defined by the 3D orientations of the head and torso in the challenging PASCAL VOC 2010 person detection dataset. Our method is robust to clutter, aspect and viewpoint variation and works even when body parts like faces and limbs are occluded or hard to localize. We combine this representation with other sources of information like interaction with objects and other people in the image and use it for action recognition. We report competitive results on the PASCAL VOC 2010 static image action classification challenge.

This thesis focuses on the area of face processing and aims at designing a reliable framework to facilitate face, age, and gender recognition. A Bag-of-Words framework has been optimized for the task of face recognition by evaluating different feature descriptors and different bag-of-words configurations. More specifically, we choose a compact set of features (e.g., descriptors, window locations, window sizes, dictionary sizes, etc.) in order to produce the highest possible rate of accuracy. Experiments on a challenging dataset shows that our framework achieves a better level of accuracy when compared to other popular approaches such as dimension reduction techniques, edge detection operators, and texture and shape feature extractors. The second contribution of this thesis is the proposition of a general framework for age and gender classification. Although the vast majority of the existing solutions focus on a single visual descriptor that often only encodes a certain characteristic of the image regions, this thesis aims at integrating multiple feature types. For this purpose, feature selection is employed to obtain more accurate and robust facial descriptors. Once descriptors have been computed, a compact set of features is chosen, which facilitates facial image processing for age and gender analysis. In addition to this, a new color descriptor (CLR-LBP) is proposed and the results obtained is shown to be comparable to those of other pre-existing color descriptors. The experimental results indicates that our age and gender framework outperforms other proposed methods when examined on two challenging databases, where face objects are present with different expressions and levels of illumination. This achievement demonstrates the effectiveness of our proposed solution and allows us to achieve a higher accuracy over the existing state-of-the-art methods.

─ This article summarizes the most important research projects developed in thearea of recognitions y stemsusing facial images. It presents the description of the principal lines of study about people identification systems using face recognition. Furthermore, it contains a synthesis of recent mathematical techniques for pattern sextraction in these systems.


Question: Analysis of the output range for deep feedforward neural networks.

This paper explores the application and advancements of deep learning, a subset of machine-learning technology that has become integral to various aspects of modern society. From web searches and social media content filtering to e-commerce recommendations and consumer electronics, deep learning techniques are increasingly prevalent. They are employed in identifying objects in images, transcribing speech to text, matching news or products with user interests, and refining search results. Unlike traditional machine-learning methods, which were limited in handling raw, natural data and required significant domain expertise to design feature extractors, deep learning enables the machine to process raw data and automatically discover the necessary representations for pattern recognition or classification.

Representation learning, a key component of deep learning, involves multiple levels of representation obtained through the composition of simple yet non-linear modules. These modules transform the representation at each level, starting from the raw input, into a higher, slightly more abstract level. Through the composition of such transformations, complex functions can be learned. For classification tasks, higher layers of representation amplify critical aspects of the input for discrimination while suppressing irrelevant variations.

For instance, in image processing, the first layer of representation often detects edges at various orientations and locations in the image. Subsequent layers then identify motifs by recognizing specific edge arrangements, regardless of minor variations in their positions. Further layers may assemble motifs into larger combinations corresponding to parts of familiar objects, and later layers detect objects as combinations of these parts. The crucial difference in deep learning is that these layers of features are not manually designed by engineers but are learned from data using a general-purpose learning procedure.

Deep learning is making significant strides in addressing long-standing challenges in the artificial intelligence community, excelling at discovering complex structures in high-dimensional data and thus applicable across various scientific, business, and governmental domains. It has outperformed other machine-learning techniques in image and speech recognition, predicting drug molecule activity, analyzing particle accelerator data, reconstructing brain circuits, and predicting the effects of mutations in non-coding DNA on gene expression and disease. Surprisingly, it has also shown promising results in natural language understanding tasks such as topic classification, sentiment analysis, question answering, and language translation.

We anticipate that deep learning will continue to achieve numerous successes in the near future due to its minimal requirement for manual engineering, allowing it to capitalize on the growing availability of computation and data. The ongoing development of new learning algorithms and architectures for deep neural networks will only accelerate this progress.

This paper introduces Non-Autonomous Input-Output Stable Network (NAIS-Net), a very deep architecture where each stacked processing block is derived from a time-invariant non-autonomous dynamical system. Non-autonomy is implemented by skip connections from the block input to each of the unrolled processing stages and allows stability to be enforced so that blocks can be unrolled adaptively to a pattern-dependent processing depth. NAIS-Net induces non-trivial, Lipschitz input-output maps, even for an infinite unroll length. We prove that the network is globally asymptotically stable so that for every initial condition there is exactly one input-dependent equilibrium assuming tanh units, and multiple stable equilibria for ReL units. An efficient implementation that enforces the stability under derived conditions for both fully-connected and convolutional layers is also presented. Experimental results show how NAIS-Net exhibits stability in practice, yielding a significant reduction in generalization gap compared to ResNets.

Efforts to reduce the numerical precision of computations in deep learning training have yielded systems that aggressively quantize weights and activations, yet employ wide high-precision accumulators for partial sums in inner-product operations to preserve the quality of convergence. The absence of any framework to analyze the precision requirements of partial sum accumulations results in conservative design choices. This imposes an upper-bound on the reduction of complexity of multiply-accumulate units. We present a statistical approach to analyze the impact of reduced accumulation precision on deep learning training. Observing that a bad choice for accumulation precision results in loss of information that manifests itself as a reduction in variance in an ensemble of partial sums, we derive a set of equations that relate this variance to the length of accumulation and the minimum number of bits needed for accumulation. We apply our analysis to three benchmark networks: CIFAR-10 ResNet 32, ImageNet ResNet 18 and ImageNet AlexNet. In each case, with accumulation precision set in accordance with our proposed equations, the networks successfully converge to the single precision floating-point baseline. We also show that reducing accumulation precision further degrades the quality of the trained network, proving that our equations produce tight bounds. Overall this analysis enables precise tailoring of computation hardware to the application, yielding areaand power-optimal systems.

Neural networks are a powerful technology for classification of visual inputs arising from documents. However, there is a confusing plethora of different neural network methods that are used in the literature and in industry. This paper describes a set of concrete best practices that document analysis researchers can use to get good results with neural networks. The most important practice is getting a training set as large as possible: we expand the training set by adding a new form of distorted data. The next most important practice is that convolutional neural networks are better suited for visual document tasks than fully connected networks. We propose that a simple “do-it-yourself” implementation of convolution with a flexible architecture is suitable for many visual document problems. This simple convolutional neural network does not require complex methods, such as momentum, weight decay, structuredependent learning rates, averaging layers, tangent prop, or even finely-tuning the architecture. The end result is a very simple yet general architecture which can yield state-of-the-art performance for document analysis. We illustrate our claims on the MNIST set of English digit

Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network’s behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding.

Neural networks have great success in many machine learning applications, but the fundamental learning theory behind them remains largely unsolved. Learning neural networks is NP-hard, but in practice, simple algorithms like stochastic gradient descent (SGD) often produce good solutions. Moreover, it is observed that overparameterization (that is, designing networks whose number of parameters is larger than statistically needed to perfectly fit the data) improves both optimization and generalization, appearing to contradict traditional learning theory. In this work, we extend the theoretical understanding of two and three-layer neural networks in the overparameterized regime. We prove that, using overparameterized neural networks, one can (improperly) learn some notable hypothesis classes, including two and three-layer neural networks with fewer parameters. Moreover, the learning process can be simply done by SGD or its variants in polynomial time using polynomially many samples. We also show that for a fixed sample size, the generalization error of the solution found by some SGD variant can be made almost independent of the number of parameters in the overparameterized network. ∗V1 appears on this date and V2 polishes writing and tightens parameters. Authors sorted in alphabetical order.

We introduce a method to train Binarized Neural Networks (BNNs) neural networks with binary weights and activations at run-time. At training-time the binary weights and activations are used for computing the parameters gradients. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which is expected to substantially improve power-efficiency. To validate the effectiveness of BNNs we conduct two sets of experiments on the Torch7 and Theano frameworks. On both, BNNs achieved nearly state-of-the-art results over the MNIST, CIFAR-10 and SVHN datasets. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available on-line.

Convolutional neural networks have recently been shown to outperform fully connected deep neural networks on several speech recognition tasks. Their superior performance is due to their convolutional structure that processes several, slightly shifted versions of the input window using the same weights, and then pools the resulting neural activations. This pooling operation makes the network less sensitive to translations. The convolutional network results published up till now used sigmoid or rectified linear neurons. However, quite recently a new type of activation function called the maxout activation has been proposed. Its operation is closely related to convolutional networks, as it applies a similar pooling step, but over different neurons evaluated on the same input. Here, we combine the two technologies, and experiment with deep convolutional neural networks built from maxout neurons. Phone recognition tests on the TIMIT database show that switching to maxout units from rectifier units decreases the phone error rate for each network configuration studied, and yields relative error rate reductions of between 2% and 6%.

We extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes. The combined system is analogous to a Turing Machine or Von Neumann architecture but is differentiable end-toend, allowing it to be efficiently trained with gradient descent. Preliminary results demonstrate that Neural Turing Machines can infer simple algorithms such as copying, sorting, and associative recall from input and output examples.

We propose a quantum generalisation of a classical neural network. The classical neurons are firstly rendered reversible by adding ancillary bits. Then they are generalised to being quantum reversible, i.e. unitary. (The classical networks we generalise are called feedforward, and have step-function activation functions.) The quantum network can be trained efficiently using gradient descent on a cost function to perform quantum generalisations of classical tasks. We demonstrate numerically that it can: (i) compress quantum states onto a minimal number of qubits, creating a quantum autoencoder, and (ii) discover quantum communication protocols such as teleportation. Our general recipe is theoretical and implementation-independent. The quantum neuron module can naturally be implemented photonically.

1 A neural network is a massively parallel distributed processor that has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects (Haykin 1998): 1. Knowledge is acquired by the network through a learning process. 2. Interconnection strengths known as synaptic weights are used to store the knowledge. Basically, learning is a process by which the free parameters (i.e., synaptic weights and bias levels) of a neural network are adapted through a continuing process of stimulation by the environment in which the network is embedded. The type of learning is determined by the manner in which the parameter changes take place. In a general sense, the learning process may be classified as follows:


Question: Analysis of the output range for deep feedforward neural networks.

This paper outlines a novel method for verifying the behavior of feed-forward neural networks that employ piece-wise linear activation functions, a common feature in deep learning models. These networks are notoriously challenging to validate using state-of-the-art satisfiability modulo theory (SMT) and integer linear programming (ILP) solvers. Our solution involves the integration of a global linear approximation of the network's overall functionality into the verification process, which aids in reasoning akin to SMT over the network's dynamics. A dedicated verification algorithm is proposed, leveraging this approximation within a search procedure. This algorithm deduces additional node phases for the non-linear nodes from partial phase assignments, mirroring the process of unit propagation in conventional SAT solving. Moreover, the paper elucidates how to derive further conflict clauses and safe node configurations from the outcomes of the analysis stages executed during the search. The efficacy of this methodology is assessed on real-world case studies, including collision avoidance systems and handwriting recognition tasks, showcasing its potential for enhancing the reliability of deep learning applications.

Summary form only given. A novel method called dynamic node creation (DNC) that attacks issues of training large networks and of testing networks with different numbers of hidden layer units is presented. DNC sequentially adds nodes one at a time to the hidden layer(s) of the network until the desired approximation accuracy is achieved. Simulation results for parity, symmetry, binary addition, and the encoder problem are presented. The procedure was capable of finding known minimal topologies in many cases, and was always within three nodes of the minimum. Computational expense for finding the solutions was comparable to training normal backpropagation (BP) networks with the same final topologies. Starting out with fewer nodes than needed to solve the problem actually seems to help find a solution. The method yielded a solution for every problem tried. BP applied to the same large networks with randomized initial weights was unable, after repeated attempts, to replicate some minimum solutions found by DNC.<<ETX>>

Misuse detection is the process of attempting to identify instances of network attacks by comparing current activity against the expected actions of an intruder. Most current approaches to misuse detection involve the use of rule-based expert systems to identify indications of known attacks. However, these techniques are less successful in identifying attacks which vary from expected patterns. Artificial neural networks provide the potential to identify and classify network activity based on limited, incomplete, and nonlinear data sources. We present an approach to the process of misuse detection that utilizes the analytical strengths of neural networks, and we provide the results from our preliminary analysis of this approach.

Extreme learning machine (ELM) is an emerging learning algorithm for the generalized single hidden layer feedforward neural networks, of which the hidden node parameters are randomly generated and the output weights are analytically computed. However, due to its shallow architecture, feature learning using ELM may not be effective for natural signals (e.g., images/videos), even with a large number of hidden nodes. To address this issue, in this paper, a new ELM-based hierarchical learning framework is proposed for multilayer perceptron. The proposed architecture is divided into two main components: 1) self-taught feature extraction followed by supervised feature classification and 2) they are bridged by random initialized hidden weights. The novelties of this paper are as follows: 1) unsupervised multilayer encoding is conducted for feature extraction, and an ELM-based sparse autoencoder is developed via ℓ1 constraint. By doing so, it achieves more compact and meaningful feature representations than the original ELM; 2) by exploiting the advantages of ELM random feature mapping, the hierarchically encoded outputs are randomly projected before final decision making, which leads to a better generalization with faster learning speed; and 3) unlike the greedy layerwise training of deep learning (DL), the hidden layers of the proposed framework are trained in a forward manner. Once the previous layer is established, the weights of the current layer are fixed without fine-tuning. Therefore, it has much better learning efficiency than the DL. Extensive experiments on various widely used classification data sets show that the proposed algorithm achieves better and faster convergence than the existing state-of-the-art hierarchical learning methods. Furthermore, multiple applications in computer vision further confirm the generality and capability of the proposed learning scheme.

One of the mysteries in the success of neural networks is randomly initialized first order methods like gradient descent can achieve zero training loss even though the objective function is non-convex and non-smooth. This paper demystifies this surprising phenomenon for two-layer fully connected ReLU activated neural networks. For anm hidden node shallow neural network with ReLU activation and n training data, we show as long asm is large enough and no two inputs are parallel, randomly initialized gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function. Our analysis relies on the following observation: over-parameterization and random initialization jointly restrict every weight vector to be close to its initialization for all iterations, which allows us to exploit a strong convexity-like property to show that gradient descent converges at a global linear rate to the global optimum. We believe these insights are also useful in analyzing deep models and other first order methods.

Sub-8-bit representation of DNNs incur some noticeable loss of accuracy despite rigorous (re)training at low-precision. Such loss of accuracy essentially makes them equivalent to a much shallower counterpart, diminishing the power of being deep networks. To address this problem of accuracy drop we introduce the notion of residual networks where we add more low-precision edges to sensitive branches of the sub-8-bit network to compensate for the lost accuracy. Further, we present a perturbation theory to identify such sensitive edges. Aided by such an elegant trade-off between accuracy and model size, the 8-2 architecture (8-bit activations, ternary weights), enhanced by residual ternary edges, turns out to be sophisticated enough to achieve similar accuracy as 8-8 representation (∼ 1% drop from our FP-32 baseline), despite ∼ 1.6× reduction in model size, ∼ 26× reduction in number of multiplications , and potentially ∼ 2× inference speed up comparing to 8-8 representation, on the state-of-the-art deep network ResNet-101 pre-trained on ImageNet dataset. Moreover, depending on the varying accuracy requirements in a dynamic environment, the deployed low-precision model can be upgraded/downgraded on-the-fly by partially enabling/disabling residual connections. For example, disabling the least important residual connections in the above enhanced network, the accuracy drop is ∼ 2% (from our FP-32 baseline), despite ∼ 1.9× reduction in model size, ∼ 32× reduction in number of multiplications, and potentially ∼ 2.3× inference speed up comparing to 8-8 representation. Finally, all the ternary connections are sparse in nature, and the residual ternary conversion can be done in a resource-constraint setting without any low-precision (re)training and without accessing the data.

Artificial neural network is a valuable tool for time series forecasting. In the case of performing multiperiodic forecasting with artificial neural networks, two methods, namely iterative and direct, can be used. In iterative method, first subsequent period information is predicted through past observations. Afterwards, the estimated value is used as an input; thereby the next period is predicted. The process is carried on until the end of the forecast horizon. In the direct forecast method, successive periods can be predicted all at once. Hence, this method is thought to yield better results as only observed data is utilized in order to predict future periods. In this study, forecasting was performed using direct and iterative methods, and results of the methods are compared using grey relational analysis to find the method which gives a better result. 2008 Elsevier Ltd. All rights reserved.

Machine learning has become mainstream across industries. Numerous examples proved the validity of it for security applications. In this work, we investigate how to reverse engineer a neural network by using only power side-channel information. To this end, we consider a multilayer perceptron as the machine learning architecture of choice and assume a non-invasive and eavesdropping attacker capable of measuring only passive side-channel leakages like power consumption, electromagnetic radiation, and reaction time. We conduct all experiments on real data and common neural net architectures in order to properly assess the applicability and extendability of those attacks. Practical results are shown on an ARM CORTEX-M3 microcontroller. Our experiments show that the side-channel attacker is capable of obtaining the following information: the activation functions used in the architecture, the number of layers and neurons in the layers, the number of output classes, and weights in the neural network. Thus, the attacker can effectively reverse engineer the network using sidechannel information. Next, we show that once the attacker has the knowledge about the neural network architecture, he/she could also recover the inputs to the network with only a single-shot measurement. Finally, we discuss several mitigations one could use to thwart such attacks.

The success of deep neural networks has inspired many to wonder whether other learners could benefit from deep, layered architectures. We present a general framework called forward thinking for deep learning that generalizes the architectural flexibility and sophistication of deep neural networks while also allowing for (i) different types of learning functions in the network, other than neurons, and (ii) the ability to adaptively deepen the network as needed to improve results. This is done by training one layer at a time, and once a layer is trained, the input data are mapped forward through the layer to create a new learning problem. The process is then repeated, transforming the data through multiple layers, one at a time, rendering a new dataset, which is expected to be better behaved, and on which a final output layer can achieve good performance. In the case where the neurons of deep neural nets are replaced with decision trees, we call the result a Forward Thinking Deep Random Forest (FTDRF). We demonstrate a proof of concept by applying FTDRF on the MNIST dataset. We also provide a general mathematical formulation, called Forward Thinking that allows for other types of deep learning problems to be considered.

A simple linear neuron model with constrained Hebbian-type synaptic modification is analyzed and a new class of unconstrained learning rules is derived. It is shown that the model neuron tends to extract the principal component from a stationary input vector sequence.

We present the soft exponential activation function for artificial neural networks that continuously interpolates between logarithmic, linear, and exponential functions. This activation function is simple, differentiable, and parameterized so that it can be trained as the rest of the network is trained. We hypothesize that soft exponential has the potential to improve neural network learning, as it can exactly calculate many natural operations that typical neural networks can only approximate, including addition, multiplication, inner product, distance, and sinusoids.


Question: Analysis of the output range for deep feedforward neural networks.

This paper unveils a groundbreaking technique for executing stealthy attacks against deep learning systems through the exploitation of adversarial examples. Adversarial examples, a form of maliciously crafted input, are designed to elicit incorrect outputs from machine learning models, particularly deep neural networks (DNNs), while appearing benign to human perception. The implications of such attacks are profound, encompassing scenarios where malware is misidentified as legitimate content or where vehicle behavior is compromised. Notably, prior adversarial attacks have relied on knowledge of the model's internal workings or the training data. In contrast, we present the first practical black-box attack that requires no such information. Our attacker's only capability is to observe the output labels produced by the DNN for selected inputs. The attack strategy hinges on training a local model as a surrogate for the target DNN, using synthetically generated inputs labeled by the DNN itself. Once the local model is trained, we employ it to generate adversarial examples, which are then found to be consistently misclassified by the targeted DNN. To validate our approach in a real-world setting, we conducted a blinded attack against a DNN hosted by MetaMind, an online deep learning API. With just 6,400 synthetic inputs labeled by the target DNN (a fraction of MetaMind's training set size), our surrogate model was able to create adversarial examples that led to misclassification rates of 84.24% on MetaMind's DNN. We further demonstrate that our strategy is not limited to DNNs but extends to various machine learning techniques, including logistic regression and support vector machines (SVMs), regardless of the ML model chosen for the surrogate. Our attack was successfully executed against models hosted by Amazon and Google, using logistic regression surrogates trained with a mere 800 label queries. The resulting adversarial examples were misclassified by Amazon and Google at rates of 96.19% and 88.94%, respectively. Intriguingly, our black-box attack method was also found to circumvent defense strategies that had previously been effective in thwarting adversarial example generation. The paper concludes with experimental insights into the reasons behind the transferability of adversarial samples between different models.

Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network’s behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding.

We recently conducted a research project for a large North American automobile insurer. This study was the most exhaustive ever undertaken by this particular insurer and lasted over an entire year. We analyzed the discriminating power of each variable used for ratemaking. We analyzed the performance of several models within five broad categories: linear regressions, generalized linear models, decision trees, neural networks and support vector machines. In this paper, we present the main results of this study. We qualitatively compare models and show how neural networks can represent high-order nonlinear dependencies with a small number of parameters, each of which is estimated on a large proportion of the data, thus yielding low variance. We thoroughly explain the purpose of the nonlinear sigmoidal transforms which are at the very heart of neural networks’ performances. The main numerical result is a statistically significant reduction in the out-of-sample meansquared error using the neural network model and our ability to substantially reduce the median premium by charging more to the highest risks. This in turn can translate into substantial savings and financial benefits for an insurer. We hope this paper goes a long way in convincing actuaries to include neural networks within their set of modeling tools for ratemaking.

Artificial Neural Network is among the most popular algorithm for supervised learning. However, Neural Networks have a well-known drawback of being a “Black Box” learner that is not comprehensible to the Users. This lack of transparency makes it unsuitable for many high risk tasks such as medical diagnosis that requires a rational justification for making a decision. Rule Extraction methods attempt to curb this limitation by extracting comprehensible rules from a trained Network. Many such extraction algorithms have been developed over the years with their respective strengths and weaknesses. They have been broadly categorized into three types based on their approach to use internal model of the Network. Eclectic Methods are hybrid algorithms that combine the other approaches to attain more performance. In this paper, we present an Eclectic method called HERETIC. Our algorithm uses Inductive Decision Tree learning combined with information of the neural network structure for extracting logical rules. Experiments and theoretical analysis show HERETIC to be better in terms of speed and performance.

Deep neural networks such as Convolutional Networks (ConvNets) and Deep Belief Networks (DBNs) represent the state-of-the-art for many machine learning and computer vision classification problems. To overcome the large computational cost of deep networks, spiking deep networks have recently been proposed, given the specialized hardware now available for spiking neural networks (SNNs). However, this has come at the cost of performance losses due to the conversion from analog neural networks (ANNs) without a notion of time, to sparsely firing, event-driven SNNs. Here we analyze the effects of converting deep ANNs into SNNs with respect to the choice of parameters for spiking neurons such as firing rates and thresholds. We present a set of optimization techniques to minimize performance loss in the conversion process for ConvNets and fully connected deep networks. These techniques yield networks that outperform all previous SNNs on the MNIST database to date, and many networks here are close to maximum performance after only 20 ms of simulated time. The techniques include using rectified linear units (ReLUs) with zero bias during training, and using a new weight normalization method to help regulate firing rates. Our method for converting an ANN into an SNN enables low-latency classification with high accuracies already after the first output spike, and compared with previous SNN approaches it yields improved performance without increased training time. The presented analysis and optimization techniques boost the value of spiking deep networks as an attractive framework for neuromorphic computing platforms aimed at fast and efficient pattern recognition.

The increasing use of deep neural networks for safety-critical applications, such as autonomous driving and flight control, raises concerns about their safety and reliability. Formal verification can address these concerns by guaranteeing that a deep learning system operates as intended, but the state-of-the-art is limited to small systems. In this work-in-progress report we give an overview of our work on mitigating this difficulty, by pursuing two complementary directions: devising scalable verification techniques, and identifying design choices that result in deep learning systems that are more amenable to verification. ACM Reference Format: Lindsey Kuper, Guy Katz, Justin Gottschlich, Kyle Julian, Clark Barrett, and Mykel J. Kochenderfer. 2018. Toward Scalable Verification for SafetyCritical Deep Networks. In Proceedings of SysML Conference (SysML). ACM, New York, NY, USA, 3 pages.

Implementing E cient, Portable Computations for Machine Learning

Past works have shown that, somewhat surprisingly, over-parametrization can help generalization in neural networks. Towards explaining this phenomenon, we adopt a margin-based perspective. We establish: 1) for multilayer feedforward relu networks, the global minimizer of a weakly-regularized cross-entropy loss has the maximum normalized margin among all networks, 2) as a result, increasing the over-parametrization improves the normalized margin and generalization error bounds for two-layer networks. In particular, an infinite-size neural network enjoys the best generalization guarantees. The typical infinite feature methods are kernel methods; we compare the neural net margin with that of kernel methods and construct natural instances where kernel methods have much weaker generalization guarantees. We validate this gap between the two approaches empirically. Finally, this infinite-neuron viewpoint is also fruitful for analyzing optimization. We show that a perturbed gradient flow on infinite-size networks finds a global optimizer in polynomial time.

Deep learning has made significant breakthroughs in various fields of artificial intelligence. Advantages of deep learning include the ability to capture highly complicated features, weak involvement of human engineering, etc. However, it is still virtually impossible to use deep learning to analyze programs since deep architectures cannot be trained effectively with pure back propagation. In this pioneering paper, we propose the “coding criterion” to build program vector representations, which are the premise of deep learning for program analysis. Our representation learning approach directly makes deep learning a reality in this new field. We evaluate the learned vector representations both qualitatively and quantitatively. We conclude, based on the experiments, the coding criterion is successful in building program representations. To evaluate whether deep learning is beneficial for program analysis, we feed the representations to deep neural networks, and achieve higher accuracy in the program classification task than “shallow” methods, such as logistic regression and the support vector machine. This result confirms the feasibility of deep learning to analyze programs. It also gives primary evidence of its success in this new field. We believe deep learning will become an outstanding technique for program analysis in the near future.

Sub-8-bit representation of DNNs incur some noticeable loss of accuracy despite rigorous (re)training at low-precision. Such loss of accuracy essentially makes them equivalent to a much shallower counterpart, diminishing the power of being deep networks. To address this problem of accuracy drop we introduce the notion of residual networks where we add more low-precision edges to sensitive branches of the sub-8-bit network to compensate for the lost accuracy. Further, we present a perturbation theory to identify such sensitive edges. Aided by such an elegant trade-off between accuracy and model size, the 8-2 architecture (8-bit activations, ternary weights), enhanced by residual ternary edges, turns out to be sophisticated enough to achieve similar accuracy as 8-8 representation (∼ 1% drop from our FP-32 baseline), despite ∼ 1.6× reduction in model size, ∼ 26× reduction in number of multiplications , and potentially ∼ 2× inference speed up comparing to 8-8 representation, on the state-of-the-art deep network ResNet-101 pre-trained on ImageNet dataset. Moreover, depending on the varying accuracy requirements in a dynamic environment, the deployed low-precision model can be upgraded/downgraded on-the-fly by partially enabling/disabling residual connections. For example, disabling the least important residual connections in the above enhanced network, the accuracy drop is ∼ 2% (from our FP-32 baseline), despite ∼ 1.9× reduction in model size, ∼ 32× reduction in number of multiplications, and potentially ∼ 2.3× inference speed up comparing to 8-8 representation. Finally, all the ternary connections are sparse in nature, and the residual ternary conversion can be done in a resource-constraint setting without any low-precision (re)training and without accessing the data.

Quantum neural networks (QNNs) have been explored as one of the best approach for improving the computational efficiency of neural networks. Because of the powerful and fantastic performance of quantum computation, some researchers have begun considering the implications of quantum computation on the field of artificial neural networks (ANNs).The purpose of this paper is to introduce an application of QNNs in construction of cryptography system in which two networks exchange their outputs (in qubits) and the key to be synchronized between two communicating parties. This system is based on multilayer qubit QNNs trained with back-propagation algorithm.


Question: Analysis of the output range for deep feedforward neural networks.

This paper introduces Reluplex, a specialized and efficient SMT solver designed to address the challenge of verifying properties within deep neural networks, particularly those employing the Rectified Linear Unit (ReLU) activation function. Deep neural networks have gained prominence in solving intricate real-world problems, yet their integration into safety-critical systems remains problematic due to the difficulty in formally certifying their behavior. Reluplex offers a novel, scalable, and efficient method to overcome this hurdle by extending the simplex method to handle the non-convex nature of ReLU, a key component in contemporary neural network architectures. Unlike other verification methods that often rely on simplifying assumptions, Reluplex directly confronts the complexity of neural networks, ensuring comprehensive and accurate analysis. Our evaluation of Reluplex on a prototype deep neural network, designed for the next-generation airborne collision avoidance system (ACAS Xu) for unmanned aircraft, highlights its capability to validate properties of networks significantly larger than those previously assessed by existing techniques, showcasing its potential to revolutionize the safety and reliability of neural network applications.

We revisit the idea of using deep neural networks for one-shot decoding of random and structured codes, such as polar codes. Although it is possible to achieve maximum a posteriori (MAP) bit error rate (BER) performance for both code families and for short codeword lengths, we observe that (i) structured codes are easier to learn and (ii) the neural network is able to generalize to codewords that it has never seen during training for structured, but not for random codes. These results provide some evidence that neural networks can learn a form of decoding algorithm, rather than only a simple classifier. We introduce the metric normalized validation error (NVE) in order to further investigate the potential and limitations of deep learning-based decoding with respect to performance and complexity.

We propose a new weight initialization suited for arbitrary nonlinearities by generalizing previous weight initializations. The initialization corrects for the influence of dropout rates and an arbitrary nonlinearity’s influence on variance through simple corrective scalars. Consequently, this initialization does not require computing mini-batch statistics nor weight pre-initialization. This simple method enables improved accuracy over previous initializations, and it allows for training highly regularized neural networks where previous initializations lead to poor convergence.

Deep Learning has attracted significant attention in recent years. Here I present a brief overview of my first Deep Learner of 1991, and its historic context, with a timeline of Deep Learning highlights. Note: As a machine learning researcher I am obsessed with proper credit assignment. This draft is the result of an experiment in rapid massive open online peer review. Since 20 September 2013, subsequent revisions published under www.deeplearning.me have absorbed many suggestions for improvements by experts. The abbreviation “TL” is used to refer to subsections of the timeline section. Figure 1: My first Deep Learning system of 1991 used a deep stack of recurrent neural networks (a Neural Hierarchical Temporal Memory) pre-trained in unsupervised fashion to accelerate subsequent supervised learning [79, 81, 82]. 1 ar X iv :1 31 2. 55 48 v1 [ cs .N E ] 1 9 D ec 2 01 3 In 2009, our Deep Learning Artificial Neural Networks became the first Deep Learners to win official international pattern recognition competitions [40, 83] (with secret test sets known only to the organisers); by 2012 they had won eight of them (TL 1.13), including the first contests on object detection in large images (ICPR 2012) [2, 16] and image segmentation (ISBI 2012) [3, 15]. In 2011, they achieved the world’s first superhuman visual pattern recognition results [20, 19]. Others have implemented very similar techniques, e.g., [51], and won additional contests or set benchmark records since 2012, e.g., (TL 1.13, TL 1.14). The field of Deep Learning research is far older though—compare the timeline (TL) further down. My first Deep Learner dates back to 1991 [79, 81, 82] (TL 1.7). It can perform credit assignment across hundreds of nonlinear operators or neural layers, by using unsupervised pre-training for a stack of recurrent neural networks (RNN) (deep by nature) as in Figure 1. Such RNN are general computers more powerful than normal feedforward NN, and can encode entire sequences of input vectors. The basic idea is still relevant today. Each RNN is trained for a while in unsupervised fashion to predict its next input. From then on, only unexpected inputs (errors) convey new information and get fed to the next higher RNN which thus ticks on a slower, self-organising time scale. It can easily be shown that no information gets lost. It just gets compressed (much of machine learning is essentially about compression). We get less and less redundant input sequence encodings in deeper and deeper levels of this hierarchical temporal memory, which compresses data in both space (like feedforward NN) and time. There also is a continuous variant [84]. One ancient illustrative Deep Learning experiment of 1993 [82] required credit assignment across 1200 time steps, or through 1200 subsequent nonlinear virtual layers. The top level code of the initially unsupervised RNN stack, however, got so compact that (previously infeasible) sequence classification through additional supervised learning became possible. There is a way of compressing higher levels down into lower levels, thus partially collapsing the hierarchical temporal memory. The trick is to retrain lower-level RNN to continually imitate (predict) the hidden units of already trained, slower, higher-level RNN, through additional predictive output neurons [81, 79, 82]. This helps the lower RNN to develop appropriate, rarely changing memories that may bridge very long time lags. The Deep Learner of 1991 was a first way of overcoming the Fundamental Deep Learning Problem identified and analysed in 1991 by my very first student (now professor) Sepp Hochreiter (TL 1.6): the problem of vanishing or exploding gradients [46, 47, 10]. The latter motivated all our subsequent Deep Learning research of the 1990s and 2000s. Through supervised LSTM RNN (1997) (e.g., [48, 32, 39, 36, 37, 40, 38, 83], TL 1.8) we could eventually perform similar feats as with the 1991 system [81, 82], overcoming the Fundamental Deep Learning Problem without any unsupervised pre-training. Moreover, LSTM could also learn tasks unlearnable by the partially unsupervised 1991 chunker [81, 82]. Particularly successful are stacks of LSTM RNN [40] trained by Connectionist Temporal Classification (CTC) [36]. On faster computers of 2009, this became the first RNN system ever to win an official international pattern recognition competition [40, 83], through the work of my PhD student and postdoc Alex Graves, e.g., [40]. To my knowledge, this also was the first Deep Learning system ever (recurrent or not) to win such a contest (TL 1.10). (In fact, it won three different ICDAR 2009 contests on connected handwriting in three different languages, e.g., [83, 40], TL 1.10.) A while ago, Alex moved on to Geoffrey Hinton’s lab (Univ. Toronto), where a stack [40] of our bidirectional LSTM RNN [39] also broke a famous TIMIT speech recognition record [38] (TL 1.14), despite thousands of man years previously spent on HMM-based speech recognition research. CTC-LSTM also helped to score first at NIST’s OpenHaRT2013 evaluation [11]. Recently, well-known entrepreneurs also got interested [43, 52] in such hierarchical temporal memories [81, 82] (TL 1.7). The expression Deep Learning actually got coined relatively late, around 2006, in the context of unsupervised pre-training for less general feedforward networks [44] (TL 1.9). Such a system reached 1.2% error rate [44] on the MNIST handwritten digits [54, 55], perhaps the most famous benchmark of Machine Learning. Our team first showed that good old backpropagation (TL 1.2) on GPUs (with training pattern distortions [6, 86] but without any unsupervised pre-training) can actually achieve a three times better result of 0.35% [17] back then, a world record (a previous standard net achieved 0.7% [86]; a backprop-trained [54, 55] Convolutional NN (CNN or convnet) [29, 30, 54, 55] got 0.39% [70](TL 1.9);

We propose a quantum generalisation of a classical neural network. The classical neurons are firstly rendered reversible by adding ancillary bits. Then they are generalised to being quantum reversible, i.e. unitary. (The classical networks we generalise are called feedforward, and have step-function activation functions.) The quantum network can be trained efficiently using gradient descent on a cost function to perform quantum generalisations of classical tasks. We demonstrate numerically that it can: (i) compress quantum states onto a minimal number of qubits, creating a quantum autoencoder, and (ii) discover quantum communication protocols such as teleportation. Our general recipe is theoretical and implementation-independent. The quantum neuron module can naturally be implemented photonically.

As book review editor of the IEEE Transactions on Neural Networks, Mohamad Hassoun has had the opportunity to assess the multitude of books on artificial neural networks that have Historically the data pairs and logical, fashion transition while associative memories commonly? This organization of such as a, clear and algorithms search simulated annealing learning algorithm. After the utility of how to, discover form. More than three different levels of the cost function will. Dean pomerleau in reinforcement and they, discovered two chapters on. This view the application for handwriting recognition this approach by arrows originating.

Quantum neural networks (QNNs) have been explored as one of the best approach for improving the computational efficiency of neural networks. Because of the powerful and fantastic performance of quantum computation, some researchers have begun considering the implications of quantum computation on the field of artificial neural networks (ANNs).The purpose of this paper is to introduce an application of QNNs in construction of cryptography system in which two networks exchange their outputs (in qubits) and the key to be synchronized between two communicating parties. This system is based on multilayer qubit QNNs trained with back-propagation algorithm.

Utilizing information diffusion method and artificial neural networks, we propose in this paper a hybrid fuzzy neural network to estimate the relationship between isoseismal area and earthquake magnitude. We focus on the study of incompleteness and contradictory nature of patterns in scanty historical earthquake records. Information diffusion method is employed to construct fuzzy relationships which are equal to the number of observations. Integration of the relationships can change the contradictory patterns into more compatible ones which, in turn, can smoothly and quickly train the feedforward neural network with backpropagation algorithm (BP) to obtain the final relationship. A practical application is employed to show the superiority of the model. @ 1999 Elsevier Science B.V. All rights reserved.

The report introduces a constructive learning algorithm for recurrent neural networks, which modifies only the weights to output units in order to achieve the learning task. key words: recurrent neural networks, supervised learning Zusammenfassung. Der Report führt ein konstruktives Lernverfahren für rekurrente neuronale Netze ein, welches zum Erreichen des Lernzieles lediglich die Gewichte der zu den Ausgabeneuronen führenden Verbindungen modifiziert. Stichwörter: rekurrente neuronale Netze, überwachtes Lernen

In many regards, tuning deep-learning networks is still more an art than it is a technique. Choosing the correct hyper-parameters and getting a complex network to learn properly can be daunting to people not well versed in that art. Taking the example of entailment classification, this work will analyze different methods and tools that can be used in order to diagnose learning problems. It will aim at providing an deep understanding of some key hyper-parameters, sketch out their effects and explain how to identify recurrent problems when tuning them.

We recently conducted a research project for a large North American automobile insurer. This study was the most exhaustive ever undertaken by this particular insurer and lasted over an entire year. We analyzed the discriminating power of each variable used for ratemaking. We analyzed the performance of several models within five broad categories: linear regressions, generalized linear models, decision trees, neural networks and support vector machines. In this paper, we present the main results of this study. We qualitatively compare models and show how neural networks can represent high-order nonlinear dependencies with a small number of parameters, each of which is estimated on a large proportion of the data, thus yielding low variance. We thoroughly explain the purpose of the nonlinear sigmoidal transforms which are at the very heart of neural networks’ performances. The main numerical result is a statistically significant reduction in the out-of-sample meansquared error using the neural network model and our ability to substantially reduce the median premium by charging more to the highest risks. This in turn can translate into substantial savings and financial benefits for an insurer. We hope this paper goes a long way in convincing actuaries to include neural networks within their set of modeling tools for ratemaking.


Question: Analysis of the output range for deep feedforward neural networks.

This paper introduces a groundbreaking learning methodology known as back-propagation, tailored for networks of units resembling neurons. This procedure iteratively refines the connection weights within the network to minimize the discrepancy between the network's actual output vector and the desired output vector. Through this iterative weight adjustment process, internal 'hidden' units, not directly connected to the input or output, evolve to represent critical characteristics of the task domain. The inherent patterns and regularities of the task are captured through the intricate interactions of these hidden units. This feature of creating valuable new features distinguishes back-propagation from earlier, more rudimentary techniques, such as the perceptron-convergence procedure.

**Question**: A Direct Search Method to solve Economic Dispatch Problem with Valve-Point Effect

**Documents**: A novel approach to solve the economic dispatch problem with valve-point effect using differential evolution
The economic dispatch problem with valve-point effect is a complex non-convex optimization problem due to the presence of non-smooth cost functions. This paper proposes a novel approach based on differential evolution (DE) algorithm to solve this problem. The proposed approach is capable of handling both smooth and non-smooth cost functions. The proposed approach is tested on a standard test system and the results are compared with those obtained by other evolutionary algorithms. The results show that the proposed approach is capable of finding the global optimum solution with a high degree of accuracy.

**Similar Documents**: This paper presents an innovative solution to address the economic dispatch problem with valve-point effect using the differential evolution (DE) algorithm. The valve-point effect introduces complexity to the problem by creating non-smooth cost functions, transforming it into a challenging non-convex optimization task. Our proposed methodology, rooted in differential evolution, adeptly manages both smooth and non-smooth cost functions, offering a robust solution. To validate the effectiveness of our approach, it was applied to a conventional test system. Comparative analysis with outcomes from alternative evolutionary algorithms underscores the superior performance of our methodology, consistently delivering the global optimum solution with exceptional precision.

A general-purpose, fully interconnected neural-net chip was used to perform computationally intensive tasks for handwritten digit recognition. The chip has nearly 3000 programmable connections, which can be set for template matching. The templates can be reprogrammed as needed during the recognition sequence. The recognition process proceeds in four major steps. First, the image is captured using a TV camera and a digital framegrab. This image is converted, using a digital computer, to either black or white pixels and scaled to fill a 16*16-pixel frame. Next, using the neural-net chip, the image is skeletonized, i.e. the image is thinned to a backbone one pixel wide. Then, the chip is programmed, and a feature map is created by template-matching stored primitive patterns on the chip with regions on the skeletonized image. Finally, recognition, based on the feature map, is achieved using any one of a variety of statistical and heuristic techniques on a digital computer. Best scores range between 90% and 99% correct classification, depending on the quality of the original handwritten digits.<<ETX>>

We consider the problem of learning from examples in layered linear feed-forward neural networks using optimization methods, such as back propagation, with respect to the usual quadratic error function E of the connection weights. Our main result is a complete description of the landscape attached to E in terms of principal component analysis. We show that E has a unique minimum corresponding to the projection onto the subspace generated by the first principal vectors of a covariance matrix associated with the training patterns. All the additional critical points of E are saddle points (corresponding to projections onto subspaces generated by higher order vectors). The auto-associative case is examined in detail. Extensions and implications for the learning algorithms are discussed.

We present an application of back-propagation networks to handwritten digit recognition. Minimal preprocessing of the data was required, but architecture of the network was highly constrained and specifically designed for the task. The input of the network consists of normalized images of isolated digits. The method has 1 % error rate and about a 9% reject rate on zipcode digits provided by the U.S. Postal Service.

This report describes DeepHyperNEAT, an extension of HyperNEAT to allow it to alter the topology of its indirectlyencoded neural network (called the substrate) so that it can continue to grow and increase in complexity over evolution. This aim is accomplished by augmenting HyperNEAT’s compositional pattern producing networks (CPPNs) with new information that allows them to represent substrate topology, and by adding three novel mutations to HyperNEAT that exploit this new information. The purpose of this report is to detail the method and validate its ability to evolve, which is shown through a simple XOR-based test. Significantly more work will be needed to analyze the algorithmic and scientific implications of these new capabilities, but by releasing this report it becomes possible for others also to explore the opportunities that such an extension opens up. A link to associated source code is also provided at the end of this document. Introduction HyperNEAT [1; 2; 3] (Hypercube-based NeuroEvolution of Augmenting Topologies) is a method for evolving neural networks (NNs) through an indirect encoding based on the NEAT method [4] that came before it. The main idea in HyperNEAT is that a special genetic representation called a compositional pattern producing network (CPPN) [5] that is evolved by NEAT generates a pattern of weights that connect a fixed set of neurons positioned within a two-dimensional or three-dimensional space called the substrate. A longstanding limitation of the conventional HyperNEAT algorithm is that while the CPPN can increase in size, the number of neurons and layers in the substrate remains fixed from the beginning of evolution. While there exists one method called evolvable substrate HyperNEAT (ES-HyperNEAT) [6] that allows HyperNEAT to increase the complexity of its substrate that indeed offers opportunities not available in the original HyperNEAT, the new method proposed here, called Deep HyperNEAT is designed explicitly to evolve substrates reminiscent of architectures seen in deep learning today, with increasing numbers of layers and separate parallel pathways through the network. Deep HyperNEAT achieves its new ability by fitting more information into the standard CPPN of HyperNEAT, which now can mutate to describe new layers and new neurons in the substrate. This paper is a preliminary technical report on the method, released to allow others to experiment with its ideas quickly. Thus many intriguing scientific questions about the opportunities created by this new capability will not be answered here; instead the focus is on introducing the technical components of the method and showing that they work correctly (i.e. perform as expected) in a simple validation experiment on the XOR problem. Rigorous scientific investigation is left for the future. The paper is organized as follows: HyperNEAT and the CPPN indirect encoding are reviewed first. The main extensions to HyperNEAT and CPPNs that constitute Deep HyperNEAT are then described, and demonstrated running in a simple XOR test. The paper concludes with thoughts on implications and future work. A public repository containing all of the code for the method is linked at the end of the paper. Background: HyperNEAT HyperNEAT [1; 2; 3] is a neuroevolution method that utilizes indirect encoding to evolve connectivity patterns in neural networks (NN). HyperNEAT historically has found applications in a wide breadth of domains, including multiagent control [7], checkers [2; 3], quadruped locomotion [8; 9], Robocup Keepaway [10], and simulated driving in traffic [11]. The motivation for HyperNEAT was to capture the ability of natural genetic encoding (DNA) to reuse information to encode massive neural structures with orders of magnitude fewer genes. The key principle in HyperNEAT that enabled this kind of meaningful reuse is that neural geometry should be able to exploit problem structure. (This principle is similar to the inspiration behind incorporating convolution in deep learning, but its realization and implications in HyperNEAT are different.) For example, two hidden neurons that process adjacent parts of an input field (e.g. a visual field) should also be roughly adjacent within the neural geometry. That way, locality within an NN becomes a heuristic for relatedness. To situate neurons in an NN within a geometric context, each neuron in HyperNEAT is assigned a Cartesian coordinate. For example, if the NN is configured as a stack of two-dimensional layers of neurons, then the position of each neuron within a layer is simply (x,y). In HyperNEAT it is conventional that the coordinate system of each layer centers at (0,0) and ranges between [-1, 1] in both x and y, thereby forming a square filter or “sheet” of neurons. That way, each neuron n within a layer l can be identified by its position (x,y). Specifying neurons in the NN as a function of the neural geometry allows for an elegant formalism for encoding the connectivity patterns between layers. The connectivity pattern between one layer and another (e.g. all possible connections and weights between neurons in some layer i and another layer j) can be encoded by a function ConnectLayers that takes the positions of the source and target neurons as input. Instead of evolving the weights of every connection individually as in past approaches to neuroevolution like NEAT [12], HyperNEAT evolves a graphical representation of a function, ConnectLayers, that describes how the neurons are connected. In this way, ConnectLayers acts as a lower-dimensional encoding for how to generate connectivity patterns. For example, we might want to query ConnectLayers for the weight of a connection between a neuron a in layer 1 with coordinates (x1,y1) and a neuron b in layer 2 with coordinates (x2,y2). ConnectLayers then returns the weight wx1,y1,x2,y2 : wx1,y1,x2,y2 = ConnectLayers(x1,y1,x2,y2). (1) The main idea in HyperNEAT is to evolve ConnectLayers. In HyperNEAT, ConnectLayers is represented by a compositional pattern producing network (CPPN) [13] (figure 1). A CPPN is an evolvable directed graph representation for functions. Much like NNs, nodes are functions and connections have weights that are applied to the outputs of nodes before passing into other nodes. Importantly, CPPN node functions are heterogeneous in any given CPPN (they are sampled from a set of possible functions usually containing sigmoid, linear, Gaussian, and cosine) and there is no notion of layers. A thorough introduction or review of CPPNs is in [5]. The CPPN begins with a simple topology (usually just its input nodes and its output node with a sparse number of connections between them) and complexifies via evolution in HyperNEAT. Evolving the CPPN this way, through complexification, allows the CPPN to be biased towards compressed representations of connectivity patterns. This approach leads to increasingly intricate yet organized connectivity patterns. In practice CPPNs have proven effective and popular in part because they are designed to evolve increasing complexity with the original NEAT algorithm, which underlies evolution in HyperNEAT. The predicate “hyper” in HyperNEAT signifies that the CPPN is in effect encoding a pattern within a four-dimensional or six-dimensional hypercube (because ConnectLayers typically takes four or six inputs), which is then painted onto the connectivity of a network. In effect, HyperNEAT provides the unique ability to evolve a compressed function that can encode the connectivity of an arbitrarily large network as a function of its geometry. That way, it is possible to produce NNs whose connections display regular and compositional structure through the reuse of motifs encoded in the CPPN. The

In this paper, a novel and effective criterion based on the estimation of the signal-to-noise-ratio figure (SNRF) is proposed to optimize the number of hidden neurons in neural networks to avoid overfitting in the function approximation. SNRF can quantitatively measure the useful information left unlearned so that overfitting can be automatically detected from the training error only without use of a separate validation set. It is illustrated by optimizing the number of hidden neurons in a multi-layer perceptron (MLP) using benchmark datasets. The criterion can be further utilized in the optimization of other parameters of neural networks when overfitting needs to be considered.

A continuous Deep Belief Network (cDBN) with two hidden layers is proposed in this 18 paper, focusing on the problem of weak feature learning ability when dealing with continuous data. 19 In cDBN, the input data is trained in an unsupervised way by using continuous version of transfer 20 functions, the contrastive divergence is designed in hidden layer training process to raise 21 convergence speed, an improved dropout strategy is then implemented in unsupervised training to 22 realize features learning by de-cooperating between the units, and then the network is fine-tuned 23 using back propagation algorithm. Besides, hyper-parameters are analysed through stability 24 analysis to assure the network can find the optimal. Finally, the experiments on Lorenz chaos series, 25 CATS benchmark and other real world like CO2 and waste water parameters forecasting show that 26 cDBN has the advantage of higher accuracy, simpler structure and faster convergence speed than 27 other methods. 28

The report introduces a constructive learning algorithm for recurrent neural networks, which modifies only the weights to output units in order to achieve the learning task. key words: recurrent neural networks, supervised learning Zusammenfassung. Der Report führt ein konstruktives Lernverfahren für rekurrente neuronale Netze ein, welches zum Erreichen des Lernzieles lediglich die Gewichte der zu den Ausgabeneuronen führenden Verbindungen modifiziert. Stichwörter: rekurrente neuronale Netze, überwachtes Lernen

Artificial neural networks’ (ANNs’) generalization powers have in recent years received admiration of finance researchers and practitioners. Their usage in such areas as bankruptcy prediction, debt-risk assessment, and security-market applications has yielded promising results. With such intensive research and proven ability of the ANN in the area of security-market application and the growing importance of the role of equity securities in Singapore, it has motivated the conceptual development of this work in using the ANN in stock selection. With their proven generalization ability, neural networks are able to infer the characteristics of performing stocks from the historical patterns. The performance of stocks is reflective of the profitability and quality of management of the underlying company. Such information is reflected in financial and technical variables. As such, the ANN is used as a tool to uncover the intricate relationships between the performance of stocks and the related financial and technical variables. Historical data, such as financial variables (inputs) and performance of the stock (output) is used in this ANN application. Experimental results obtained thus far have been very encouraging. IDEA GROUP PUBLISHING This paper appears in the publication, Artificial Neural Networks in Finance and Manufacturing edited by Joarder Kamruzzaman, Rezaul Begg, and Ruhul Sarker© 2006, Idea Group Inc. 701 E. Chocolate Avenue, Suite 200, Hershey PA 17033-1240, USA Tel: 717/533-8845; Fax 717/533-8661; URL-http://www.idea-group.com ITB13014 Improving Returns on Stock Investment through Neural Network Selection 153 Copyright © 2006, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. Introduction With the growing importance in the role of equities to both international and local investors, the selection of attractive stocks is of utmost importance to ensure a good return. Therefore, a reliable tool in the selection process can be of great assistance to these investors. An effective and efficient tool/system gives the investor the competitive edge over others as he/she can identify the performing stocks with minimum effort. In assisting the investors in their decision-making process, both the academics and practitioners have devised trading strategies, rules, and concepts based on fundamental and technical analysis. Innovative investors opt to employ information technology to improve the efficiency in the process. This is done through transforming trading strategies into computer-known language so as to exploit the logical processing power of the computer. This greatly reduces the time and effort in short-listing the list of attractive stocks. In the age where information technology is dominant, such computerized rule-based expert systems have severe limitations that will affect their effectiveness and efficiency. In particular, their inability in handling nonlinear relationships between financial variables and stock prices has been a major shortcoming. However, with the significant advancement in the field of ANNs, these limitations have found a solution. In this work, the generalization ability of the ANN is being harnessed in creating an effective and efficient tool for stock selection. Results of the research in this field have so far been very encouraging. Application of Neural Network in Stock Investment One of the earliest studies was by Halquist and Schmoll (1989), who used a neural network model to predict trends in the S&P 500 index. They found that the model was able to predict the trends 61% of the time. This was followed by Trippi and DeSieno (1992) and Grudnitski and Osburn (1993). Trippi and DeSieno (1992) devised an S&P 500 trading system that consisted of several trained neural networks and a set of rules for combining the network results to generate a composite recommended trading strategy. The trading system was used to predict S&P 500 index futures and the results showed that this system significantly outperformed the passive buy-and-hold strategy. Grudnitski and Osburn (1993) used a neural network to predict the monthly price changes and trading return in the S&P 500 index futures. The results showed that the neural network was able to predict correctly 75% of the time and gave a positive return above risk. Another work on predicting S&P 500 index futures was by Tsaih, Hsu, and Lai (1998). Similar to Trippi and DeSieno (1992), Tsaih et al. (1998) also integrated a rule-based system technique with a neural network to produce a trading system. However, in the Tsaih et al. (1998) study, they used reasoning neural networks instead of the backpropagation method used by Trippi and Desieno (1992). Empirical results in the daily prediction of price changes in the S&P 500 index futures showed that this hybrid artificialintelligence (AI) approach outperformed the passive buy-and-hold investment strategy. 11 more pages are available in the full version of this document, which may be purchased using the "Add to Cart" button on the publisher's webpage: www.igi-global.com/chapter/improving-returns-stockinvestment-through/5354

In this paper, we will consider the approximation properties of a recently introduced neural network model called graph neural network (GNN), which can be used to process-structured data inputs, e.g., acyclic graphs, cyclic graphs, and directed or undirected graphs. This class of neural networks implements a function tau(G, n) isin R m that maps a graph G and one of its nodes n onto an m-dimensional Euclidean space. We characterize the functions that can be approximated by GNNs, in probability, up to any prescribed degree of precision. This set contains the maps that satisfy a property called preservation of the unfolding equivalence, and includes most of the practically useful functions on graphs; the only known exception is when the input graph contains particular patterns of symmetries when unfolding equivalence may not be preserved. The result can be considered an extension of the universal approximation property established for the classic feedforward neural networks (FNNs). Some experimental examples are used to show the computational capabilities of the proposed model.

Large multilayer neural networks trained with backpropagation have recently achieved state-ofthe-art results in a wide range of problems. However, using backprop for neural net learning still has some disadvantages, e.g., having to tune a large number of hyperparameters to the data, lack of calibrated probabilistic predictions, and a tendency to overfit the training data. In principle, the Bayesian approach to learning neural networks does not have these problems. However, existing Bayesian techniques lack scalability to large dataset and network sizes. In this work we present a novel scalable method for learning Bayesian neural networks, called probabilistic backpropagation (PBP). Similar to classical backpropagation, PBP works by computing a forward propagation of probabilities through the network and then doing a backward computation of gradients. A series of experiments on ten real-world datasets show that PBP is significantly faster than other techniques, while offering competitive predictive abilities. Our experiments also show that PBP provides accurate estimates of the posterior variance on the network weights.