Question: A compliance control strategy using contact force for a trotting quadruped robot.

This paper introduces a novel concept, the Capture Point, which is a crucial step towards enabling humanoid robots to recover from pushes and maintain balance. The challenge of determining when and where to take a step to avoid falling has remained largely unresolved. In pursuit of addressing this, we propose methods to calculate the Capture Point and the Capture Region, which is the area on the ground where a humanoid must step to come to a halt after being pushed. The intersection of the Capture Region with the base of support dictates the appropriate strategy for the robot to successfully regain balance under given conditions. Computing the Capture Region for a humanoid is typically complex, but by simplifying the model of walking, the task becomes more manageable. We expand upon the established linear inverted pendulum model by incorporating a flywheel body, demonstrating how to accurately determine the Capture Region for this enhanced model. The addition of rotational inertia allows the humanoid to manage its centroidal angular momentum, akin to human behavior, thereby significantly increasing the Capture Region. We showcase simulations of a basic planar biped that can recover from pushes by stepping into the Capture Region and utilizing internal angular momentum. Our ongoing research involves applying the simplified model's solution as an approximation for more sophisticated simulations of bipedal walking, including a 3D biped with distributed mass, to further refine the Capture Point concept for practical humanoid push recovery.

In this paper we focus on developing a control algorithm for multi-terrain tracked robots with flippers using a reinforcement learning (RL) approach. The work is based on the deep deterministic policy gradient (DDPG) algorithm, proven to be very successful in simple simulation environments. The algorithm works in an end-to-end fashion in order to control the continuous position of the flippers. This end-to-end approach makes it easy to apply the controller to a wide array of circumstances, but the huge flexibility comes to the cost of an increased difficulty of solution. The complexity of the task is enlarged even more by the fact that real multi-terrain robots move in partially observable environments. Notwithstanding these complications, being able to smoothly control a multi-terrain robot can produce huge benefits in impaired people daily lives or in search and rescue situations. Manuscript received 22 September 2017 ar X iv :1 70 9. 08 43 0v 1 [ cs .R O ] 2 5 Se p 20 17

The basic concepts for exoskeletal systems have been suggested for some time with applications ranging from construction, manufacturing and mining to rescue and emergency services. In recent years, research has been driven by possible uses in medical/rehabilitation and military applications. Yet there are still significant barriers to the effective use and exploitation of this technology. Among the most pertinent of these factors is the power and actuation system and its impact of control, strength, speed and, perhaps most critically, safety. This work describes the design, construction and testing of an ultra low-mass, full-body exoskeleton system having seven degrees of freedom (DOFs) for the upper limbs and five degrees of freedom (DOFs) for each of the lower limbs. This low mass is primarily due to the use of a new range of pneumatic muscle actuators as the power source for the system. The work presented will show how the system takes advantage of the inherent controllable compliance to produce a unit that is powerful, providing a wide range of functionality (motion and forces over an extended range) in a manner that has high safety integrity for the user. The general layout of both the upper and the lower body exoskeleton is presented together with results from preliminary experiments to demonstrate the potential of the device in limb retraining, rehabilitation and power assist (augmentation) operations.

The subject of this paper is the design, control, and evaluation of a biped–climbing robot featuring a new hybrid hip joint. The hybrid hip provides both prismatic and revolute motion, discretely, to the robot, using a single actuator. This is intended to improve its adaptability in confined environments and its capability to maneuver over and around obstacles. Optimization of the hybrid hip joint relative to robot size, weight, and actuation limits is considered while maximizing range of motion. The mechanical structure of the robot is discussed, as well as forward and inverse kinematics for motion planning. Reachable robot workspace in both revolute and prismatic phases of the hybrid hip joint is analyzed. Different robot locomotion strides are developed and dynamic controller requirements are considered. Experimental evaluation of the robot walking and climbing on and between surfaces with different inclinations is conducted to evaluate performance and repeatability of the system. KEY WORDS—biped robot, climbing robot, robot design, kinematics, locomotion strides

A novel surgical palpation probe installing a miniature 6-axis force/torque (F/T) sensor is presented for robot-assisted minimally invasive surgery. The 6-axis F/T sensor is developed by using the capacitive transduction principle and a novel capacitance sensing method. The sensor consists of only three parts, namely a sensing printed circuit board, a deformable part, and a base part. The simple configuration leads to simpler manufacturing and assembly processes in conjunction with high durability and low weight. In this study, a surgical instrument installed with a surgical palpation probe is implemented. The 6-axis F/T sensing capability of the probe has been experimentally validated by comparing it with a reference 6-axis F/T sensor. Finally, a vivo tissue palpation task is performed in a simulated surgical environment with an animal organ and a relatively hard simulated cancer buried under the surface of the organ.

This contribution presents the first results of the development of an unmanned aerial vehicle (UAV) which is capable of applying force to a wall while maintaining flight stability. This is a novel idea since UAVs are used so far only for tasks without physical contact to the surrounding objects. The basis for the work presented is a quadrotor system which is stabilized with an inertial measurement unit. As a new approach an additional actuator was added to generate forces in physical contact while the UAV stays horizontal. A control architecture based on ultrasonic distance sensors and a CMOS-camera is proposed. The performance of the system was proved by several flight tests. Potential applications of the system can be physical tasks at high places like cleaning windows or walls as well as rescue or maintenance tasks.

In most 6 DOF robot arms, considerable amounts of gravitational torques due to the robot's own weight are applied to pitch joints of the robot, which causes most arms to use high capacity motors and speed reducers. A spring-based counterbalance mechanism can compensate for this gravitational torque, thus leading to a significant reduction in the effect of gravity. However, a simple installation of counterbalance mechanisms at each pitch joint does not work properly because the gravitational torque at each joint is dependent also on the other joints. To achieve multi-DOF counterbalancing, we propose a parallelogram linkage combined with dual counterbalance mechanisms, each being composed of a slider-crank mechanism and springs. Simulations and experimental results showed that the counterbalance robot arm based on the proposed counterbalance mechanisms effectively reduced the torques required to support the robot mass, thus allowing the prospective use of much smaller motors and speed reducers than traditional industrial robots.

Exoskeletons have potential to improve human quality of life by relieving loads on the human musculoskeletal system or by helping motor rehabilitation. Controllers that were initially developed for industrial applications are also applied to control an exoskeleton, despite of major differences in requirements. Nevertheless, which controller has better performance for this specific application remains an open question. This paper presents a comparison between sliding mode control and feedback linearization control. The implementation of the sliding mode controller assumes a complete measurement of the biological system's dynamics. On the other hand, the feedback linearization method does not need any information from the biological system. Our results indicate that a model combining biological control and dynamics could improve several characteristics of robotic exoskeleton, such as energy consumption.

In this article, we propose a new method for providing assistance during cyclical movements. This method is trajectory-free, in the sense that it provides user assistance irrespective of the performed movement, and requires no other sensing than the assisting robot’s own encoders. The approach is based on adaptive oscillators, i.e., mathematical tools that are capable of learning the high level features (frequency, envelope, etc.) of a periodic input signal. Here we present two experiments that we recently conducted to validate our approach: a simple sinusoidal movement of the elbow, that we designed as a proof-of-concept, and a walking experiment. In both cases, we collected evidence illustrating that our approach indeed assisted healthy subjects during movement execution. Owing to the intrinsic periodicity of daily life movements involving the lower-limbs, we postulate that our approach holds promise for the design of innovative rehabilitation and assistance protocols for the lower-limb, requiring little to no user-specific calibration.

Functional electrical stimulation (FES) and robotic exoskeletons are two important technologies widely used for physical rehabilitation of paraplegic patients. We developed a hybrid rehabilitation system (FEXO Knee) that combined FES and an exoskeleton for swinging movement control of human knee joints. This study proposed a novel cooperative control strategy, which could realize arbitrary distribution of torque generated by FES and exoskeleton, and guarantee harmonic movements. The cooperative control adopted feedfoward control for FES and feedback control for exoskeleton. A parameter regulator was designed to update key parameters in real time to coordinate FES controller and exoskeleton controller. Two muscle groups (quadriceps and hamstrings) were stimulated to generate active torque for knee joint in synchronization with torque compensation from exoskeleton. The knee joint angle and the interactive torque between exoskeleton and shank were used as feedback signals for the control system. Central pattern generator (CPG) was adopted that acted as a phase predictor to deal with phase confliction of motor patterns, and realized synchronization between the two different bodies (shank and exoskeleton). Experimental evaluation of the hybrid FES-exoskeleton system was conducted on five healthy subjects and four paraplegic patients. Experimental results and statistical analysis showed good control performance of the cooperative control on torque distribution, trajectory tracking, and phase synchronization.

This paper presents a locomotion control method based on central pattern generator (CPG) for hexapod walking robot to achieve gait generation with smooth transition. By deriving an analytical limit cycle approximation of the Van der Pol oscillator, a simple diffusive coupling scheme is proposed to construct a ring-shape CPG network with phase-locked behavior. The stability of the proposed network is proved using synchronization analysis with guaranteed uniform ultimate boundedness of the synchronous errors. In contrast to conventional numerical methods in tuning parameters of the CPG network, our method provides an explicit result that could analytically determine the network parameters to yield the rhythmic waveforms with prescribed frequency, amplitude, and phase relationship among neurons. Employing the proposed network to govern the swing/stance phase according to the profile of the resulting CPG signals, a locomotion control strategy for the hexapod robot is further developed to manipulate the leg movements during the gait cycle. By simply adjusting the phase lags of the CPG network, the proposed control strategy is capable of generating stable walking gaits for hexapod robots and achieving smooth transition among the generated gaits. The simulation and experimental results have demonstrated the effectiveness of the proposed locomotion control method.


Question: Convergence analysis of the online gradient method for pi-sigma neural networks incorporating inner-penalty terms.

This paper presents an innovative penalty-function approach to prune feedforward neural networks through weight elimination. The proposed penalty function comprises dual components: one to inhibit the utilization of redundant connections and another to curb the weights of connections from attaining excessively high values. Straightforward guidelines for the removal of weights from the network architecture are also provided. The efficacy of this penalty function is evaluated on a trio of benchmark problems: the contiguity problem, the parity problems, and the monks problems. The pruned networks obtained for these challenges exhibit a lower number of connections compared to previous reports in the scientific literature, showcasing the efficiency of the proposed approach.

A quantitative and practical Bayesian framework is described for learning of mappings in feedforward networks. The framework makes possible (1) objective comparisons between solutions using alternative network architectures, (2) objective stopping rules for network pruning or growing procedures, (3) objective choice of magnitude and type of weight decay terms or additive regularizers (for penalizing large weights, etc.), (4) a measure of the effective number of well-determined parameters in a model, (5) quantified estimates of the error bars on network parameters and on network output, and (6) objective comparisons with alternative learning and interpolation models such as splines and radial basis functions. The Bayesian "evidence" automatically embodies "Occam's razor," penalizing overflexible and overcomplex models. The Bayesian approach helps detect poor underlying assumptions in learning models. For learning models well matched to a problem, a good correlation between generalization ability and the Bayesian evidence is obtained.

This article introduces a class of incremental learning procedures specialized for prediction-that is, for using past experience with an incompletely known system to predict its future behavior. Whereas conventional prediction-learning methods assign credit by means of the difference between predicted and actual outcomes, the new methods assign credit by means of the difference between temporally successive predictions. Although such temporal-difference methods have been used in Samuel's checker player, Holland's bucket brigade, and the author's Adaptive Heuristic Critic, they have remained poorly understood. Here we prove their convergence and optimality for special cases and relate them to supervised-learning methods. For most real-world prediction problems, temporal-difference methods require less memory and less peak computation than conventional methods and they produce more accurate predictions. We argue that most problems to which supervised learning is currently applied are really prediction problems of the sort to which temporal-difference methods can be applied to advantage.

A momentum term is usually included in the simulations of connectionist learning algorithms. Although it is well known that such a term greatly improves the speed of learning, there have been few rigorous studies of its mechanisms. In this paper, I show that in the limit of continuous time, the momentum parameter is analogous to the mass of Newtonian particles that move through a viscous medium in a conservative force field. The behavior of the system near a local minimum is equivalent to a set of coupled and damped harmonic oscillators. The momentum term improves the speed of convergence by bringing some eigen components of the system closer to critical damping. Similar results can be obtained for the discrete time case used in computer simulations. In particular, I derive the bounds for convergence on learning-rate and momentum parameters, and demonstrate that the momentum term can increase the range of learning rate over which the system converges. The optimal condition for convergence is also analyzed.

Gradient descent optimization algorithms, while increasingly popular, are often used as black-box optimizers, as practical explanations of their strengths and weaknesses are hard to come by. This article aims to provide the reader with intuitions with regard to the behaviour of different algorithms that will allow her to put them to use. In the course of this overview, we look at different variants of gradient descent, summarize challenges, introduce the most common optimization algorithms, review architectures in a parallel and distributed setting, and investigate additional strategies for optimizing gradient descent.

We present a novel l1 regularized off-policy convergent TD-learning method (termed RO-TD), which is able to learn sparse representations of value functions with low computational complexity. The algorithmic framework underlying ROTD integrates two key ideas: off-policy convergent gradient TD methods, such as TDC, and a convex-concave saddle-point formulation of non-smooth convex optimization, which enables first-order solvers and feature selection using online convex regularization. A detailed theoretical and experimental analysis of RO-TD is presented. A variety of experiments are presented to illustrate the off-policy convergence, sparse feature selection capability and low computational cost of the RO-TD algorithm.

Received Nov 5, 2016 Revised Jan 9, 2017 Accepted Feb 18, 2017 Artificial Neural Networks have earned popularity in recent years because of their ability to approximate nonlinear functions. Training a neural network involves minimizing the mean square error between the target and network output. The error surface is nonconvex and highly multimodal. Finding the minimum of a multimodal function is a NP complete problem and cannot be solved completely. Thus application of heuristic global optimization algorithms that computes a good global minimum to neural network training is of interest. This paper reviews the various heuristic global optimization algorithms used for training feedforward neural networks and recurrent neural networks. The training algorithms are compared in terms of the learning rate, convergence speed and accuracy of the output produced by the neural network. The paper concludes by suggesting directions for novel ANN training algorithms based on recent advances in global optimization. Keyword:

The multilayer perceptron, when trained as a classifier using backpropagation, is shown to approximate the Bayes optimal discriminant function. The result is demonstrated for both the two-class problem and multiple classes. It is shown that the outputs of the multilayer perceptron approximate the a posteriori probability functions of the classes being trained. The proof applies to any number of layers and any type of unit activation function, linear or nonlinear.

When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This overfitting is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random “dropout” gives big improvements on many benchmark tasks and sets new records for speech and object recognition. A feedforward, artificial neural network uses layers of non-linear hidden units between its inputs and its outputs. By adapting the weights on the incoming connections of these hidden units it learns feature detectors that enable it to predict the correct output when given an input vector [15]. If the relationship between the input and the correct output is complicated and the network has enough hidden units to model it accurately, there will typically be many different settings of the weights that can model the training set almost perfectly, especially if there is only a limited amount of labeled training data. Each of these weight vectors will make different predictions on held-out test data and almost all of them will do worse on the test data than on the training data because the feature detectors have been tuned to work well together on the training data but not on the test data. Overfitting can be reduced by using “dropout” to prevent complex co-adaptations on the training data. On each presentation of each training case, each hidden unit is randomly omitted from the network with a probability of 0.5, so a hidden unit cannot rely on other hidden units being present. Another way to view the dropout procedure is as a very efficient way of performing model averaging with neural networks. A good way to reduce the error on the test set is to average the predictions produced by a very large number of different networks. The standard way to do this is to train many separate networks and then to apply each of these networks to the test data, but this is computationally expensive during both training and testing. Random dropout makes it possible to train a huge number of different networks in a reasonable time. There is almost certainly a different network for each presentation of each training case but all of these networks share the same weights for the hidden units that are present. We use the standard, stochastic gradient descent procedure for training the dropout neural networks on mini-batches of training cases, but we modify the penalty term that is normally used to prevent the weights from growing too large. Instead of penalizing the squared length (L2 norm) of the whole weight vector, we set an upper bound on the L2 norm of the incoming weight vector for each individual hidden unit. If a weight-update violates this constraint, we renormalize the weights of the hidden unit by division. Using a constraint rather than a penalty prevents weights from growing very large no matter how large the proposed weight-update is. This makes it possible to start with a

Optimizing deep neural networks (DNNs) often suffers from the ill-conditioned problem. We observe that the scaling-based weight space symmetry property in rectified nonlinear network will cause this negative effect. Therefore, we propose to constrain the incoming weights of each neuron to be unit-norm, which is formulated as an optimization problem over Oblique manifold. A simple yet efficient method referred to as projection based weight normalization (PBWN) is also developed to solve this problem. PBWN executes standard gradient updates, followed by projecting the updated weight back to Oblique manifold. This proposed method has the property of regularization and collaborates well with the commonly used batch normalization technique. We conduct comprehensive experiments on several widely-used image datasets including CIFAR-10, CIFAR-100, SVHN and ImageNet for supervised learning over the state-of-the-art convolutional neural networks, such as Inception, VGG and residual networks. The results show that our method is able to improve the performance of DNNs with different architectures consistently. We also apply our method to Ladder network for semi-supervised learning on permutation invariant MNIST dataset, and our method outperforms the state-of-the-art methods: we obtain test errors as 2.52%, 1.06%, and 0.91% with only 20, 50, and 100 labeled samples, respectively.

Initial weight choice has been recognized to be an important aspect of the training methodology for sigmoidal feedforward neural networks. In this paper, a new mechanism for weight initialization is proposed. The mechanism distributes the initial input to output weights in a manner that all weights (including thresholds) leading into a hidden layer are uniformly distributed in a region and the center of the region from which the weights are sampled are such that no region overlaps for two distinct hidden nodes. The proposed method is compared against random weight initialization routines on five function approximation tasks using the Resilient Backpropagation (RPROP) algorithm for training. The proposed method is shown to lead to about twice as fast convergence to a pre-specifled goal for training as compared to any of the random weight initialization methods. Moreover, it is shown that at least for these problems the networks reach a deeper minima of the error functional during training and generalizes better than the networks trained whose weights were initialized by random weight initialization methods.


Question: Convergence analysis of the online gradient method for pi-sigma neural networks incorporating inner-penalty terms.

This document explores an innovative technique, referred to as Optimal Brain Damage, which employs information-theoretic principles to develop practical and highly effective methods for adjusting the scale of neural networks. The core concept is to eliminate weights that have minimal impact on the network's performance, leading to several advantages: enhanced generalization, reduced need for training examples, and accelerated learning and classification speeds. The methodology is grounded in utilizing second-derivative data to strike a balance between network complexity and the error rate on the training set. The efficacy of these strategies is substantiated through experimental results on a real-world application, affirming their utility in neural network optimization.

This article introduces a class of incremental learning procedures specialized for prediction-that is, for using past experience with an incompletely known system to predict its future behavior. Whereas conventional prediction-learning methods assign credit by means of the difference between predicted and actual outcomes, the new methods assign credit by means of the difference between temporally successive predictions. Although such temporal-difference methods have been used in Samuel's checker player, Holland's bucket brigade, and the author's Adaptive Heuristic Critic, they have remained poorly understood. Here we prove their convergence and optimality for special cases and relate them to supervised-learning methods. For most real-world prediction problems, temporal-difference methods require less memory and less peak computation than conventional methods and they produce more accurate predictions. We argue that most problems to which supervised learning is currently applied are really prediction problems of the sort to which temporal-difference methods can be applied to advantage.

Feedforward neural networks trained by error backpropagation are examples of nonparametric regression estimators. We present a tutorial on nonparametric inference and its relation to neural networks, and we use the statistical viewpoint to highlight strengths and weaknesses of neural models. We illustrate the main points with some recognition experiments involving artificial data as well as handwritten numerals. In way of conclusion, we suggest that current-generation feedforward neural networks are largely inadequate for difficult problems in machine perception and machine learning, regardless of parallel-versus-serial hardware or other implementation issues. Furthermore, we suggest that the fundamental challenges in neural modeling are about representation rather than learning per se. This last point is supported by additional experiments with handwritten numerals.

Gradient-based approaches to direct policy search in reinf orcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-f unction methods. In this paper we introduceGPOMDP, a simulation-based algorithm for generating a bi sedestimate of the gradient of theaverage rewardin Partially Observable Markov Decision Processes ( POMDPs) controlled by parameterized stochastic policies. A similar algorithm wa s proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm’s chief advantages are tha t i requires storage of only twice the number of policy parameters, uses one free parameter 2 [0; 1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowl edge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter is related to the mixing timeof the controlledPOMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control s paces, multiple-agents, higher-order derivatives, and a version for training stochastic policie s with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient e stimates generated by GPOMDP can be used in both a traditional stochastic gradient algori thm and a conjugate-gradient procedure to find local optima of the average reward.

-Taking advantage of techniques developed by Kolmogorov, we give a direct proof of the universal approximation capabilities of perceptron type networks with two hidden layers. From our proof we derive estimates of numbers of hidden units based on properties of the function being approximated and the accuracy of its approximation. Keywords--Feedforward neural networks, Multilayer perceptron type networks, Sigmoidal activation function, Approximations of continuous functions, Uniform approximation, Universal approximation capabilities, Estimates of number of hidden units, Modulus of continuity.

This study proposes a modified convolutional neural network (CNN) algorithm that is based on dropout and the stochastic gradient descent (SGD) optimizer (MCNN-DS), after analyzing the problems of CNNs in extracting the convolution features, to improve the feature recognition rate and reduce the time-cost of CNNs. The MCNN-DS has a quadratic CNN structure and adopts the rectified linear unit as the activation function to avoid the gradient problem and accelerate convergence. To address the overfitting problem, the algorithm uses an SGD optimizer, which is implemented by inserting a dropout layer into the all-connected and output layers, to minimize cross entropy. This study used the datasets MNIST, HCL2000, and EnglishHand as the benchmark data, analyzed the performance of the SGD optimizer under different learning parameters, and found that the proposed algorithm exhibited good recognition performance when the learning rate was set to [0.05, 0.07]. The performances of WCNN, MLP-CNN, SVM-ELM, and MCNN-DS were compared. Statistical results showed the following: (1) For the benchmark MNIST, the MCNN-DS exhibited a high recognition rate of 99.97%, and the time-cost of the proposed algorithm was merely 21.95% of MLP-CNN, and 10.02% of SVM-ELM; (2) Compared with SVM-ELM, the average improvement in the recognition rate of MCNN-DS was 2.35% for the benchmark HCL2000, and the time-cost of MCNN-DS was only 15.41%; (3) For the EnglishHand test set, the lowest recognition rate of the algorithm was 84.93%, the highest recognition rate was 95.29%, and the average recognition rate was 89.77%.

-By replacing the sigmoid activation function often used in neural networks with an exponential function, a probabilistic neural network ( PNN) that can compute nonlinear decision boundaries which approach the Bayes optimal is formed. Alternate activation functions having similar properties are also discussed. A fourlayer neural network of the type proposed can map any input pattern to any number of classifications. The decision boundaries can be modified in real-time using new data as they become available, and can be implemented using artificial hardware "'neurons" that operate entirely in parallel. Provision is also made for estimating the probability and reliability of a classification as well as making the decision. The technique offers a tremendous speed advantage for problems in which the incremental adaptation time of back propagation is a significant fraction of the total computation time. For one application, the PNN paradigm was 200,000 times faster than back-propagation. Keywords--Neural network, Probability density function, Parallel processor, "Neuron", Pattern recognition, Parzen window, Bayes strategy, Associative memory.

This paper aims to investigate efficient and scalable machine learning algorithms for resolving Non-negative Matrix Factorization (NMF), which is important for many real-world applications, particularly for collaborative filtering and recommender systems. Unlike traditional batch learning methods, a recently proposed online learning technique named "NN-PA" tackles NMF by applying the popular Passive-Aggressive (PA) online learning, and found promising results. Despite its simplicity and high efficiency, NN-PA falls short in at least two critical limitations: (i) it only exploits the first-order information and thus may converge slowly especially at the beginning of online learning tasks; (ii) it is sensitive to some key parameters which are often difficult to be tuned manually, particularly in a practical online learning system. In this work, we present a novel family of online Adaptive Passive-Aggressive (APA) learning algorithms for NMF, named "NN-APA", which overcomes two critical limitations of NN-PA by (i) exploiting second-order information to enhance PA in making more informative updates at each iteration; and (ii) achieving the parameter auto-selection by exploring the idea of online learning with expert advice in deciding the optimal combination of the key parameters in NMF. We theoretically analyze the regret bounds of the proposed method and show its advantage over the state-of-the-art NN-PA method, and further validate the efficacy and scalability of the proposed technique through an extensive set of experiments on a variety of large-scale real recommender systems datasets.

In this paper, we study the global convergence of majorization minimization (MM) algorithms for solving nonconvex regularized optimization problems. MM algorithms have received great attention in machine learning. However, when applied to nonconvex optimization problems, the convergence of MM algorithms is a challenging issue. We introduce theory of the KurdykaLojasiewicz inequality to address this issue. In particular, we show that many nonconvex problems enjoy the KurdykaLojasiewicz property and establish the global convergence result of the corresponding MM procedure. We also extend our result to a well known method that called CCCP (concave-convex procedure).

Efforts to reduce the numerical precision of computations in deep learning training have yielded systems that aggressively quantize weights and activations, yet employ wide high-precision accumulators for partial sums in inner-product operations to preserve the quality of convergence. The absence of any framework to analyze the precision requirements of partial sum accumulations results in conservative design choices. This imposes an upper-bound on the reduction of complexity of multiply-accumulate units. We present a statistical approach to analyze the impact of reduced accumulation precision on deep learning training. Observing that a bad choice for accumulation precision results in loss of information that manifests itself as a reduction in variance in an ensemble of partial sums, we derive a set of equations that relate this variance to the length of accumulation and the minimum number of bits needed for accumulation. We apply our analysis to three benchmark networks: CIFAR-10 ResNet 32, ImageNet ResNet 18 and ImageNet AlexNet. In each case, with accumulation precision set in accordance with our proposed equations, the networks successfully converge to the single precision floating-point baseline. We also show that reducing accumulation precision further degrades the quality of the trained network, proving that our equations produce tight bounds. Overall this analysis enables precise tailoring of computation hardware to the application, yielding areaand power-optimal systems.

This paper presents a recurrent neural circuit for solving quadratic programming problems. The objective is tominimize a quadratic cost function subject to linearconstraints. The proposed circuit employs non-linearfeedback, in the form of unipolar comparators, to introducetranscendental terms in the energy function ensuring fastconvergence to the solution. The proof of validity of the energy function is also provided. The hardware complexity of the proposed circuit comparesfavorably with other proposed circuits for the same task. PSPICE simulation results arepresented for a chosen optimization problem and are foundto agree with the algebraic solution.


Question: Convergence analysis of the online gradient method for pi-sigma neural networks incorporating inner-penalty terms.

This paper presents the Cascade-Correlation Learning Architecture, a novel approach to artificial neural network design and supervised learning. Unlike traditional methods that merely adjust weights within a fixed network topology, Cascade-Correlation starts with a minimal network and dynamically trains and adds new hidden units sequentially, thereby organically building a multi-layered structure. Once a hidden unit is integrated into the network, its input-side weights are locked, transforming the unit into a permanent feature detector. These units can then be utilized for generating outputs or for the creation of more sophisticated feature detectors. The Cascade-Correlation architecture boasts several advantages over conventional algorithms: it exhibits rapid learning, the network autonomously determines its size and topology, it retains its structure even when the training set is altered, and it operates without the need for back-propagation of error signals through the network's connections. This research was partially funded by the National Science Foundation and the Defense Advanced Research Projects Agency (DARPA). It is important to note that the views and conclusions presented in this document are solely those of the authors and do not necessarily reflect the official policies of DARPA or the US Government.

The Cascade-Correlation learning algorithm was conceived to address the limitations of the widely-used back-propagation learning algorithm, particularly its sluggish learning pace. Even on simple benchmark problems, a back-propagation network may require thousands of epochs to learn the desired behavior from examples. Two primary issues, referred to as the step-size problem and the moving target problem, contribute to this sluggishness. The step-size problem arises because the standard back-propagation method calculates only the first derivative of the error function with respect to each weight in the network. This gradient descent approach, while effective in infinitesimal steps, is impractical for fast learning as it requires re-computing the gradient after each tiny step. To achieve faster learning, larger steps are desirable, but choosing a reasonable step size is challenging without information on the higher-order derivatives of the error function. Several strategies have been proposed to tackle the step-size problem, including the use of momentum, conjugate gradient methods, dynamic step-size adjustment, and the computation of an approximation to the second derivative of the error function. Fahlman's quickprop algorithm is one such successful method for handling the step-size problem in back-propagation systems. Quickprop employs a second-order method, akin to Newton's method, to update weights based on two approximations: the assumption that small changes in one weight have minimal impact on the error gradient at other weights, and that the error function with respect to

Hessian-Free optimization (HF) is an approach for unconstrained minimization of real-valued smooth objective functions. Like standard Newton’s method, it uses local quadratic approximations to generate update proposals. It belongs to the broad class of approximate Newton methods that are practical for problems of very high dimensionality, such as the training objectives of large neural networks. Different algorithms that use many of the same key principles have appeared in the literatures of various communities under different names such as Newton-CG, CG-Steihaug, Newton-Lanczos, and Truncated Newton [27, 28], but applications to machine learning and especially neural networks, have been limited or non-existent until recently. With the work of Martens [22] and later Martens and Sutskever [23] it has been demonstrated that such an approach, if carefully designed and implemented, can work very well for optimizing nonconvex functions such as the training objective for deep neural networks and recurrent neural networks (RNNs), given sensible random initializations. This was significant because gradient descent methods have been observed to be very slow and sometimes completely ineffective [16, 4, 17] on these problems, unless special non-random initializations schemes like layer-wise pre-training [16, 15, 3] are used. HF, which is a general optimization-based approach, can be used in conjunction with or as an alternative to existing pre-training methods and is more widely applicable, since it relies on fewer assumptions about the specific structure of the network. In this report we will first describe the basic HF approach, and then examine well-known general purpose performance-improving techniques as well as others that are specific to HF (versus other Truncated-Newton type approaches) or to neural networks. We will also provide practical tips for creating efficient and correct implementations, and discuss the pitfalls which may arise when designing and using an HF-based approach in a particular application.

The long short-term memory (LSTM) network trained by gradient descent solves difficult problems which traditional recurrent neural networks in general cannot. We have recently observed that the decoupled extended Kalman filter training algorithm allows for even better performance, reducing significantly the number of training steps when compared to the original gradient descent training algorithm. In this paper we present a set of experiments which are unsolvable by classical recurrent networks but which are solved elegantly and robustly and quickly by LSTM combined with Kalman filters.

Stochastic gradient algorithms are the main focus of large-scale optimization problems and led to important successes in the recent advancement of the deep learning algorithms. The convergence of SGD depends on the careful choice of learning rate and the amount of the noise in stochastic estimates of the gradients. In this paper, we propose an adaptive learning rate algorithm, which utilizes stochastic curvature information of the loss function for automatically tuning the learning rates. The information about the element-wise curvature of the loss function is estimated from the local statistics of the stochastic first order gradients. We further propose a new variance reduction technique to speed up the convergence. In our experiments with deep neural networks, we obtained better performance compared to the popular stochastic gradient algorithms.1

In this paper we present a new approach for constructing subgradient schemes for different types of nonsmooth problems with convex structure. Our methods are primaldual since they are always able to generate a feasible approximation to the optimum of an appropriately formulated dual problem. Besides other advantages, this useful feature provides the methods with a reliable stopping criterion. The proposed schemes differ from the classical approaches (divergent series methods, mirror descent methods) by presence of two control sequences. The first sequence is responsible for aggregating the support functions in the dual space, and the second one establishes a dynamically updated scale between the primal and dual spaces. This additional flexibility allows to guarantee a boundedness of the sequence of primal test points even in the case of unbounded feasible set. We present the variants of subgradient schemes for nonsmooth convex minimization, minimax problems, saddle point problems, variational inequalities, and stochastic optimization. In all situations our methods are proved to be optimal from the view point of worst-case black-box lower complexity bounds.

Variational methods have been previously explored as a tractable approximation to Bayesian inference for neural networks. However the approaches proposed so far have only been applicable to a few simple network architectures. This paper introduces an easy-to-implement stochastic variational method (or equivalently, minimum description length loss function) that can be applied to most neural networks. Along the way it revisits several common regularisers from a variational perspective. It also provides a simple pruning heuristic that can both drastically reduce the number of network weights and lead to improved generalisation. Experimental results are provided for a hierarchical multidimensional recurrent neural network applied to the TIMIT speech corpus.

This paper aims to investigate efficient and scalable machine learning algorithms for resolving Non-negative Matrix Factorization (NMF), which is important for many real-world applications, particularly for collaborative filtering and recommender systems. Unlike traditional batch learning methods, a recently proposed online learning technique named "NN-PA" tackles NMF by applying the popular Passive-Aggressive (PA) online learning, and found promising results. Despite its simplicity and high efficiency, NN-PA falls short in at least two critical limitations: (i) it only exploits the first-order information and thus may converge slowly especially at the beginning of online learning tasks; (ii) it is sensitive to some key parameters which are often difficult to be tuned manually, particularly in a practical online learning system. In this work, we present a novel family of online Adaptive Passive-Aggressive (APA) learning algorithms for NMF, named "NN-APA", which overcomes two critical limitations of NN-PA by (i) exploiting second-order information to enhance PA in making more informative updates at each iteration; and (ii) achieving the parameter auto-selection by exploring the idea of online learning with expert advice in deciding the optimal combination of the key parameters in NMF. We theoretically analyze the regret bounds of the proposed method and show its advantage over the state-of-the-art NN-PA method, and further validate the efficacy and scalability of the proposed technique through an extensive set of experiments on a variety of large-scale real recommender systems datasets.

Despite their great success, there is still no comprehensive theoretical understanding of learning with Deep Neural Networks (DNNs) or their inner organization. Previous work [Tishby and Zaslavsky (2015)] proposed to analyze DNNs in the Information Plane; i.e., the plane of the Mutual Information values that each layer preserves on the input and output variables. They suggested that the goal of the network is to optimize the Information Bottleneck (IB) tradeoff between compression and prediction, successively, for each layer. In this work we follow up on this idea and demonstrate the effectiveness of the InformationPlane visualization of DNNs. Our main results are: (i) most of the training epochs in standard DL are spent on compression of the input to efficient representation and not on fitting the training labels. (ii) The representation compression phase begins when the training errors becomes small and the Stochastic Gradient Decent (SGD) epochs change from a fast drift to smaller training error into a stochastic relaxation, or random diffusion, constrained by the training error value. (iii) The converged layers lie on or very close to the Information Bottleneck (IB) theoretical bound, and the maps from the input to any hidden layer and from this hidden layer to the output satisfy the IB self-consistent equations. This generalization through noise mechanism is unique to Deep Neural Networks and absent in one layer networks. (iv) The training time is dramatically reduced when adding more hidden layers. Thus the main advantage of the hidden layers is computational. This can be explained by the reduced relaxation time, as this it scales super-linearly (exponentially for simple diffusion) with the information compression from the previous layer. (v) As we expect critical slowing down of the stochastic relaxation near phase transitions on the IB curve, we expect the hidden layers to converge to such critical points.1

Model-free control” and the corresponding “intelligent” PID controllers (iPIDs), which already had many successful concrete applications, are presented here for the first time in an unified manner, where the new advances are taken into account. The basics of model-free control is now employing some old functional analysis and some elementary differential algebra. The estimation techniques become quite straightforward via a recent online parameter identification approach. The importance of iPIs and especially of iPs is deduced from the presence of friction. The strange industrial ubiquity of classic PID’s and the great difficulty for tuning them in complex situations is deduced, via an elementary sampling, from their connections with iPIDs. Several numerical simulations are presented which include some infinite-dimensional systems. They demonstrate not only the power of our intelligent controllers but also the great simplicity for tuning them.

The training of deep neural networks is a high-dimension optimization problem with respect to the loss function of a model. Unfortunately, these functions are of high dimension and non-convex and hence difficult to characterize. In this paper, we empirically investigate the geometry of the loss functions for state-of-the-art networks with multiple stochastic optimization methods. We do this through several experiments that are visualized on polygons to understand how and when these stochastic optimization methods find local minima.

Total variation (TV) is a one of the most popular regularizers for stabilizing the solution of ill-posed inverse problems. This paper proposes a novel proximal-gradient algorithm for minimizing TV regularized least-squares cost functionals. Unlike traditional methods that require nested iterations for computing the proximal step of TV, our algorithm approximates the latter with several simple proximals that have closed form solutions. We theoretically prove that the proposed parallel proximal method achieves the TV solution with arbitrarily high precision at a global rate of converge that is equivalent to the fast proximal-gradient methods. The results in this paper have the potential to enhance the applicability of TV for solving very large-scale imaging inverse problems.


Question: Convergence analysis of the online gradient method for pi-sigma neural networks incorporating inner-penalty terms.

This paper introduces a pioneering technique, Dynamic Node Creation (DNC), aimed at addressing the complexities of training extensive backpropagation networks and optimizing networks with variable hidden layer unit counts. DNC progressively incorporates nodes into the network's hidden layers in a sequential manner until the targeted approximation accuracy is attained. The paper showcases DNC's efficacy through simulations involving parity, symmetry, binary addition, and encoder problem scenarios. Results indicate that DNC not only frequently discovers minimal topologies but also remains within a three-node margin of the absolute minimum. Interestingly, the computational cost associated with DNC is on par with that of conventional backpropagation (BP) networks with equivalent final architectures. A noteworthy observation is that initiating with a lesser number of nodes than required for problem-solving actually facilitates the discovery of a solution. DNC successfully delivered a solution for each tested problem, whereas BP, when applied to similarly large networks with randomized initial weights, failed to replicate certain minimum solutions found by DNC, despite multiple attempts.

This paper studies an adaptive tracking control for a class of nonlinear stochastic systems with unknown functions. The considered systems are in the nonaffine pure-feedback form, and it is the first to control this class of systems with stochastic disturbances. The fuzzy-neural networks are used to approximate unknown functions. Based on the backstepping design technique, the controllers and the adaptation laws are obtained. Compared to most of the existing stochastic systems, the proposed control algorithm has fewer adjustable parameters and thus, it can reduce online computation load. By using Lyapunov analysis, it is proven that all the signals of the closed-loop system are semiglobally uniformly ultimately bounded in probability and the system output tracks the reference signal to a bounded compact set. The simulation example is given to illustrate the effectiveness of the proposed control algorithm.

It is clear that the learning speed of feedforward neural networks is in general far slower than required and it has been a major bottleneck in their applications for past decades. Two key reasons behind may be: (1) the slow gradient-based learning algorithms are extensively used to train neural networks, and (2) all the parameters of the networks are tuned iteratively by using such learning algorithms. Unlike these conventional implementations, this paper proposes a new learning algorithm called extreme learning machine (ELM) for single-hidden layer feedforward neural networks (SLFNs) which randomly chooses hidden nodes and analytically determines the output weights of SLFNs. In theory, this algorithm tends to provide good generalization performance at extremely fast learning speed. The experimental results based on a few artificial and real benchmark function approximation and classification problems including very large complex applications show that the new algorithm can produce good generalization performance in most cases and can learn thousands of times faster than conventional popular learning algorithms for feedforward neural networks. r 2006 Elsevier B.V. All rights reserved.

The multilayer perceptron, when working in auto-association mode, is sometimes considered as an interesting candidate to perform data compression or dimensionality reduction of the feature space in information processing applications. The present paper shows that, for auto-association, the nonlinearities of the hidden units are useless and that the optimal parameter values can be derived directly by purely linear techniques relying on singular value decomposition and low rank matrix approximation, similar in spirit to the well-known Karhunen-Loève transform. This approach appears thus as an efficient alternative to the general error back-propagation algorithm commonly used for training multilayer perceptrons. Moreover, it also gives a clear interpretation of the rôle of the different parameters.

In this paper, we propose a method for solving the distributed optimization problem in which the objective function is the sum of separable convex functions with linear constraints. In our approach, the primal variable is partially updated to make the Method of Multiplier algorithm distributed which is based on the suitable scaling of constraint matrix. The algorithm is then viewed as a dynamical system the convergence analysis of which is done using the passivity concepts of nonlinear control theory. The convexity of the function is related to the passivity of the non-linear functions which is in feedback with the positive real linear system.

Stochastic gradient algorithms are the main focus of large-scale optimization problems and led to important successes in the recent advancement of the deep learning algorithms. The convergence of SGD depends on the careful choice of learning rate and the amount of the noise in stochastic estimates of the gradients. In this paper, we propose an adaptive learning rate algorithm, which utilizes stochastic curvature information of the loss function for automatically tuning the learning rates. The information about the element-wise curvature of the loss function is estimated from the local statistics of the stochastic first order gradients. We further propose a new variance reduction technique to speed up the convergence. In our experiments with deep neural networks, we obtained better performance compared to the popular stochastic gradient algorithms.1

Stochastic gradient optimization is a class of widely used algorithms for training machine learning models. To optimize an objective, it uses the noisy gradient computed from the random data samples instead of the true gradient computed from the entire dataset. However, when the variance of the noisy gradient is large, the algorithm might spend much time bouncing around, leading to slower convergence and worse performance. In this paper, we develop a general approach of using control variate for variance reduction in stochastic gradient. Data statistics such as low-order moments (pre-computed or estimated online) is used to form the control variate. We demonstrate how to construct the control variate for two practical problems using stochastic gradient optimization. One is convex—the MAP estimation for logistic regression, and the other is non-convex—stochastic variational inference for latent Dirichlet allocation. On both problems, our approach shows faster convergence and better performance than the classical approach.

The performance of neural network classifiers is determined by a number of hyperparameters, including learning rate, batch size, and depth. A number of attempts have been made to explore these parameters in the literature, and at times, to develop methods for optimizing them. However, exploration of parameter spaces has often been limited. In this note, I report the results of large scale experiments exploring these different parameters and their interactions. 1 Datasets and Libraries All experiments reported here were carried out using the Torch library [1] and CUDA (some of the experiments have been reproduced on a smaller scale with other libraries). The dataset for all the experiments is MNIST [3, 2]. Characters were deskewed prior to all experiments. Deskewing significantly reduces error rates in nearest neighbor classifiers. Skew corresponds to a simple one-parameter family of linear transformations in feature space and causes decision regions to become highly anisotropic. Without deskewing, differences in performance between different architectures might primarily reduce to their ability to “learn deskewing”. With deskewing, MNIST character classification become more of an instance of a typical classification problem. Prior results on classifying deskewed MNIST data both with neural networks and with other methods are shown in the table below. 2 Logistic vs Softmax Outputs Multi-Layer Perceptrons (MLPs) used for classification usually attempt to approximate posterior probabilities and use those as their discriminant function. Two common approaches to this are the use of least square regression with logistic output units trained with a least square error measure (“logistic outputs”) 1 ar X iv :1 50 8. 02 78 8v 1 [ cs .N E ] 1 2 A ug 2 01 5 Method Test Error Preprocessing Reference Reduced Set SVM deg 5 polynomial 1 deskewing LeCun et al. 1998 SVM deg 4 polynomial 1.1 deskewing LeCun et al. 1998 K-nearest-neighbors, L3 1.22 deskewing, noise removal, blurring, 2 pixel shift Kenneth Wilder, U. Chicago K-nearest-neighbors, L3 1.33 deskewing, noise removal, blurring, 1 pixel shift Kenneth Wilder, U. Chicago 2-layer NN, 300 HU 1.6 deskewing LeCun et al. 1998 Table 1: Other previously reported results on the MNIST database. and a softmax output layer (“softmax outputs”). In the limit of infinite amounts of training data, both approaches converge to true posterior probability estimates. Softmax output layers have the property that they are guaranteed to produce a normalized posterior probability distribution across all classes, while least square regression with logistic output units generates independent probability estimates for each class membership without any guarantees that these probabilities sum up to one. Softmax is often preferred, although there is no obvious theoretical reason why it should yield better discriminant functions or lower classification error for finite training sets. In OCR and speech recognition, some practitioners have observed that logistic outputs yield better posterior probability estimates and better results when combined with probabilistic language models. In addition, when the sum of the posterior probability estimates derived from logistic outputs differs significantly from unity, that is a strong indication that the input lies outside the training set and should be rejected. Figure 1 shows a scatterplot of test vs training error for a large number of MLPs with one hidden layer at different learning rates, different number of hidden units, and different batch sizes. Such scatterplots show what error rates are achievable by the different architectures, hyperparameter choices, initializations, and order of sample presentations. The lowest points in the vertical direction indicate the lowest test set error achievable by the architecture in this set of experiments. The scatterplot shows that logistic outputs achieve test set error rates of about 1.0% vs 1.1% for softmax outputs. At the same time, logistic outputs never achieve zero percent training set error, while softmax outputs frequently do. In order to ascertain that the difference in test set error between the two

This paper aims to investigate efficient and scalable machine learning algorithms for resolving Non-negative Matrix Factorization (NMF), which is important for many real-world applications, particularly for collaborative filtering and recommender systems. Unlike traditional batch learning methods, a recently proposed online learning technique named "NN-PA" tackles NMF by applying the popular Passive-Aggressive (PA) online learning, and found promising results. Despite its simplicity and high efficiency, NN-PA falls short in at least two critical limitations: (i) it only exploits the first-order information and thus may converge slowly especially at the beginning of online learning tasks; (ii) it is sensitive to some key parameters which are often difficult to be tuned manually, particularly in a practical online learning system. In this work, we present a novel family of online Adaptive Passive-Aggressive (APA) learning algorithms for NMF, named "NN-APA", which overcomes two critical limitations of NN-PA by (i) exploiting second-order information to enhance PA in making more informative updates at each iteration; and (ii) achieving the parameter auto-selection by exploring the idea of online learning with expert advice in deciding the optimal combination of the key parameters in NMF. We theoretically analyze the regret bounds of the proposed method and show its advantage over the state-of-the-art NN-PA method, and further validate the efficacy and scalability of the proposed technique through an extensive set of experiments on a variety of large-scale real recommender systems datasets.

We consider deep neural networks, in which the output of each node is a quadratic function of its inputs. Similar to other deep architectures, these networks can compactly represent any function on a finite training set. The main goal of this paper is the derivation of an efficient layer-by-layer algorithm for training such networks, which we denote as the Basis Learner. The algorithm is a universal learner in the sense that the training error is guaranteed to decrease at every iteration, and can eventually reach zero under mild conditions. We present practical implementations of this algorithm, as well as preliminary experimental results. We also compare our deep architecture to other shallow architectures for learning polynomials, in particular kernel learning.

Abstr,,ct. In this paper we demonstrate that finite linear combinations of compositions of a fixed, univariate function and a set ofaffine functionals can uniformly approximate any continuous function of n real variables with support in the unit hypercube; only mild conditions are imposed on the univariate function. Our results settle an open question about representability in the class of single bidden layer neural networks. In particular, we show that arbitrary decision regions can be arbitrarily well approximated by continuous feedforward neural networks with only a single internal, hidden layer and any continuous sigmoidal nonlinearity. The paper discusses approximation properties of other possible types of nonlinearities that might be implemented by artificial neural networks.


Question: Convergence analysis of the online gradient method for pi-sigma neural networks incorporating inner-penalty terms.

This document presents an efficient technique for pruning neural networks trained via back-propagation, focusing on assessing the global error function's sensitivity to the presence or absence of each synapse within the artificial neural network architecture. The method introduces shadow arrays to monitor the incremental adjustments to synaptic weights during a single iteration of back-propagation learning. Synapses are subsequently ranked based on decreasing sensitivity values, enabling the network's streamlined reduction by eliminating the least significant connections from the sorted list. Distinct from prior methods, this straightforward pruning procedure avoids altering the cost function, does not disrupt the learning process, and imposes a minimal computational burden, making it a practical solution for optimizing neural network efficiency without compromising performance.

-Taking advantage of techniques developed by Kolmogorov, we give a direct proof of the universal approximation capabilities of perceptron type networks with two hidden layers. From our proof we derive estimates of numbers of hidden units based on properties of the function being approximated and the accuracy of its approximation. Keywords--Feedforward neural networks, Multilayer perceptron type networks, Sigmoidal activation function, Approximations of continuous functions, Uniform approximation, Universal approximation capabilities, Estimates of number of hidden units, Modulus of continuity.

Optimizing task-related mathematical model is one of the most fundamental methodologies in statistic and learning areas. However, generally designed schematic iterations may hard to investigate complex data distributions in real-world applications. Recently, training deep propagations (i.e., networks) has gained promising performance in some particular tasks. Unfortunately, existing networks are often built in heuristic manners, thus lack of principled interpretations and solid theoretical supports. In this work, we provide a new paradigm, named Propagation and Optimization based Deep Model (PODM), to bridge the gaps between these different mechanisms (i.e., model optimization and deep propagation). On the one hand, we utilize PODM as a deeply trained solver for model optimization. Different from these existing network based iterations, which often lack theoretical investigations, we provide strict convergence analysis for PODM in the challenging nonconvex and nonsmooth scenarios. On the other hand, by relaxing the model constraints and performing end-to-end training, we also develop a PODM based strategy to integrate domain knowledge (formulated as models) and real data distributions (learned by networks), resulting in a generic ensemble framework for challenging real-world applications. Extensive experiments verify our theoretical results and demonstrate the superiority of PODM against these state-of-the-art approaches.

In this survey we discuss various approximation-theoretic problems that arise in the multilayer feedforward perceptron (MLP) model in neural networks. The MLP model is one of the more popular and practical of the many neural network models. Mathematically it is also one of the simpler models. Nonetheless the mathematics of this model is not well understood, and many of these problems are approximation-theoretic in character. Most of the research we will discuss is of very recent vintage. We will report on what has been done and on various unanswered questions. We will not be presenting practical (algorithmic) methods. We will, however, be exploring the capabilities and limitations of this model. In the rst two sections we present a brief introduction and overview of neural networks and the multilayer feedforward perceptron model. In Section 3 we discuss in great detail the question of density. When does this model have the theoretical ability to approximate any reasonable function arbritrarily well? In Section 4 we present conditions for simultaneously approximating a function and its derivatives. Section 5 considers the interpolation capability of this model. In Section 6 we study upper and lower bounds on the order of approximation of this model. The material presented in Sections 3{6 treats the single hidden layer MLP model. In Section 7 we discuss some of the diierences that arise when considering more than one hidden layer. The lengthy list of references includes many papers not cited in the text, but relevant to the subject matter of this survey.

This paper presents evolutionary optimization of the linear quadratic regulator (LQR) for a voltage-source inverter with an LC output filter. The procedure involves particle-swarm-based search for the best weighting factors in the quadratic cost function. It is common practice that the weights in the cost function are set using the guess-and-check method. However, it becomes quite challenging, and usually very time-consuming, if there are many auxiliary states added to the system. In order to immunize the system against unbalanced and nonlinear loads, oscillatory terms are incorporated into the control scheme, and this significantly increases the number of weights to be guessed. All controller gains are determined altogether in one LQR procedure call, and the originality reported here refers to evolutionary tuning of the weighting matrix. There is only one penalty factor to be set by the designer during the controller synthesis procedure. This coefficient enables shaping the dynamics of the closed-loop system by penalizing the dynamics of control signals instead of selecting individual weighting factors for augmented state vector components. Simulational tuning and experimental verification (the physical converter at the level of 21 kVA) are included.

The true online TD(λ) algorithm has recently been proposed (van Seijen and Sutton, 2014) as a universal replacement for the popular TD(λ) algorithm, in temporal-difference learning and reinforcement learning. True online TD(λ) has better theoretical properties than conventional TD(λ), and the expectation is that it also results in faster learning. In this paper, we put this hypothesis to the test. Specifically, we compare the performance of true online TD(λ) with that of TD(λ) on challenging examples, random Markov reward processes, and a real-world myoelectric prosthetic arm. We use linear function approximation with tabular, binary, and non-binary features. We assess the algorithms along three dimensions: computational cost, learning speed, and ease of use. Our results confirm the strength of true online TD(λ): 1) for sparse feature vectors, the computational overhead with respect to TD(λ) is minimal; for non-sparse features the computation time is at most twice that of TD(λ), 2) across all domains/representations the learning speed of true online TD(λ) is often better, but never worse than that of TD(λ), and 3) true online TD(λ) is easier to use, because it does not require choosing between trace types, and it is generally more stable with respect to the step-size. Overall, our results suggest that true online TD(λ) should be the first choice when looking for an efficient, general-purpose TD method.

The multilayer perceptron, when trained as a classifier using backpropagation, is shown to approximate the Bayes optimal discriminant function. The result is demonstrated for both the two-class problem and multiple classes. It is shown that the outputs of the multilayer perceptron approximate the a posteriori probability functions of the classes being trained. The proof applies to any number of layers and any type of unit activation function, linear or nonlinear.

Nonconvex and nonsmooth problems have recently received considerable attention in signal/image processing, statistics and machine learning. However, solving the nonconvex and nonsmooth optimization problems remains a big challenge. Accelerated proximal gradient (APG) is an excellent method for convex programming. However, it is still unknown whether the usual APG can ensure the convergence to a critical point in nonconvex programming. In this paper, we extend APG for general nonconvex and nonsmooth programs by introducing a monitor that satisfies the sufficient descent property. Accordingly, we propose a monotone APG and a nonmonotone APG. The latter waives the requirement on monotonic reduction of the objective function and needs less computation in each iteration. To the best of our knowledge, we are the first to provide APG-type algorithms for general nonconvex and nonsmooth problems ensuring that every accumulation point is a critical point, and the convergence rates remain O ( 1 k2 ) when the problems are convex, in which k is the number of iterations. Numerical results testify to the advantage of our algorithms in speed.

Training deep neural network is a high dimensional and a highly non-convex optimization problem. In this paper, we revisit Hessian-free optimization method for deep networks with negative curvature direction detection. We also develop its distributed variant and demonstrate superior scaling potential to SGD, which allows more efficiently utilizing larger computing resources thus enabling large models and faster time to obtain desired solution. We show that these techniques accelerate the training process for both the standard MNIST dataset and also the TIMIT speech recognition problem, demonstrating robust performance with upto an order of magnitude larger batch sizes. This increased scaling potential is illustrated with near l inear speed-up on upto 32 CPU nodes for a simple 4-layer network.

In this paper we explore acceleration techniques for large scale nonconvex optimization problems with special focuses on deep neural networks. The extrapolation scheme is a classical approach for accelerating stochastic gradient descent for convex optimization, but it does not work well for nonconvex optimization typically. Alternatively, we propose an interpolation scheme to accelerate nonconvex optimization and call the method Interpolatron. We explain motivation behind Interpolatron and conduct a thorough empirical analysis. Empirical results on DNNs of great depths (e.g., 98-layer ResNet and 200-layer ResNet) on CIFAR-10 and ImageNet show that Interpolatron can converge much faster than the stateof-the-art methods such as the SGD with momentum and Adam. Furthermore, Anderson’s acceleration, in which mixing coefficients are computed by least-squares estimation, can also be used to improve the performance. Both Interpolatron and Anderson’s acceleration are easy to implement and tune. We also show that Interpolatron has linear convergence rate under certain regularity assumptions.

This paper introduces a novel algorithm to approximate the matrix with minimum nuclear norm among all matrices obeying a set of convex constraints. This problem may be understood as the convex relaxation of a rank minimization problem and arises in many important applications as in the task of recovering a large matrix from a small subset of its entries (the famous Netflix problem). Off-the-shelf algorithms such as interior point methods are not directly amenable to large problems of this kind with over a million unknown entries. This paper develops a simple first-order and easy-to-implement algorithm that is extremely efficient at addressing problems in which the optimal solution has low rank. The algorithm is iterative, produces a sequence of matrices {Xk ,Y k}, and at each step mainly performs a soft-thresholding operation on the singular values of the matrix Y k. There are two remarkable features making this attractive for low-rank matrix completion problems. The first is that the soft-thresholding operation is applied to a sparse matrix; the second is that the rank of the iterates {Xk} is empirically nondecreasing. Both these facts allow the algorithm to make use of very minimal storage space and keep the computational cost of each iteration low. On the theoretical side, we provide a convergence analysis showing that the sequence of iterates converges. On the practical side, we provide numerical examples in which 1, 000 × 1, 000 matrices are recovered in less than a minute on a modest desktop computer. We also demonstrate that our approach is amenable to very large scale problems by recovering matrices of rank about 10 with nearly a billion unknowns from just about 0.4% of their sampled entries. Our methods are connected with the recent literature on linearized Bregman iterations for 1 minimization, and we develop a framework in which one can understand these algorithms in terms of well-known Lagrange multiplier algorithms.


Question: Simulation and evaluation of a single-phase AC-DC boost PFC converter with a passive snubber for enhancing power quality.

A Novel Soft-Switched Continuous-Conduction-Mode (CCM) Boost Converter for High-Power Applications

This article presents a new soft-switched continuous-conduction-mode (CCM) boost converter, specifically designed for high-power applications, including power factor correction, hybrid electric vehicles, and fuel cell power conversion systems. The innovative converter ensures zero-voltage-switched (ZVS) turn-on for active switches in CCM and zero-current-switched turn-off for diodes, significantly reducing reverse-recovery losses. Compared to the conventional zero-voltage-transition converter, the voltage ratings of components and the energy volumes of passive components are notably reduced. The voltage conversion ratio is nearly doubled compared to a standard boost converter. The article also explores the extension of this concept to develop multiphase dc-dc converters. The effectiveness of the proposed converter is substantiated through experimental results obtained from a 1.5-kW prototype. This novel approach not only enhances efficiency but also offers significant improvements in power quality, making it a promising solution for high-power applications.

The self-oscillating flyback converter is a popular circuit for cost-sensitive applications due to its simplicity and low component count. This paper presents self-oscillating fly-back converter for mobile batteries charging applications. The converter is designed to step-down the high voltage input to an acceptable level from 220 VAC to 7VDC. By adding indictor and capacitor the power factor and input currant increased, and became 500mA, where the old input currant was 250mA, Diode was connected between Emitter and collector, so the output voltage was improved, and has less ripple, in addition to the soft switching was obtained, simulation model is developed by using MULTISIM software. The operation principle of this converter is described using some operating waveforms based on simulation results and a detailed circuit operation analysis. Analytical results are verified by laboratory experiment. With an input of 220 VAC, 500mA and an output of 7V DC, the efficiency of circuit was75%.

Based on the combination of a three-phase diode bridge and a dc/dc boost converter, a new three-phase threeswitch three-level pulsewidth modulated (PWM) rectifier system is developed. It can be characterized by sinusoidal mains current consumption, controlled output voltage, and low-blocking voltage stress on the power transistors. The application could be, e.g., for feeding the dc link of a telecommunications power supply module. The stationary operational behavior, the control of the mains currents, and the control of the output voltage are analyzed. Finally, the stresses on the system components are determined by digital simulation and compared to the stresses in a conventional six-switch two-level PWM rectifier system.

For a three-phase buck-type pulsewidth modulation rectifier input stage of a high-power telecommunications power supply module, a differential-mode (DM) electromagnetic compatibility (EMC) filter is designed for compliance to CISPR 22 Class B in the frequency range of 150 kHz-30 MHz. The design is based on a harmonic analysis of the rectifier input current and a mathematical model of the measurement procedure including the line impedance stabilization network (LISN) and the test receiver. Guidelines for a successful filter design are given, and components for a 5-kW rectifier prototype are selected. Furthermore, formulas for the estimation of the quasi-peak detector output based on the LISN output voltage spectrum are provided. The damping of filter resonances is optimized for a given attenuation in order to facilitate a higher stability margin for system control. Furthermore, the dependence of the filter input and output impedances and the attenuation characteristic on the inner mains impedance are discussed. As experimentally verified by using a three-phase common-/Differential-Mode separator, this procedure allows accurate prediction of the converter DM conducted emission levels and therefore could be employed in the design process of the rectifier system to ensure compliance to relevant EMC standards

AC/DC converters for worldwide deployment or multiple applications are often required to operate with single-or three-phase AC connections at a variety of voltages. Electric vehicle charging is an excellent example: a typical AC/DC converter optimized for a 400 V three-phase European connection will only produce about 35% of rated power when operating on a 240 V single-phase connection in North America. This paper proposes a reconfigurable AC-DC converter with integral DC-DC stage that allows the single-phase AC power to roughly double compared to a standard configuration, up to 2/3 of the DC/DC stage rating. The system is constructed with two conventional back-to-back three-phase two-level bridges. When operating in single-phase mode, rather than leaving the DC side under-utilized, our proposed design repurposes one of the DC phase legs and the unused AC phase to create two parallel single-phase connections. This roughly doubles the AC output current and hence doubles the single phase AC power.

In order to obtain a high step-up voltage gain, high-efficiency converter, this paper proposed a dual switches dc/dc converter with three-winding-coupled inductor and charge pump. The proposed converter composed of dual switches structure, three-winding-coupled inductor, and charge pump. This combination facilitates realization of high step-up voltage gain with a low voltage/current stress on the power switches. Meanwhile, the voltage across the diodes is low and the diode reverse-recovery problem is alleviated by the leakage inductance of the three-winding-coupled inductor. Taking all these into consideration, the efficiency can be high. This paper illustrated the operation principle of the proposed converter; discussed the effect of leakage inductance on voltage gain; the conditions of zero current shutting off of the diodes are illustrated; the voltage and current stress of the power devices are shown; a comparison between the performance of the proposed converter and previous high step-up converters was conducted. Finally, a prototype rated at 500 W has been established, and the experimental results verify the correctness of the analysis.

Bidirectional bridgeless PFC is an advantageous topology for high power application among many bridgeless PFCs due to small EMI filter size and no reverse recovery problem. However due to the large junction capacitance of secondary leg's diodes, the dead angle is shown in the zero-crossing area. Thus, the input current distortion is intensified, and the unnecessary switching loss decreases the efficiency. In this paper, a simple gate skipping technique is implemented to the zero-crossing area of the bidirectional bridgeless PFC. As a result, the unnecessary switching loss of boost switches and the input current distortion are reduced. The improvement of the proposed method is verified by experimental results with the high line input and 800W(400V/2A) output prototype.

This paper proposes a single-phase, single-stage buck–boost inverter for photovoltaic (PV) systems. The presented topology has one common terminal in input and output ports which eliminates common mode leakage current problem in the grid-connected PV applications. Although it uses four switches, its operation is bimodal and only two switches receive high-frequency pulse width modulation signals in each mode. Its principle of operation is described in detail with the help of equivalent circuits. Its dynamic model is presented, based on which a bimodal controller is designed. Experimental results, in stand-alone and grid-connected mode, obtained with a 300-W laboratory prototype are presented to validate its performance.

Uninterruptible power supplies (UPSs) are widely used to supply critical loads, such as airline computers and life-support systems in hospitals, providing protection against power failure or anomalies of power-line voltage. In general, there are two types of traditional single phase UPSs. The first one couples a battery bank to a half or full-bridge inverter with a low-frequency transformer. In this type of UPSs, the ac output voltage is higher than that of the battery bank; thus, a step-up transformer is required to boost voltage. .The second one couples a battery bank to a dc/dc booster with a half or full bridge inverter. In this type of UPSs, the additional booster is needed, leading to high cost and low efficiency. The controlling of the switches in the booster also complicates the system. Due to the presence of the step-up transformer, the inverter current is much higher than the load current, causing high current stress on the switches of the inverter The dead time in the pulse width-modulation (PWM) signals to prevent the upper and lower switches at the same phase leg from shooting through has to be provided in the aforementioned two types of UPSs, and it distorts the voltage waveform of the ac output voltage. In order to overcome the above problems in the traditional UPSs, new topology of the UPS is proposed by using a voltage source inverter and Zsource inverter. With this new topology, the proposed UPS offers the advantages over the traditional UPSs such as the dc/dc booster and the inverter have been combined into one single-stage power conversion and the distortion of the ac output-voltage waveform is reduced in the absence of dead time in the PWM signals. The effectiveness of the proposed method is implemented by using MATLAB/SIMULINK software.

This paper is mainly concerned with a high frequency soft-switching PWM inverter suitable for consumer induction heating system. Proposed system is composed of soft switching chopper based voltage boost PFC input stage and phase shifted PWM controlled full bridge ZCZVS high frequency inverter stage. Its fundamentals of operating performances are illustrated and evaluated on the experimental results. Its effectiveness is substantially proved on the basis of the experimental results from a practical point of view.

This paper investigates the soft start-up for high frequency LLC resonant converter with optimal trajectory control. Two methods are proposed to realize soft start-up for high frequency LLC converter by commercial low-cost microcontrollers (MCU). Both methods can achieve soft start-up with minimum stress and optimal energy delivery. One method is mixed-signal implementation by sensing resonant tank to minimize the digital delay. Another method is digital implementation with look-up table. Experimental results are demonstrated on 500kHz 1kW 400V/12V LLC converter.