Question: A swarm intelligence optimized wavelet neural network method for computer-assisted detection of breast cancer in mammograms.

The escalating global cancer burden is significantly influenced by demographic changes and the adoption of cancer-inducing behaviors, particularly smoking, in economically emerging nations. According to the GLOBOCAN 2008 data, an estimated 12.7 million cancer cases and 7.6 million cancer fatalities occurred globally in 2008, with the majority, 56% of the cases and 64% of the deaths, concentrated in the developing world. Breast cancer, accounting for 23% of total cancer cases and 14% of cancer deaths, stands out as the most prevalent cancer and the leading cause of cancer mortality among women. For men, lung cancer is the primary cancer site, making up 17% of new cancer cases and 23% of total cancer deaths. Notably, breast cancer has become the leading cause of cancer death among women in economically developing nations, a shift from the preceding decade where cervical cancer was the primary cause. Moreover, lung cancer mortality among women in developing countries is on par with cervical cancer, each accounting for 11% of total female cancer deaths. Despite the cancer incidence rates in developing countries being half of those in developed countries for both sexes, the cancer mortality rates are strikingly similar. Poorer cancer survival rates in developing nations are largely attributed to late-stage diagnosis and limited access to timely and standard treatment. A significant portion of the global cancer burden could be averted through the implementation of established cancer control strategies and initiatives focusing on tobacco control, vaccination (for liver and cervical cancers), early detection, treatment, and public health campaigns promoting physical activity and a healthier diet. Clinicians, public health officials, and policymakers have a pivotal role in accelerating the global application of these interventions.

Statistical process control (SPC) is a method for improving the quality o f products. Control charting plays a most important role in SPC. SPC control charts arc used for monitoring and detecting unnatural process behaviour. Unnatural patterns in control charts indicate unnatural causes for variations. Control chart pattern recognition is therefore important in SPC. Past research shows that although certain types o f charts, such as the CUSUM chart, might have powerful detection ability, they lack robustness and do not function automatically. In recent years, neural network techniques have been applied to automatic pattern recognition. Spiking Neural Networks (SNNs) belong to the third generation o f artificial neural networks, with spiking neurons as processing elements. In SNNs, time is an important feature for information representation and processing. This thesis proposes the application o f SNN techniques to control chart pattern recognition. It is designed to present an analysis o f the existing learning algorithms o f SNN for pattern recognition and to explain how and why spiking neurons have more computational power in comparison to the previous generation o f neural networks. This thesis focuses on the architecture and the learning procedure o f the network. Four new learning algorithms are presented with their specific architecture: Spiking Learning Vector Quantisation (S-LVQ), Enhanced-Spiking Learning Vector Quantisation (NS-LVQ), S-LVQ with Bees and NS-LVQ with Bees. The latter two algorithms employ a new intelligent swarm-based optimisation called the Bees Algorithm to optimise the LVQ pattern recognition networks. Overall, the aim o f the research is to develop a simple architecture for the proposed network as well as to develop a network that is efficient for application to control chart pattern recognition. Experiments show that the proposed architecture and the learning procedure give high pattern recognition accuracies.

In past years, deep convolutional neural networks (DCNN) have achieved big successes in image classification and object detection, as demonstrated on ImageNet in academic field. However, There are some unique practical challenges remain for real-world image recognition applications, e.g., small size of the objects, imbalanced data distributions, limited labeled data samples, etc. In this work, we are making efforts to deal with these challenges through a computational framework by incorporating latest developments in deep learning. In terms of two-stage detection scheme, pseudo labeling, data augmentation, cross-validation and ensemble learning, the proposed framework aims to achieve better performances for practical image recognition applications as compared to using standard deep learning methods. The proposed framework has recently been deployed as the key kernel for several image recognition competitions organized by Kaggle. The performance is promising as our final private scores were ranked 4 out of 2293 teams for fish recognition on the challenge "The Nature Conservancy Fisheries Monitoring" and 3 out of 834 teams for cervix recognition on the challenge "Intel &MobileODT Cervical Cancer Screening", and several others. We believe that by sharing the solutions, we can further promote the applications of deep learning techniques.

This paper presents evolvable block-based neural networks (BbNNs) for personalized ECG heartbeat pattern classification. A BbNN consists of a 2-D array of modular component NNs with flexible structures and internal configurations that can be implemented using reconfigurable digital hardware such as field-programmable gate arrays (FPGAs). Signal flow between the blocks determines the internal configuration of a block as well as the overall structure of the BbNN. Network structure and the weights are optimized using local gradient-based search and evolutionary operators with the rates changing adaptively according to their effectiveness in the previous evolution period. Such adaptive operator rate update scheme ensures higher fitness on average compared to predetermined fixed operator rates. The Hermite transform coefficients and the time interval between two neighboring R-peaks of ECG signals are used as inputs to the BbNN. A BbNN optimized with the proposed evolutionary algorithm (EA) makes a personalized heartbeat pattern classifier that copes with changing operating environments caused by individual difference and time-varying characteristics of ECG signals. Simulation results using the Massachusetts Institute of Technology/Beth Israel Hospital (MIT-BIH) arrhythmia database demonstrate high average detection accuracies of ventricular ectopic beats (98.1%) and supraventricular ectopic beats (96.6%) patterns for heartbeat monitoring, being a significant improvement over previously reported electrocardiogram (ECG) classification results.

An in-depth analysis of signals from horn antennas used in breast cancer detection research has been carried out. It was found that following the excitation signal, both the double-ridged and the quad-ridged ultra-wideband horn antennas radiate additional unwanted signals that persistently oscillate. Undesirable signal oscillations sources were identified as the horn antenna cavity resonance and inherent antenna LC resonance. These signals interfere with the tumor’s signal response and need to be eliminated for successful detection of the cancerous growth. This paper proposes solutions to remove or minimize these signals without affecting antenna parameters such as bandwidth, gain, ports isolation, and polarization isolation. Modification of the antenna cavity successfully suppressed the unwanted cavity oscillation. Modification of the antenna waveguide reduced inductance and consequently mitigated LC oscillation. The resulting time and frequency domain horn antenna signals demonstrate the effectiveness of the proposed methods. Finally, a breast phantom with a tumor is simulated using signals from the original and a modified horn antenna. The delay and sum method is used to create images. The breast images demonstrate enhanced image quality through the reduction of clutter using the proposed techniques. The signal-to-clutter ratios are 0.448 and 1.6823 dB for the images produced by using the original and modified antennas, respectively.

-A neural network learning procedure has been applied to the classification ~/sonar returns [kom two undersea targets, a metal cylinder and a similarly shaped rock. Networks with an intermediate layer ~/ hidden processing units achieved a classification accuracy as high as 100% on a training set of l04 returns. These net~orks correctly classified up to 90.4% of 104 test returns not contained in the training set. This perfi~rmance was better than that of a nearest neighbor classifier, which was 82.7%. and was close to that of an optimal Bayes classifie~ Specific signal features extracted by hidden units in a trained network were identified and related to coding schemes in the pattern of connection strengths between the input and the hidden units. Network perlbrmance and class[/~cation strategy was comparable to that of trained human listeners. Keywords--Learning algorithms, Hidden units. Multilayered neural network, Sonar, Signal processing.

Early detection of lung nodules in thoracic Computed Tomography (CT) scans is of great importance for the successful diagnosis and treatment of lung cancer. Due to improvements in screening technologies, and an increased demand for their use, radiologists are required to analyze an ever increasing amount of image data, which can affect the quality of their diagnoses. Computer-Aided Detection (CADe) systems are designed to assist radiologists in this endeavor. Here, we present a CADe system for the detection of lung nodules in thoracic CT images. Our system is based on (1) the publicly available Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI) database, which contains 1018 thoracic CT scans with nodules of different shape and size, and (2) a deep Convolutional Neural Network (CNN), which is trained, using the back-propagation algorithm, to extract valuable volumetric features from the input data and detect lung nodules in sub-volumes of CT images. Considering only those test nodules that have been annotated by four radiologists, our CADe system achieves a sensitivity (true positive rate) of 78.9% with 20 false positives (FPs) per scan, or a sensitivity of 71.2% with 10 FPs per scan. This is achieved without using any segmentation or additional FP reduction procedures, both of which are commonly used in other CADe systems. Furthermore, our CADe system is validated on a larger number of lung nodules compared to other studies, which increases the variation in their appearance, and therefore, makes their detection by a CADe system more challenging.

This paper introduces a new methodology for dynamic learning, visualization, and classification of functional magnetic resonance imaging (fMRI) as spatiotemporal brain data. The method is based on an evolving spatiotemporal data machine of evolving spiking neural networks (SNNs) exemplified by the NeuCube architecture [1]. The method consists of several steps: mapping spatial coordinates of fMRI data into a 3-D SNN cube (SNNc) that represents a brain template; input data transformation into trains of spikes; deep, unsupervised learning in the 3-D SNNc of spatiotemporal patterns from data; supervised learning in an evolving SNN classifier; parameter optimization; and 3-D visualization and model interpretation. Two benchmark case study problems and data are used to illustrate the proposed methodology—fMRI data collected from subjects when reading affirmative or negative sentences and another one—on reading a sentence or seeing a picture. The learned connections in the SNNc represent dynamic spatiotemporal relationships derived from the fMRI data. They can reveal new information about the brain functions under different conditions. The proposed methodology allows for the first time to analyze dynamic functional and structural connectivity of a learned SNN model from fMRI data. This can be used for a better understanding of brain activities and also for online generation of appropriate neurofeedback to subjects for improved brain functions. For example, in this paper, tracing the 3-D SNN model connectivity enabled us for the first time to capture prominent brain functional pathways evoked in language comprehension. We found stronger spatiotemporal interaction between left dorsolateral prefrontal cortex and left temporal while reading a negated sentence. This observation is obviously distinguishable from the patterns generated by either reading affirmative sentences or seeing pictures. The proposed NeuCube-based methodology offers also a superior classification accuracy when compared with traditional AI and statistical methods. The created NeuCube-based models of fMRI data are directly and efficiently implementable on high performance and low energy consumption neuromorphic platforms for real-time applications.

Visual inspection of neuroimagery is susceptible to human eye limitations. Computerized methods have been shown to be equally or more e↵ective than human clinicians in diagnosing dementia from neuroimages. Nevertheless, much of the work involves the use of domain expertise to extract hand–crafted features. The key technique in this paper is the use of cross–domain features to represent MRI data. We used a sparse autoencoder to learn a set of bases from natural images and then applied convolution to extract features from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset. Using this new representation, we classify MRI instances into three categories: Alzheimer’s Disease (AD), Mild Cognitive Impairment (MCI) and Healthy Control (HC). Our approach, in spite of being very simple, achieved high classification performance, which is competitive with or better than other approaches.

Due to the distributed nature of Denial-of-Service attacks, it is tremendously challenging to identify such malicious behavior using traditional intrusion detection systems in Wireless Sensor Networks (WSNs). In the current paper, a bio-inspired method is introduced, namely the cooperative-based fuzzy artificial immune system (Co-FAIS). It is a modular-based defense strategy derived from the danger theory of the human immune system. The agents synchronize and work with one another to calculate the abnormality of sensor behavior in terms of context antigen value (CAV) or attackers and update the fuzzy activation threshold for security response. In such a multi-node circumstance, the sniffer module adapts to the sink node to audit data by analyzing the packet components and sending the log file to the next layer. The fuzzy misuse detector module (FMDM) integrates with a danger detector module to identify the sources of danger signals. The infected sources are transmitted to the fuzzy Q-learning vaccination modules (FQVM) in order for particular, required action to enhance system abilities. The Cooperative Decision Making Modules (Co-DMM) incorporates danger detector module with the fuzzy Q-learning vaccination module to produce optimum defense strategies. To evaluate the performance of the proposed model, the Low Energy Adaptive Clustering Hierarchy (LEACH) was simulated using a network simulator. The model was subsequently compared against other existing soft computing methods, such as fuzzy logic controller (FLC), artificial immune system (AIS), and fuzzy Q-learning (FQL), in terms of detection accuracy, counter-defense, network lifetime and energy consumption, to demonstrate its efficiency and viability. The proposed method improves detection accuracy and successful defense rate performance against attacks compared to conventional empirical methods. & 2014 Elsevier Ltd. All rights reserved.

Indexing and classification tools for Content Based Visual Information Retrieval (CBVIR) have been penetrating the universe of medical image analysis. They have been recently investigated for Alzheimer’s disease (AD) diagnosis. This is a normal “knowledge diffusion” process, when methodologies developed for multimedia mining penetrate a new application area. The latter brings its own specificities requiring an adjustment of methodologies on the basis of domain knowledge. In this paper, we develop an automatic classification framework for AD recognition in structural Magnetic Resonance Images (MRI). The main contribution of this work consists in considering visual features from the most involved region in AD (hippocampal area) and in using a late fusion to increase precision results. Our approach has been first evaluated on the baseline MR images of 218 subjects from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database and then tested on a 3T weighted contrast MRI obtained from a subsample of a large French epidemiological study: “Bordeaux dataset”. The experimental results show that our classification of patients with AD versus NC (Normal Control) subjects achieves the accuracies of 87 % and 85 % for ADNI subset and “Bordeaux dataset” respectively. For the most challenging group of subjects with the Mild Cognitive Impairment (MCI), we reach accuracies of 78.22 % and 72.23 % for MCI versus NC and MCI versus AD respectively on ADNI. The late fusion scheme improves classification results by 9 % in average for these three categories. Results demonstrate very promising classification performance and simplicity compared to the state-of-the-art volumetric AD diagnosis methods.


Question: A wavelet neural network approach for computer-aided detection of breast cancer on mammograms, optimized by swarm intelligence.

In the realm of skin cancer detection, the computer-aided diagnosis of melanoma has gained considerable attention due to advancements in skin imaging technology and the evolution of image processing techniques. Border detection in dermoscopy images, which plays a pivotal role in this process, has become a critical focus area, as the precision of subsequent diagnostic steps hinges on its accuracy.

This paper introduces a novel, unsupervised method for border detection in dermoscopy images of pigmented skin lesions, utilizing the statistical region merging (SRM) algorithm. The SRM algorithm is particularly well-suited for this task due to its speed and ability to efficiently merge regions based on statistical criteria, ensuring a robust segmentation of the lesion borders.

To assess the effectiveness of the proposed method, a comprehensive evaluation was conducted on a dataset consisting of 90 dermoscopy images. The accuracy of border detection was quantified using a metric that incorporated three sets of borders delineated by dermatologists, serving as the ground-truth. This comparison not only validated the proposed method's performance but also provided a benchmark against which to measure its accuracy.

Furthermore, the method was benchmarked against four contemporary automated approaches: orientation-sensitive fuzzy c-means, dermatologist-like tumor extraction algorithm, meanshift clustering, and the modified JSEG method. The comparative analysis revealed that the proposed SRM-based method excelled in terms of both speed and accuracy, demonstrating its potential as a valuable tool in the computer-aided diagnosis of melanoma.

In conclusion, the integration of the SRM algorithm into border detection for dermoscopy images offers a promising avenue for enhancing the accuracy and efficiency of skin cancer diagnosis. Its fast and accurate performance highlights the potential of SRM as a cornerstone in the development of advanced computer-aided diagnostic systems for melanoma detection.

Bio-inspired evolutionary algorithms are probabilistic search methods that mimic natural biological evolution. They show the behavior of the biological entities interacting locally with one another or with their environment to solve complex problems. This paper aims to analyze the most predominantly used bio-inspired optimization techniques that have been used for stock market prediction and hence present a comparative study between them.

Statistical process control (SPC) is a method for improving the quality o f products. Control charting plays a most important role in SPC. SPC control charts arc used for monitoring and detecting unnatural process behaviour. Unnatural patterns in control charts indicate unnatural causes for variations. Control chart pattern recognition is therefore important in SPC. Past research shows that although certain types o f charts, such as the CUSUM chart, might have powerful detection ability, they lack robustness and do not function automatically. In recent years, neural network techniques have been applied to automatic pattern recognition. Spiking Neural Networks (SNNs) belong to the third generation o f artificial neural networks, with spiking neurons as processing elements. In SNNs, time is an important feature for information representation and processing. This thesis proposes the application o f SNN techniques to control chart pattern recognition. It is designed to present an analysis o f the existing learning algorithms o f SNN for pattern recognition and to explain how and why spiking neurons have more computational power in comparison to the previous generation o f neural networks. This thesis focuses on the architecture and the learning procedure o f the network. Four new learning algorithms are presented with their specific architecture: Spiking Learning Vector Quantisation (S-LVQ), Enhanced-Spiking Learning Vector Quantisation (NS-LVQ), S-LVQ with Bees and NS-LVQ with Bees. The latter two algorithms employ a new intelligent swarm-based optimisation called the Bees Algorithm to optimise the LVQ pattern recognition networks. Overall, the aim o f the research is to develop a simple architecture for the proposed network as well as to develop a network that is efficient for application to control chart pattern recognition. Experiments show that the proposed architecture and the learning procedure give high pattern recognition accuracies.

The advance of deep learning has made huge changes in computer vision and produced various off-the-shelf trained models. Particularly, Convolutional Neural Network (CNN) has been widely used to build image classification model which allow researchers transfer the pre-trained learning model for other classifications. We propose a transfer learning method to detect breast cancer using histopathology images based on Google's Inception v3 model which were initially trained for the classification of non-medical images. The pilot study shows the feasibility of transfer learning in the detection of breast cancer with AUC of 0.93.

It is challenging to develop an intelligent agent-based or robotic system to conduct long-term automatic health monitoring and robust efficient disease diagnosis as autonomous e-Carers in real-world applications. In this research, we aim to deal with such challenges by presenting an intelligent decision support system for skin lesion recognition as the initial step, which could be embedded into an intelligent service robot for health monitoring in home environments to promote early diagnosis. The system is developed to identify benign and malignant skin lesions using multiple steps, including pre-processing such as noise removal, segmentation, feature extraction from lesion regions, feature selection and classification. After extracting thousands of raw shape, colour and texture features from the lesion areas, a Genetic Algorithm (GA) is used to identify the most discriminating significant feature subsets for healthy and cancerous cases. A Support Vector Machine classifier has been employed to perform benign and malignant lesion recognition. Evaluated with 1300 images from the Dermofit dermoscopy image database, the empirical results indicate that our approach achieves superior performance in comparison to other related research reported in the literature.

Artificial bee colony algorithm is a new population-based evolutionary method based on the intelligent behavior of honey bee swarm. It has shown more effective than other biological-inspired algorithms. However, there are still insufficiencies in ABC algorithm, which is good at exploration but poor at exploitation and its convergence speed is also an issue in some cases. For these insufficiencies, we propose a novel artificial bee colony algorithm (NABC) for numerical optimization problems in this paper to improve the exploitation capability by incorporating the current best solution into the search procedure. Experiments are conducted on a set of unimodal/multimodal benchmark functions. The experiments results of NABC have been compared with Gbest-guided artificial bee colony algorithm (G-ABC), improved artificial bee colony algorithm (I-ABC), Elitist artificial bee colony algorithm (E-ABC). The results show that NABC is superior to those algorithms in most of the tested functions.

The development of data-mining applications such as classification and clustering has shown the need for machine learning algorithms to be applied to large scale data. In this paper we present the comparison of different classification techniques using Waikato Environment for Knowledge Analysis or in short, WEKA. WEKA is an open source software which consists of a collection of machine learning algorithms for data mining tasks. The aim of this paper is to investigate the performance of different classification or clustering methods for a set of large data. The algorithm or methods tested are Bayes Network, Radial Basis Function, Pruned Tree, Single Conjunctive Rule Learner and Nearest Neighbors Algorithm. A fundamental review on the selected technique is presented for introduction purposes. The data breast cancer data with a total data of 6291 and a dimension of 699 rows and 9 columns will be used to test and justify the differences between the classification methods or algorithms. Subsequently, the classification technique that has the potential to significantly improve the common or conventional methods will be suggested for use in large scale data, bioinformatics or other general applications. Keywords— Machine Learning, Data Mining, WEKA, Classification, Bioinformatics.

Soft computing in the field of agriculture science is being employed with computer vision techniques in order to detect the diseases in crops to increase the overall yield. A Modified Rotation Kernel Transformation(MRKT) based directional feature extraction scheme is presents to resolve the issues occurring due to shape, color or other deceptive features during plant disease recognition. The MRKT based scheme is used to calculate the directional features and histograms for plant parts like leaf and fruit of Mango crop. These histograms and the directional feature set with use of artificial neural network lead to a better recognition technique of Anthracnose disease which occurs in form of black spots on fruits and leaves of mango plant. The results obtained using proposed MRKT directional feature set have shows the better results with accuracy up to 98%.

Convolutional Neural Networks (CNNs) are typically trained in the RGB color space. However, in medical imaging, we believe that pixel stain quantities offer a fundamental view of the interaction between tissues and stain chemicals. Since the optical density (OD) colorspace allows to compute pixel stain quantities from pixel RGB intensities using the Beer-Lambert’s law, we propose a stain deconvolutional layer, hereby named as SD-Layer, affixed at the front of CNN that performs two functions: (1) it transforms the input RGB microscopic images to Optical Density (OD) space and (2) this layer deconvolves OD image with the stain basis learned through backpropagation and provides tissue-specific stain absorption quantities as input to the following CNN layers. With the introduction of only nine additional learnable parameters in the proposed SD-Layer, we obtain a considerably improved performance on two standard CNN architectures: AlexNet and T-CNN. Using the T-CNN architecture prefixed with the proposed SD-Layer, we obtain 5-fold crossvalidation accuracy of 93.2% in the problem of differentiating malignant immature White Blood Cells (WBCs) from normal immature WBCs for cancer detection.

In this paper, a multilevel thresholding method which allows the determination of the appropriate number of thresholds as well as the adequate threshold values is proposed. This method combines a genetic algorithm with a wavelet transform. First, the length of the original histogram is reduced by using the wavelet transform. Based on this lower resolution version of the histogram, the number of thresholds and the threshold values are determined by using a genetic algorithm. The thresholds are then projected onto the original space. In this step, a refinement procedure may be added to detect accurate threshold values. Experiments and comparative results with multilevel thresholding methods over a synthetic histogram and real images show the efficiency of the proposed method. 2007 Elsevier Inc. All rights reserved.

In this work, a proposed hybrid dragonfly algorithm (DA) with extreme learning machine (ELM) system for prediction problem is presented. ELM model is considered a promising method for data regression and classification problems. It has fast training advantage, but it always requires a huge number of nodes in the hidden layer. The usage of a large number of nodes in the hidden layer increases the test/evaluation time of ELM. Also, there is no guarantee of optimality of weights and biases settings on the hidden layer. DA is a recently promising optimization algorithm that mimics the moving behavior of moths. DA is exploited here to select less number of nodes in the hidden layer to speed up the performance of the ELM. It also is used to choose the optimal hidden layer weights and biases. A set of assessment indicators is used to evaluate the proposed and compared methods over ten regression data sets from the UCI repository. Results prove the capability of the proposed DA-ELM model in searching for optimal feature combinations in feature space to enhance ELM generalization ability and prediction accuracy. The proposed model was compared against the set of commonly used optimizers and regression systems. These optimizers are namely, particle swarm optimization (PSO) and genetic algorithm (GA). The proposed DA-ELM model proved an advance overall compared methods in both accuracy and generalization ability.


Question: A wavelet neural network approach for the computer-aided detection of breast cancer on mammograms, optimized by swarm intelligence.

This paper provides an in-depth review of contemporary approaches to lesion border detection in dermoscopy images, a critical aspect of diagnosing melanoma and other pigmented skin conditions. Given the complexities and subjective nature of human interpretation, the automated analysis of dermoscopy images has emerged as a significant field of research. The paper emphasizes the pivotal role of automated lesion border detection in this process.

A detailed survey of recent methodologies for border detection is presented, with a focus on computational efficiency and evaluation metrics. The review highlights common challenges faced by current techniques, such as issues with image acquisition, dataset size, diagnostic diversity, and the evaluation of detection accuracy. Moreover, it notes that dermatologists' ability to delineate lesion borders often relies on higher-level medical knowledge, suggesting that integrating such domain-specific knowledge into automated systems could enhance their performance, particularly in diverse diagnostic scenarios.

In light of these findings, the paper underscores the need for improved methodologies that not only address the computational and evaluation challenges but also incorporate medical expertise to achieve more accurate and reliable border detection in dermoscopy images. This could significantly enhance the diagnostic capabilities of computerized systems in the field of dermatology, potentially leading to earlier and more precise diagnoses of skin lesions.

This paper looks into efficient implementation of one dimensional low pass Finite Impulse Response filters using certain commonly used and state-of-the-art optimization techniques. Methods like Parks-McClellan (PM) equiripple design, Quantum-behaved Particle Swarm Optimization (QPSO) and Cuckoo Search Algorithm (CSA) with Levy Flight are employed and overall performance is further improved by hybridization and adaptive step size update. Various performance metrics are analyzed with a focus on increasing convergence speed to reach global optima faster. It is seen that the improved search methods used in this work, i.e., Simulated Annealing based Weighted Mean Best QPSO (SAWQPSO) and Adaptive CSA (ACSA) effect significant reductions in convergence time with ACSA proving to be the faster one. The designed filter is used in the receiver stage of a Frequency Modulated Radio Transmission model using a Quadrature-Phase Shift Keyed (QPSK) Modulator and Demodulator. Its efficiency is validated by obtaining near perfect correlation between the message and recovered signals.

In this paper, a prototype wavelet-based neuralnetwork classifier for recognizing power-quality disturbances is implemented and tested under various transient events. The discrete wavelet transform (DWT) technique is integrated with the probabilistic neural-network (PNN) model to construct the classifier. First, the multiresolution-analysis technique of DWT and the Parseval’s theorem are employed to extract the energy distribution features of the distorted signal at different resolution levels. Then, the PNN classifies these extracted features to identify the disturbance type according to the transient duration and the energy features. Since the proposed methodology can reduce a great quantity of the distorted signal features without losing its original property, less memory space and computing time are required. Various transient events tested, such as momentary interruption, capacitor switching, voltage sag/swell, harmonic distortion, and flicker show that the classifier can detect and classify different power disturbance types efficiently.

This study proposes the use of deep learning algorithms to detect the presence of skin cancer, specifically melanoma, from images of skin lesions taken by a standard camera. Skin cancer is the most prevalent form of cancer in the US where 3.3 million people get treated each year. The 5-year survival rate of melanoma is 98% when detected and treated early yet over 10,000 people are lost each year due mostly to late-stage diagnoses [2]. Thus, there is a need to make melanoma screening and diagnoses methods cheaper, quicker, simpler, and more accessible. This study aims to produce an inexpensive and fast computer-vision based machine learning tool that can be used by doctors and patients to track and classify suspicious skin lesions as benign or malignant with adequate accuracy using only a cell phone camera. The data set was trained on 3 separate learning models with increasingly improved classification accuracy. The 3 models included logistic regression, a deep neural network, and a fine-tuned, pre-trained, VGG-16 Convolutional Neural Network (CNN) [7]. Preliminary results show the developed algorithm’s ability to segment moles from images with 70% accuracy and classify skin lesions as melanoma with 78% balanced accuracy using a fine-tuned VGG-16 CNN.

We investigate a parallel structure of cat swarm optimization (CSO) in this paper, and we call it parallel cat swarm optimization (PCSO). In the experiments, we compare particle swarm optimization (PSO) with CSO and PCSO. The experimental results indicate that both CSO and PCSO perform well. Moreover, PCSO is an effective scheme to improve the convergent speed of cat swarm optimization in case the population size is small and the whole iteration is less.

This paper compared various MLP activation functions for classification problems. The most well-known (Artificial Neural Network) ANN architecture is the Multilayer Perceptron (MLP) network which is widely used for solving problems related to data classifications. Selection of the activation functions in the MLP network plays an essential role on the network performance. A lot of studies have been conducted by reseachers to investigate special activation function to solve different kind of problems. Therefore, this paper intends to investigate the activation functions in MLP networks in terms of the accuracy performances. The activation functions under investigation are sigmoid, hyperbolic tangent, neuronal, logarithmic, sinusoidal and exponential. Medical diagnosis data from two case studies, thyroid disease classification and breast cancer classification, have been used to test the performance of the MLP network. The MLP networks are trained using Back Propagation learning algorithm. The performance of the MLP networks are calculated based on the percentage of correct classificition. The results show that the hyperbolic tangent function in MLP network had the capability to produce the highest accuracy for classifying breast cancer data. Meanwhile, for thyroid disease classification, neuronal function is the most suitable function that performed the highest accuracy in MLP network.

Decision support systems have been utilised since 1960, providing physicians with fast and accurate means towards more accurate diagnoses and increased tolerance when handling missing or incomplete data. This paper describes the application of neural network models for classification of electroencephalogram (EEG) signals. Decision making was performed in two stages: initially, a feature extraction scheme using the wavelet transform (WT) has been applied and then a learning-based algorithm classifier performed the classification. The performance of the neural model was evaluated in terms of training performance and classification accuracies and the results confirmed that the proposed scheme has potential in classifying the EEG signals

Extreme Learning Machine (ELM) is an elegant technique for training Single-hidden Layer Feedforward Networks (SLFNs) with extremely fast speed that attracts significant interest recently. One potential weakness of ELM is the random generation of the input weights and hidden biases, which may deteriorate the classification accuracy. In this paper, we propose a new Memetic Algorithm (MA) based Extreme Learning Machine (M-ELM) for classification problems. M-ELM uses Memetic Algorithm which is a combination of population-based global optimization technique and individual-based local heuristic search method to find optimal network parameters for ELM. The optimized network parameters will enhance the classification accuracy and generalization performance of ELM. Experiments and comparisons on 22 benchmark data sets demonstrate that M-ELM is able to provide highly competitive results compared with other state-of-the-art varieties of ELM algorithms.

The paper provides an insight into the improved novel metaheuristics of the Firefly Algorithm for constrained continuous optimization tasks. The presented technique is inspired by social behavior of fireflies and the phenomenon of bioluminescent communication. The first part of the paper is devoted to the detailed description of the existing algorithm. Then some suggestions for extending the simple scheme of the technique under consideration are presented. Subsequent sections concentrate on the performed experimental parameter studies and a comparison with existing Particle Swarm Optimization strategy based on existing benchmark instances. Finally some concluding remarks on possible algorithm extensions are given, as well as some properties of the presented approach and comments on its performance in the constrained continuous optimization tasks.

In this work, we examine the strength of deep learning approaches for pathology detection in chest radiographs. Convolutional neural networks (CNN) deep architecture classification approaches have gained popularity due to their ability to learn mid and high level image representations. We explore the ability of CNN learned from a non-medical dataset to identify different types of pathologies in chest x-rays. We tested our algorithm on a 433 image dataset. The best performance was achieved using CNN and GIST features. We obtained an area under curve (AUC) of 0.87-0.94 for the different pathologies. The results demonstrate the feasibility of detecting pathology in chest x-rays using deep learning approaches based on non-medical learning. This is a first-of-its-kind experiment that shows that Deep learning with ImageNet, a large scale non-medical image database may be a good substitute to domain specific representations, which are yet to be available, for general medical image recognition tasks.

The back propagation neural network (BPNN) can easily fall into the local minimum point in time series forecasting. A hybrid approach that combines the adaptive differential evolution (ADE) algorithm with BPNN, called ADE–BPNN, is designed to improve the forecasting accuracy of BPNN. ADE is first applied to search for the global initial connection weights and thresholds of BPNN. Then, BPNN is employed to thoroughly search for the optimal weights and thresholds. Two comparative real-life series data sets are used to verify the feasibility and effectiveness of the hybrid method. The proposed ADE–BPNN can effectively improve forecasting accuracy relative to basic BPNN, autoregressive integrated moving average model (ARIMA), and other hybrid models. 2014 Elsevier Ltd. All rights reserved.


Question: A wavelet neural network approach for computer-aided detection of breast cancer on mammograms, optimized by swarm intelligence.

This paper presents a novel method for segmenting digitized dermatoscopic images using a two-dimensional color clustering approach. The RGB image is initially transformed into the L*u*v* color space for processing. A 2D histogram is generated based on the two principal components and then subjected to smoothing through a Gaussian low-pass filter. The method identifies the maxima locations and computes several features from the histogram's contour lines, including the number of enclosed pixels, the base surface area, and the height of the maximum. These features facilitate the identification of valid clusters, which in turn determine the number of classes for segmentation. The image segmentation process employs an enhanced version of the fuzzy c-means (FCM) clustering technique, which incorporates cluster orientation information. Subsequently, mathematical morphology is applied to clean the segmented image, smoothing the borders of the regions and eliminating small components, resulting in a refined segmentation output.

0957-4174/$ see front matter 2011 Elsevier Ltd. A doi:10.1016/j.eswa.2011.02.075 ⇑ Corresponding author. Tel.: +4773597119. E-mail addresses: hanhong@stud.ntnu.no (H (Y. Wang), Kesheng.wang@ntnu.no (K. Wang), (Y. Chen). One of the most studied problems in the financial investment expert system is the intractability of portfolios. The non-linear constrained portfolio optimization problem with multi-objective functions cannot be efficiently solved using traditionally approaches. This paper presents a meta-heuristic approach to portfolio optimization problem using Particle Swarm Optimization (PSO) technique. The model is tested on various restricted and unrestricted risky investment portfolios and a comparative study with Genetic Algorithms is implemented. The PSO model demonstrates high computational efficiency in constructing optimal risky portfolios. Preliminary results show that the approach is very promising and achieves results comparable or superior with the state of the art solvers. 2011 Elsevier Ltd. All rights reserved.

In the Bacteria Foraging Optimization Algorithm (BFAO), the chemotactic process is randomly set, imposing that the bacteria swarm together and keep a safe distance from each other. In hybrid bacteria foraging optimization algorithm and particle swarm optimization (hBFOA-PSO) algorithm the principle of swarming is introduced in the framework of BFAO. The hBFOA-PSO algorithm is based on the adjustment of each bacterium position according to the neighborhood environment. In this paper, the effectiveness of the hBFOA-PSO algorithm has been tested for Automatic Generation Control (AGC) of an interconnected power system. A widely used linear model of two area non-reheat thermal system equipped with Proportional-Integral (PI) controller is considered initially for the design and analysis purpose. At first, a conventional Integral Time multiply Absolute Error (ITAE) based objective function is considered and the performance of hBFOA-PSO algorithm is compared with PSO, BFOA and GA. Further a modified objective function using ITAE, damping ratio of dominant eigenvalues and settling time with appropriate weight coefficients is proposed to increase the performance of the controller. Further, robustness analysis is carried out by varying the operating load condition and time constants of speed governor, turbine, tieline power in the range of +50% to -50% as well as size and position of step load perturbation to demonstrate the robustness of the proposed hBFOA-PSO optimized PI controller. The proposed approach is also extended to a nonlinear power system model by considering the effect of governor dead band non-linearity and the superiority of the proposed approach is shown by comparing the results of Craziness based Particle Swarm Optimization (CRAZYPSO) approach for the identical interconnected power system. Finally, the study is extended to a three area system considering both thermal and hydro units with different PI coefficients and comparison between ANFIS and proposed approach has been provided.

In this study, a new metaheuristic optimization algorithm, called cuckoo search (CS), is introduced for solving structural optimization tasks. The new CS algorithm in combination with Lévy flights is first verified using a benchmark nonlinear constrained optimization problem. For the validation against structural engineering optimization problems, CS is subsequently applied to 13 design problems reported in the specialized literature. The performance of the CS algorithm is further compared with various algorithms representative of the state of the art in the area. The optimal solutions obtained by CS are mostly far better than the best solutions obtained by the existing methods. The unique search features used in CS and the implications for future research are finally discussed in detail.

Receiver operating characteristic (ROC) analysis is an established method of measuring diagnostic performance in medical imaging studies. Traditionally, artificial neural networks (ANN's) have been applied as a classifier to find one "best" detection rate. Recently researchers have begun to report ROC curve results for ANN classifiers. The current standard method of generating ROC curves for an ANN is to vary the output node threshold for classification. Here, the authors propose a different technique for generating ROC curves for a two class ANN classifier. They show that this new technique generates better ROC curves in the sense of having greater area under the ROC curve (AUC), and in the sense of being composed of a better distribution of operating points.

We propose a robust wavelet domain method for noise filtering in medical images. The proposed method adapts itself to various types of image noise as well as to the preference of the medical expert; a single parameter can be used to balance the preservation of (expert-dependent) relevant details against the degree of noise reduction. The algorithm exploits generally valid knowledge about the correlation of significant image features across the resolution scales to perform a preliminary coefficient classification. This preliminary coefficient classification is used to empirically estimate the statistical distributions of the coefficients that represent useful image features on the one hand and mainly noise on the other. The adaptation to the spatial context in the image is achieved by using a wavelet domain indicator of the local spatial activity. The proposed method is of low complexity, both in its implementation and execution time. The results demonstrate its usefulness for noise suppression in medical ultrasound and magnetic resonance imaging. In these applications, the proposed method clearly outperforms single-resolution spatially adaptive algorithms, in terms of quantitative performance measures as well as in terms of visual quality of the images.

An experimental comparison of evolutionary algorithms and gradient-based methods for the optimal control problem is carried out. The problem is solved separately by Particle swarm optimization, Grey wolf optimizer, Fast gradient descent method, Marquardt method and Adam method. The simulation is performed on a jet aircraft model. The results of each algorithm performance are compared according to the best found value of the fitness function, the mean value and the standard deviation.

Breast cancer leads the list of cancer that act on women worldwide. It starts when cells in the breast begin to build up beyond control. These cells normally create a tumour that can usually be seen on an x-ray or felt as a lump. Analysing and grading the tumour will take up much of a pathologist time. Pathologists have been largely diagnosing disease the same way for the past years, by manually reviewing images under a microscope. Thus, to help the pathologists improve accuracy and significantly change the way breast cancer been diagnosed, this paper presents an automated classification program. BreakHis dataset was used which build of 7909 breast tumor images gathered from 82 patients. This system is developed in order to categorize the cancer cells into two classes of cancer which are benign and malignant. The classification system compared different types of feature extractors using k-nearest neighbours classifier to efficiently observe the performance of the classification system. An extensive set of experiments showed that the overall accuracy rates range from 83% to 86%.

This paper presents a systematic approach for earlier diagnosis of Thyroid disease using back propagation algorithm used in neural network. Back propagation algorithm is a widely used algorithm in this field. ANN has been developed based on back propagation of error used for earlier prediction of disease. ANN was subsequently trained with experimental data and testing is carried out using data that was not used during training process. Results show that outcome of ANN is in good agreement with experimental data; this indicates that developed neural network can be used as an alternative for earlier prediction of a disease. Keywords—Back propagation, decision tree, gradient descent, prediction, Supervised Learning

Internet of Things (IoT) and its relevant technologies have been attracting the attention of researchers from academia, industry, and government in recent years. However, since the requirements of the IoT are quite different from what the Internet today can offer, several innovative techniques have been gradually developed and incorporated into IoT, which is referred to as the Future Internet of Things (FIoT). Among them, how to extract ‘‘data’’ and transfer them into ‘‘knowledge’’ from sensing layer to application layer has become a vital issue. This paper begins with an overview of IoT and FIoT, followed by discussions on how to apply data mining and computational intelligence to FIoT. An intelligent data management framework inspired by swarm optimization will then given. Finally, open issues and future trends of this field

Various meta-heuristic optimization approaches have been recently created and applied in different areas. Many of these approaches are inspired by swarm behaviors in the nature. This paper studies the solving optimization problems using Black Hole Algorithm (BHA) which is a population-based algorithm. Since the performance of this algorithm was not tested in mathematical functions, we have studied this issue using some standard functions. The results of the BHA are compared with the results of GA and PSO algorithms which indicate that the performance of BHA is better than the other two mentioned algorithms.


Question: Analog CMOS-designed resistive processing unit for training deep neural networks.

This paper presents a framework, DistBelief, designed to facilitate the training of deep neural networks with an unprecedented scale of parameters, utilizing extensive CPU resources across thousands of cores. The research underscores the significance of training large models in enhancing performance, particularly in the domains of unsupervised feature learning and deep learning. DistBelief is engineered to leverage computing clusters consisting of thousands of machines to train models of considerable size. Two key algorithms are introduced within this framework: Downpour SGD and Sandblaster.

Downpour SGD is an asynchronous stochastic gradient descent technique that supports a substantial number of model replicas, significantly enhancing the scale and velocity of deep network training. Sandblaster, on the other hand, is a versatile framework that accommodates various distributed batch optimization methods, including a distributed implementation of the L-BFGS algorithm. Both Downpour SGD and Sandblaster L-BFGS contribute to the acceleration of deep network training, enabling the training of a deep network that is 30 times larger than previously reported in academic literature.

To illustrate the efficacy of these methods, the paper highlights their application in achieving state-of-the-art performance on ImageNet, a benchmark task in visual object recognition that encompasses 16 million images and 21k categories. Additionally, the same techniques are demonstrated to drastically expedite the training of a moderately sized deep network for a commercial speech recognition service. While the focus of the paper is on the application of these methods to train large neural networks, it is worth noting that the underlying algorithms are broadly applicable to any gradient-based machine learning algorithm.

Deep convolutional neural networks (CNNs) show extraordinary abilities in artificial intelligence applications, but their large scale of computation usually limits their uses on resource-constrained devices. For CNN’s acceleration, exploiting the data reuse of CNNs is an effective way to reduce bandwidth and energy consumption. Row-stationary (RS) dataflow of Eyeriss is one of the most energy-efficient state-of-the-art hardware architectures, but has redundant storage usage and data access, so the data reuse has not been fully exploited. It also requires complex control and is intrinsically unable to skip over zero-valued inputs in timing. In this paper, we present COSY (CNN on Systolic Array), an energy-efficient hardware architecture based on the systolic array for CNNs. COSY adopts the method of systolic array to achieve the storage sharing between processing elements (PEs) in RS dataflow at the RF level, which reduces low-level energy consumption and on-chip storage. Multiple COSY arrays sharing the same storage can execute multiple 2-D convolutions in parallel, further increasing the data reuse in the low-level storage and improving throughput. To compare the energy consumption of COSY and Eyeriss running actual CNN models, we build a process-based energy consumption evaluation system according to the hardware storage hierarchy. The result shows that COSY can achieve an over 15% reduction in energy consumption under the same constraints, improving the theoretical Energy-Delay Product (EDP) and Energy-Delay Squared Product (ED^2P) by 1.33 x on average. In addition, we prove that COSY has the intrinsic ability for zero-skipping, which can further increase the improvements to 2.25× and 3.83× respectively.

Deployment of convolutional neural networks (ConvNets) in always-on Internet of Everything (IoE) edge devices is severely constrained by the high memory energy consumption of hardware ConvNet implementations. Leveraging the error resilience of ConvNets by accepting bit errors at reduced voltages presents a viable option for energy savings, but few implementations utilize this due to the limited quantitative understanding of how bit errors affect performance. This paper demonstrates the efficacy of SRAM voltage scaling in a 9-layer CIFAR-10 binarized ConvNet processor, achieving memory energy savings of 3.12× with minimal accuracy degradation (∼99% of nominal). Additionally, we quantify the effect of bit error accumulation in a multi-layer network and show that further energy savings are possible by splitting weight and activation voltages. Finally, we compare the measured error rates for the CIFAR-10 binarized ConvNet against MNIST networks to demonstrate the difference in bit error requirements across varying complexity in network topologies and classification tasks.

This paper describes a transmitter driver based on a CMOS inverter with a resistive feedback. By employing the proposed driver topology, the pre-driver can be greatly simplified, resulting in a remarkable reduction of the overall driver power consumption. It also offers another advantage that the implementation of equalization is straightforward, compared with a conventional voltage-mode driver. Furthermore, the output impedance remains relatively constant while the data is being transmitted, resulting in good signal integrity. For evaluation of the driver performance, a fully functional 20 Gb/s transmitter is implemented, including a PRBS generator, a serializer, and a half-rate clock generator. In order to enhance the overall speed of the digital circuits for 20 Gb/s data transmission, the resistive feedback is applied to the time-critical inverters, which enables shorter rise/fall times. The prototype chip is fabricated in a 65 nm CMOS technology. The implemented driver circuit operates up to the data rate of 20 Gb/s, exhibiting an energy efficiency of 0.4 pJ/b for the output swing of 250 mVpp,diff.

This paper explores the use of MOS current-mode logic (MCML) as a fast and low noise alternative to static CMOS circuits in microprocessors, thereby improving the performance, energy efficiency, and signal integrity of future computer systems. The power and ground noise generated by an MCML circuit is typically 10-100× smaller than the noise generated by a static CMOS circuit. Unlike static CMOS, whose dominant dynamic power is proportional to the frequency, MCML circuits dissipate a constant power independent of clock frequency. Although these traits make MCML highly energy efficient when operating at high speeds, the constant static power of MCML poses a challenge for a microarchitecture that operates at the modest clock rate and with a low activity factor. To address this challenge, a single-core microarchitecture for MCML is explored that exploits the C-slow retiming technique, and operates at a high frequency with low complexity to save energy. This design principle contrasts with the contemporary multicore design paradigm for static CMOS that relies on a large number of gates operating in parallel at the modest speeds. The proposed architecture generates 10-40× lower power and ground noise, and operates within 13% of the performance (i.e., 1/ExecutionTime) of a conventional, eight-core static CMOS processor while exhibiting 1.6× lower energy and 9% less area. Moreover, the operation of an MCML processor is robust under both systematic and random variations in transistor threshold voltage and effective channel length.

A hallmark of human intelligence and cognition is its flexibility. One of the longstanding goals in AI research is to replicate this flexibility in a learning machine. In this work we describe a mechanism by which artificial neural networks can learn rapid-temporal adaptation – the ability to adapt quickly to new environments or tasks – that we call adaptive neurons. Adaptive neurons modify their activations with task-specific values retrieved from a working memory. On standard metalearning and few-shot learning benchmarks in both vision and language domains, models augmented with adaptive neurons achieve state-of-the-art results.

As applications such as Apple Siri, Google Now, Microsoft Cortana, and Amazon Echo continue to gain traction, web-service companies are adopting large deep neural networks (DNN) for machine learning challenges such as image processing, speech recognition, natural language processing, among others. A number of open questions arise as to the design of a server platform specialized for DNN and how modern warehouse scale computers (WSCs) should be outfitted to provide DNN as a service for these applications.
 In this paper, we present DjiNN, an open infrastructure for DNN as a service in WSCs, and Tonic Suite, a suite of 7 end-to-end applications that span image, speech, and language processing. We use DjiNN to design a high throughput DNN system based on massive GPU server designs and provide insights as to the varying characteristics across applications. After studying the throughput, bandwidth, and power properties of DjiNN and Tonic Suite, we investigate several design points for future WSC architectures. We investigate the total cost of ownership implications of having a WSC with a disaggregated GPU pool versus a WSC composed of homogeneous integrated GPU servers. We improve DNN throughput by over 120x for all but one application (40x for Facial Recognition) on an NVIDIA K40 GPU. On a GPU server composed of 8 NVIDIA K40s, we achieve near-linear scaling (around 1000x throughput improvement) for 3 of the 7 applications. Through our analysis, we also find that GPU-enabled WSCs improve total cost of ownership over CPU-only designs by 4-20x, depending on the composition of the workload

A new two-terminal circuit element-called the memrirtorcharacterized by a relationship between the charge q(t) s St% i(7J d7 and the flux-linkage (p(t) = J-‘-m vfrj d T is introduced os the fourth boric circuit element. An electromagnetic field interpretation of this relationship in terms of a quasi-static expansion of Maxwell’s equations is presented. Many circuit~theoretic properties of memdstorr are derived. It is shown that this element exhibiis some peculiar behavior different from that exhibited by resistors, inductors, or capacitors. These properties lead to a number of unique applications which cannot be realized with RLC networks alone. I + ” -3 nl Although a physical memristor device without internal power supply has not yet been discovered, operational laboratory models have been built with the help of active circuits. Experimental results ore presented to demonstrate the properties and potential applications of memristors. (a) I. 1NTR00~cnoN I + Y -3 T HIS PAPER presents the logical and scientific basis for the existence of a new two-terminal circuit element called the memristor (a contraction for memory (b) resistor) which has every right to be as basic as the three classical circuit elements already in existence, namely, the resistor, inductor, and capacitor. Although the existence of a memristor in the form of a physical device without internal power supply has not yet been discovered, its laboratory realization in the form of active circuits will be presented in Section II.’ Many interesting circuit-theoretic properties possessed by the memristor, the most important of which is perhaps the passivity property which provides the circuit-theoretic basis for its physical realizability, will be derived in Section III. An electromagnetic field interpretation of the memristor characterization will be presented in Section IV with the help of a quasi-static expansion of Maxwell’s equations. Finally, some novel applications of memristors will be presented in Section V. 1 + ” -3

The goal of this paper is to demonstrate a method for tensorizing neural networks based upon an efficient way of approximating scale invariant quantum states, the Multi-scale Entanglement Renormalization Ansatz (MERA). We employ MERA as a replacement for linear layers in a neural network and test this implementation on the CIFAR-10 dataset. The proposed method outperforms factorization using tensor trains, providing greater compression for the same level of accuracy and greater accuracy for the same level of compression. We demonstrate MERAlayers with 3900 times fewer parameters and a reduction in accuracy of less than 1% compared to the equivalent fully connected layers.

Training deep neural networks requires a large amount of memory, making very deep neural networks difficult to fit on accelerator memories. In order to overcome this limitation, we present a method to reduce the amount of memory for training a deep neural network. The method enables to suppress memory increase during the backward pass, by reusing the memory regions allocated for the forward pass. Experimental results exhibit our method reduced the occupied memory size in training by 44.7% on VGGNet with no accuracy affection. Our method also enabled training speedup by increasing the mini batch size up to double.

The first successful FPGA implementation [1] of artificial neural networks (ANNs) was published a little over a decade ago. It is timely to review the progress that has been made in this research area. This brief survey provides a taxonomy for classifying FPGA implementations of ANNs. Different implementation techniques and design issues are discussed. Future research trends are also presented.


Question: Analog CMOS-designed resistive processing unit for training deep neural networks.

This paper introduces an innovative learning mechanism, back-propagation, tailored for networks composed of units mimicking neuronal functions. The procedure iteratively refines the weights of connections within the network, aiming to reduce the discrepancy between the network's actual output and the targeted output. Through this adjustment process, 'hidden' units, not directly involved in input or output, evolve to encode critical features of the task domain. The network captures the underlying patterns of the task through the interactions of these hidden units. This feature creation capability sets back-propagation apart from previous, more rudimentary methods, such as the perceptron-convergence technique. By enabling the network to learn and utilize these new features, back-propagation significantly enhances the network's adaptability and performance in complex tasks.

Received Nov 5, 2016 Revised Jan 9, 2017 Accepted Feb 18, 2017 Artificial Neural Networks have earned popularity in recent years because of their ability to approximate nonlinear functions. Training a neural network involves minimizing the mean square error between the target and network output. The error surface is nonconvex and highly multimodal. Finding the minimum of a multimodal function is a NP complete problem and cannot be solved completely. Thus application of heuristic global optimization algorithms that computes a good global minimum to neural network training is of interest. This paper reviews the various heuristic global optimization algorithms used for training feedforward neural networks and recurrent neural networks. The training algorithms are compared in terms of the learning rate, convergence speed and accuracy of the output produced by the neural network. The paper concludes by suggesting directions for novel ANN training algorithms based on recent advances in global optimization. Keyword:

Training deep neural networks requires a large amount of memory, making very deep neural networks difficult to fit on accelerator memories. In order to overcome this limitation, we present a method to reduce the amount of memory for training a deep neural network. The method enables to suppress memory increase during the backward pass, by reusing the memory regions allocated for the forward pass. Experimental results exhibit our method reduced the occupied memory size in training by 44.7% on VGGNet with no accuracy affection. Our method also enabled training speedup by increasing the mini batch size up to double.

Binary weight convolutional neural networks (BCNNs) can achieve near state-of-the-art classification accuracy and have far less computation complexity compared with traditional CNNs using high-precision weights. Due to their binary weights, BCNNs are well suited for vision-based Internet-of-Things systems being sensitive to power consumption. BCNNs make it possible to achieve very high throughput with moderate power dissipation. In this paper, an energy-efficient architecture for BCNNs is proposed. It fully exploits the binary weights and other hardware-friendly characteristics of BCNNs. A judicious processing schedule is proposed so that off-chip I/O access is minimized and activations are maximally reused. To significantly reduce the critical path delay, we introduce optimized compressor trees and approximate binary multipliers with two novel compensation schemes. The latter is able to save significant hardware resource, and almost no computation accuracy is compromised. Taking advantage of error resiliency of BCNNs, an innovative approximate adder is developed, which significantly reduces the silicon area and data path delay. Thorough error analysis and extensive experimental results on several data sets show that the approximate adders in the data path cause negligible accuracy loss. Moreover, algorithmic transformations for certain layers of BCNNs and a memory-efficient quantization scheme are incorporated to further reduce the energy cost and on-chip storage requirement. Finally, the proposed BCNN hardware architecture is implemented with the SMIC 130-nm technology. The postlayout results demonstrate that our design can achieve an energy efficiency over 2.0TOp/s/W when scaled to 65 nm, which is more than two times better than the prior art.

We report a 3-D-stackable 1S1R passive cross-point resistive random access memory (RRAM). The sneak (leakage) current challenge in the cross-point RRAM integration has been overcome utilizing a field-assisted superlinear threshold selector. The selector offers high selectivity of >107, sharp switching slope of <;5 mV/decade, ability to tune the threshold voltage, stable operation at 125°C, and endurance of >1011. Furthermore, we demonstrate 1S1R integration in which the selector-subthreshold current is <;10 pA while offering >102 memory ON/OFF ratio and >106 selectivity during cycling. Combined with self-current-controlled RRAM, the 1S1R enables high-density and high-performance memory applications.

This paper analyzes the behavior of stackaugmented recurrent neural network (RNN) models. Due to the architectural similarity between stack RNNs and pushdown transducers, we train stack RNN models on a number of tasks, including string reversal, contextfree language modelling, and cumulative XOR evaluation. Examining the behavior of our networks, we show that stack-augmented RNNs can discover intuitive stack-based strategies for solving our tasks. However, stack RNNs are more difficult to train than classical architectures such as LSTMs. Rather than employ stack-based strategies, more complex networks often find approximate solutions by using the stack as unstructured memory.

The formation of ordered structures, in particular Turing patterns, in complex spatially extended systems has been observed in many different contexts, spanning from natural sciences (chemistry, physics, and biology) to technology (mechanics and electronics). In this paper, it is shown that the use of memristors in a simple cell of a spatially-extended circuit architecture allows us to design systems able to generate Turing patterns. In addition, the memristor parameters play a key role in the selection of the type and characteristics of the emerging pattern, which is also influenced by the initial conditions. The problem of finding the regions of parameters where Turing patterns may emerge in the proposed cellular architecture is solved in an analytic way, and numerical results are shown to illustrate the system behavior with respect to its parameters.

We present in this paper an overview of circuit techniques dedicated to design reliable low-voltage (1-V and below) analog functions in deep submicron standard CMOS processes. The challenges of designing such low-voltage and reliable analog building blocks are addressed both at circuit and physical layout levels. State-ofthe-art circuit topologies and techniques (input level shifting, bulk and current driven, DTMOS), used to build main analog modules (operational amplifier, analog CMOS switches) are covered with the implementation of MOS capacitors.

Crossbar arrays based on non-volatile resistive devices such as resistive RAM and phase change memory have become an important technology due to the applications to memory systems. The energy consumption of integrated circuits has become a primary issue due to thermal constraints in high performance systems and limited battery time in mobile and IoT applications. In this paper, the energy efficiency of a crossbar array of a one-selector-one-resistor (1S1R) configuration during a write operation is explored for the V/2 and V/3 bias schemes. The characteristics that affect the most energy efficient bias scheme are demystified. The write energy of a crossbar array is modeled in terms of the array size, number of selected cells, and the nonlinearity factor. For a specific array size and selector technology, the number of selected cells during a write operation can affect the choice of bias scheme. Moreover, the effect of leakage current due to partially biased unselected cells is explored.

This article provides next step towards solving speed bottleneck of any system that intensively uses convolutions operations (e.g. CNN). Method described in the article is applied on deformable part models (DPM) algorithm. Method described here is based on multidimensional tensors and provides efficient tradeoff between DPM performance and accuracy. Experiments on various databases, including Pascal VOC, show that the proposed method allows decreasing a number of convolutions up to 4.5 times compared with DPM v.5, while maintaining similar accuracy. If insignificant accuracy degradation is allowable, higher computational gain can be achieved. The method consists of filters tensor decomposition and convolutions shortening using the decomposed filter. Mathematical overview of the proposed method as well as simulation results are provided.

As technology scales, the improved speed and energy efficiency make the successiveapproximation-register (SAR) architecture an attractive alternative for applications that require high-speed and high-accuracy analog-to-digital converters (ADCs). In SAR ADCs, the key linearity and speed limiting factors are capacitor mismatch and incomplete digital-to-analog converter (DAC)/reference voltage settling. In this thesis, a sub-radix-2 SAR ADC is presented with several new contributions. The main contributions include investigation of using digital error correction (redundancy) in SAR ADCs for dynamic error correction and speed improvement, development of two new calibration algorithms to digitally correct for manufacturing mismatches, design of new architecture to incorporate redundancy within the architecture itself while achieving 94% better energy efficiency compared to conventional switching algorithm, development of a new capacitor DAC structure to improve the SNR by four times with improved matching, joint design of the analog and digital circuits to create an asynchronous platform in order to reach the targeted performance, and analysis of key circuit blocks to enable the design to meet noise, power and timing requirements. The design is fabricated in standard 1P9M 65nm CMOS technology with 1.2V supply. The active die area is 0.083mm with full rail-to-rail input swing of 2.4VP−P . A 67.4dB SNDR, 78.1dB SFDR, +1.0/-0.9 LSB12 INL and +0.5/-0.7 LSB12 DNL are achieved at 50MS/s at Nyquist rate. The total power consumption, including the estimated calibration and reference power, is 2.1mW, corresponding to 21.9fJ/conv.step FoM. This ADC achieves the best FoM of any ADCs with greater than 10b ENOB and 10MS/s sampling rate. Thesis Supervisor: Duane S. Boning Title: Professor of Electrical Engineering and Computer Science Thesis Supervisor: Hae-Seung Lee Title: Professor of Electrical Engineering and Computer Science


Question: Analog CMOS-designed resistive processing unit for training deep neural networks.

This paper explores the transformative impact of deep learning, a subset of machine learning techniques, on various facets of modern technology and society. From web search algorithms to content filtering on social networks, recommendation systems on e-commerce platforms, and features in consumer electronics like cameras and smartphones, deep learning is playing a pivotal role. Its capabilities in processing raw, natural data sets it apart from conventional machine learning methods, which often required intricate engineering and domain expertise to design feature extractors that transformed raw input into a form suitable for detection or classification.

Representation learning, a key component of deep learning, enables machines to automatically discover the representations necessary for pattern recognition or classification from raw data. Deep learning, as a form of representation learning, involves multiple levels of representation, achieved through the composition of simple yet non-linear modules. These modules transform representations at one level (originating from raw input) into representations at higher, more abstract levels. This process allows for the learning of complex functions by amplifying important aspects of input data for discrimination and suppressing irrelevant variations, especially in classification tasks.

A prime example is in image processing, where the first layer of representation typically detects edges at particular orientations and locations, the second layer identifies motifs by recognizing specific arrangements of edges, and subsequent layers assemble these motifs into larger combinations that correspond to recognizable objects or parts of objects. The critical feature of deep learning is that these hierarchical representations are learned from data, rather than being manually designed by engineers.

Deep learning is achieving significant breakthroughs in solving long-standing problems in artificial intelligence, demonstrating exceptional proficiency in high-dimensional data analysis across various scientific, business, and governmental domains. It has set new benchmarks in image and speech recognition, drug molecule activity prediction, particle accelerator data analysis, brain circuit reconstruction, and the impact of non-coding DNA mutations on gene expression and disease. Notably, it has also shown remarkable promise in natural language understanding tasks such as topic classification, sentiment analysis, question answering, and language translation.

The future prospects of deep learning are bright due to its minimal requirement for human engineering, allowing it to capitalize on the growing availability of computational resources and data. Ongoing advancements in learning algorithms and architectures for deep neural networks are expected to further accelerate its progress, suggesting that deep learning will continue to achieve notable successes across diverse fields.

In summary, deep learning, with its capacity for automatic feature discovery and complex function learning, is reshaping the landscape of artificial intelligence, with implications spanning from image recognition to drug discovery and natural language processing. Its potential for further development is vast, underpinned by its inherent

In a memristor crossbar array, a memristor is positioned on each row-column intersection, and its resistance, low or high, represents two logical states. The state of every memristor can be sensed by the current flowing through the memristor. In this work, we study the sneak path problem in crossbars arrays, in which current can sneak through other cells, resulting in reading a wrong state of the memristor. Our main contributions are a new characterization of arrays free of sneak paths, and efficient methods to read the array cells while avoiding sneak paths. To each read method we match a constraint on the array content that guarantees sneak-path free readout, and calculate the resulting capacity.

This paper presents a recursive digital low-dropout (RLDO) regulator that improves response time, quiescent power, and load regulation dynamic range over prior digital LDO designs by 1–2 orders of magnitude. The proposed RLDO enables a practical digital replacement to analog LDOs by using an SAR-like binary search algorithm in a coarse loop and a sub-LSB pulse width modulation duty control scheme in a fine loop. A proportional-derivative compensation scheme is employed to ensure stable operation independent of load current, the size of the output decoupling capacitor, and clock frequency. Implemented in 0.0023 mm2 in 65 nm CMOS, the 7-bit RLDO achieves, at a 0.5-V input, a response time of 15.1 ns with a figure of merit of 199.4 ps, along with stable operation across a 20 000 $\times $  dynamic load range.

In this paper, we propose a novel Spin Orbit Torque Magnetic Random Access Memory (SOT-MRAM) array design that could simultaneously work as non-volatile memory and implement a reconfigurable in-memory logic (AND, OR) without add-on logic circuits to memory chip as in traditional logic-in-memory designs. The computed logic output could be simply read out like a normal MRAM bit-cell using the shared memory peripheral circuits. Such intrinsic in-memory logic could be used to process data within memory to greatly reduce power-hungry and long distance data communication in conventional Von-Neumann computing systems. We further employ in-memory data encryption using Advanced Encryption Standard (AES) algorithm as a case study to demonstrate the efficiency of the proposed design. The device to architecture co-simulation results show that the proposed design can achieve 70.15% and 80.87% lower energy consumption compared to CMOS-ASIC and CMOL-AES implementations, respectively. It offers almost similar energy consumption as recent DW-AES implementation, but with 60.65% less area overhead.

A key challenge for neural modeling is to explain how a continuous stream of multimodal input from a rapidly changing environment can be processed by stereotypical recurrent circuits of integrate-and-fire neurons in real time. We propose a new computational model for real-time computing on time-varying input that provides an alternative to paradigms based on Turing machines or attractor neural networks. It does not require a task-dependent construction of neural circuits. Instead, it is based on principles of high-dimensional dynamical systems in combination with statistical learning theory and can be implemented on generic evolved or found recurrent circuitry. It is shown that the inherent transient dynamics of the high-dimensional dynamical system formed by a sufficiently large and heterogeneous neural circuit may serve as universal analog fading memory. Readout neurons can learn to extract in real time from the current state of such recurrent neural circuit information about current and past inputs that may be needed for diverse tasks. Stable internal states are not required for giving a stable output, since transient internal states can be transformed by readout neurons into stable target outputs due to the high dimensionality of the dynamical system. Our approach is based on a rigorous computational model, the liquid state machine, that, unlike Turing machines, does not require sequential transitions between well-defined discrete internal states. It is supported, as the Turing machine is, by rigorous mathematical results that predict universal computational power under idealized conditions, but for the biologically more realistic scenario of real-time processing of time-varying inputs. Our approach provides new perspectives for the interpretation of neural coding, the design of experiments and data analysis in neurophysiology, and the solution of problems in robotics and neurotechnology.

Convolutional Neural Networks (CNN) are verycomputation-intensive. Recently, a lot of CNN accelerators based on the CNN intrinsic parallelism are proposed. However, we observed that there is a big mismatch between the parallel types supported by computing engine and the dominant parallel types of CNN workloads. This mismatch seriously degrades resource utilization of existing accelerators. In this paper, we propose aflexible dataflow architecture (FlexFlow) that can leverage the complementary effects among feature map, neuron, and synapse parallelism to mitigate the mismatch. We evaluated our design with six typical practical workloads, it acquires 2-10x performance speedup and 2.5-10x power efficiency improvement compared with three state-of-the-art accelerator architectures. Meanwhile, FlexFlow is highly scalable with growing computing engine scale.

Deep neural networks (DNNs) continue to make significant advances, solving tasks from image classification to translation or reinforcement learning. One aspect of the field receiving considerable attention is efficiently executing deep models in resource-constrained environments, such as mobile or embedded devices. This paper focuses on this problem, and proposes two new compression methods, which jointly leverage weight quantization and distillation of larger networks, called “teachers,” into compressed “student” networks. The first method we propose is called quantized distillation and leverages distillation during the training process, by incorporating distillation loss, expressed with respect to the teacher network, into the training of a smaller student network whose weights are quantized to a limited set of levels. The second method, differentiable quantization, optimizes the location of quantization points through stochastic gradient descent, to better fit the behavior of the teacher model. We validate both methods through experiments on convolutional and recurrent architectures. We show that quantized shallow students can reach similar accuracy levels to state-of-the-art full-precision teacher models, while providing up to order of magnitude compression, and inference speedup that is almost linear in the depth reduction. In sum, our results enable DNNs for resource-constrained environments to leverage architecture and accuracy advances developed on more powerful devices.

A simulation procedure is described for making feasible large-scale simulations of recurrent neural networks of spiking neurons and plastic synapses. The procedure is applicable if the dynamic variables of both neurons and synapses evolve deterministically between any two successive spikes. Spikes introduce jumps in these variables, and since spike trains are typically noisy, spikes introduce stochasticity into both dynamics. Since all events in the simulation are guided by the arrival of spikes, at neurons or synapses, we name this procedure event-driven. The procedure is described in detail, and its logic and performance are compared with conventional (synchronous) simulations. The main impact of the new approach is a drastic reduction of the computational load incurred upon introduction of dynamic synaptic efficacies, which vary organically as a function of the activities of the pre- and postsynaptic neurons. In fact, the computational load per neuron in the presence of the synaptic dynamics grows linearly with the number of neurons and is only about 6 more than the load with fixed synapses. Even the latter is handled quite efficiently by the algorithm. We illustrate the operation of the algorithm in a specific case with integrate-and-fire neurons and specific spike-driven synaptic dynamics. Both dynamical elements have been found to be naturally implementable in VLSI. This network is simulated to show the effects on the synaptic structure of the presentation of stimuli, as well as the stability of the generated matrix to the neural activity it induces.

We propose a quantum generalisation of a classical neural network. The classical neurons are firstly rendered reversible by adding ancillary bits. Then they are generalised to being quantum reversible, i.e. unitary. (The classical networks we generalise are called feedforward, and have step-function activation functions.) The quantum network can be trained efficiently using gradient descent on a cost function to perform quantum generalisations of classical tasks. We demonstrate numerically that it can: (i) compress quantum states onto a minimal number of qubits, creating a quantum autoencoder, and (ii) discover quantum communication protocols such as teleportation. Our general recipe is theoretical and implementation-independent. The quantum neuron module can naturally be implemented photonically.

Inspired by biophysical principles underlying nonlinear dendritic computation in neural circuits, we develop a scheme to train deep neural networks to make them robust to adversarial attacks. Our scheme generates highly nonlinear, saturated neural networks that achieve state of the art performance on gradient based adversarial examples on MNIST, despite never being exposed to adversarially chosen examples during training. Moreover, these networks exhibit unprecedented robustness to targeted, iterative schemes for generating adversarial examples, including second-order methods. We further identify principles governing how these networks achieve their robustness, drawing on methods from information geometry. We find these networks progressively create highly flat and compressed internal representations that are sensitive to very few input dimensions, while still solving the task. Moreover, they employ highly kurtotic weight distributions, also found in the brain, and we demonstrate how such kurtosis can protect even linear classifiers from adversarial attack.

This paper involves designing a small scale battery charging system which is powered via a photovoltaic panel. This work aims at the usage of solar energy for charging the battery and optimizing the power of the system. Implementation is done using Artificial Neural Network (ANN) on FPGA. To develop this system an Artificial Neural Network is trained and its result is further used for the PWM technique. PWM pulse generation has been done using Papilio board which is based on XILINX Spartan 3E FPGA. The ANN with PWM technique is ported on FPGA which is programmed using VHDL. This able to automatically control the whole charging system operation without requirement of external sensory unit. The simulation results are achieved by using MATLAB and XILINX. These results allowed demonstrating the charging of the battery using proposed ANN and PWM technique. KeywordsPhotovoltaic battery charger, PWM, ANN, FPGA