Question: AMP-inspired deep networks for solving sparse linear inverse problems.

This paper proposes a novel strategy to integrate the strengths of model-based methods and deep neural networks while mitigating their respective weaknesses. Model-based approaches allow for the seamless integration of domain knowledge through constraints, albeit at the cost of complex inference processes. Conversely, deterministic deep neural networks facilitate straightforward inference but sacrifice the ease of incorporating domain-specific insights. Our aim is to synthesize the benefits of both paradigms by transforming the iterative inference of a model-based method into a layered structure akin to a neural network. This transformation, termed "unfolding," involves decoupling the model parameters across layers, resulting in innovative architectures that can be trained using gradient-based methods. The outcome is a formula that combines the representational power of deep networks with the structured approach of model-based methods, enabling inference within a fixed number of optimized layers. We demonstrate this framework's application to non-negative matrix factorization, yielding a new non-negative deep neural network structure that can be trained with a multiplicative backpropagation-style algorithm. Through experiments in speech enhancement, we illustrate that our model surpasses conventional neural networks in performance while requiring a significantly reduced number of parameters. This enhancement, we posit, is attributed to the framework's capacity to embed problem-level assumptions into the deep network's architecture. arXiv.org This work may not be duplicated or reproduced without permission. Permission to copy in part or whole for non-commercial educational and research purposes is granted, provided that all copies acknowledge the permission granted by Mitsubishi Electric Research Laboratories, Inc., recognize the authors and their contributions, and retain all applicable portions of the copyright notice. Reproduction or republishing for any other purpose requires a license and payment of fees to Mitsubishi Electric Research Laboratories, Inc. All rights are reserved. Copyright © Mitsubishi Electric Research Laboratories, Inc., 2014. 201 Broadway, Cambridge, Massachusetts 02139.

Matrix operations are a fundamental building block of many computational tasks in fields as diverse as scientific computing, machine learning, and data mining. Matrix inversion is an important matrix operation, but it is difficult to implement in today's popular parallel dataflow programming systems, such as MapReduce. The reason is that each element in the inverse of a matrix depends on multiple elements in the input matrix, so the computation is not easily partitionable. In this paper, we present a scalable and efficient technique for matrix inversion in MapReduce. Our technique relies on computing the LU decomposition of the input matrix and using that decomposition to compute the required matrix inverse. We present a technique for computing the LU decomposition and the matrix inverse using a pipeline of MapReduce jobs. We also present optimizations of this technique in the context of Hadoop. To the best of our knowledge, our technique is the first matrix inversion technique using MapReduce. We show experimentally that our technique has good scalability, enabling us to invert a 10^5 x 10^5 matrix in 5 hours on Amazon EC2. We also show that our technique outperforms ScaLAPACK, a state-of-the-art linear algebra package that uses MPI.

Designing and implementing efficient, provably correct parallel machine learning (ML) algorithms is challenging. Existing high-level parallel abstractions like MapReduce are insufficiently expressive while low-level tools like MPI and Pthreads leave ML experts repeatedly solving the same design challenges. By targeting common patterns in ML, we developed GraphLab, which improves upon abstractions like MapReduce by compactly expressing asynchronous iterative algorithms with sparse computational dependencies while ensuring data consistency and achieving a high degree of parallel performance. We demonstrate the expressiveness of the GraphLab framework by designing and implementing parallel versions of belief propagation, Gibbs sampling, Co-EM, Lasso and Compressed Sensing. We show that using GraphLab we can achieve excellent parallel performance on large scale real-world problems.

To what extent is the success of deep visualization due to the training? Could we do deep visualization using untrained, random weight networks? To address this issue, we explore new and powerful generative models for three popular deep visualization tasks using untrained, random weight convolutional neural networks. First we invert representations in feature spaces and reconstruct images from white noise inputs. The reconstruction quality is statistically higher than that of the same method applied on well trained networks with the same architecture. Next we synthesize textures using scaled correlations of representations in multiple layers and our results are almost indistinguishable with the original natural texture and the synthesized textures based on the trained network. Third, by recasting the content of an image in the style of various artworks, we create artistic images with high perceptual quality, highly competitive to the prior work of Gatys et al. on pretrained networks. To our knowledge this is the first demonstration of image representations using untrained deep neural networks. Our work provides a new and fascinating tool to study the representation of deep network architecture and sheds light on new understandings on deep visualization.

With advances of recent technologies, augmented reality systems and autonomous vehicles gained a lot of interest from academics and industry. Both these areas rely on scene geometry understanding, which usually requires depth map estimation. However, in case of systems with limited computational resources, such as smartphones or autonomous robots, high resolution dense depth map estimation may be challenging. In this paper, we study the problem of semi-dense depth map interpolation along with low resolution depth map upsampling. We present an end-to-end learnable residual convolutional neural network architecture that achieves fast interpolation of semi-dense depth maps with different sparse depth distributions: uniform, sparse grid and along intensity image gradient. We also propose a loss function combining classical mean squared error with perceptual loss widely used in intensity image super-resolution and style transfer tasks. We show that with some modifications, this architecture can be used for depth map super-resolution. Finally, we evaluate our results on both synthetic and real data, and consider applications for autonomous vehicles and creating AR/MR video games.

Multi-modal sparse coding has played an important role in many multimedia applications, where data are usually with multiple modalities. Recently, various multi-modal sparse coding approaches have been proposed to learn sparse codes of multi-modal data, which assume that data appear in all modalities, or at least there is one modality containing all data. However, in real applications, it is often the case that some modalities of the data may suffer from missing information and thus result in partial multi-modality data. In this paper, we propose to solve the partial multi-modal sparse coding problem via multi-modal similarity structure regularization. Specifically, we propose a partial multi-modal sparse coding framework termed Adaptive Partial Multi-Modal Similarity Structure Regularization for Sparse Coding (AdaPM2SC), which preserves the similarity structure within the same modality and between different modalities. Experimental results conducted on two real-world datasets demonstrate that AdaPM2SC significantly outperforms the state-of-the-art methods under partial multi-modality scenario.

We present a new learning algorithm for Boltzmann machines that contain many layers of hidden variables. Data-dependent expectations are estimated using a variational approximation that tends to focus on a single mode, and dataindependent expectations are approximated using persistent Markov chains. The use of two quite different techniques for estimating the two types of expectation that enter into the gradient of the log-likelihood makes it practical to learn Boltzmann machines with multiple hidden layers and millions of parameters. The learning can be made more efficient by using a layer-by-layer “pre-training” phase that allows variational inference to be initialized with a single bottomup pass. We present results on the MNIST and NORB datasets showing that deep Boltzmann machines learn good generative models and perform well on handwritten digit and visual object recognition tasks.

We present an approach for the verification of feed-forward neural networks in which all nodes have a piece-wise linear activation function. Such networks are often used in deep learning and have been shown to be hard to verify for modern satisfiability modulo theory (SMT) and integer linear programming (ILP) solvers. The starting point of our approach is the addition of a global linear approximation of the overall network behavior to the verification problem that helps with SMT-like reasoning over the network behavior. We present a specialized verification algorithm that employs this approximation in a search process in which it infers additional node phases for the non-linear nodes in the network from partial node phase assignments, similar to unit propagation in classical SAT solving. We also show how to infer additional conflict clauses and safe node fixtures from the results of the analysis steps performed during the search. The resulting approach is evaluated on collision avoidance and handwritten digit recognition case studies.

One of the mysteries in the success of neural networks is randomly initialized first order methods like gradient descent can achieve zero training loss even though the objective function is non-convex and non-smooth. This paper demystifies this surprising phenomenon for two-layer fully connected ReLU activated neural networks. For anm hidden node shallow neural network with ReLU activation and n training data, we show as long asm is large enough and no two inputs are parallel, randomly initialized gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function. Our analysis relies on the following observation: over-parameterization and random initialization jointly restrict every weight vector to be close to its initialization for all iterations, which allows us to exploit a strong convexity-like property to show that gradient descent converges at a global linear rate to the global optimum. We believe these insights are also useful in analyzing deep models and other first order methods.

Demosaicking and denoising are the key first stages of the digital imaging pipeline but they are also a severely ill-posed problem that infers three color values per pixel from a single noisy measurement. Earlier methods rely on hand-crafted filters or priors and still exhibit disturbing visual artifacts in hard cases such as moiré or thin edges. We introduce a new data-driven approach for these challenges: we train a deep neural network on a large corpus of images instead of using hand-tuned filters. While deep learning has shown great success, its naive application using existing training datasets does not give satisfactory results for our problem because these datasets lack hard cases. To create a better training set, we present metrics to identify difficult patches and techniques for mining community photographs for such patches. Our experiments show that this network and training procedure outperform state-of-the-art both on noisy and noise-free data. Furthermore, our algorithm is an order of magnitude faster than the previous best performing techniques.


Question: AMP-inspired deep networks for solving sparse linear inverse problems.

This document details the development of TensorFlow, a versatile system designed to facilitate large-scale machine learning tasks on a diverse array of distributed computing environments. The system, engineered by a team of researchers and engineers from Google, is specifically architected to handle the complexities of machine learning algorithms across heterogeneous platforms. TensorFlow's capabilities extend beyond conventional frameworks, offering a robust solution for training and deploying machine learning models in various settings, from single machines to large-scale distributed systems. The system's design allows for the efficient execution of computational graphs, which represent machine learning models, on a wide range of hardware configurations, including CPUs, GPUs, and specialized accelerators. This flexibility and scalability make TensorFlow an ideal choice for researchers and practitioners aiming to solve complex machine learning problems, particularly those involving large datasets and complex models. The paper outlines the system's architecture, its key features, and how it supports the development and deployment of machine learning models across different computing environments, emphasizing its ability to handle the computational demands of deep learning and other advanced machine learning techniques.

Deep learning thrives with large neural networks and large datasets. However, larger networks and larger datasets result in longer training times that impede research and development progress. Distributed synchronous SGD offers a potential solution to this problem by dividing SGD minibatches over a pool of parallel workers. Yet to make this scheme efficient, the per-worker workload must be large, which implies nontrivial growth in the SGD minibatch size. In this paper, we empirically show that on the ImageNet dataset large minibatches cause optimization difficulties, but when these are addressed the trained networks exhibit good generalization. Specifically, we show no loss of accuracy when training with large minibatch sizes up to 8192 images. To achieve this result, we adopt a linear scaling rule for adjusting learning rates as a function of minibatch size and develop a new warmup scheme that overcomes optimization challenges early in training. With these simple techniques, our Caffe2-based system trains ResNet50 with a minibatch size of 8192 on 256 GPUs in one hour, while matching small minibatch accuracy. Using commodity hardware, our implementation achieves ∼90% scaling efficiency when moving from 8 to 256 GPUs. This system enables us to train visual recognition models on internetscale data with high efficiency.

Regularized training of an autoencoder typically results in hidden unit biases that take on large negative values. We show that negative biases are a natural result of using a hidden layer whose responsibility is to both represent the input data and act as a selection mechanism that ensures sparsity of the representation. We then show that negative biases impede the learning of data distributions whose intrinsic dimensionality is high. We also propose a new activation function that decouples the two roles of the hidden layer and that allows us to learn representations on data with very high intrinsic dimensionality, where standard autoencoders typically fail. Since the decoupled activation function acts like an implicit regularizer, the model can be trained by minimizing the reconstruction error of training data, without requiring any additional regularization.

Sparse representation based modeling has been successfully used in many image-related inverse problems such as deblurring, super-resolution and compressive sensing. The heart of sparse representations lies on how to find a space (spanned by a dictionary of atoms) where the local image patch exhibits high sparsity and how to determine the image local sparsity. To identify the locally varying sparsity, it is necessary to locally adapt the dictionary learning process and the sparsity-regularization parameters. However, spatial adaptation alone runs into the risk of over-fitting the data because variation and invariance are two sides of the same coin. In this work, we propose two sets of complementary ideas for regularizing image reconstruction process: (1) the sparsity regularization parameters are locally estimated for each coefficient and updated along with adaptive learning of PCA-based dictionaries; (2) a nonlocal self-similarity constraint is introduced into the overall cost functional to improve the robustness of the model. An efficient alternative minimization algorithm is present to solve the proposed objective function and then an effective image reconstruction algorithm is presented. The experimental results on image deblurring, super-resolution and compressive sensing demonstrate that the proposed image reconstruct method outperforms many existing image reconstruction methods in both PSNR and visual quality assessment. & 2012 Elsevier B.V. All rights reserved.

Extreme learning machine (ELM) has become a popular topic in machine learning in recent years. ELM is a new kind of single-hidden layer feedforward neural network with an extremely low computational cost. ELM, however, has two evident drawbacks: 1) the output weights solved by Moore-Penrose generalized inverse is a least squares minimization issue, which easily suffers from overfitting and 2) the accuracy of ELM is drastically sensitive to the number of hidden neurons so that a large model is usually generated. This brief presents a sparse Bayesian approach for learning the output weights of ELM in classification. The new model, called Sparse Bayesian ELM (SBELM), can resolve these two drawbacks by estimating the marginal likelihood of network outputs and automatically pruning most of the redundant hidden neurons during learning phase, which results in an accurate and compact model. The proposed SBELM is evaluated on wide types of benchmark classification problems, which verifies that the accuracy of SBELM model is relatively insensitive to the number of hidden neurons; and hence a much more compact model is always produced as compared with other state-of-the-art neural network classifiers.

A one-layer recurrent neural network with a discontinuous activation function is proposed for linear programming. The number of neurons in the neural network is equal to that of decision variables in the linear programming problem. It is proven that the neural network with a sufficiently high gain is globally convergent to the optimal solution. Its application to linear assignment is discussed to demonstrate the utility of the neural network. Several simulation examples are given to show the effectiveness and characteristics of the neural network.

We consider the problem of estimating sparse graphs by a lasso penalty applied to the inverse covariance matrix. Using a coordinate descent procedure for the lasso, we develop a simple algorithm--the graphical lasso--that is remarkably fast: It solves a 1000-node problem ( approximately 500,000 parameters) in at most a minute and is 30-4000 times faster than competing methods. It also provides a conceptual link between the exact problem and the approximation suggested by Meinshausen and Bühlmann (2006). We illustrate the method on some cell-signaling data from proteomics.

Multiple problems, including high computational cost, spurious local minima, and solutions with no geologic sense, have prevented widespread application of full waveform inversion (FWI), especially FWI of seismic reflections. These problems are fundamentally related to a large number of model parameters and to the absence of low frequencies in recorded seismograms. Instead of inverting for all the parameters in a dense model, image-guided full waveform inversion inverts for a sparse model space that contains far fewer parameters. We represent a model with a sparse set of values, and from these values, we use image-guided interpolation (IGI) and its adjoint operator to compute finelyand uniformly-sampled models that can fit recorded data in FWI. Because of this sparse representation, image-guided FWI updates more blocky models, and this blockiness in the model space mitigates the absence of low frequencies in recorded data. Moreover, IGI honors imaged structures, so image-guided FWI built in this way yields models that are geologically sensible.

Training deep neural networks requires a large amount of memory, making very deep neural networks difficult to fit on accelerator memories. In order to overcome this limitation, we present a method to reduce the amount of memory for training a deep neural network. The method enables to suppress memory increase during the backward pass, by reusing the memory regions allocated for the forward pass. Experimental results exhibit our method reduced the occupied memory size in training by 44.7% on VGGNet with no accuracy affection. Our method also enabled training speedup by increasing the mini batch size up to double.

Generative Adversarial Nets (GANs) and Variational Auto-Encoders (VAEs) provide impressive image generations from Gaussian white noise, but the underlying mathematics are not well understood. We compute deep convolutional network generators by inverting a fixed embedding operator. Therefore, they do not require to be optimized with a discriminator or an encoder. The embedding is Lipschitz continuous to deformations so that generators transform linear interpolations between input white noise vectors into deformations between output images. This embedding is computed with a wavelet Scattering transform. Numerical experiments demonstrate that the resulting Scattering generators have similar properties as GANs or VAEs, without learning a discriminative network or an encoder.

Modern convolutional networks, incorporating rectifiers and max-pooling, are neither smooth nor convex; standard guarantees therefore do not apply. Nevertheless, methods from convex optimization such as gradient descent and Adam are widely used as building blocks for deep learning algorithms. This paper provides the first convergence guarantee applicable to modern convnets, which furthermore matches a lower bound for convex nonsmooth functions. The key technical tool is the neural Taylor approximation – a straightforward application of Taylor expansions to neural networks – and the associated Taylor loss. Experiments on a range of optimizers, layers, and tasks provide evidence that the analysis accurately captures the dynamics of neural optimization. The second half of the paper applies the Taylor approximation to isolate the main difficulty in training rectifier nets – that gradients are shattered – and investigates the hypothesis that, by exploring the space of activation configurations more thoroughly, adaptive optimizers such as RMSProp and Adam are able to converge to better solutions.


Question: AMP-inspired deep networks for solving sparse linear inverse problems.

This paper introduces a novel approach to channel estimation in massive MIMO systems, specifically addressing the issue of pilot contamination. Pilot contamination is a critical challenge in massive MIMO due to the inability to accurately estimate the channel for all links, particularly those from adjacent cells causing interference. To overcome this limitation, we propose a method that focuses on estimating the parameters of the desired links within a target cell, while ignoring the interference links. This estimation problem is inherently underdetermined, yet by capitalizing on the unique propagation characteristics of massive MIMO systems, it becomes feasible to accurately estimate these channel parameters. Our approach is grounded in the observation that in cellular networks, the channel from user equipment to a base station is typically composed of a limited number of clustered paths in space. With a large antenna array, signals can be sharply localized in space, leading to observations that, when transformed into the beam domain through Fourier transform, exhibit sparsity. In other words, the channel matrix is predominantly composed of small values, with a few significant components. Leveraging this sparsity, we employ Gaussian-mixture Bayesian learning methods to reconstruct the sparse channel components using a reduced number of observations. Experimental results demonstrate that our method significantly outperforms conventional estimators in terms of channel estimation accuracy and achievable rates, even in scenarios where pilot contamination is present.

This paper presents a Quadtree Convolutional Neural Network (QCNN) for efficiently learning from image datasets representing sparse data such as handwriting, pen strokes, freehand sketches, etc. Instead of storing the sparse sketches in regular dense tensors, our method decomposes and represents the image as a linear quadtree that is only refined in the non-empty portions of the image. The actual image data corresponding to non-zero pixels is stored in the finest nodes of the quadtree. Convolution and pooling operations are restricted to the sparse pixels, leading to better efficiency in computation time as well as memory usage. Specifically, the computational and memory costs in QCNN grow linearly in the number of non-zero pixels, as opposed to traditional CNNs where the costs are quadratic in the number of pixels. This enables QCNN to learn from sparse images much faster and process high resolution images without the memory constraints faced by traditional CNNs. We study QCNN on four sparse image datasets for sketch classification and simplification tasks. The results show that QCNN can obtain comparable accuracy with large reduction in computational and memory costs.

Over the past years there has been considerable interest in statistically optimal reconstruction of cross-sectional images from tomographic data. In particular, a variety of such algorithms have been proposed for maximum a posteriori (MAP) reconstruction from emission tomographic data. While MAP estimation requires the solution of an optimization problem, most existing reconstruction algorithms take an indirect approach based on the expectation maximization (EM) algorithm. We propose a new approach to statistically optimal image reconstruction based on direct optimization of the MAP criterion. The key to this direct optimization approach is greedy pixel-wise computations known as iterative coordinate decent (ICD). We propose a novel method for computing the ICD updates, which we call ICD/Newton-Raphson. We show that ICD/Newton-Raphson requires approximately the same amount of computation per iteration as EM-based approaches, but the new method converges much more rapidly (in our experiments, typically five to ten iterations). Other advantages of the ICD/Newton-Raphson method are that it is easily applied to MAP estimation of transmission tomograms, and typical convex constraints, such as positivity, are easily incorporated.

In this paper, we propose a new model called deep network cascade (DNC) to gradually upscale low-resolution images layer by layer, each layer with a small scale factor. DNC is a cascade of multiple stacked collaborative local auto-encoders. In each layer of the cascade, non-local self-similarity search is first performed to enhance high-frequency texture details of the partitioned patches in the input image. The enhanced image patches are then input into a collaborative local auto-encoder (CLA) to suppress the noises as well as collaborate the compatibility of the overlapping patches. By closing the loop on non-local self-similarity search and CLA in a cascade layer, we can refine the super-resolution result, which is further fed into next layer until the required image scale. Experiments on image super-resolution demonstrate that the proposed DNC can gradually upscale a low-resolution image with the increase of network layers and achieve more promising results in visual quality as well as quantitative performance.

We consider the problem of estimating sparse graphs by a lasso penalty applied to the inverse covariance matrix. Using a coordinate descent procedure for the lasso, we develop a simple algorithm--the graphical lasso--that is remarkably fast: It solves a 1000-node problem ( approximately 500,000 parameters) in at most a minute and is 30-4000 times faster than competing methods. It also provides a conceptual link between the exact problem and the approximation suggested by Meinshausen and Bühlmann (2006). We illustrate the method on some cell-signaling data from proteomics.

We consider the problem of recovering two unknown vectors, w and x, of length L from their circular convolution. We make the structural assumption that the two vectors are members of known subspaces, one with dimension N and the other with dimension K. Although the observed convolution is nonlinear in both w and x, it is linear in the rank-1 matrix formed by their outer product wx*. This observation allows us to recast the deconvolution problem as low-rank matrix recovery problem from linear measurements, whose natural convex relaxation is a nuclear norm minimization program. We prove the effectiveness of this relaxation by showing that, for “generic” signals, the program can deconvolve w and x exactly when the maximum of N and K is almost on the order of L. That is, we show that if x is drawn from a random subspace of dimension N, and w is a vector in a subspace of dimension K whose basis vectors are spread out in the frequency domain, then nuclear norm minimization recovers wx* without error. We discuss this result in the context of blind channel estimation in communications. If we have a message of length N, which we code using a random L x N coding matrix, and the encoded message travels through an unknown linear time-invariant channel of maximum length K, then the receiver can recover both the channel response and the message when L ≳ N + K, to within constant and log factors.

Modern deep neural networks have a large number of parameters, making them very hard to train. We propose DSD, a dense-sparse-dense training flow, for regularizing deep neural networks and achieving better optimization performance. In the first D (Dense) step, we train a dense network to learn connection weights and importance. In the S (Sparse) step, we regularize the network by pruning the unimportant connections with small weights and retraining the network given the sparsity constraint. In the final D (re-Dense) step, we increase the model capacity by removing the sparsity constraint, re-initialize the pruned parameters from zero and retrain the whole dense network. Experiments show that DSD training can improve the performance for a wide range of CNNs, RNNs and LSTMs on the tasks of image classification, caption generation and speech recognition. On ImageNet, DSD improved the Top1 accuracy of GoogLeNet by 1.1%, VGG-16 by 4.3%, ResNet-18 by 1.2% and ResNet-50 by 1.1%, respectively. On the WSJ’93 dataset, DSD improved DeepSpeech and DeepSpeech2 WER by 2.0% and 1.1%. On the Flickr-8K dataset, DSD improved the NeuralTalk BLEU score by over 1.7. DSD is easy to use in practice: at training time, DSD incurs only one extra hyper-parameter: the sparsity ratio in the S step. At testing time, DSD doesn’t change the network architecture or incur any inference overhead. The consistent and significant performance gain of DSD experiments shows the inadequacy of the current training methods for finding the best local optimum, while DSD effectively achieves superior optimization performance for finding a better solution. DSD models are available to download at https://songhan.github.io/DSD.

Recently, deep learning has demonstrated great success in computer vision with the capability to learn powerful image features from a large training set. However, most of the published work has been confined to solving 2D problems, with a few limited exceptions that treated the 3D space as a composition of 2D orthogonal planes. The challenge of 3D deep learning is due to a much larger input vector, compared to 2D, which dramatically increases the computation time and the chance of over-fitting, especially when combined with limited training samples (hundreds to thousands), typical for medical imaging applications. To address this challenge, we propose an efficient and robust deep learning algorithm capable of full 3D detection in volumetric data. A two-step approach is exploited for efficient detection. A shallow network (with one hidden layer) is used for the initial testing of all voxels to obtain a small number of promising candidates, followed by more accurate classification with a deep network. In addition, we propose two approaches, i.e., separable filter decomposition and network sparsification, to speed up the evaluation of a network. To mitigate the over-fitting issue, thereby increasing detection robustness, we extract small 3D patches from a multi-resolution image pyramid. The deeply learned image features are further combined with Haar wavelet features to increase the detection accuracy. The proposed method has been quantitatively evaluated for carotid artery bifurcation detection on a head-neck CT dataset from 455 patients. Compared to the state-ofthe-art, the mean error is reduced by more than half, from 5.97 mm to 2.64 mm, with a detection speed of less than 1 s/volume.

A constrained optimization type of numerical algorithm for removing noise from images is presented. The total variation of the image is minimized subject to constraints involving the statistics of the noise. The constraints are imposed using Lagrange multipliers. The solution is obtained using the gradient-projection method. This amounts to solving a time dependent partial differential equation on a manifold determined by the constraints. As t → ∞ the solution converges to a steady state which is the denoised image. The numerical algorithm is simple and relatively fast. The results appear to be state-of-the-art for very noisy images. The method is noninvasive, yielding sharp edges in the image. The technique could be interpreted as a first step of moving each level set of the image normal to itself with velocity equal to the curvature of the level set divided by the magnitude of the gradient of the image, and a second step which projects the image back onto the constraint set.

Many image restoration algorithms in recent years are based on patch processing. The core idea is to decompose the target image into fully overlapping patches, restore each of them separately, and then merge the results by a plain averaging. This concept has been demonstrated to be highly effective, leading often times to the state-of-the-art results in denoising, inpainting, deblurring, segmentation, and other applications. While the above is indeed effective, this approach has one major flaw: the prior is imposed on intermediate (patch) results, rather than on the final outcome, and this is typically manifested by visual artifacts. The expected patch log likelihood (EPLL) method by Zoran and Weiss was conceived for addressing this very problem. Their algorithm imposes the prior on the patches of the final image, which in turn leads to an iterative restoration of diminishing effect. In this paper, we propose to further extend and improve the EPLL by considering a multi-scale prior. Our algorithm imposes the very same prior on different scale patches extracted from the target image. While all the treated patches are of the same size, their footprint in the destination image varies due to subsampling. Our scheme comes to alleviate another shortcoming existing in patch-based restoration algorithms-the fact that a local (patch-based) prior is serving as a model for a global stochastic phenomenon. We motivate the use of the multi-scale EPLL by restricting ourselves to the simple Gaussian case, comparing the aforementioned algorithms and showing a clear advantage to the proposed method. We then demonstrate our algorithm in the context of image denoising, deblurring, and super-resolution, showing an improvement in performance both visually and quantitatively.

A 3 AIS-Impute (proposed algorithm). Require: partially observed matrix O, parameter λ, decay parameter ν ∈ (0, 1), threshold ; 1: [U0, λ0, V0] = rank-1 SVD(PΩ(O)); 2: c = 1, ̃0 = ‖PΩ(O)‖F , X0 = X1 = λ0U0V > 0 ; 3: for t = 1, 2, . . . do 4: ̃t = ν̃0; θt = (c− 1)/(c+ 2); 5: λt = ν(λ0 − λ) + λ; 6: Yt = Xt + θt(Xt −Xt−1); 7: Z̃t = Yt + PΩ(O − Yt); 8: Vt−1 = Vt−1 − Vt(VtVt−1), remove zero columns; 9: Rt = QR([Vt, Vt−1]); 10: [Ut+1,Σt+1, Vt+1] = approx-SVT(Z̃t, Rt, λt, ̃t); 11: if F (Ut+1Σt+1V > t+1) > F (UtΣtV > t ); c = 1 else c = c+ 1; 12: end for 13: return Xt+1 = Ut+1Σt+1V > t+1.


Question: AMP-inspired deep networks for solving sparse linear inverse problems.

This paper explores Bayesian Compressive Sensing, a framework designed for the accurate reconstruction of signals from a limited set of measurements. The signals, represented as N-dimensional real vectors, are assumed to be compressible in a specific linear basis B, meaning that the signal can be closely approximated using a significantly smaller subset of M coefficients (M << N) associated with this basis. Compressive sensing leverages this compressibility by directly measuring a linear combination of the original signal, resulting in a reduced number of measurements compared to the full N-dimensional signal. The goal is to estimate the underlying signal f from these compressive-sensing measurements g, while accounting for the under-determined linear relationship between f and g and the compressibility of f in basis B.

Our Bayesian approach to compressive sensing estimation offers several advantages: Firstly, it provides not only an estimate of the underlying signal f but also estimates of the uncertainty or "error bars," offering a measure of confidence in the reconstructed signal. Secondly, by utilizing the error bars, our method offers a systematic way to determine when an adequate number of compressive-sensing measurements have been collected, ensuring optimal data usage. Thirdly, the framework naturally supports adaptive optimization of the compressive sensing measurements, moving away from random determination, which can lead to more efficient and effective signal reconstruction. Lastly, our approach accounts for additive noise in the compressive-sensing measurements and provides an estimate of the noise variance, crucial for real-world applications where noise is inevitable.

The paper details the theoretical foundations of our Bayesian compressive sensing framework, presents an associated algorithm, and showcases example results. We also conduct comparisons with other compressive-sensing inversion algorithms from the literature to demonstrate the superiority and unique capabilities of our Bayesian approach in addressing sparse linear inverse problems.

A constrained optimization type of numerical algorithm for removing noise from images is presented. The total variation of the image is minimized subject to constraints involving the statistics of the noise. The constraints are imposed using Lagrange multipliers. The solution is obtained using the gradient-projection method. This amounts to solving a time dependent partial differential equation on a manifold determined by the constraints. As t → ∞ the solution converges to a steady state which is the denoised image. The numerical algorithm is simple and relatively fast. The results appear to be state-of-the-art for very noisy images. The method is noninvasive, yielding sharp edges in the image. The technique could be interpreted as a first step of moving each level set of the image normal to itself with velocity equal to the curvature of the level set divided by the magnitude of the gradient of the image, and a second step which projects the image back onto the constraint set.

Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them. But also generic hardware and software implementations of deep learning run more efficiently on sparse networks. Several methods exist for pruning connections of a neural network after it was trained without connectivity constraints. We present an algorithm, DEEP R, that enables us to train directly a sparsely connected neural network. DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total number is all the time strictly bounded. We demonstrate that DEEP R can be used to train very sparse feedforward and recurrent neural networks on standard benchmark tasks with just a minor loss in performance. DEEP R is based on a rigorous theoretical foundation that views rewiring as stochastic sampling of network configurations from a posterior.

Designing and implementing efficient, provably correct parallel machine learning (ML) algorithms is challenging. Existing high-level parallel abstractions like MapReduce are insufficiently expressive while low-level tools like MPI and Pthreads leave ML experts repeatedly solving the same design challenges. By targeting common patterns in ML, we developed GraphLab, which improves upon abstractions like MapReduce by compactly expressing asynchronous iterative algorithms with sparse computational dependencies while ensuring data consistency and achieving a high degree of parallel performance. We demonstrate the expressiveness of the GraphLab framework by designing and implementing parallel versions of belief propagation, Gibbs sampling, Co-EM, Lasso and Compressed Sensing. We show that using GraphLab we can achieve excellent parallel performance on large scale real-world problems.

This paper proposes a general weighted l2-l0 norms energy minimization model to remove mixed noise such as Gaussian-Gaussian mixture, impulse noise, and Gaussian-impulse noise from the images. The approach is built upon maximum likelihood estimation framework and sparse representations over a trained dictionary. Rather than optimizing the likelihood functional derived from a mixture distribution, we present a new weighting data fidelity function, which has the same minimizer as the original likelihood functional but is much easier to optimize. The weighting function in the model can be determined by the algorithm itself, and it plays a role of noise detection in terms of the different estimated noise parameters. By incorporating the sparse regularization of small image patches, the proposed method can efficiently remove a variety of mixed or single noise while preserving the image textures well. In addition, a modified K-SVD algorithm is designed to address the weighted rank-one approximation. The experimental results demonstrate its better performance compared with some existing methods.

Modern convolutional networks, incorporating rectifiers and max-pooling, are neither smooth nor convex; standard guarantees therefore do not apply. Nevertheless, methods from convex optimization such as gradient descent and Adam are widely used as building blocks for deep learning algorithms. This paper provides the first convergence guarantee applicable to modern convnets, which furthermore matches a lower bound for convex nonsmooth functions. The key technical tool is the neural Taylor approximation – a straightforward application of Taylor expansions to neural networks – and the associated Taylor loss. Experiments on a range of optimizers, layers, and tasks provide evidence that the analysis accurately captures the dynamics of neural optimization. The second half of the paper applies the Taylor approximation to isolate the main difficulty in training rectifier nets – that gradients are shattered – and investigates the hypothesis that, by exploring the space of activation configurations more thoroughly, adaptive optimizers such as RMSProp and Adam are able to converge to better solutions.

High-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors. Gradient descent can be used for fine-tuning the weights in such "autoencoder" networks, but this works well only if the initial weights are close to a good solution. We describe an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data.

Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers - 8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

Model-based methods and deep neural networks have both been tremendously successful paradigms in machine learning. In model-based methods, we can easily express our problem domain knowledge in the constraints of the model at the expense of difficulties during inference. Deterministic deep neural networks are constructed in such a way that inference is straightforward, but we sacrifice the ability to easily incorporate problem domain knowledge. The goal of this paper is to provide a general strategy to obtain the advantages of both approaches while avoiding many of their disadvantages. The general idea can be summarized as follows: given a model-based approach that requires an iterative inference method, we unfold the iterations into a layer-wise structure analogous to a neural network. We then de-couple the model parameters across layers to obtain novel neural-network-like architectures that can easily be trained discriminatively using gradient-based methods. The resulting formula combines the expressive power of a conventional deep network with the internal structure of the model-based approach, while allowing inference to be performed in a fixed number of layers that can be optimized for best performance. We show how this framework can be applied to the non-negative matrix factorization to obtain a novel non-negative deep neural network architecture, that can be trained with a multiplicative back-propagation-style update algorithm. We present experiments in the domain of speech enhancement, where we show that the resulting model is able to outperform conventional neural network while only requiring a fraction of the number of parameters. We believe this is due to the ability afforded by our framework to incorporate problem level assumptions into the architecture of the deep network. arXiv.org This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Research Laboratories, Inc.; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Research Laboratories, Inc. All rights reserved. Copyright c ©Mitsubishi Electric Research Laboratories, Inc., 2014 201 Broadway, Cambridge, Massachusetts 02139

Sparse Gaussian elimination with partial pivoting computes the factorization PAQ = LU of a sparse matrix A, where the row ordering P is selected during factorization using standard partial pivoting with row interchanges. The goal is to select a column preordering, Q, based solely on the nonzero pattern of A, that limits the worst-case number of nonzeros in the factorization. The fill-in also depends on P, but Q is selected to reduce an upper bound on the fill-in for any subsequent choice of P. The choice of Q can have a dramatic impact on the number of nonzeros in L and U. One scheme for determining a good column ordering for A is to compute a symmetric ordering that reduces fill-in in the Cholesky factorization of ATA. A conventional minimum degree ordering algorithm would require the sparsity structure of ATA to be computed, which can be expensive both in terms of space and time since ATA may be much denser than A. An alternative is to compute Q directly from the sparsity structure of A; this strategy is used by MATLAB's COLMMD preordering algorithm. A new ordering algorithm, COLAMD, is presented. It is based on the same strategy but uses a better ordering heuristic. COLAMD is faster and computes better orderings, with fewer nonzeros in the factors of the matrix.

The purpose of this article is to introduce the readers to the emerging technologies enabled by deep learning and to review the research work conducted in this area that is of direct relevance to signal processing. We also point out, in our view, the future research directions that may attract interests of and require efforts from more signal processing researchers and practitioners in this emerging area for advancing signal and information processing technology and applications.


Question: AMP-inspired deep neural networks for solving sparse linear inverse problems.

In the realm of large-scale distributed deep learning, our research has unveiled significant advancements in training massive models with billions of parameters across thousands of CPU cores. To this end, we have engineered a robust software framework, coined as DistBelief, designed to harness the power of large-scale computing clusters for the purpose of training extensive deep learning models. This framework is equipped with two sophisticated algorithms tailored for distributed training: (i) Downpour Stochastic Gradient Descent (SGD), which facilitates asynchronous SGD operations for multiple model replicas, and (ii) Sandblaster, a comprehensive framework that supports a plethora of distributed batch optimization methods, notably including a distributed implementation of the Limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) algorithm.

Both Downpour SGD and Sandblaster L-BFGS significantly enhance the scale and velocity of deep network training. Our system has successfully trained a deep network that is 30 times larger than any previously reported in the literature, achieving state-of-the-art performance on ImageNet, a complex visual object recognition task encompassing 16 million images and 21k categories. We further demonstrate that these techniques substantially expedite the training of a comparably smaller deep network for a commercial speech recognition application.

While our primary focus and reported outcomes are centered around the application of these methods to the training of large neural networks, the underlying algorithms are versatile and can be applied to any gradient-based machine learning algorithm. This opens up new avenues for scaling up various machine learning models, thus broadening the scope of problems that can be effectively addressed using deep learning techniques.

In this paper we have discussed what appears to be a superior implementation of the Algebraic Reconstruction Technique (ART). The method is based on 1) simultaneous application of the error correction terms as computed by ART for all rays in a given projection; 2) longitudinal weighting of the correction terms back-distributed along the rays; and 3) using bilinear elements for discrete approximation to the ray integrals of a continuous image. Since this implementation generates a good reconstruction in only one iteration, it also appears to have a computational advantage over the more traditional implementation of ART. Potential applications of this implementation include image reconstruction in conjunction with ray tracing for ultrasound and microwave tomography in which the curved nature of the rays leads to a non-uniform ray density across the image.

High-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors. Gradient descent can be used for fine-tuning the weights in such "autoencoder" networks, but this works well only if the initial weights are close to a good solution. We describe an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data.

Deep directed generative models have attracted much attention recently due to their generative modeling nature and powerful data representation ability. In this article, we review different structures of deep directed generative models and the learning and inference algorithms associated with the structures. We focus on a specific structure that consists of layers of Bayesian networks (BNs) due to the property of capturing inherent and rich dependencies among latent variables. The major difficulty of learning and inference with deep directed models with many latent variables is the intractable inference due to the dependencies among the latent variables and the exponential number of latent variable configurations. Current solutions use variational methods, often through an auxiliary network, to approximate the posterior probability inference. In contrast, inference can also be performed directly without using any auxiliary network to maximally preserve the dependencies among the latent variables. Specifically, by exploiting the sparse representation with the latent space, max-max instead of maxsum operation can be used to overcome the exponential number of latent configurations. Furthermore, the max-max operation and augmented coordinate ascent (AugCA) are applied to both supervised and unsupervised learning as well as to various inference. Quantitative evaluations on benchmark data sets of different models are given for both data representation and feature-learning tasks.

The predominant methodology in training deep learning advocates the use of stochastic gradient descent methods (SGDs). Despite its ease of implementation, SGDs are difficult to tune and parallelize. These problems make it challenging to develop, debug and scale up deep learning algorithms with SGDs. In this paper, we show that more sophisticated off-the-shelf optimization methods such as Limited memory BFGS (L-BFGS) and Conjugate gradient (CG) with line search can significantly simplify and speed up the process of pretraining deep algorithms. In our experiments, the difference between LBFGS/CG and SGDs are more pronounced if we consider algorithmic extensions (e.g., sparsity regularization) and hardware extensions (e.g., GPUs or computer clusters). Our experiments with distributed optimization support the use of L-BFGS with locally connected networks and convolutional neural networks. Using L-BFGS, our convolutional network model achieves 0.69% on the standard MNIST dataset. This is a state-of-theart result on MNIST among algorithms that do not use distortions or pretraining.

Inverse problems correspond to a certain type of optimization problems formulated over appropriate input distributions. Recently, there has been a growing interest in understanding the computational hardness of these optimization problems, not only in the worst case, but in an average-complexity sense under this same input distribution.In this revised note, we are interested in studying another aspect of hardness, related to the ability to learn how to solve a problem by simply observing a collection of previously solved instances. These ‘planted solutions’ are used to supervise the training of an appropriate predictive model that parametrizes a broad class of algorithms, with the hope that the resulting model will provide good accuracy-complexity tradeoffs in the average sense.We illustrate this setup on the Quadratic Assignment Problem, a fundamental problem in Network Science. We observe that data-driven models based on Graph Neural Networks offer intriguingly good performance, even in regimes where standard relaxation based techniques appear to suffer.

We show that the output of a (residual) convolutional neural network (CNN) with an appropriate prior over the weights and biases is a Gaussian process (GP) in the limit of infinitely many convolutional filters, extending similar results for dense networks. For a CNN, the equivalent kernel can be computed exactly and, unlike “deep kernels”, has very few parameters: only the hyperparameters of the original CNN. Further, we show that this kernel has two properties that allow it to be computed efficiently; the cost of evaluating the kernel for a pair of images is similar to a single forward pass through the original CNN with only one filter per layer. The kernel equivalent to a 32-layer ResNet obtains 0.84% classification error on MNIST, a new record for GPs with a comparable number of parameters. 1

By reducing optimization to a sequence of small subproblems, working set methods achieve fast convergence times for many challenging problems. Despite excellent performance, theoretical understanding of working sets is limited, and implementations often resort to heuristics to determine subproblem size, makeup, and stopping criteria. We propose BLITZ, a fast working set algorithm accompanied by useful guarantees. Making no assumptions on data, our theory relates subproblem size to progress toward convergence. This result motivates methods for optimizing algorithmic parameters and discarding irrelevant variables as iterations progress. Applied to `1-regularized learning, BLITZ convincingly outperforms existing solvers in sequential, limited-memory, and distributed settings. BLITZ is not specific to `1-regularized learning, making the algorithm relevant to many applications involving sparsity or constraints.

Model-based reconstruction is a powerful framework for solving a variety of inverse problems in imaging. In recent years, enormous progress has been made in the problem of denoising, a special case of an inverse problem where the forward model is an identity operator. Similarly, great progress has been made in improving model-based inversion when the forward model corresponds to complex physical measurements in applications such as X-ray CT, electron-microscopy, MRI, and ultrasound, to name just a few. However, combining state-of-the-art denoising algorithms (i.e., prior models) with state-of-the-art inversion methods (i.e., forward models) has been a challenge for many reasons. In this paper, we propose a flexible framework that allows state-of-the-art forward models of imaging systems to be matched with state-of-the-art priors or denoising models. This framework, which we term as Plug-and-Play priors, has the advantage that it dramatically simplifies software integration, and moreover, it allows state-of-the-art denoising methods that have no known formulation as an optimization problem to be used. We demonstrate with some simple examples how Plug-and-Play priors can be used to mix and match a wide variety of existing denoising models with a tomographic forward model, thus greatly expanding the range of possible problem solutions.

One popular approach for blind deconvolution is to formulate a maximum a posteriori (MAP) problem with sparsity priors on the gradients of the latent image, and then alternatingly estimate the blur kernel and the latent image. While several successful MAP based methods have been proposed, there has been much controversy and confusion about their convergence, because sparsity priors have been shown to prefer blurry images to sharp natural images. In this paper, we revisit this problem and provide an analysis on the convergence of MAP based approaches. We first introduce a slight modification to a conventional joint energy function for blind deconvolution. The reformulated energy function yields the same alternating estimation process, but more clearly reveals how blind deconvolution works. We then show the energy function can actually favor the right solution instead of the no-blur solution under certain conditions, which explains the success of previous MAP based approaches. The reformulated energy function and our conditions for the convergence also provide a way to compare the qualities of different blur kernels, and we demonstrate its applicability to automatic blur kernel size selection, blur kernel estimation using light streaks, and defocus estimation.

The `1-regularized maximum likelihood estimation problem has recently become a topic of great interest within the machine learning, statistics, and optimization communities as a method for producing sparse inverse covariance estimators. In this paper, a proximal gradient method (G-ISTA) for performing `1-regularized covariance matrix estimation is presented. Although numerous algorithms have been proposed for solving this problem, this simple proximal gradient method is found to have attractive theoretical and numerical properties. G-ISTA has a linear rate of convergence, resulting in an O(log ε) iteration complexity to reach a tolerance of ε. This paper gives eigenvalue bounds for the G-ISTA iterates, providing a closed-form linear convergence rate. The rate is shown to be closely related to the condition number of the optimal point. Numerical convergence results and timing comparisons for the proposed method are presented. G-ISTA is shown to perform very well, especially when the optimal point is well-conditioned.


Question: Applying classification techniques for the diagnosis of liver cancer in medical practice.

This study explores the comparative efficacy of various classification techniques using the Waikato Environment for Knowledge Analysis (WEKA) in the context of diagnosing liver cancer. The proliferation of data-mining applications, including classification and clustering, has underscored the importance of applying robust machine learning algorithms to extensive datasets. Here, we employ WEKA, an open-source software suite that houses a variety of machine learning algorithms for data mining tasks. Our objective is to evaluate the performance of distinct classification and clustering methodologies on a sizable dataset. The algorithms scrutinized in this study include the Bayes Network, Radial Basis Function, Pruned Tree, Single Conjunctive Rule Learner, and Nearest Neighbors Algorithm. A comprehensive overview of these selected techniques is provided to acquaint readers with their foundational principles. We utilize a liver cancer dataset, comprising a total of 6291 records, with a matrix size of 699 rows and 9 columns, to assess and validate the comparative merits of these classification methods. Based on our findings, we will propose the classification technique that holds the most promise for enhancing conventional methods in the analysis of large datasets, with potential applications in bioinformatics and beyond. Keywords—Machine Learning, Data Mining, WEKA, Classification, Bioinformatics.

Half of the polyps surgically removed during conventional colonoscopy are benign with no malignant potential. Our purpose was to develop a CADx system for distinction between neoplastic and non-neoplastic lesions in CTC to reduce "unnecessary" colonoscopic polypectomy. Although computer aided detection (CADe) systems have been developed, less attention was given to the development of CADx systems. Our CADx system consists of shape-index-based coarse segmentation of lesions, 3D volume growing and sub-voxel refinement for fine segmentation of lesions, morphologic and texture feature analysis, Wilks' lambda-based stepwise feature selection, linear discriminant analysis for providing an integrated imaging biomarker for diagnosis of neoplastic lesions. Our database contained biopsy-confirmed 54 neoplastic lesions in 29 patients and 14 non-neoplastic lesions in 10 patients. Our CADx system integrating the selected features was able to determine an accurate likelihood of being a neoplasm and distinguish 87% (47/54) neoplastic lesions from 57% (8/14) non-neoplastic lesions correctly only using computed tomography (CT) images, achieving an area under the receiver operating characteristic curve (AUC) of 0.82. This study showed the potential of the use of CTC as a diagnostic tool beyond already accepted detection, thus, CTC with CADx would be potentially useful for reducing "unnecessary" polypectomy.

In thispaperweinvestigateanew approach to theclassificationof mammographic imagesaccording to breasttype. The classificationof breastdensityin this studyis motivatedby its useasprior knowledge in theimageprocessingpipeline.By utilising thisknowledgeat differentstagesincludingenhanceme nt, segmentation and featureextraction, its applicationaims to increasethe sensiti vity of detectingbreastcancer . Our implementeddiscriminationof breastdensityis basedon the underlyingtexture containedwithin the breast tissueapparent on a digital mammogramandrealisedby utilising four approachesto quantifyingthe texture. Following featureextraction,we adopta variationon bootstrapaggregation(’bagging’) to meetthe assumptionsof independencein datarepresentationof theinputdataset,necessaryfor classifiercombination. Multiple classifierscomprisingfeed-forward Artifi cial NeuralNetwork (ANN) aresubseque ntly trainedwith the differentperturbedinput dataspacesusing10-fold cross-validation. Thesetof classifieroutputs,expressedin a probabilistic framework, aresubsequently combinedusingsix differentclassifiercombinationrulesandtheresultscompared.In this studywe examinetwo differentclassificationtasks;a four-classclassificationproblem differentiatingbetweenfatty, partly fatty, denseandextremelydensebreasttypesanda two-classproblems, differentiatingbetweendenseandfatty breasttypes. The datasetusedin this study is the Digital Database of ScreeningMammograms(DDSM) containingMedio-LateralOblique(MLO) views for eachbreastfor 377 patients. For both tasksthe bestcombinationstrategy was found using the productrule giving an average recognition rateon testof 71.4%for thefour-classproblemand96.7%for thetwo-classproblem.

The paper presents an automated system for classification of fruits. A dataset containing five different fruits was constructed using an ordinary camera. All the fruits were analyzed on the basis of their color (RGB space), shape and texture and then classified using different classifiers to find the classifier that gives the best accuracy. Gray Level Cooccurance Matrix(GLCM) is used to calculate texture features .Best accuracy was achieved by support vector machine (svm). All the processing was carried out in Matlab. Keywords—computer vision, pattern recognition, support vector machine, texture features.

Modern healthcare service records, called Claims, record the medical treatments by a Provider (Doctor/Clinic), medication advised etc., along with the charges, and payments to be made by the patient and the Payer (insurance provider). Denial and rejection of healthcare claims is a significant administrative burden and source of loss to various healthcare providers and payers as well. Automating the identification of Claims prone to denial by reason, source, cause and other deciding factors is critical to lowering this burden of rework. We present classification methods based on Machine Learning (ML) to fully automate identification of such claims prone to rejection or denial with high accuracy, investigate the reasons for claims denial and recommend methods to engineer features using Claim Adjustment Reason Codes (CARC) as features with high Information Gain. The ML engine reported is first of its kind in Claims risk identification and represents a novel, significant enhancement to the state of practice of using ML for automating and containing claims denial risks.

medical fields has led to the automated processing ofpictorial data. Here, a device called the cytocomputer searches for genetic mutations. A computer revolution has occurred not only in technical fields but also in medicine, where vast amounts of information must be processed quickly and accurately. Nowhere is the need for image processing techniques more apparent than in clinical diagnosis or mass screening applications where data take the form of digital images. New high-resolution scanning techniques such as computed tomography, nuclear magnetic resonance, po-sitron emission tomography, and digital radiography produce images containing immense amounts of relevant information for medical analysis. But as these scanning techniques become more vital to clinical diagnosis, the work for specialists who must visually examine the resultant images increases. In many cases, quantitative data in the form of measurements and counts are needed to supplement nonimage patient data, and the manual extraction of these data is a time-consuming and costly step in an otherwise automated process. Furthermore, subtle variants of shade and shape can be the earliest clues to a diagnosis, placing the additional burden of complete thoroughness on the examining specialist. For the last five years, the University of Michigan and the Environmental Research Institute of Michigan have conducted a unique series of studies that involve the processing of biomedical imagery on a highly parallel computer specifically designed for image processing. System designers have incorporated the requirements of extracting a verifiable answer from an image in a reasonable time into an integrated approach to hardware and software design. The system includes a parallel pipelined image processor, called a cytocomputer, and a high-level language specifically created for image processing, C-3PL, the cytocomputer parallel picture processing language. These studies have involved a great many people from both the medical and engineering communities and have highlighted the interdisciplinary aspects of biomedical image processing. The methods have been tested in anatomy, developmental biology, nuclear medicine, car-diology, and transplant rejection. The general consensus is that quantification by automated image analysis not only increases diagnostic accuracy but also provides significant data not obtainable from qualitative analysis alone. One study in particular, on which descriptions in this article are based, involves a joint effort by the University of Michigan's human genetics and electrical and computer engineering departments and is supported by a grant from the National Cancer Institute. Basically, automated image analysis is being applied via sophisticated biochemical and computer techniques to derive …

Ischaemic heart disease is one of the world's most important causes of mortality, so improvements and rationalization of diagnostic procedures would be very useful. The four diagnostic levels consist of evaluation of signs and symptoms of the disease and ECG (electrocardiogram) at rest, sequential ECG testing during the controlled exercise, myocardial scintigraphy, and finally coronary angiography (which is considered to be the reference method). Machine learning methods may enable objective interpretation of all available results for the same patient and in this way may increase the diagnostic accuracy of each step. We conducted many experiments with various learning algorithms and achieved the performance level comparable to that of clinicians. We also extended the algorithms to deal with non-uniform misclassification costs in order to perform ROC analysis and control the trade-off between sensitivity and specificity. The ROC analysis shows significant improvements of sensitivity and specificity compared to the performance of the clinicians. We further compare the predictive power of standard tests with that of machine learning techniques and show that it can be significantly improved in this way.

This manuscript briefly describes an algorithm developed for the ISIC 2017 Skin Lesion Classification Competition. In this task, participants are asked to complete two independent binary image classification tasks that involve three unique diagnoses of skin lesions (melanoma, nevus, and seborrheic keratosis). In the first binary classification task, participants are asked to distinguish between (a) melanoma and (b) nevus and seborrheic keratosis. In the second binary classification task, participants are asked to distinguish between (a) seborrheic keratosis and (b) nevus and melanoma. The other phases of the competition are not considered. Our proposed algorithm consists of three steps: preprocessing, classification using VGG-NET [2] and Random Forests [3], and calculation of a final score.

Diagnosis of any diseases earlier is always preferable .diabetes is one such diseases.it reaches to the epidemic proportional in many developing countries and also reaching to industrialized nations. In this study, we investigate a programmable method to analyze diabetes diseases based on BFO and ANN.ANN is a powerful tool and solves the problems such as classification and prediction. Its performance is highly dependent on its architecture and weights. To gain high efficiency in ANN, an appropriate Architecture and learning Algorithm is preferable. The experiments on dataset occupied from undertaking repository to verify the potential of BFO algorithm and showed that BFO-ANN is more accurate than traditional ANN in terms of classification. General Terms-Artificial Neural Network, BacterialForaging Optimization Algorithm.

The rising cost of health care is one of the world’s most important problems. Accordingly, predicting such costs with accuracy is a significant first step in addressing this problem. Since the 1980s, there has been research on the predictive modeling of medical costs based on (health insurance) claims data using heuristic rules and regression methods. These methods, however, have not been appropriately validated using populations that the methods have not seen. We utilize modern data-mining methods, specifically classification trees and clustering algorithms, along with claims data from over 800,000 insured individuals over three years, to provide rigorously validated predictions of health-care costs in the third year, based on medical and cost data from the first two years. We quantify the accuracy of our predictions using unseen (out-of-sample) data from over 200,000 members. The key findings are: (a) our data-mining methods provide accurate predictions of medical costs and represent a powerful tool for prediction of health-care costs, (b) the pattern of past cost data is a strong predictor of future costs, and (c) medical information only contributes to accurate prediction of medical costs of high-cost members.

Decision support systems have been utilised since 1960, providing physicians with fast and accurate means towards more accurate diagnoses and increased tolerance when handling missing or incomplete data. This paper describes the application of neural network models for classification of electroencephalogram (EEG) signals. Decision making was performed in two stages: initially, a feature extraction scheme using the wavelet transform (WT) has been applied and then a learning-based algorithm classifier performed the classification. The performance of the neural model was evaluated in terms of training performance and classification accuracies and the results confirmed that the proposed scheme has potential in classifying the EEG signals


Question: Classification techniques applied to medical diagnosis for detecting liver cancer.

This study aims to evaluate and contrast the effectiveness of Artificial Neural Network (ANN) and Support Vector Machine (SVM) algorithms in diagnosing liver cancer. Utilizing the BUPA Liver Disorder Dataset, the models' performance is assessed through various metrics, including accuracy, sensitivity, specificity, and the Area under the Curve (AUC). Our findings reveal that SVM exhibits superior classification capabilities compared to ANN. Specifically, SVM demonstrates an accuracy of 63.11%, a specificity of 100%, and an AUC of 68.34%, whereas ANN achieves an accuracy of 57.28%, a specificity of 32.56%, and an AUC of 53.78%. These results suggest that SVM could serve as a more reliable alternative to existing or emerging classification algorithms in the context of liver cancer diagnosis, potentially enhancing the accuracy and reliability of medical assessments in this area.

The diagnosis of diseases is a significant and tedious task in medicine. The detection of heart disease from various factors or symptoms is a multi-layered issue which is not free from false presumptions often accompanied by unpredictable effects. Thus the effort to utilize knowledge and experience of numerous specialists and clinical screening data of patients collected in databases to facilitate the diagnosis process is considered a valuable option. The healthcare industry gathers enormous amounts of heart disease data that regrettably, are not “mined” to determine concealed information for effective decision making by healthcare practitioners. In this paper, we have proposed an efficient approach for the extraction of significant patterns from the heart disease warehouses for heart attack prediction. Initially, the data warehouse is preprocessed to make it appropriate for the mining process. After preprocessing, the heart disease warehouse is clustered using the K-means clustering algorithm, which will extract the data relevant to heart attack from the warehouse. Subsequently the frequent patterns are mined from the extracted data, relevant to heart disease, using the MAFIA algorithm. Then the significant weightage of the frequent patterns are calculated. Further, the patterns significant to heart attack prediction are chosen based on the calculated significant weightage. These significant patterns can be used in the development of heart attack prediction system.

Skin diseases are very common in people’s daily life. Each year, millions of people in American are affected by all kinds of skin disorders. Diagnosis of skin diseases sometimes requires a high-level of expertise due to the variety of their visual aspects. As human judgment are often subjective and hardly reproducible, to achieve a more objective and reliable diagnosis, a computer aided diagnostic system should be considered. In this paper, we investigate the feasibility of constructing a universal skin disease diagnosis system using deep convolutional neural network (CNN). We train the CNN architecture using the 23,000 skin disease images from the Dermnet dataset and test its performance with both the Dermnet and OLE, another skin disease dataset, images. Our system can achieve as high as 73.1% Top-1 accuracy and 91.0% Top-5 accuracy when testing on the Dermnet dataset. For the test on the OLE dataset, Top-1 and Top-5 accuracies are 31.1% and 69.5%. We show that these accuracies can be further improved if more training images are used.

Keywords: Image segmentation Plant leaf Complicated background Watershed segmentation Hu geometric moments Zernike moment Moving center hypersphere (MCH) classifier a b s t r a c t Classifying plant leaves has so far been an important and difficult task, especially for leaves with complicated background where some interferents and overlapping phenomena may exist. In this paper, an efficient classification framework for leaf images with complicated background is proposed. First, a so-called automatic marker-controlled watershed segmen-tation method combined with pre-segmentation and morphological operation is introduced to segment leaf images with complicated background based on the prior shape information. Then, seven Hu geometric moments and sixteen Zernike moments are extracted as shape features from segmented binary images after leafstalk removal. In addition , a moving center hypersphere (MCH) classifier which can efficiently compress feature data is designed to address obtained mass high-dimensional shape features. Finally, experimental results on some practical plant leaves show that proposed classification framework works well while classifying leaf images with complicated background. There are twenty classes of practical plant leaves successfully classified and the average correct classification rate is up to 92.6%. Plants play the most important part in the cycle of nature. They are the primary producers that sustain all other life forms including people. This is because plants are the only organisms that can convert light energy from the sun into food. Animals, incapable of making their own food, depend directly or indirectly on plants for their supply of food. Moreover, all of the oxygen available for living organisms comes from plants. Plants are also the primary habitat for thousands of other organisms. In addition, many of the fuel people use today, such as coal, natural gas and gasoline, were made from plants that lived millions of years ago. However, in recent years, people have been seriously destroying the natural environments, so that many plants constantly die and even die out every year. Conversely, the resulting ecological crisis has brought many serious consequences including land desertion, climate anomaly, land flood, and so on, which have menaced the survival of human being and the development of society. Now, people have realized the importance and urgency of protecting plant resource. Besides taking effective measures to protect plants, it is important for ordinary people to know or classify plants which can further enhance the public's consciousness of plant protection. Therefore, besides professional botanists, many non-professional researchers have paid more …

Automated image processing techniques have the ability to assist in the early detection of diabetic retinopathy disease which can be regarded as a manifestation of diabetes on the retina. Blood vessel segmentation is the basic foundation while developing retinal screening systems, since vessels serve as one of the main retinal landmark features. This paper proposes an automated method for identification of blood vessels in color images of the retina. For every image pixel, a feature vector is computed that utilizes properties of scale and orientation selective Gabor filters. The extracted features are then classified using generative Gaussian mixture model and discriminative support vector machines classifiers. Experimental results demonstrate that the area under the receiver operating characteristic (ROC) curve reached a value 0.974, which is highly comparable and, to some extent, higher than the previously reported ROCs that range from 0.787 to 0.961. Moreover, this method gives a sensitivity of 96.50% with a specificity of 97.10% for identification of blood vessels. Keywords– Retinal blood vessels, Gabor filters, support vector machines, vessel segmentation

This paper proposes a classification technique using conjunction of Machine Learning Algorithms and ECG Diagnostic Criteria which improves the accuracy of detecting Arrhythmia using Electrocardiogram (ECG) data. ECG is the most widely used first line clinical instrument to record the electrical activities of the heart. The data-set from UC Irvine (UCI) Machine Learning Repository was used to implement a multi-class classification for different types of heart abnormalities. After implementing rigorous data preprocessing and feature selection techniques,different machine learning algorithms such as Neural Networks, Decision trees, Random Forest, Gradient Boosting and Support Vector Machines were used. Maximum experimental accuracy of 84.82% was obtained via the conjunction of SVM and Gradient Boosting. A further improvement in accuracy was obtained by validating the factors which were important for doctors to decide between normal and abnormal heart conditions.The performance of classification is evaluated using measures such as confusion matrix, kappa-score, confidence interval, Area under curve (AUC) and overall-accuracy. Key–Words: Machine Learning, Arrhythmia Classification, ECG, Neural Networks, SVM, Gradient Boosting

ISSN 2250 – 1061 | © 2011 Bonfring Abstract--In machine learning, classification refers to a step by step procedure for designating a given piece of input data into any one of the given categories. There are many classification problem occurs and need to be solved. Different types are classification algorithms like tree-based, rule-based, etc are widely used. This work studies the effectiveness of Rule-Based classifiers for classification by taking a sample data set from UCI machine learning repository using the open source machine learning tool. A comparison of different rulebased classifiers used in Data Mining and a practical guideline for selecting the most suited algorithm for a classification is presented and some empirical criteria for describing and evaluating the classifiers are given.

OBJECTIVES
The intensive care environment generates a wealth of critical care data suited to developing a well-calibrated prediction tool. This study was done to develop an intensive care unit (ICU) mortality prediction model built on University of Kentucky Hospital (UKH)'s data and to assess whether the performance of various data mining techniques, such as the artificial neural network (ANN), support vector machine (SVM) and decision trees (DT), outperform the conventional logistic regression (LR) statistical model.

METHODS
The models were built on ICU data collected regarding 38,474 admissions to the UKH between January 1998 and September 2007. The first 24 hours of the ICU admission data were used, including patient demographics, admission information, physiology data, chronic health items, and outcome information.

RESULTS
Only 15 study variables were identified as significant for inclusion in the model development. The DT algorithm slightly outperformed (AUC, 0.892) the other data mining techniques, followed by the ANN (AUC, 0.874), and SVM (AUC, 0.876), compared to that of the APACHE III performance (AUC, 0.871).

CONCLUSIONS
With fewer variables needed, the machine learning algorithms that we developed were proven to be as good as the conventional APACHE III prediction.

One of the disease that require X-ray diagnosis is scoliosis. Early detection of scoliosis is important to do for anyone. From the early detection information, the doctor may take the firts step to further treatment quickly. Determination of spinal curvature is a first step method that used to measure how severe the degree of scoliosis. The severity degree of scoliosis can be assess by using Cobb angle. Therefore, by approximate the spinal curvature, we can approximate the cobb angle too. From previous work that interobserver measurement value may reach 11.8° and intraobserver measurement error is 6°. So, as far as the cobb angle measuring, the subjectivity aspect is the natural thing and can be tolerated until now. This research propose an algorithm how to define spinal curvature with the aid of a computer in digital X-ray image quickly but has a standard error that can be tolerated. The preprocessing has been done by canny edge detection. The k-means clustering algorithm can detect the centroid point after segmentation preprocessing of the spinal segment and polynomial curve fitting will be used in the process for determining the spinal curve. From the spinal curvature information, the scoliosis curve can be classified into 4 condition, normal, mild, moderate, and severe scoliosis.

In most of the food industries, farms and fruit markets the quality sorting of the fruits is done manually; only a few automated sorting systems are developed for this. Each fruit changes its color in its life span; including stages not ripe, semi ripped, completely ripped or at the end rotten. Hence, we can predict the quality of the fruits by processing its color images and then applying pattern recognition techniques on those images. By quality of fruit we mean size, ripeness, sweetness, longetivity, diseased/ rotten etc. Some advance technologies which are used for classification of fruits are Infrared Imaging, Magnetic Resonance Imaging, Thermal Sensing, Electronic Nose sensing etc; but these are costlier as compared to color image processing and pattern recognition techniques. This paper summarizes and compares various techniques of Pattern Recognition which are used effectively for fruit sorting. All the fruits have their unique ways of ripening stages; hence the pattern recognition techniques used for one fruit does not necessarily match for the other fruit. This paper reviews the high quality research work done for the quality classification of Apples, Dates, Mangoes, Cranberry, Jatropha, Tomatoes, and some other fruits using pattern recognition techniques. KeywordsClassification & predication, Pattern Recognition, Image Processing, Fruit Quality __________________________________________________*****_________________________________________________

Data mining techniques have been extensively applied in bioinformatics to analyze biomedical data. In this paper, we choose the Rapid-I¿s RapidMiner as our tool to analyze a Pima Indians Diabetes Data Set, which collects the information of patients with and without developing diabetes. The discussion follows the data mining process. The focus will be on the data preprocessing, including attribute identification and selection, outlier removal, data normalization and numerical discretization, visual data analysis, hidden relationships discovery, and a diabetes prediction model construction.