Question: Utilizing recurrent neural networks for the identification of surgical activities.

This paper delves into the challenge of modeling complex polyphonic music sequences represented in a piano-roll format, focusing on the intricate task of capturing temporal dependencies within these high-dimensional data. We propose a novel probabilistic framework that leverages a recurrent neural network (RNN) to conditionally estimate distributions, enabling the model to discern and learn the temporal dynamics inherent in musical sequences. This approach not only surpasses conventional models in terms of performance but also demonstrates its efficacy across a spectrum of real-world polyphonic music datasets. Furthermore, we illustrate the potential of our music language model as a symbolic prior, significantly enhancing the accuracy of polyphonic music transcription tasks. By serving as a powerful tool for understanding and generating musical sequences, our model opens new avenues for research in music information retrieval and musicology.

**Question**: A Comparative Study of Machine Learning Techniques for Intrusion Detection Systems

**Documents**: A Comparative Study of Machine Learning Techniques for Intrusion Detection Systems
This paper presents a comparative study of different machine learning techniques for intrusion detection systems (IDS). The study covers a wide range of machine learning methods including decision trees, support vector machines, and neural networks. The performance of these techniques is evaluated using standard datasets, and the results are compared based on various metrics such as accuracy, precision, recall, and F1-score. The findings indicate that support vector machines and neural networks perform better than decision trees in terms of overall accuracy and F1-score, making them more suitable for IDS applications.

**Similar Documents**: This paper offers an in-depth comparative analysis of various machine learning methodologies in the context of intrusion detection systems (IDS). It explores a comprehensive array of techniques, encompassing decision trees, support vector machines (SVMs), and neural networks, to determine their efficacy in IDS. Utilizing benchmark datasets, the performance of these methods is meticulously assessed through a set of key metrics, including accuracy, precision, recall, and F1-score. The results reveal that SVMs and neural networks exhibit superior performance compared to decision trees, particularly in terms of overall accuracy and F1-score. This suggests that these techniques are more adept at handling the complex and dynamic nature of intrusion detection, thus making them highly suitable for modern IDS applications.

Exploring Deep Learning Methods for discovering features in speech signals. Navdeep Jaitly Doctor of Philosophy Graduate Department of Computer Science University of Toronto 2014 This thesis makes three main contributions to the area of speech recognition with Deep Neural Network Hidden Markov Models (DNN-HMMs). Firstly, we explore the effectiveness of features learnt from speech databases using Deep Learning for speech recognition. This contrasts with prior works that have largely confined themselves to using traditional features such as Mel Cepstral Coefficients and Mel log filter banks for speech recognition. We start by showing that features learnt on raw signals using Gaussian-ReLU Restricted Boltzmann Machines can achieve accuracy close to that achieved with the best traditional features. These features are, however, learnt using a generative model that ignores domain knowledge. We develop methods to discover features that are endowed with meaningful semantics that are relevant to the domain using capsules. To this end, we extend previous work on transforming autoencoders and propose a new autoencoder with a domain-specific decoder to learn capsules from speech databases. We show that capsule instantiation parameters can be combined with Mel log filter banks to produce improvements in phone recognition on TIMIT. On WSJ the word error rate does not improve, even though we get strong gains in classification accuracy. We speculate this may be because of the mismatched objectives of word error rate over an utterance and frame error rate on the sub-phonetic class for a frame. Secondly, we develop a method for data augmentation in speech datasets. Such methods result in strong gains in object recognition, but have largely been ignored in speech recognition. Our data augmentation encourages the learning of invariance to vocal tract length of speakers. The method is shown to improve the phone error rate on TIMIT and the word error rate on a 14 hour subset of WSJ. Lastly, we develop a method for learning and using a longer range model of targets, conditioned on the input. This method predicts the labels for multiple frames together and uses a geometric average of these predictions during decoding. It produces state of the art results on phone recognition with TIMIT and also produces significant gains on WSJ.

Human activity is one of the most important ingredients of context information. In wearable computing scenarios, activities such as walking, standing and sitting can be inferred from data provided by body-worn acceleration sensors. In such settings, most approaches use a single set of features, regardless of which activity to be recognized. In this paper we show that recognition rates can be improved by careful selection of individual features for each activity. We present a systematic analysis of features computed from a real-world data set and show how the choice of feature and the window length over which the feature is computed affects the recognition rates for different activities. Finally, we give a recommendation of suitable features and window lengths for a set of common activities.

Most current speech recognition systems use hidden Markov models (HMMs) to deal with the temporal variability of speech and Gaussian mixture models to determine how well each state of each HMM fits a frame or a short window of frames of coefficients that represents the acoustic input. An alternative way to evaluate the fit is to use a feedforward neural network that takes several frames of coefficients as input and produces posterior probabilities over HMM states as output. Deep neural networks with many hidden layers, that are trained using new methods have been shown to outperform Gaussian mixture models on a variety of speech recognition benchmarks, sometimes by a large margin. This paper provides an overview of this progress and represents the shared views of four research groups who have had recent successes in using deep neural networks for acoustic modeling in speech recognition.

What if we could effectively read the mind and transfer human visual capabilities to computer vision methods? In this paper, we aim at addressing this question by developing the first visual object classifier driven by human brain signals. In particular, we employ EEG data evoked by visual object stimuli combined with Recurrent Neural Networks (RNN) to learn a discriminative brain activity manifold of visual categories in a reading the mind effort. Afterward, we transfer the learned capabilities to machines by training a Convolutional Neural Network (CNN)–based regressor to project images onto the learned manifold, thus allowing machines to employ human brain–based features for automated visual classification. We use a 128-channel EEG with active electrodes to record brain activity of several subjects while looking at images of 40 ImageNet object classes. The proposed RNN-based approach for discriminating object classes using brain signals reaches an average accuracy of about 83%, which greatly outperforms existing methods attempting to learn EEG visual object representations. As for automated object categorization, our human brain–driven approach obtains competitive performance, comparable to those achieved by powerful CNN models and it is also able to generalize over different visual datasets.

In this paper, we present a real-time algorithm for automatic recognition of not only physical activities, but also, in some cases, their intensities, using five triaxial wireless accelerometers and a wireless heart rate monitor. The algorithm has been evaluated using datasets consisting of 30 physical gymnasium activities collected from a total of 21 people at two different labs. On these activities, we have obtained a recognition accuracy performance of 94.6% using subject-dependent training and 56.3% using subject-independent training. The addition of heart rate data improves subject-dependent recognition accuracy only by 1.2% and subject-independent recognition only by 2.1%. When recognizing activity type without differentiating intensity levels, we obtain a subject-independent performance of 80.6%. We discuss why heart rate data has such little discriminatory power.

Recurrent neural networks have been used for time-s eri s prediction with good results. In this dissertation we compare recurrent neural networks with time-delayed feed forward networks, feed forward networks and li near regression models to see which architecture that can make the most accurate predictions. The data used in all experiments is real-world sales data containing two kinds of segments: campaign segments and non-campaign segments. The task is to make predictions of sales under campaigns and with this in mind we evaluate if we c an get more accurate predictions if we only use the campaign segments when modeling the data. Throughout the entire project the KDD process have been used to give a st ructured work-process. The results showed that recurrent network is not better han the other evaluated algorithms, in fact, the time-delayed feed forward neural network showed to give the best predictions. The results also showed that we c ould get more accurate predictions when only using information from campaign segments.

We introduce a novel Recurrent Neural Network-based algorithm for future video feature generation and action anticipation called feature mapping RNN . Our novel RNN architecture builds upon three effective principles of machine learning, namely parameter sharing, Radial Basis Function kernels and adversarial training. Using only some of the earliest frames of a video, the feature mapping RNN is able to generate future features with a fraction of the parameters needed in traditional RNN. By feeding these future features into a simple multilayer perceptron facilitated with an RBF kernel layer, we are able to accurately predict the action in the video. In our experiments, we obtain 18% improvement on JHMDB-21 dataset, 6% on UCF101-24 and 13% improvement on UT-Interaction datasets over prior stateof-the-art for action anticipation.

In this work1, we present a method to represent a video with a sequence of words, and learn the temporal sequencing of such words as the key information for predicting and recognizing human actions. We leverage core concepts from the Natural Language Processing (NLP) literature used in sentence classification to solve the problems of action prediction and action recognition. Each frame is converted into a word that is represented as a vector using the Bag of Visual Words (BoW) encoding method. The words are then combined into a sentence to represent the video, as a sentence. The sequence of words in different actions are learned with a simple but effective Temporal Convolutional Neural Network (T-CNN) that captures the temporal sequencing of information in a video sentence. We demonstrate that a key characteristic of the proposed method is its low-latency, i.e. its ability to predict an action accurately with a partial sequence (sentence). Experiments on two datasets, UCF101 and HMDB51 show that the method on average reaches 95% of its accuracy within half the video frames. Results, also demonstrate that our method achieves compatible state-of-the-art performance in action recognition (i.e. at the completion of the sentence) in addition to action prediction.

Understanding natural human activity involves not only identifying the action being performed, but also locating the semantic elements of the scene and describing the person's interaction with them. We present a system that is able to recognize complex, fine-grained human actions involving the manipulation of objects in realistic action sequences. Our method takes advantage of recent advances in sensors and pose trackers in learning an action model that draws on successful discriminative techniques while explicitly modeling both pose trajectories and object manipulations. By combining these elements in a single model, we are able to simultaneously recognize actions and track the location and manipulation of objects. To showcase this ability, we introduce a novel Cooking Action Dataset that contains video, depth readings, and pose tracks from a Kinect sensor. We show that our model outperforms existing state of the art techniques on this dataset as well as the VISINT dataset with only video sequences.

The capability of processing and digesting raw data is one of the key features of a humanlike artificial intelligence system. For instance, real-time machine translation should be able to process and understand spoken natural language, and autonomous driving relies on the comprehension of visual inputs. Representation learning is a class of machine learning techniques that autonomously learn to derive latent features from raw data. These new features are expected to represent the data instances in a vector space that facilitates the machine learning task. This thesis studies two specific data situations that require efficient representation learning: knowledge graph data and high dimensional sequences. In the first part of this thesis, we first review multiple relational learning models based on tensor decomposition for knowledge graphs. We point out that relational learning is in fact a means of learning representations through one-hot mapping of entities. Furthermore, we generalize this mapping function to consume a feature vector that encodes all known facts about each entity. It enables the relational model to derive the latent representation instantly for a new entity, without having to re-train the tensor decomposition. In the second part, we focus on learning representations from high dimensional sequential data. Sequential data often pose the challenge that they are of variable lengths. Electronic health records, for instance, could consist of clinical event data that have been collected at subsequent time steps. But each patient may have a medical history of variable length. We apply recurrent neural networks to produce fixed-size latent representations from the raw feature sequences of various lengths. By exposing a prediction model to these learned representations instead of the raw features, we can predict the therapy prescriptions more accurately as a means of clinical decision support. We further propose Tensor-Train recurrent neural networks. We give a detailed introduction to the technique of tensorizing and decomposing large weight matrices into a few smaller tensors. We demonstrate the specific algorithms to perform the forward-pass and the back-propagation in this setting. Then we apply this approach to the input-to-hidden weight matrix in recurrent neural networks. This novel architecture can process extremely high dimensional sequential features such as video data. The model also provides a promising solution to processing sequential features with high sparsity. This is, for instance, the case with electronic health records, since they are often of categorical nature and have to be binary-coded. We incorporate a statistical survival model with this representation learning model, which shows superior prediction quality.


Question: Theoretical analysis and practical considerations for nonlinear camera response functions in image deblurring.

This paper delves into the critical issue of managing outliers in the non-blind image deconvolution process, a fundamental aspect of image deblurring systems. Existing deconvolution techniques are predicated on a linear blur model, assuming that the blurred image results from a linear convolution between the original image and the blur kernel. However, this assumption frequently breaks down in real-world scenarios due to the presence of outliers during the imaging process. Such outliers can lead to significant distortions in the deconvolved image, manifesting as severe ringing artifacts even when the kernel is accurately estimated. We identify and analyze several typical sources of outliers that undermine the performance of existing deconvolution methods, including pixel saturation and non-Gaussian noise. In response, we propose an innovative blur model that directly accounts for these outlier scenarios, upon which we construct a robust non-blind deconvolution algorithm. This algorithm is designed to mitigate the visual artifacts induced by outliers, significantly enhancing the quality of deblurred images. The efficacy of our approach is substantiated through a series of experiments on both simulated and real-world data, showcasing its superior performance in managing outlier-induced distortions during the image deblurring process.

In this article, we introduce variational image restoration and segmentation models that incorporate the L1 data-fidelity measure and a nonsmooth, nonconvex regularizer. The L1 fidelity term allows us to restore or segment an image with low contrast or outliers, and the nonconvex regularizer enables homogeneous regions of the objective function (a restored image or an indicator function of a segmented region) to be efficiently smoothed while edges are well preserved. To handle the nonconvexity of the regularizer, a multistage convex relaxation method is adopted. This provides a better solution than the classical convex total variation regularizer, or than the standard L1 convex relaxation method. Furthermore, we design fast and efficient optimization algorithms that can handle the non-differentiability of both the fidelity and regularization terms. The proposed iterative algorithms asymptotically solve the original nonconvex problems. Our algorithms output a restored image or segmented regions in the image, as well as an edge indicator that characterizes the edges of the output, similar to Mumford–Shah-like regularizing functionals. Numerical examples demonstrate the promising results of the proposed restoration and segmentation models.

In this paper, a fast image restoration method is proposed to restore the true image from an atmospheric turbulence degraded video. A non-rigid image registration algorithm is employed to register all the frames of the video to a reference frame and determine the shift maps. The First Register Then Average And Subtract-variant (FRTAASv) method is applied to correct the geometric distortion of the reference frame. A performance comparison is presented between the proposed restoration method and the earlier Minimum Sum of Squared Differences (MSSD) image registration based FRTAASv method, in terms of processing time and accuracy. Simulation results show that the proposed method requires shorter processing time to achieve the same geometric accuracy.

PRNU-based techniques guarantee a good forgery detection performance irrespective of the specific type of forgery. The presence or absence of the camera PRNU pattern is detected by a correlation test. Given the very low power of the PRNU signal, however, the correlation must be averaged over a pretty large window, reducing the algorithm's ability to reveal small forgeries. To improve resolution, we estimate correlation with a spatially adaptive filtering technique, with weights computed over a suitable pilot image. Implementation efficiency is achieved by resorting to the recently proposed guided filters. Experiments prove that the proposed filtering strategy allows for a much better detection performance in the case of small forgeries.

Underwater images are degraded mainly by light scattering and color distortion. Light scattering occurs because the light is reflected and deflected multiple times by the suspended particles in the water before reaching the camera. Color distortion is caused due to the absorption degrees vary according to the light wavelength. At last, the underwater images are dominated by a bluish tone. In this paper, we propose a novel underwater image enhancement approach based on dark channel prior and luminance adjustment. The dehazed underwater image obtained by dark channel prior characterizes with low brightness. Then, we utilize a novel luminance adjustment method to enhance image contrast. The enhanced image shows improved global contrast and better preserved image details and edges. Experimental results demonstrate the effectiveness of our method compared to the state-of-the-art methods.

We present a method to jointly estimate scene depth and recover the clear latent image from a foggy video sequence. In our formulation, the depth cues from stereo matching and fog information reinforce each other, and produce superior results than conventional stereo or defogging algorithms. We first improve the photo-consistency term to explicitly model the appearance change due to the scattering effects. The prior matting Laplacian constraint on fog transmission imposes a detail-preserving smoothness constraint on the scene depth. We further enforce the ordering consistency between scene depth and fog transmission at neighboring points. These novel constraints are formulated together in an MRF framework, which is optimized iteratively by introducing auxiliary variables. The experiment results on real videos demonstrate the strength of our method.

A 352 288 pixel CMOS image sensor chip with perpixel single-slope ADC and dynamic memory in a standard digital 0.18m CMOS process is described. The chip performs “snapshot” image acquisition, parallel 8-bit A/D conversion, and digital readout at continuous rate of 10 000 frames/s or 1 Gpixels/s with power consumption of 50 mW. Each pixel consists of a photogate circuit, a three-stage comparator, and an 8-bit 3T dynamic memory comprising a total of 37 transistors in 9.4 9.4 m with a fill factor of 15%. The photogate quantum efficiency is 13.6%, and the sensor conversion gain is 13.1 V/e . At 1000 frames/s, measured integral nonlinearity is 0.22% over a 1-V range, rms temporal noise with digital CDS is 0.15%, and rms FPN with digital CDS is 0.027%. When operated at low frame rates, on-chip power management circuits permit complete powerdown between each frame conversion and readout. The digitized pixel data is read out over a 64-bit (8-pixel) wide bus operating at 167 MHz, i.e., over 1.33 GB/s. The chip is suitable for general high-speed imaging applications as well as for the implementation of several still and standard video rate applications that benefit from high-speed capture, such as dynamic range enhancement, motion estimation and compensation, and image stabilization.

1 Consultant, Maplewood, MN USA 4 * Correspondence: sahyun@infionline.net; 1-(651)-927-9686 5 6 Abstract: We examined a series of real-world, pictorial photographs with varying 7 characteristics, along with their modification by noise addition and unsharp masking. As 8 response metrics we used three different versions of the aesthetic measure originally 9 proposed by Birkhoff. The first aesthetic measure, which has been used in other studies, 10 and which we used in our previous work as well, showed a preference for the least 11 complex of the images. It provided no justification for noise addition, but did reveal 12 enhancement on unsharp masking. Optimum level of unsharp masking varied with the 13 image, but was predictable from the individual image’s GIF compressibility. We expect this 14 result to be useful for guiding the processing of pictorial photographic imagery. The 15 second aesthetic measure, that of informational aesthetics based on entropy alone failed 16 to provide useful discrimination among the images or the conditions of their modification. 17 A third measure, derived from the concepts of entropy maximization, as well as the 18 hypothesized preference of observers for “simpler”, i.e., more compressible, images, 19 yielded qualitatively the same results as the more traditional version of the measure. 20 Differences among the photographs and the conditions of their modification were more 21 clearly defined with this metric, however. 22

This paper presents a low-light image enhancement method using the variational-optimization-based Retinex algorithm. The proposed enhancement method first estimates the initial illumination and uses its gamma corrected version to constrain the illumination component. Next, the variational-based minimization is iteratively performed to separate the reflectance and illumination components. The color assignment of the estimated reflectance component is then performed to restore the color component using the input RGB color channels. Experimental results show that the proposed method can provide better enhanced result without saturation, noise amplification or color distortion.

Intrinsic image decomposition refers to recover the albedo and shading from images, which is an ill-posed problem in signal processing. As realistic labeled data are severely lacking, it is difficult to apply learning methods in this issue. In this letter, we propose using a synthesized dataset to facilitate the solving of this problem. A physically based renderer is used to generate color images and their underlying ground-truth albedo and shading from three-dimensional models. Additionally, we render a Kinect-like noisy depth map for each instance. We utilize this synthetic dataset to train a deep neural network for intrinsic image decomposition and further fine-tune it for real-world images. Our model supports both RGB and RGB-D as input, and it employs both high-level and low-level features to avoid blurry outputs. Experimental results verify the effectiveness of our model on realistic images.

A constrained optimization type of numerical algorithm for removing noise from images is presented. The total variation of the image is minimized subject to constraints involving the statistics of the noise. The constraints are imposed using Lagrange multipliers. The solution is obtained using the gradient-projection method. This amounts to solving a time dependent partial differential equation on a manifold determined by the constraints. As t → ∞ the solution converges to a steady state which is the denoised image. The numerical algorithm is simple and relatively fast. The results appear to be state-of-the-art for very noisy images. The method is noninvasive, yielding sharp edges in the image. The technique could be interpreted as a first step of moving each level set of the image normal to itself with velocity equal to the curvature of the level set divided by the magnitude of the gradient of the image, and a second step which projects the image back onto the constraint set.


Question: Theoretical analysis and practical considerations for image deblurring using nonlinear camera response functions.

This paper introduces a novel method for the radiometric calibration of a camera's response function using just a single image, addressing a common limitation in computer vision applications where multiple registered images captured under varying exposure settings are often required. Our approach leverages the measured RGB distributions at color edges within the single image to perform the calibration. It automatically identifies and selects relevant edge information for processing, and applies a Bayesian framework to calculate the calibration parameters. Through comprehensive experimental validation, we have confirmed that our method can achieve highly accurate calibration results, even when utilizing only a single input image, thereby offering a significant advancement in the field of camera calibration techniques.

Video and images acquired by a visual system are seriously degraded under hazy and foggy weather, which will affect the detection, tracking, and recognition of targets. Thus, restoring the true scene from such a foggy video or image is of significance. The main goal of this paper was to summarize current video and image defogging algorithms. We first presented a review of the detection and classification method of a foggy image. Then, we summarized existing image defogging algorithms, including image restoration algorithms, image contrast enhancement algorithms, and fusion-based defogging algorithms. We also presented current video defogging algorithms. We summarized objective image quality assessment methods that have been widely used for the comparison of different defogging algorithms, followed by an experimental comparison of various classical image defogging algorithms. Finally, we presented the problems of video and image defogging which need to be further studied. The code of all algorithms will be available at <;uri xlink:href="http://www.yongxu.org/lunwen.html" xlink:type="simple">http://www.yongxu.org/lunwen.html<;/uri>.

This paper proposes a method for transferring the RGB color spectrum to near-infrared (NIR) images using deep multi-scale convolutional neural networks. A direct and integrated transfer between NIR and RGB pixels is trained. The trained model does not require any user guidance or a reference image database in the recall phase to produce images with a natural appearance. To preserve the rich details of the NIR image, its high frequency features are transferred to the estimated RGB image. The presented approach is trained and evaluated on a real-world dataset containing a large amount of road scene images in summer. The dataset was captured by a multi-CCD NIR/RGB camera, which ensures a perfect pixel to pixel registration.

We propose a multi-focus image fusion method. The energy of Laplacian of input images is obtained to decide which portions of the input images are in better focus, and the majority filter offers a means of spreading the focused regions to neighborhood. The edge-preserving guided image filter is applied to reduce the block effect. We conduct several experiments which demonstrate that it provided a better performance than the state of the art fusion methods in visual and quantitative evaluations.

A regression-based model was developed relating aerial image quality, expressed in terms of the National Imagery Interpretability Rating Scale (NIIRS), to fundamental image attributes. The General Image-Quality Equation (GIQE) treats three main attributes: scale, expressed as the ground-sampled distance; sharpness, measured from the system modulation transfer function; and the signal-to-noise ratio. The GIQE can be applied to any visible sensor and predicts NIIRS ratings with a standard error of 0.3 NIIRS. The image attributes treated by the GIQE are influenced by system design and operation parameters. The GIQE allows system designers and operators to perform trade-offs for the optimization of image quality.

The most successful approaches for filtering Monte Carlo noise use feature-based filters (e.g., cross-bilateral and cross non-local means filters) that exploit additional scene features such as world positions and shading normals. However, their main challenge is finding the optimal weights for each feature in the filter to reduce noise but preserve scene detail. In this paper, we observe there is a complex relationship between the noisy scene data and the ideal filter parameters, and propose to learn this relationship using a nonlinear regression model. To do this, we use a multilayer perceptron neural network and combine it with a matching filter during both training and testing. To use our framework, we first train it in an offline process on a set of noisy images of scenes with a variety of distributed effects. Then at run-time, the trained network can be used to drive the filter parameters for new scenes to produce filtered images that approximate the ground truth. We demonstrate that our trained network can generate filtered images in only a few seconds that are superior to previous approaches on a wide range of distributed effects such as depth of field, motion blur, area lighting, glossy reflections, and global illumination.

This paper proposes a collaborative color calibration meth od for multi-camera systems. The multi-camera color calibr t on problem is formulated as an overdetermined linear system, i n which a dynamic range shaping is incorporated to ensure the high contrasts for captured images. The cameras are calibrated w ith the parameters obtained by solving the linear system. Fo r non-planar multi-camera systems, we design a novel omnidirectional co lor checker and present a method for establishing global cor respondences to facilitate automatic color calibration without m anual adjustment. According to experimental results on bot h synthetic and real-system datasets, the proposed method shows high perfo rmance in achieving inter-camera color consistency and hig h dynamic range. Thanks to the generality of the linear system formula tion and the flexibility of the designed color checker, the pr oposed method is applicable to various multi-camera systems.

The security problem of screen image leakage from a display unit has become serious with the rapid speed of signal transmission technology. This paper presents a novel investigation on the characteristics of the power line compromising channel. Moreover, a measurement system has been actually developed for the leakage signal analyzing and image reconstruction. In order to overcome the degradation of reconstructed motion images and enhance the reconstructed image quality, a multi-image blind deconvolution method was proposed and test experiments were carried out to verify the effectiveness of the multi-image blind deconvolution algorithm based on the conducted signal from the power line.

This paper proposes a two-phase scheme for removing salt-and-pepper impulse noise. In the first phase, an adaptive median filter is used to identify pixels which are likely to be contaminated by noise (noise candidates). In the second phase, the image is restored using a specialized regularization method that applies only to those selected noise candidates. In terms of edge preservation and noise suppression, our restored images show a significant improvement compared to those restored by using just nonlinear filters or regularization methods only. Our scheme can remove salt-and-pepper-noise with a noise level as high as 90%.

Document digitization is faster and more affordable using digital cameras than scanners. On the other hand, if we aim at extending the basic digital camera functionalities for such a purpose, post-processings will be of first importance, at least to improve the text legibility. In this paper, we address the specific problem of the virtual flattening of curved documents, as for example the pages of an opened book lying on its spine. In order to compute the document shape, we use the shape from shading technique and discuss why, in some cases, it is more suitable than other 3D single-view reconstruction techniques. We extend the seminal work by Wada et al. (Proceedings of the IAPR Workshop on machine vision and applications, Tokyo, Japan, pp. 591–594, 1992) and consecutive papers, reformulating the problem in terms of perspective shape from shading. Finally, we design a complete post-processing algorithm and test it on real images. Even if the documents are much curved, it is shown that the restored images are almost identical to scanned images of the flattened documents.

A model based approach for rectifying the camera image of the bound document has been developed, i.e., the surface of the document is represented by a general cylindrical surface. The principle of using the model to unwrap the image is discussed. Practically, the skeleton of each horizontal text line is extracted to help estimate the parameter of the model, and rectify the images. To use the model, only a few priori is required, and no more auxiliary device is necessary. Experiment results are given to demonstrate the feasibility and the stability of the method.


Question: Theoretical analysis and practical application of nonlinear camera response functions in image deblurring.

This paper introduces an advanced radiometric calibration method designed to address the inconsistencies in image brightness that often arise due to nonlinear camera response functions, exposure variations, and vignetting effects. These factors can significantly impact the visual quality of image mosaics and textures used in 3D models, leading to color inconsistencies and visible boundaries. Our proposed algorithm comprehensively calibrates the radiometric response, accounting for exposure adjustments and vignetting. By separating the vignetting effect from the response function estimation, we ensure robustness against noise and outliers. Through rigorous testing with both synthetic and real datasets, we demonstrate the superiority of our algorithm over existing techniques. The results of our radiometric calibration are utilized to seamlessly align images for creating mosaics and enhancing 3D model textures. Additionally, our method enables the creation of high dynamic range (HDR) mosaics that more accurately reflect the true scene characteristics compared to standard mosaics.

This paper presents a new document image dewarping method that removes geometric distortions in camera-captured document images. The proposed method does not directly use the text-line which has been the most widely used feature for the document dewarping. Instead, we use the discrete representation of text-lines and text-blocks which are the sets of connected components. Also, we model the geometric distortions caused by page curl and perspective view as the generalized cylindrical surfaces and camera rotation respectively. With these distortion models and the discrete representation of the features, we design a cost function whose minimization yields the parameters of the distortion model. In the cost function, we encode the properties of the pages such as text-block alignment, linespacing, and the straightness of text-lines. By describing the text features using the sets of discrete points, the cost function can be easily defined and efficiently solved by Levenberg–Marquadt algorithm. Experiments show that the proposed method works well for the various layouts and curved surfaces, and compares favorably with the conventional methods on the standard dataset. & 2015 Elsevier Ltd. All rights reserved.

This paper represents a twofold approach. On the one hand, we introduce a simple method that encompasses an ensemble of image processing algorithms with a multilevel color transfer approach at its core. On the other hand, the method is applied for providing an artistic look to standard images. The approach proposes a multilevel color transfer in a chromatic channel of the CIELAB color space. Once converted from red, green and blue, a specific channel on both images, input and target (reference), is thresholded in a number of levels. Later, the color transfer is performed between regions from corresponding levels using a classical color transfer method. In the application phase, the color palette of a recognized artwork of the Fauve movement is mapped to the input image, emulating the sight of the artist, characterized by the use of vivid colors. Filtering techniques are applied to the outcome, in order to emulate the basic brushstrokes of the artist. Experimental results are shown, visualizing and comparing the input images with the outcomes.

The human face is one of the most interesting subjects involved in numerous applications. Significant progress has been made towards the image deblurring problem, however, existing generic deblurring methods are not able to achieve satisfying results on blurry face images. The success of the state-of-the-art image deblurring methods stems mainly from implicit or explicit restoration of salient edges for kernel estimation. When there is not much texture in the blurry image (e.g., face images), existing methods are less effective as only few edges can be used for kernel estimation. Moreover, recent methods are usually jeopardized by selecting ambiguous edges, which are imaged from the same edge of the object after blur, for kernel estimation due to local edge selection strategies. In this paper, we address these problems of deblurring face images by exploiting facial structures. We propose a maximum a posteriori (MAP) deblurring algorithm based on an exemplar dataset, without using the coarse-to-fine strategy or ad-hoc edge selections. Extensive evaluations against state-of-the-art methods demonstrate the effectiveness of the proposed algorithm for deblurring face images. We also show the extendability of our method to other specific deblurring tasks.

The accurate and quick modelling of inverse lens distortion to rectify images or predict the image coordinates of real world objects has long been a challenge. This work investigates the use of artificial neural networks to perform this modelling. Several architectures are investigated and multiple methods of training the networks are used to achieve the best results. The error is expressed as a physical displacement on the imaging chip so that a fair comparison can be made between other published results which are based on cameras with different resolutions. It is shown that the novel application of locally optimised neural networks to residual lens calibration data yields an inverse distortion modelling that is highly competitive with prior published results.

This paper introduces a new compositional framework for classifying color correction methods according to their two main computational units. The framework was used to dissect fifteen among the best color correction algorithms and the computational units so derived, with the addition of four new units specifically designed for this work, were then reassembled in a combinatorial way to originate about one hundred distinct color correction methods, most of which never considered before. The above color correction methods were tested on three different existing datasets, including both real and artificial color transformations, plus a novel dataset of real image pairs categorized according to the kind of color alterations induced by specific acquisition setups. Differently from previous evaluations, special emphasis was given to effectiveness in real world applications, such as image mosaicing and stitching, where robustness with respect to strong image misalignments and light scattering effects is required. Experimental evidence is provided for the first time in terms of the most recent perceptual image quality metrics, which are known to be the closest to human judgment. Comparative results show that combinations of the new computational units are the most effective for real stitching scenarios, regardless of the specific source of color alteration. On the other hand, in the case of accurate image alignment and artificial color alterations, the best performing methods either use one of the new computational units, or are made up of fresh combinations of existing units.

This paper proposes a two-phase scheme for removing salt-and-pepper impulse noise. In the first phase, an adaptive median filter is used to identify pixels which are likely to be contaminated by noise (noise candidates). In the second phase, the image is restored using a specialized regularization method that applies only to those selected noise candidates. In terms of edge preservation and noise suppression, our restored images show a significant improvement compared to those restored by using just nonlinear filters or regularization methods only. Our scheme can remove salt-and-pepper-noise with a noise level as high as 90%.

Example learning-based single image super-resolution (SR) is a promising method for reconstructing a high-resolution (HR) image from a single-input low-resolution (LR) image. Lots of popular SR approaches are more likely either time-or space-intensive, which limit their practical applications. Hence, some research has focused on a subspace view and delivered state-of-the-art results. In this paper, we utilize an effective way with mixture prior models to transform the large nonlinear feature space of LR images into a group of linear subspaces in the training phase. In particular, we first partition image patches into several groups by a novel selective patch processing method based on difference curvature of LR patches, and then learning the mixture prior models in each group. Moreover, different prior distributions have various effectiveness in SR, and in this case, we find that student-t prior shows stronger performance than the well-known Gaussian prior. In the testing phase, we adopt the learned multiple mixture prior models to map the input LR features into the appropriate subspace, and finally reconstruct the corresponding HR image in a novel mixed matching way. Experimental results indicate that the proposed approach is both quantitatively and qualitatively superior to some state-of-the-art SR methods.

Recent advances in computer technology have made digital image tampering more and more common. In this paper, we propose an authentic vs. spliced image classification method making use of geometry invariants in a semi-automatic manner. For a given image, we identify suspicious splicing areas, compute the geometry invariants from the pixels within each region, and then estimate the camera response function (CRF) from these geometry invariants. The cross-fitting errors are fed into a statistical classifier. Experiments show a very promising accuracy, 87%, over a large data set of 363 natural and spliced images. To the best of our knowledge, this is the first work detecting image splicing by verifying camera characteristic consistency from a single-channel image

In order to simultaneously sharpen image details and attenuate noise, we propose to combine the recent blockmatching and 3D Þltering (BM3D) denoising approach, based on 3D transform-domain collaborative Þltering, with alpha-rooting, a transform-domain sharpening technique. The BM3D exploits grouping of similar image blocks into 3D arrays (groups) on which collaborative Þltering (by hard-thresholding) is applied. We propose two approaches of sharpening by alpha-rooting; the Þrst applies alpharooting individually on the 2D transform spectra of each grouped block; the second applies alpha-rooting on the 3D-transform spectra of each 3D array in order to sharpen Þne image details shared by all grouped blocks and further enhance the interblock differences. The conducted experiments with the proposed method show that it can preserve and sharpen Þne image details and effectively attenuate noise.

In this paper we compare the performance of a number of representative instrumental models for image dissimilarity with respect to their ability to predict both image dissimilarity and image quality, as perceived by human subjects. Two sets of experimental data, one for images degraded by noise and blur, and one for JPEG-coded images, are used in the comparison. ( 1998 Elsevier Science B.V. All rights reserved.


Question: Theoretical analysis and practical application of nonlinear camera response functions in image deblurring techniques.

This supplementary material elaborates on the techniques used for fast image deconvolution, focusing on the hyper-Laplacian priors approach as detailed in our NIPS 2009 paper [1]. In addressing the w sub-problem (Eqn. 5) in [1], we opt for a lookup table (LUT) method or an analytical solution for certain values of α. Nonetheless, various numerical alternatives are viable. Among the simplest and fastest is the Newton-Raphson (NR) method for finding the roots of the polynomials outlined in Eqn. 10 and Eqn. 11 of [1]. As indicated in Table 1, four iterations of NR yield results comparable in time to our analytical solution. However, this numerical approach exhibits poor stability, especially when the polynomials nearly touch the x-axis. In practical applications, we observed erroneous solutions approximately 1-2% of the time, a frequency that is unacceptable in the context of our iterative algorithm. Direct application of NR to Eqn. 5 in [1] also proves unreliable. While more stable alternatives to NR are available, they come at a higher computational cost. For instance, we experimented with Matlab’s roots command, which solves the problem by finding eigenvalues, but this method was significantly slower than our analytical strategy.

In this paper, we attempt a challenging task to unify two important complementary operations, i.e. contrast enhancement and denoising, that is required in most image processing applications. The proposed method is implemented using practical analog circuit configurations that can lead to near real-time processing capabilities useful to be integrated with vision sensors. Metrics used for performance includes estimation of Residual Noise Level (RNL), Structural Similarity Index Measure (SSIM), Output-to-Input Contrast Ratio (CRo_i), and its combined score (SCD). The class of contrast stretching methods has resulted in higher noise levels (RNL ≥ 7) along with increased contrast measures (CRo-i ≥ eight times than that of the input image) and SSIM ≤ 0.52. Denoising methods generates images with lesser noise levels (RNL ≤ 0.2308), poor contrast enhancements (CRo-i ≤ 1.31) and with best structural similarity (SSIM ≥ 0.85). In contrast, the proposed model offers best contrast stretching (CRo-i = 5.83), least noise (RNL = 0.02), a descent structural similarity (SSIM = 0.6453) and the highest combined score (SCD = 169).

Photometric stereo is one technique for 3D shape determination that has been implemented in a variety of experimental settings and that has produced consistently good results. The idea is to use intensity values recorded from multiple images obtained from the same viewpoint but under different conditions of illumination. The resulting radiometric constraint makes it possible to obtain local estimates of both surface orientation and surface curvature without requiring global smoothness assumptions and without requiring prior image segmentation. This paper moves photometric stereo one step closer to practical viability by describing an experimental setting in which surface gradient estimation is achieved on full frame video data at near video frame rates (i.e., 15Hz). The implementation uses commercially available hardware. Reflectance is modeled empirically using measurements obtained from a calibration sphere. Estimation of the gradient, (p, q), requires only simple table lookup. Curvature estimation uses, in addition, the reflectance map, R(p, q). The required lookup table and reflectance maps are derived during calibration. Because reflectance is modeled empirically, no prior physical model of the reflectance characteristics of the objects to be analyzed is assumed. At the same time, if a good physical model is available, it can be retrofit to the method for implementation purposes. Photometric stereo is subject to error in the presence of cast shadows and interreflection. No purely local technique can succeed since these phenomena are inherently non-local. Nevertheless, this paper demonstrates that one can exploit the redundancy in three light source photometric stereo to, in most cases, locally detect the presence of cast shadows and interreflection. Detection is facilitated by explicitly including a local confidence estimate in the lookup table used for gradient estimation.

Guided image filter (GIF) is a well-known local filter for its edge-preserving property and low computational complexity. Unfortunately, the GIF may suffer from halo artifacts, because the local linear model used in the GIF cannot represent the image well near some edges. In this paper, a gradient domain GIF is proposed by incorporating an explicit first-order edge-aware constraint. The edge-aware constraint makes edges be preserved better. To illustrate the efficiency of the proposed filter, the proposed gradient domain GIF is applied for single-image detail enhancement, tone mapping of high dynamic range images and image saliency detection. Both theoretical analysis and experimental results prove that the proposed gradient domain GIF can produce better resultant images, especially near the edges, where halos appear in the original GIF.

Contrary to the normal belief we show that selfocclusion is present in any real aperture image and we present a method on how we can take care of the occlusion while recovering the depth using the defocus as the cue. The spacevariant blur is modeled as an MRF and the MAP estimates are obtained f o r both the depth map and the everywhere f o cused intensity image. The blur kernel is adjusted in the regions where occlusion is present, particularly at the regions of discontinuities in the scene. The performance of the proposed algorithm is tested over synthetic data and the estimates are found to be better than the earlier schemes where such subtle effects were ignored.

Wavelet frame systems are known to be effective in capturing singularities from noisy and degraded images. In this paper, we introduce a new edge driven wavelet frame model for image restoration by approximating images as piecewise smooth functions. With an implicit representation of image singularities sets, the proposed model inflicts different strength of regularization on smooth and singular image regions and edges. The proposed edge driven model is robust to both image approximation and singularity estimation. The implicit formulation also enables an asymptotic analysis of the proposed models and a rigorous connection between the discrete model and a general continuous variational model. Finally, numerical results on image inpainting and deblurring show that the proposed model is compared favorably against several popular image restoration models.

This is Part II of the paper, "Gray-Level Grouping (GLG): an Automatic Method for Optimized Image Contrast Enhancement". Part I of this paper introduced a new automatic contrast enhancement technique: gray-level grouping (GLG). GLG is a general and powerful technique, which can be conveniently applied to a broad variety of low-contrast images and outperforms conventional contrast enhancement techniques. However, the basic GLG method still has limitations and cannot enhance certain classes of low-contrast images well, e.g., images with a noisy background. The basic GLG also cannot fulfill certain special application purposes, e.g., enhancing only part of an image which corresponds to a certain segment of the image histogram. In order to break through these limitations, this paper introduces an extension of the basic GLG algorithm, selective gray-level grouping (SGLG), which groups the histogram components in different segments of the grayscale using different criteria and, hence, is able to enhance different parts of the histogram to various extents. This paper also introduces two new preprocessing methods to eliminate background noise in noisy low-contrast images so that such images can be properly enhanced by the (S)GLG technique. The extension of (S)GLG to color images is also discussed in this paper. SGLG and its variations extend the capability of the basic GLG to a larger variety of low-contrast images, and can fulfill special application requirements. SGLG and its variations not only produce results superior to conventional contrast enhancement techniques, but are also fully automatic under most circumstances, and are applicable to a broad variety of images.

There is a close connection between the calibration of a single camera and the epipolar transformation obtained when the camera undergoes a displacement. The epipolar transformation imposes two algebraic constraints on the camera calibration. If two epipolar transformations, arising from different camera displacements, are available then the compatible camera calibrations are parameterized by an algebraic curve of genus four. The curve can be represented either by a space curve of degree seven contained in the intersection of two cubic surfaces, or by a curve of degree six in the dual of the image plane. The curve in the dual plane has one singular point of order three and three singular points of order two. If three epipolar transformations are available, then two curves of degree six can be obtained in the dual plane such that one of the real intersections of the two yields the correct camera calibration. The two curves have a common singular point of order three. Experimental results are given to demonstrate the feasibility of camera calibration based on the epipolar transformation. The real intersections of the two dual curves are found by locating the zeros of a function defined on the interval [0, 2π]. The intersection yielding the correct camera calibration is picked out by referring back to the three epipolar transformations.

The depth images acquired by consumer depth sensors (e.g., Kinect and ToF) usually are of low resolution and insufficient quality. One natural solution is to incorporate with high resolution RGB camera for exploiting their statistical correlation. However, most existing methods are intuitive and limited in characterizing the complex and dynamic dependency between intensity and depth images. To address these limitations, we propose a weighted analysis representation model for guided depth image enhancement, which advances the conventional methods in two aspects: (i) task driven learning and (ii) dynamic guidance. First, we generalize the analysis representation model by including a guided weight function for dependency modeling. And the task-driven learning formulation is introduced to obtain the optimized guidance tailored to specific enhancement task. Second, the depth image is gradually enhanced along with the iterations, and thus the guidance should also be dynamically adjusted to account for the updating of depth image. To this end, stage-wise parameters are learned for dynamic guidance. Experiments on guided depth image upsampling and noisy depth image restoration validate the effectiveness of our method.

Images captured under hazy conditions have low contrast and poor color. This is primarily due to air-light which degrades image quality according to the transmission map. The approach to enhance these hazy images we introduce here is based on the `Dark-Channel Prior' method with image refinement by the `Weighted Least Square' based edge-preserving smoothing. Local contrast is further enhanced by multi-scale tone manipulation. The proposed method improves the contrast, color and detail for the entire image domain effectively. In the experiment, we compare the proposed method with conventional methods to validate performance.

Solid freeform fabrication (SFF) processes based on mask image projection such as digital micro-mirror devices (DMD) have the potential to be fast and inexpensive. More and more research and commercial systems have been developed based on such digital devices. However, a digital light processing (DLP) projector based on DMD has limited resolution and certain image blurring. In order to use a DLP projector in the large-area mask projection stereolithography, it is critical to plan mask images in order to achieve high accuracy and resolution. Based on our previous work on optimized pixel blending, we present a calibration method for capturing the non-uniformity of a projection image by a low cost off-the-shelf DLP projector. Our method is based on two calibration systems, a geometric calibration system that can calibrate the position, shape, size, and orientation of a pixel and an energy calibration system that can calibrate the light intensity of a pixel. Based on both results, the light intensity at various grayscale levels can be approximated for each pixel. Developing a library of such approximation functions is critical for the optimized pixel blending to generate a better mask image plan. Experimental results verify our calibration results.


Question: Theoretical analysis and practical considerations for nonlinear camera response functions in image deblurring.

This paper introduces a novel approach for enhancing the performance of motion deblurring algorithms in the presence of significant noise, a common issue in low-light imaging scenarios. Existing state-of-the-art methods are often inadequate when confronted with high noise levels in blurred images. To address this, we propose an innovative technique that integrates denoising and deblurring processes to improve image quality. Our method leverages the estimated motion blur kernels to optimize denoising, ensuring that the denoised image remains consistent with the camera's motion (i.e., eliminating high-frequency noise that does not align with the motion blur). This refined denoising process facilitates more accurate blur kernel estimation and enhances deblurring outcomes. The denoising and deblurring operations are performed iteratively, resulting in superior image quality compared to methods that either regularize deblurring to suppress noise or apply denoising as a pre-processing step. This enhancement is substantiated through experimental evidence, both quantitatively and qualitatively, using a variety of image samples.

1 Consultant, Maplewood, MN USA 4 * Correspondence: sahyun@infionline.net; 1-(651)-927-9686 5 6 Abstract: We examined a series of real-world, pictorial photographs with varying 7 characteristics, along with their modification by noise addition and unsharp masking. As 8 response metrics we used three different versions of the aesthetic measure originally 9 proposed by Birkhoff. The first aesthetic measure, which has been used in other studies, 10 and which we used in our previous work as well, showed a preference for the least 11 complex of the images. It provided no justification for noise addition, but did reveal 12 enhancement on unsharp masking. Optimum level of unsharp masking varied with the 13 image, but was predictable from the individual image’s GIF compressibility. We expect this 14 result to be useful for guiding the processing of pictorial photographic imagery. The 15 second aesthetic measure, that of informational aesthetics based on entropy alone failed 16 to provide useful discrimination among the images or the conditions of their modification. 17 A third measure, derived from the concepts of entropy maximization, as well as the 18 hypothesized preference of observers for “simpler”, i.e., more compressible, images, 19 yielded qualitatively the same results as the more traditional version of the measure. 20 Differences among the photographs and the conditions of their modification were more 21 clearly defined with this metric, however. 22

With the proliferation of cameras on mobile devices there is an increased desire to image document pages as an alternative to scanning. However, the quality of captured document images is often lower than its scanned equivalent due to hardware limitations and stability issues. In this context, automatic assessment of the quality of captured images is useful for many applications. Although there has been a lot of work on developing computational methods and creating standard datasets for natural scene image quality assessment, until recently quality estimation of camera captured document images has not been given much attention. One traditional quality indicator for document images is the Optical Character Recognition (OCR) accuracy. In this work, we present a dataset of camera captured document images containing varying levels of focal-blur introduced manually during capture. For each image we obtained the character level OCR accuracy. Our dataset can be used to evaluate methods for predicting OCR quality of captured documents as well as enhancements. In order to make the dataset publicly and freely available, originals from two existing datasets University of Washington dataset and Tobacco Database were selected. We present a case study with three recent methods for predicting the OCR quality of images on our dataset.

Advancements in deep learning have ignited an explosion of research on efficient hardware for embedded computer vision. Hardware vision acceleration, however, does not address the cost of capturing and processing the image data that feeds these algorithms. We examine the role of the image signal processing (ISP) pipeline in computer vision to identify opportunities to reduce computation and save energy. The key insight is that imaging pipelines should be be configurable: to switch between a traditional photography mode and a low-power vision mode that produces lower-quality image data suitable only for computer vision. We use eight computer vision algorithms and a reversible pipeline simulation tool to study the imaging system's impact on vision performance. For both CNN-based and classical vision algorithms, we observe that only two ISP stages, demosaicing and gamma compression, are critical for task performance. We propose a new image sensor design that can compensate for these stages. The sensor design features an adjustable resolution and tunable analog-to-digital converters (ADCs). Our proposed imaging system's vision mode disables the ISP entirely and configures the sensor to produce subsampled, lower-precision image data. This vision mode can save ~75% of the average energy of a baseline photography mode with only a small impact on vision task accuracy.

The method described here for recovering the shape of a surface from a shaded image can deal with complex, wrinkled surfaces. Integrability can be enforced easily because both surface height and gradient are represented. (A gradient field is integrable if it is the gradient of some surface height function.) The robustness of the method stems in part from linearization of the reflectance map about the current estimate of the surface orientation at each picture cell. (The reflectance map gives the dependence of scene radiance on surface orientation.) The new scheme can find an exact solution of a given shape-from-shading problem even though a regularizing term is included. The reason is that the penalty term is needed only to stabilize the iterative scheme when it is far from the correct solution; it can be turned off as the solution is approached. This is a reflection of the fact that shape-from-shading problems are not ill posed when boundary conditions are available, or when the image contains singular points. This article includes a review of previous work on shape from shading and photoclinometry. Novel features of the new scheme are introduced one at a time to make it easier to see what each contributes. Included is a discussion of implementation details that are important if exact algebraic solutions of synthetic shape-from-shading problems are to be obtained. The hope is that better performance on synthetic data will lead to better performance on real data.

Solid freeform fabrication (SFF) processes based on mask image projection such as digital micro-mirror devices (DMD) have the potential to be fast and inexpensive. More and more research and commercial systems have been developed based on such digital devices. However, a digital light processing (DLP) projector based on DMD has limited resolution and certain image blurring. In order to use a DLP projector in the large-area mask projection stereolithography, it is critical to plan mask images in order to achieve high accuracy and resolution. Based on our previous work on optimized pixel blending, we present a calibration method for capturing the non-uniformity of a projection image by a low cost off-the-shelf DLP projector. Our method is based on two calibration systems, a geometric calibration system that can calibrate the position, shape, size, and orientation of a pixel and an energy calibration system that can calibrate the light intensity of a pixel. Based on both results, the light intensity at various grayscale levels can be approximated for each pixel. Developing a library of such approximation functions is critical for the optimized pixel blending to generate a better mask image plan. Experimental results verify our calibration results.

We present novel wavelet-based inpainting algorithms. Applying ideas from anisotropic regularization and diffusion, our models can better handle degraded pixels at edges. We interpret our algorithms within the framework of forward-backward splitting methods in convex analysis and prove that the conditions for ensuring their convergence are fulfilled. Numerical examples illustrate the good performance of our algorithms.

Many image restoration algorithms in recent years are based on patch processing. The core idea is to decompose the target image into fully overlapping patches, restore each of them separately, and then merge the results by a plain averaging. This concept has been demonstrated to be highly effective, leading often times to the state-of-the-art results in denoising, inpainting, deblurring, segmentation, and other applications. While the above is indeed effective, this approach has one major flaw: the prior is imposed on intermediate (patch) results, rather than on the final outcome, and this is typically manifested by visual artifacts. The expected patch log likelihood (EPLL) method by Zoran and Weiss was conceived for addressing this very problem. Their algorithm imposes the prior on the patches of the final image, which in turn leads to an iterative restoration of diminishing effect. In this paper, we propose to further extend and improve the EPLL by considering a multi-scale prior. Our algorithm imposes the very same prior on different scale patches extracted from the target image. While all the treated patches are of the same size, their footprint in the destination image varies due to subsampling. Our scheme comes to alleviate another shortcoming existing in patch-based restoration algorithms-the fact that a local (patch-based) prior is serving as a model for a global stochastic phenomenon. We motivate the use of the multi-scale EPLL by restricting ourselves to the simple Gaussian case, comparing the aforementioned algorithms and showing a clear advantage to the proposed method. We then demonstrate our algorithm in the context of image denoising, deblurring, and super-resolution, showing an improvement in performance both visually and quantitatively.

The last decade has seen an astronomical shift from imaging with DSLR and point-and-shoot cameras to imaging with smartphone cameras. Due to the small aperture and sensor size, smartphone images have notably more noise than their DSLR counterparts. While denoising for smartphone images is an active research area, the research community currently lacks a denoising image dataset representative of real noisy images from smartphone cameras with high-quality ground truth. We address this issue in this paper with the following contributions. We propose a systematic procedure for estimating ground truth for noisy images that can be used to benchmark denoising performance for smartphone cameras. Using this procedure, we have captured a dataset - the Smartphone Image Denoising Dataset (SIDD) - of ~30,000 noisy images from 10 scenes under different lighting conditions using five representative smartphone cameras and generated their ground truth images. We used this dataset to benchmark a number of denoising algorithms. We show that CNN-based methods perform better when trained on our high-quality dataset than when trained using alternative strategies, such as low-ISO images used as a proxy for ground truth data.

The most effective superresolution methods proposed in the literature require precise knowledge of the so-called point spread function of the imager, while in practice its accurate estimation is nearly impossible. This paper presents a new superresolution method, whose main feature is its ability to account for the scant knowledge of the imager point spread function. This ability is based on representing this imprecise knowledge via a non-additive neighborhood function. The superresolution reconstruction algorithm transfers this imprecise knowledge to output by producing an imprecise (interval-valued) high-resolution image. We propose some experiments illustrating the robustness of the proposed method with respect to the imager point spread function. These experiments also highlight its high performance compared with very competitive earlier approaches. Finally, we show that the imprecision of the high-resolution interval-valued reconstructed image is a reconstruction error marker.

The image gradient is a commonly computed image feature and a potentially predictive factor for image quality assessment (IQA). Indeed, it has been successfully used for both fulland noreference image quality prediction. However, the gradient orientation has not been deeply explored as a predictive source of information for image quality assessment. Here we seek to amend this by studying the quality relevance of the relative gradient orientation, viz., the gradient orientation relative to the surround. We also deploy a relative gradient magnitude feature which accounts for perceptual masking and utilize an AdaBoosting back-propagation (BP) neural network to map the image features to image quality. The generalization of the AdaBoosting BP neural network results in an effective and robust quality prediction model. The new model, called Oriented Gradients Image Quality Assessment (OG-IQA), is shown to deliver highly competitive image quality prediction performance as compared with the most popular IQA approaches. Furthermore, we show that OG-IQA has good database independence properties and a low complexity. & 2015 Elsevier B.V. All rights reserved.


Question: StochasticNet: Creating deep neural networks through stochastic connectivity mechanisms.

This paper presents a novel technique for regularizing deep convolutional neural networks (CNNs) through stochastic connectivity, termed StochasticNet. Our method innovatively replaces traditional deterministic pooling mechanisms with a probabilistic approach, introducing randomness in the pooling phase. Instead of deterministically selecting the maximum or average value from each pooling region, StochasticNet randomly selects activations based on a multinomial distribution, directly influenced by the activities within that region. This stochastic process inherently regularizes the network without the need for additional hyperparameters. Moreover, StochasticNet can be seamlessly integrated with other regularization strategies, such as dropout and data augmentation, to further enhance its effectiveness. Demonstrating the superiority of our method, StochasticNet achieves state-of-the-art performance on multiple image datasets, outperforming competing approaches that do not employ data augmentation techniques.

We describe a neural network architecture designed to learn the musical structure of jazz melodies over chord progressions, then to create new melodies over arbitrary chord progressions from the resulting connectome (representation of neural network structure). Our architecture consists of two sub-networks, the interval expert and the chord expert, each being LSTM (long short-term memory) recurrent networks. These two sub-networks jointly learn to predict a probability distribution over future notes conditioned on past notes in the melody. We describe a training procedure for the network and an implementation as part of the opensource Impro-Visor (Improvisation Advisor) application, and demonstrate our method by providing improvised melodies based on a variety of training sets.

Software frameworks for neural networks play key roles in the development and application of deep learning methods. However, as new types of deep learning models are developed, existing frameworks designed for convolutional neural networks are becoming less useful. In this paper, we introduce Chainer, a Pythonbased, standalone open source framework for deep learning models. Chainer provides a flexible, intuitive, and high performance means of implementing a full range of deep learning models, including state-of-the-art models such as recurrent neural networks and variational autoencoders.

To what extent is the success of deep visualization due to the training? Could we do deep visualization using untrained, random weight networks? To address this issue, we explore new and powerful generative models for three popular deep visualization tasks using untrained, random weight convolutional neural networks. First we invert representations in feature spaces and reconstruct images from white noise inputs. The reconstruction quality is statistically higher than that of the same method applied on well trained networks with the same architecture. Next we synthesize textures using scaled correlations of representations in multiple layers and our results are almost indistinguishable with the original natural texture and the synthesized textures based on the trained network. Third, by recasting the content of an image in the style of various artworks, we create artistic images with high perceptual quality, highly competitive to the prior work of Gatys et al. on pretrained networks. To our knowledge this is the first demonstration of image representations using untrained deep neural networks. Our work provides a new and fascinating tool to study the representation of deep network architecture and sheds light on new understandings on deep visualization.

Neural networks have great success in many machine learning applications, but the fundamental learning theory behind them remains largely unsolved. Learning neural networks is NP-hard, but in practice, simple algorithms like stochastic gradient descent (SGD) often produce good solutions. Moreover, it is observed that overparameterization (that is, designing networks whose number of parameters is larger than statistically needed to perfectly fit the data) improves both optimization and generalization, appearing to contradict traditional learning theory. In this work, we extend the theoretical understanding of two and three-layer neural networks in the overparameterized regime. We prove that, using overparameterized neural networks, one can (improperly) learn some notable hypothesis classes, including two and three-layer neural networks with fewer parameters. Moreover, the learning process can be simply done by SGD or its variants in polynomial time using polynomially many samples. We also show that for a fixed sample size, the generalization error of the solution found by some SGD variant can be made almost independent of the number of parameters in the overparameterized network. ∗V1 appears on this date and V2 polishes writing and tightens parameters. Authors sorted in alphabetical order.

Generalization ability is widely acknowledged as one of the most important criteria to evaluate the quality of unsupervised models. The objective of our research is to find a better dropout method to improve the generalization ability of convolutional deep belief network (CDBN), an unsupervised learning model for vision tasks. In this paper, the phenomenon of low feature diversity during the training process is investigated. The attention mechanism of human visual system is more focused on rare events and depresses well-known facts. Inspired by this mechanism, No Feature Left Behind Dropout (NFLB Dropout), an adaptive dropout method is firstly proposed to automatically adjust the dropout rate feature-wisely. In the proposed method, the algorithm drops well-trained features and keeps poorly-trained ones with a high probability during training iterations. In addition, we apply two approximations of the quality of features, which are inspired by theory of saliency and optimization. Compared with the model trained by standard dropout, experiment results show that our NFLB Dropout method improves not only the accuracy but the convergence speed as well.

A neural network model for a mechanism of visual pattern recognition is proposed in this paper. The network is self-organized by “learning without a teacher”, and acquires an ability to recognize stimulus patterns based on the geometrical similarity (Gestalt) of their shapes without affected by their positions. This network is given a nickname “neocognitron”. After completion of self-organization, the network has a structure similar to the hierarchy model of the visual nervous system proposed by Hubel and Wiesel. The network consits of an input layer (photoreceptor array) followed by a cascade connection of a number of modular structures, each of which is composed of two layers of cells connected in a cascade. The first layer of each module consists of “S-cells”, which show characteristics similar to simple cells or lower order hypercomplex cells, and the second layer consists of “C-cells” similar to complex cells or higher order hypercomplex cells. The afferent synapses to each S-cell have plasticity and are modifiable. The network has an ability of unsupervised learning: We do not need any “teacher” during the process of self-organization, and it is only needed to present a set of stimulus patterns repeatedly to the input layer of the network. The network has been simulated on a digital computer. After repetitive presentation of a set of stimulus patterns, each stimulus pattern has become to elicit an output only from one of the C-cell of the last layer, and conversely, this C-cell has become selectively responsive only to that stimulus pattern. That is, none of the C-cells of the last layer responds to more than one stimulus pattern. The response of the C-cells of the last layer is not affected by the pattern's position at all. Neither is it affected by a small change in shape nor in size of the stimulus pattern.

Recent efforts to combine Representation Learning with Formal Methods, commonly known as Neuro-Symbolic Methods, have given rise to a new trend of applying rich neural architectures to solve classical combinatorial optimization problems. In this paper, we propose a neural framework that can learn to solve the Circuit Satisfiability problem. Our framework is built upon two fundamental contributions: a rich embedding architecture that encodes the problem structure, and an end-to-end differentiable training procedure that mimics Reinforcement Learning and trains the model directly toward solving the SAT problem. The experimental results show the superior out-of-sample generalization performance of our framework compared to the recently developed NeuroSAT method.

Mounting concerns over variability, defects, and noise motivate a new approach for digital circuitry: stochastic logic, that is to say, logic that operates on probabilistic signals and so can cope with errors and uncertainty. Techniques for probabilistic analysis of circuits and systems are well established. We advocate a strategy for synthesis. In prior work, we described a methodology for synthesizing stochastic logic, that is to say logic that operates on probabilistic bit streams. In this paper, we apply the concept of stochastic logic to a reconfigurable architecture that implements processing operations on a datapath. We analyze cost as well as the sources of error: approximation, quantization, and random fluctuations. We study the effectiveness of the architecture on a collection of benchmarks for image processing. The stochastic architecture requires less area than conventional hardware implementations. Moreover, it is much more tolerant of soft errors (bit flips) than these deterministic implementations. This fault tolerance scales gracefully to very large numbers of errors.

Many modern neural simulators focus on the simulation of networks of spiking neurons on parallel hardware. Another important framework in computational neuroscience, rate-coded neural networks, is mostly difficult or impossible to implement using these simulators. We present here the ANNarchy (Artificial Neural Networks architect) neural simulator, which allows to easily define and simulate rate-coded and spiking networks, as well as combinations of both. The interface in Python has been designed to be close to the PyNN interface, while the definition of neuron and synapse models can be specified using an equation-oriented mathematical description similar to the Brian neural simulator. This information is used to generate C++ code that will efficiently perform the simulation on the chosen parallel hardware (multi-core system or graphical processing unit). Several numerical methods are available to transform ordinary differential equations into an efficient C++code. We compare the parallel performance of the simulator to existing solutions.

We describe a new learning procedure, back-propagation, for networks of neurone-like units. The procedure repeatedly adjusts the weights of the connections in the network so as to minimize a measure of the difference between the actual output vector of the net and the desired output vector. As a result of the weight adjustments, internal ‘hidden’ units which are not part of the input or output come to represent important features of the task domain, and the regularities in the task are captured by the interactions of these units. The ability to create useful new features distinguishes back-propagation from earlier, simpler methods such as the perceptron-convergence procedure1.


Question: StochasticNet: Creating deep neural networks through stochastic connectivity mechanisms.

In this study, we present an advanced deep convolutional neural network (CNN) architecture designed to classify the extensive ImageNet LSVRC-2010 dataset, which comprises over 1.2 million high-resolution images across 1000 distinct categories. Our model achieved remarkable performance, with top-1 and top-5 error rates of 37.5% and 17.0% respectively, significantly outperforming previous state-of-the-art approaches. The CNN architecture, featuring 60 million parameters and 650,000 neurons, is composed of five convolutional layers, interspersed with max-pooling layers for dimensionality reduction, followed by three fully connected layers culminating in a 1000-way softmax layer for classification. To expedite the training process, we utilized non-saturating neurons and an optimized GPU-based convolution operation. To combat overfitting in the fully connected layers, we implemented a novel regularization technique known as "dropout," which proved highly effective. A variant of this model was also submitted to the ILSVRC-2012 competition, where it secured the top position with a top-5 test error rate of 15.3%, markedly lower than the 26.2% achieved by the runner-up.

**Question**: A Comparative Study of Deep Learning Models for Financial Time Series Forecasting

**Documents**: Forecasting Financial Time Series Using Deep Learning Models
We explore the use of deep learning models for forecasting financial time series. Specifically, we consider two deep learning models: Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU). We compare their performance with traditional models such as Autoregressive Integrated Moving Average (ARIMA) and Support Vector Regression (SVR). We use daily stock prices of S&P 500 companies for the period 2000-2017 as our dataset. We find that both LSTM and GRU outperform ARIMA and SVR in terms of forecasting accuracy. However, LSTM performs slightly better than GRU. We also find that the performance of the deep learning models is sensitive to the choice of hyperparameters. We use grid search to find the optimal hyperparameters for each model. Our results suggest that deep learning models can be useful tools for forecasting financial time series.

**Similar Documents**: This paper investigates the effectiveness of deep learning architectures in predicting financial time series data. We focus on two prominent deep learning models: Long Short-Term

Stochastic gradient algorithms are the main focus of large-scale optimization problems and led to important successes in the recent advancement of the deep learning algorithms. The convergence of SGD depends on the careful choice of learning rate and the amount of the noise in stochastic estimates of the gradients. In this paper, we propose an adaptive learning rate algorithm, which utilizes stochastic curvature information of the loss function for automatically tuning the learning rates. The information about the element-wise curvature of the loss function is estimated from the local statistics of the stochastic first order gradients. We further propose a new variance reduction technique to speed up the convergence. In our experiments with deep neural networks, we obtained better performance compared to the popular stochastic gradient algorithms.1

A neural network model for a mechanism of visual pattern recognition is proposed in this paper. The network is self-organized by “learning without a teacher”, and acquires an ability to recognize stimulus patterns based on the geometrical similarity (Gestalt) of their shapes without affected by their positions. This network is given a nickname “neocognitron”. After completion of self-organization, the network has a structure similar to the hierarchy model of the visual nervous system proposed by Hubel and Wiesel. The network consits of an input layer (photoreceptor array) followed by a cascade connection of a number of modular structures, each of which is composed of two layers of cells connected in a cascade. The first layer of each module consists of “S-cells”, which show characteristics similar to simple cells or lower order hypercomplex cells, and the second layer consists of “C-cells” similar to complex cells or higher order hypercomplex cells. The afferent synapses to each S-cell have plasticity and are modifiable. The network has an ability of unsupervised learning: We do not need any “teacher” during the process of self-organization, and it is only needed to present a set of stimulus patterns repeatedly to the input layer of the network. The network has been simulated on a digital computer. After repetitive presentation of a set of stimulus patterns, each stimulus pattern has become to elicit an output only from one of the C-cell of the last layer, and conversely, this C-cell has become selectively responsive only to that stimulus pattern. That is, none of the C-cells of the last layer responds to more than one stimulus pattern. The response of the C-cells of the last layer is not affected by the pattern's position at all. Neither is it affected by a small change in shape nor in size of the stimulus pattern.

In this paper we introduce deep Gaussian process (GP) models. Deep GPs are a deep belief network based on Gaussian process mappings. The data is modeled as the output of a multivariate GP. The inputs to that Gaussian process are then governed by another GP. A single layer model is equivalent to a standard GP or the GP latent variable model (GP-LVM). We perform inference in the model by approximate variational marginalization. This results in a strict lower bound on the marginal likelihood of the model which we use for model selection (number of layers and nodes per layer). Deep belief networks are typically applied to relatively large data sets using stochastic gradient descent for optimization. Our fully Bayesian treatment allows for the application of deep models even when data is scarce. Model selection by our variational bound shows that a five layer hierarchy is justified even when modelling a digit data set containing only 150 examples.

This paper introduces the Deep Recurrent Attentive Writer (DRAW) neural network architecture for image generation. DRAW networks combine a novel spatial attention mechanism that mimics the foveation of the human eye, with a sequential variational auto-encoding framework that allows for the iterative construction of complex images. The system substantially improves on the state of the art for generative models on MNIST, and, when trained on the Street View House Numbers dataset, it generates images that cannot be distinguished from real data with the naked eye.

This paper introduces a new methodology for dynamic learning, visualization, and classification of functional magnetic resonance imaging (fMRI) as spatiotemporal brain data. The method is based on an evolving spatiotemporal data machine of evolving spiking neural networks (SNNs) exemplified by the NeuCube architecture [1]. The method consists of several steps: mapping spatial coordinates of fMRI data into a 3-D SNN cube (SNNc) that represents a brain template; input data transformation into trains of spikes; deep, unsupervised learning in the 3-D SNNc of spatiotemporal patterns from data; supervised learning in an evolving SNN classifier; parameter optimization; and 3-D visualization and model interpretation. Two benchmark case study problems and data are used to illustrate the proposed methodology—fMRI data collected from subjects when reading affirmative or negative sentences and another one—on reading a sentence or seeing a picture. The learned connections in the SNNc represent dynamic spatiotemporal relationships derived from the fMRI data. They can reveal new information about the brain functions under different conditions. The proposed methodology allows for the first time to analyze dynamic functional and structural connectivity of a learned SNN model from fMRI data. This can be used for a better understanding of brain activities and also for online generation of appropriate neurofeedback to subjects for improved brain functions. For example, in this paper, tracing the 3-D SNN model connectivity enabled us for the first time to capture prominent brain functional pathways evoked in language comprehension. We found stronger spatiotemporal interaction between left dorsolateral prefrontal cortex and left temporal while reading a negated sentence. This observation is obviously distinguishable from the patterns generated by either reading affirmative sentences or seeing pictures. The proposed NeuCube-based methodology offers also a superior classification accuracy when compared with traditional AI and statistical methods. The created NeuCube-based models of fMRI data are directly and efficiently implementable on high performance and low energy consumption neuromorphic platforms for real-time applications.

A proposed sparse recurrent neural network with flexible topology is used for trajectory prediction of the Atlantic hurricanes. For prediction of the future trajectories of a target hurricane, the most similar hurricanes to the target hurricane are found by comparing directions of the hurricanes. Then, the first and second differences of their positions over their life time are used for training the proposed network. Comparison of the obtained predictions with actual trajectories of Sandy and Humberto hurricanes show that our approach is quite promising for this aim.

To better understand deep architectures and unsupervised learning, uncluttered by hardware details, we develop a general autoencoder framework for the comparative study of autoencoders, including Boolean autoencoders. We derive several results regarding autoencoders and autoencoder learning, including results on learning complexity, vertical and horizontal composition, and fundamental connections between critical points and clustering. Possible implications for the theory of deep architectures are discussed.

We propose a quantum generalisation of a classical neural network. The classical neurons are firstly rendered reversible by adding ancillary bits. Then they are generalised to being quantum reversible, i.e. unitary. (The classical networks we generalise are called feedforward, and have step-function activation functions.) The quantum network can be trained efficiently using gradient descent on a cost function to perform quantum generalisations of classical tasks. We demonstrate numerically that it can: (i) compress quantum states onto a minimal number of qubits, creating a quantum autoencoder, and (ii) discover quantum communication protocols such as teleportation. Our general recipe is theoretical and implementation-independent. The quantum neuron module can naturally be implemented photonically.

Several approaches to implementing symbol-like representations in neurally plausible models have been proposed. These approaches include binding through synchrony (Shastri & Ajjanagadde, ), "mesh" binding (van der Velde & de Kamps, ), and conjunctive binding (Smolensky, ). Recent theoretical work has suggested that most of these methods will not scale well, that is, that they cannot encode structured representations using any of the tens of thousands of terms in the adult lexicon without making implausible resource assumptions. Here, we empirically demonstrate that the biologically plausible structured representations employed in the Semantic Pointer Architecture (SPA) approach to modeling cognition (Eliasmith, ) do scale appropriately. Specifically, we construct a spiking neural network of about 2.5 million neurons that employs semantic pointers to successfully encode and decode the main lexical relations in WordNet, which has over 100,000 terms. In addition, we show that the same representations can be employed to construct recursively structured sentences consisting of arbitrary WordNet concepts, while preserving the original lexical structure. We argue that these results suggest that semantic pointers are uniquely well-suited to providing a biologically plausible account of the structured representations that underwrite human cognition.

Despite their massivesize, successful deep artificial neural networkscan exhibit a remarkably small differencebetween training and test performance. Conventional wisdom attributessmall generalization error either to propertiesof themodel family, or to the regularization techniquesused during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experimentsestablish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization, and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simpledepth two neural networksalready haveperfect finitesampleexpressivity assoon as thenumber of parameters exceeds thenumber of datapointsas it usually does in practice. We interpret our experimental findingsby comparison with traditional models.