Question: A generative adversarial network for photo-realistic single image super-resolution.

This paper introduces Batch Normalization, a technique aimed at accelerating the training of deep neural networks by addressing the issue of internal covariate shift. Internal covariate shift occurs when the distribution of each layer's inputs changes during training due to the alterations in the parameters of the preceding layers. This phenomenon complicates the training process, necessitating smaller learning rates and meticulous parameter initialization, particularly when dealing with models that employ saturating nonlinearities. To mitigate this challenge, we propose normalizing the layer inputs as a core component of the model architecture, specifically tailored for each training mini-batch. By integrating normalization within the network, Batch Normalization enables the use of significantly higher learning rates, reduces the sensitivity to initialization, and in certain scenarios, diminishes the reliance on Dropout. When applied to a cutting-edge image classification model, Batch Normalization achieves comparable accuracy with a remarkable 14-fold reduction in training steps, outperforming the original model by a substantial margin. Furthermore, by leveraging an ensemble of batch-normalized networks, we set a new benchmark on ImageNet classification, achieving a top-5 test error of 4.82%, surpassing human-level accuracy.

We explore methods of producing adversarial examples on deep generative models such as the variational autoencoder (VAE) and the VAE-GAN. Deep learning architectures are known to be vulnerable to adversarial examples, but previous work has focused on the application of adversarial examples to classification tasks. Deep generative models have recently become popular due to their ability to model input data distributions and generate realistic examples from those distributions. We present three classes of attacks on the VAE and VAE-GAN architectures and demonstrate them against networks trained on MNIST, SVHN and CelebA. Our first attack leverages classification-based adversaries by attaching a classifier to the trained encoder of the target generative model, which can then be used to indirectly manipulate the latent representation. Our second attack directly uses the VAE loss function to generate a target reconstruction image from the adversarial example. Our third attack moves beyond relying on classification or the standard loss for the gradient and directly optimizes against differences in source and target latent representations. We also motivate why an attacker might be interested in deploying such techniques against a target generative network.

While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on ImageNet classification has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called "perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new dataset of human perceptual similarity judgments. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by large margins on our dataset. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.

Face recognition based on the analysis of 3D scans has been an active research subject over the last few years. However, the impact of the resolution of 3D scans on the recognition process has not been addressed explicitly yet being of primal importance after the introduction of a new generation of low cost 4D scanning devices. These devices are capable of combined depth/rgb acquisition over time with a low resolution compared to the 3D scanners typically used in 3D face recognition benchmarks. In this paper, we define a super-resolution model for 3D faces by which a sequence of low-resolution 3D scans can be processed to extract a higher resolution 3D face model, namely the superface model. The proposed solution relies on the Scaled ICP procedure to align the low-resolution 3D models with each other and estimate the value of the high-resolution 3D model based on the statistics of values of the lowresolution scans in corresponding points. The approach is validated on a data set that includes, for each subject, one sequence of low-resolution 3D face scans and one ground-truth high-resolution 3D face model acquired through a high-resolution 3D scanner. In this way, results of the super-resolution process are evaluated qualitatively and quantitatively by measuring the error between the superface and the ground-truth.

Deep residual networks [1] have emerged as a family of extremely deep architectures showing compelling accuracy and nice convergence behaviors. In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. A series of ablation experiments support the importance of these identity mappings. This motivates us to propose a new residual unit, which makes training easier and improves generalization. We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62% error) and CIFAR-100, and a 200-layer ResNet on ImageNet. Code is available at: https://github.com/KaimingHe/ resnet-1k-layers.

A fundamental problem in computer vision is that of inferring the intrinsic, 3D structure of the world from flat, 2D images of that world. Traditional methods for recovering scene properties such as shape, reflectance, or illumination rely on multiple observations of the same scene to overconstrain the problem. Recovering these same properties from a single image seems almost impossible in comparison-there are an infinite number of shapes, paint, and lights that exactly reproduce a single image. However, certain explanations are more likely than others: surfaces tend to be smooth, paint tends to be uniform, and illumination tends to be natural. We therefore pose this problem as one of statistical inference, and define an optimization problem that searches for the most likely explanation of a single image. Our technique can be viewed as a superset of several classic computer vision problems (shape-from-shading, intrinsic images, color constancy, illumination estimation, etc) and outperforms all previous solutions to those constituent problems.

Texture synthesis is important for many applications in computer graphics, vision, and image processing. However, it remains difficult to design an algorithm that is both efficient and capable of generating high quality results. In this paper, we present an efficient algorithm for realistic texture synthesis. The algorithm is easy to use and requires only a sample texture as input. It generates textures with perceived quality equal to or better than those produced by previous techniques, but runs two orders of magnitude faster. This permits us to apply texture synthesis to problems where it has traditionally been considered impractical. In particular, we have applied it to constrained synthesis for image editing and temporal texture generation. Our algorithm is derived from Markov Random Field texture models and generates textures through a deterministic searching process. We accelerate this synthesis process using tree-structured vector quantization.

Discriminative deep learning approaches have shown impressive results for problems where human-labeled ground truth is plentiful, but what about tasks where labels are difficult or impossible to obtain? This paper tackles one such problem: establishing dense visual correspondence across different object instances. For this task, although we do not know what the ground-truth is, we know it should be consistent across instances of that category. We exploit this consistency as a supervisory signal to train a convolutional neural network to predict cross-instance correspondences between pairs of images depicting objects of the same category. For each pair of training images we find an appropriate 3D CAD model and render two synthetic views to link in with the pair, establishing a correspondence flow 4-cycle. We use ground-truth synthetic-to-synthetic correspondences, provided by the rendering engine, to train a ConvNet to predict synthetic-to-real, real-to-real and real-to-synthetic correspondences that are cycle-consistent with the ground-truth. At test time, no CAD models are required. We demonstrate that our end-to-end trained ConvNet supervised by cycle-consistency outperforms state-of-the-art pairwise matching methods in correspondence-related tasks.

In this paper, we use deep neural networks for inverting face sketches to synthesize photorealistic face images. We first construct a semi-simulated dataset containing a very large number of computergenerated face sketches with different styles and corresponding face images by expanding existing unconstrained face data sets. We then train models achieving state-of-the-art results on both computer-generated sketches and hand-drawn sketches by leveraging recent advances in deep learning such as batch normalization, deep residual learning, perceptual losses and stochastic optimization in combination with our new dataset. We finally demonstrate potential applications of our models in fine arts and forensic arts. In contrast to existing patch-based approaches, our deep-neuralnetwork-based approach can be used for synthesizing photorealistic face images by inverting face sketches in the wild.

Generative Adversarial Networks (GAN) have shown great promise in tasks like synthetic image generation, image inpainting, style transfer, and anomaly detection. However, generating discrete data is a challenge. This work presents an adversarial training based correlated discrete data (CDD) generation model. It also details an approach for conditional CDD generation. The results of our approach are presented over two datasets; job-seeking candidates skill set (private dataset) and MNIST (public dataset). From quantitative and qualitative analysis of these results, we show that our model performs better as it leverages inherent correlation in the data, than an existing model that overlooks correlation.

Generative adversarial networks (GANs) have given us a great tool to fit implicit generative models to data. Implicit distributions are ones we can sample from easily, and take derivatives of samples with respect to model parameters. These models are highly expressive and we argue they can prove just as useful for variational inference (VI) as they are for generative modelling. Several papers have proposed GAN-like algorithms for inference, however, connections to the theory of VI are not always well understood. This paper provides a unifying review of existing algorithms establishing connections between variational autoencoders, adversarially learned inference, operator VI, GAN-based image reconstruction, and more. Secondly, the paper provides a framework for building new algorithms: depending on the way the variational bound is expressed we introduce prior-contrastive and jointcontrastive methods, and show practical inference algorithms based on either density ratio estimation or denoising.


Question: A generative adversarial network for photo-realistic single image super-resolution.

This document details the successful implementation of a deep convolutional neural network (CNN) for ImageNet classification, achieving significant advancements in accuracy over previous methods. The model, which was trained on 1.2 million high-resolution images across 1000 distinct categories, attained top-1 and top-5 error rates of 37.5% and 17.0%, respectively, outperforming the existing state-of-the-art significantly. The architecture, featuring 60 million parameters and 650,000 neurons, is composed of five convolutional layers, interspersed with max-pooling layers, followed by three fully connected layers culminating in a 1000-way softmax classification. To expedite training, the network employs non-saturating neurons and an optimized GPU convolution operation. To mitigate overfitting in the fully connected layers, the model adopts a novel regularization technique, "dropout", which proved highly effective. A modified version of this model was submitted to the ILSVRC-2012 competition, securing a winning top-5 test error rate of 15.3%, markedly lower than the second-best entry's 26.2%. This demonstrates the model's superior performance in complex image classification tasks.

If I provide you a face image of mine (without telling you the actual age when I took the picture) and a large amount of face images that I crawled (containing labeled faces of different ages but not necessarily paired), can you show me what I would look like when I am 80 or what I was like when I was 5? The answer is probably a No. Most existing face aging works attempt to learn the transformation between age groups and thus would require the paired samples as well as the labeled query image. In this paper, we look at the problem from a generative modeling perspective such that no paired samples is required. In addition, given an unlabeled image, the generative model can directly produce the image with desired age attribute. We propose a conditional adversarial autoencoder (CAAE) that learns a face manifold, traversing on which smooth age progression and regression can be realized simultaneously. In CAAE, the face is first mapped to a latent vector through a convolutional encoder, and then the vector is projected to the face manifold conditional on age through a deconvolutional generator. The latent vector preserves personalized face features (i.e., personality) and the age condition controls progression vs. regression. Two adversarial networks are imposed on the encoder and generator, respectively, forcing to generate more photo-realistic faces. Experimental results demonstrate the appealing performance and flexibility of the proposed framework by comparing with the state-of-the-art and ground truth.

The applicability of computer vision to real paintings and artworks has been rarely investigated, even though a vast heritage would greatly benefit from techniques which can understand and process data from the artistic domain. This is partially due to the small amount of annotated artistic data, which is not even comparable to that of natural images captured by cameras. In this paper, we propose a semantic-aware architecture which can translate artworks to photo-realistic visualizations, thus reducing the gap between visual features of artistic and realistic data. Our architecture can generate natural images by retrieving and learning details from real photos through a similarity matching strategy which leverages a weakly-supervised semantic understanding of the scene. Experimental results show that the proposed technique leads to increased realism and to a reduction in domain shift, which improves the performance of pre-trained architectures for classification, detection, and segmentation. Code will be made publicly available.

The task of data generation using Generative Models has recently gained more and more attention from the scientific community, as the number of applications in which these models work surprisingly well is constantly increasing. Some examples are image and video generation, speech synthesis and style transfer, pose guided image generation, cross-domain transfer and super resolution. Contrarily to such tasks generating data coming from the banking domain poses a different challenge, due to its atypical structure when compared with traditional data and its limited availability due to privacy restrictions. In this work, we analyze the feasibility of generating spiking time series patterns appearing in the banking environment using Generative Adversarial Networks. We develop a novel end-to-end framework for training, testing and comparing different generative models using both quantitative and qualitative metrics. Finally, we propose a novel approach that combines Variational Autoencoders with Generative Adversarial Networks in order to learn a loss function for datasets in which good similarity metrics are difficult to define.

Hand gesture-to-gesture translation in the wild is a challenging task since hand gestures can have arbitrary poses, sizes, locations and self-occlusions. Therefore, this task requires a high-level understanding of the mapping between the input source gesture and the output target gesture. To tackle this problem, we propose a novel hand Gesture Generative Adversarial Network (GestureGAN). GestureGAN consists of a single generator G and a discriminator D, which takes as input a conditional hand image and a target hand skeleton image. GestureGAN utilizes the hand skeleton information explicitly, and learns the gesture-to-gesture mapping through two novel losses, the color loss and the cycle-consistency loss. The proposed color loss handles the issue of "channel pollution" while back-propagating the gradients. In addition, we present the Frechet ResNet Distance (FRD) to evaluate the quality of generated images. Extensive experiments on two widely used benchmark datasets demonstrate that the proposed GestureGAN achieves state-of-the-art performance on the unconstrained hand gesture-to-gesture translation task. Meanwhile, the generated images are in high-quality and are photo-realistic, allowing them to be used as data augmentation to improve the performance of a hand gesture classifier. Our model and code are available at https://github.com/Ha0Tang/GestureGAN.

We propose a new fast algorithm for solving one of the standard formulations of image restoration and reconstruction which consists of an unconstrained optimization problem where the objective includes an l2 data-fidelity term and a nonsmooth regularizer. This formulation allows both wavelet-based (with orthogonal or frame-based representations) regularization or total-variation regularization. Our approach is based on a variable splitting to obtain an equivalent constrained optimization formulation, which is then addressed with an augmented Lagrangian method. The proposed algorithm is an instance of the so-called alternating direction method of multipliers, for which convergence has been proved. Experiments on a set of image restoration and reconstruction benchmark problems show that the proposed algorithm is faster than the current state of the art methods.

Human 3D pose estimation from a single image is a challenging task with numerous applications. Convolutional Neural Networks (CNNs) have recently achieved superior performance on the task of 2D pose estimation from a single image, by training on images with 2D annotations collected by crowd sourcing. This suggests that similar success could be achieved for direct estimation of 3D poses. However, 3D poses are much harder to annotate, and the lack of suitable annotated training images hinders attempts towards end-to-end solutions. To address this issue, we opt to automatically synthesize training images with ground truth pose annotations. Our work is a systematic study along this road. We find that pose space coverage and texture diversity are the key ingredients for the effectiveness of synthetic training data. We present a fully automatic, scalable approach that samples the human pose space for guiding the synthesis procedure and extracts clothing textures from real images. Furthermore, we explore domain adaptation for bridging the gap between our synthetic training images and real testing photos. We demonstrate that CNNs trained with our synthetic images out-perform those trained with real photos on 3D pose estimation tasks.

Generative adversarial networks (GANs) are recently highly successful in generative applications involving images and start being applied to time series data. Here we describe EEG-GAN as a framework to generate electroencephalographic (EEG) brain signals. We introduce a modification to the improved training of Wasserstein GANs to stabilize training and investigate a range of architectural choices critical for time series generation (most notably upand down-sampling). For evaluation we consider and compare different metrics such as Inception score, Frechet inception distance and sliced Wasserstein distance, together showing that our EEG-GAN framework generated naturalistic EEG examples. It thus opens up a range of new generative application scenarios in the neuroscientific and neurological context, such as data augmentation in brain-computer interfacing tasks, EEG super-sampling, or restoration of corrupted data segments. The possibility to generate signals of a certain class and/or with specific properties may also open a new avenue for research into the underlying structure of brain signals.

Current methods for training convolutional neural networks depend on large amounts of labeled samples for supervised training. In this paper we present an approach for training a convolutional neural network using only unlabeled data. We train the network to discriminate between a set of surrogate classes. Each surrogate class is formed by applying a variety of transformations to a randomly sampled ’seed’ image patch. We find that this simple feature learning algorithm is surprisingly successful when applied to visual object recognition. The feature representation learned by our algorithm achieves classification results matching or outperforming the current state-of-the-art for unsupervised learning on several popular datasets (STL-10, CIFAR-10, Caltech-101).

Convolutional Neural Networks have been highly successful in performing a host of computer vision tasks such as object recognition, object detection, image segmentation and texture synthesis. In 2015, Gatys et. al [7] show how the style of a painter can be extracted from an image of the painting and applied to another normal photograph, thus recreating the photo in the style of the painter. The method has been successfully applied to a wide range of images and has since spawned multiple applications and mobile apps. In this paper, the neural style transfer algorithm is applied to fashion so as to synthesize new custom clothes. We construct an approach to personalize and generate new custom clothes based on a user’s preference and by learning the user’s fashion choices from a limited set of clothes from their closet. The approach is evaluated by analyzing the generated images of clothes and how well they align with the user’s fashion style.

With recent progress in graphics, it has become more tractable to train models on synthetic images, potentially avoiding the need for expensive annotations. However, learning from synthetic images may not achieve the desired performance due to a gap between synthetic and real image distributions. To reduce this gap, we propose Simulated+Unsupervised (S+U) learning, where the task is to learn a model to improve the realism of a simulators output using unlabeled real data, while preserving the annotation information from the simulator. We develop a method for S+U learning that uses an adversarial network similar to Generative Adversarial Networks (GANs), but with synthetic images as inputs instead of random vectors. We make several key modifications to the standard GAN algorithm to preserve annotations, avoid artifacts, and stabilize training: (i) a self-regularization term, (ii) a local adversarial loss, and (iii) updating the discriminator using a history of refined images. We show that this enables generation of highly realistic images, which we demonstrate both qualitatively and with a user study. We quantitatively evaluate the generated images by training models for gaze estimation and hand pose estimation. We show a significant improvement over using synthetic images, and achieve state-of-the-art results on the MPIIGaze dataset without any labeled real data.


Question: A generative adversarial network for photo-realistic single image super-resolution.

This work introduces an innovative edge-directed interpolation technique for enhancing the resolution of natural images. The core concept is to estimate local covariance coefficients from a low-resolution image and employ these estimates to adjust the interpolation at a higher resolution. This adjustment is based on the geometric duality between the low-resolution covariance and the high-resolution covariance, effectively aligning the interpolation with the orientation of edges in the image. The adaptive nature of covariance-based interpolation enables it to dynamically adjust the coefficients to accurately represent edges of any orientation. To minimize computational complexity, a hybrid method is proposed, alternating between bilinear interpolation and covariance-based adaptive interpolation. The effectiveness of this new interpolation algorithm is demonstrated in two key applications: improving the resolution of grayscale images and reconstructing color images from CCD samples. Experimental results show that the proposed algorithm significantly enhances the visual quality of interpolated images compared to standard linear interpolation techniques.

Generative Adversarial Net is a frontier method of generative models for images, audios and videos. In this paper, we focus on conditional image generation and introduce conditional Feature-Matching Generative Adversarial Net to generate images from category labels. By visualizing state-of-art discriminative conditional generative models, we find these networks do not gain clear semantic concepts. Thus we design the loss function in the light of metric learning to measure semantic distance. The proposed model is evaluated on several well-known datasets. It is shown to be of higher perceptual quality and better diversity then existing generative models.

Convolutional neural networks (CNNs) have demonstrated superior performance in super-resolution (SR). However, most CNN-based SR methods neglect the different importance among feature channels or fail to take full advantage of the hierarchical features. To address these issues, this paper presents a novel recursive unit. Firstly, at the beginning of each unit, we adopt a compact channel attention mechanism to adaptively recalibrate the channel importance of input features. Then, the multi-level features, rather than only deep-level features, are extracted and fused. Additionally, we find that it will force our model to learn more details by using the learnable upsampling method (i.e., transposed convolution) only on residual branch (instead of using it both on residual branch and identity branch) while using the bicubic interpolation on the other branch. Analytic experiments show that our method achieves competitive results compared with the state-of-the-art methods and maintains faster speed as well.

We introduce a deep network architecture called DerainNet for removing rain streaks from an image. Based on the deep convolutional neural network (CNN), we directly learn the mapping relationship between rainy and clean image detail layers from data. Because we do not possess the ground truth corresponding to real-world rainy images, we synthesize images with rain for training. In contrast to other common strategies that increase depth or breadth of the network, we use image processing domain knowledge to modify the objective function and improve deraining with a modestly sized CNN. Specifically, we train our DerainNet on the detail (high-pass) layer rather than in the image domain. Though DerainNet is trained on synthetic data, we find that the learned network translates very effectively to real-world images for testing. Moreover, we augment the CNN framework with image enhancement to improve the visual results. Compared with the state-of-the-art single image de-raining methods, our method has improved rain removal and much faster computation time after network training.

Cross-domain mapping has been a very active topic in recent years. Given one image, its main purpose is to translate it to the desired target domain, or multiple domains in the case of multiple labels. This problem is highly challenging due to three main reasons: (i) unpaired datasets, (ii) multiple attributes, and (iii) the multimodality associated with the translation. Most of the existing state-of-theart has focused only on two reasons, i.e. producing disentangled representations from unpaired datasets in a one-toone domain translation or producing multiple unimodal attributes from unpaired datasets. In this work, we propose a joint framework of diversity and multi-mapping image-toimage translations, using a single generator to conditionally produce countless and unique fake images that hold the underlying characteristics of the source image. Extensive experiments over different datasets demonstrate the effectiveness of our proposed approach with comparisons to the state-of-the-art in both multi-label and multimodal problems. Additionally, our method is able to generalize under different scenarios: continuous style interpolation, continuous label interpolation, and multi-label mapping.

Natural image generation is currently one of the most actively explored fields in Deep Learning. Many approaches, e.g. for state-of-the-art artistic style transfer or natural texture synthesis, rely on the statistics of hierarchical representations in supervisedly trained deep neural networks. It is, however, unclear what aspects of this feature representation are crucial for natural image generation: is it the depth, the pooling or the training of the features on natural images? We here address this question for the task of natural texture synthesis and show that none of the above aspects are indispensable. Instead, we demonstrate that natural textures of high perceptual quality can be generated from networks with only a single layer, no pooling and random filters.

Different methods for image denoising have complementary strengths and can be combined to improve image denoising performance, as has been noted by several authors [11, 7]. Mosseri et al. [11] distinguish between internal and external methods depending whether they exploit internal or external statistics [13]. They also propose a rule-based scheme (PatchSNR) to combine these two classes of algorithms. In this paper, we test the underlying assumptions and show that many images might not be easily split into regions where internal methods or external methods are preferable. Instead we propose a learning based approach using a neural network, that automatically combines denoising results from an internal and from an external method. This approach outperforms both other combination methods and state-of-the-art stand-alone image denoising methods, hereby further closing the gap to the theoretically achievable performance limits of denoising [9]. Our denoising results can be replicated with a publicly available toolbox.

Conditional GANs are at the forefront of natural image synthesis. The main drawback of such models is the necessity for labelled data. In this work we exploit two popular unsupervised learning techniques, adversarial training and self-supervision, to close the gap between conditional and unconditional GANs. In particular, we allow the networks to collaborate on the task of representation learning, while being adversarial with respect to the classic GAN game. The role of self-supervision is to encourage the discriminator to learn meaningful feature representations which are not forgotten during training. We test empirically both the quality of the learned image representations, and the quality of the synthesized images. Under the same conditions, the self-supervised GAN attains a similar performance to stateof-the-art conditional counterparts. Finally, we show that this approach to fully unsupervised learning can be scaled to attain an FID of 33 on unconditional IMAGENET generation.

Aerial images are subject to various types of noise, which restricts the recognition and analysis of images, target monitoring, and search services. At present, deep learning is successful in image recognition. However, traditional convolutional neural networks (CNNs) extract the main features of an image to predict directly and are limited by the requirements of the training sample size (i.e., a small data size is not successful enough). In this paper, using a small sample size, we propose an aerial-image denoising recognition model based on CNNs with a multi-scale residual learning approach. The proposed model has the following three advantages: (1) Instead of directly learning latent clean images, the proposed model learns the noise from noisy images and then subtracts the learned residual from the noisy images to obtain reconstructed (denoised) images; (2) The developed image denoising recognition model is beneficial to small training datasets; (3) We use multi-scale residual learning as a learning approach, and dropout is introduced into the model architecture to force the network to learn to generalize well enough. Our experimental results on aerial-image denoising recognition reveal that the proposed approach is highly superior to the other state-of-the-art methods.

Recently, image-to-image translation has seen a significant success. Among many approaches, image translation based on an exemplar image, which contains the target style information, has been popular, owing to its capability to handle multimodality as well as its suitability for practical use. However, most of the existing methods extract the style information from an entire exemplar and apply it to the entire input image, which introduces excessive image translation in irrelevant image regions. In response, this paper proposes a novel approach that jointly extracts out the local masks of the input image and the exemplar as targeted regions to be involved for image translation. In particular, the main novelty of our model lies in (1) co-segmentation networks for local mask generation and (2) the local mask-based highway adaptive instance normalization technique. We demonstrate the quantitative and the qualitative evaluation results to show the advantages of our proposed approach. Finally, the code is available at https://github.com/AnonymousIclrAuthor/ Highway-Adaptive-Instance-Normalization.

We present an autoencoder that leverages learned representations to better measure similarities in data space. By combining a variational autoencoder with a generative adversarial network we can use learned feature representations in the GAN discriminator as basis for the VAE reconstruction objective. Thereby, we replace element-wise errors with feature-wise errors to better capture the data distribution while offering invariance towards e.g. translation. We apply our method to images of faces and show that it outperforms VAEs with element-wise similarity measures in terms of visual fidelity. Moreover, we show that the method learns an embedding in which high-level abstract visual features (e.g. wearing glasses) can be modified using simple arithmetic.


Question: A generative adversarial network for photo-realistic single image super-resolution.

In this work, we introduce an innovative technique for image super-resolution that leverages a distinctive image characteristic known as the gradient profile prior. This prior represents a parametric model encapsulating the form and acuteness of image gradients. By harnessing a gradient profile prior derived from a substantial collection of natural images, we impose a constraint on image gradients during the estimation of a high-resolution image from its low-resolution counterpart. This straightforward yet highly impactful prior enables us to generate outcomes that are on par with the current state of the art. The resulting high-resolution image exhibits sharpness and clarity, with minimal occurrence of ringing or jagged artifacts, thereby enhancing visual quality and fidelity.

We train a generative convolutional neural network which is able to generate images of objects given object type, viewpoint, and color. We train the network in a supervised manner on a dataset of rendered 3D chair models. Our experiments show that the network does not merely learn all images by heart, but rather finds a meaningful representation of a 3D chair model allowing it to assess the similarity of different chairs, interpolate between given viewpoints to generate the missing ones, or invent new chair styles by interpolating between chairs from the training set. We show that the network can be used to find correspondences between different chairs from the dataset, outperforming existing approaches on this task.

Transferring the style of an example image to a content image opens the door of artistic creation for end users. However, it is especially challenging for portrait photos since human vision system is sensitive to the slight artifacts on portraits. Previous methods use facial landmarks to densely align the content face with the style face to reduce the artifacts. However, they can only handle the facial region. As for the whole image, building the dense correspondence is difficult and may easily introduce errors. In this paper, we propose a robust approach for portrait style transfer that gets rid of dense correspondence. Our approach is based on the guided image synthesis framework. We propose three novel guidance maps for the synthesis process. Contrary to former methods, these maps do not require the dense correspondence between content image and style image, which allows our method to handle the whole portrait photo instead of facial region only. In comparison with recent neural style transfer methods, our method achieves more pleasing results and preserves more texture details. Extensive experiments demonstrate our advantage over former methods on portrait style transfer.

Suffering from the extreme training data imbalance between seen and unseen classes, most of existing state-of-the-art approaches fail to achieve satisfactory results for the challenging generalized zero-shot learning task. To circumvent the need for labeled examples of unseen classes, we propose a novel generative adversarial network (GAN) that synthesizes CNN features conditioned on class-level semantic information, offering a shortcut directly from a semantic descriptor of a class to a class-conditional feature distribution. Our proposed approach, pairing a Wasserstein GAN with a classification loss, is able to generate sufficiently discriminative CNN features to train softmax classifiers or any multimodal embedding method. Our experimental results demonstrate a significant boost in accuracy over the state of the art on five challenging datasets - CUB, FLO, SUN, AWA and ImageNet - in both the zero-shot learning and generalized zero-shot learning settings.

This paper investigates unified feature learning and classifier training approaches for image aesthetics assessment . Existing methods built upon handcrafted or generic image features and developed machine learning and statistical modeling techniques utilizing training examples. We adopt a novel deep neural network approach to allow unified feature learning and classifier training to estimate image aesthetics. In particular, we develop a double-column deep convolutional neural network to support heterogeneous inputs, i.e., global and local views, in order to capture both global and local characteristics of images . In addition, we employ the style and semantic attributes of images to further boost the aesthetics categorization performance . Experimental results show that our approach produces significantly better results than the earlier reported results on the AVA dataset for both the generic image aesthetics and content -based image aesthetics. Moreover, we introduce a 1.5-million image dataset (IAD) for image aesthetics assessment and we further boost the performance on the AVA test set by training the proposed deep neural networks on the IAD dataset.

This paper introduces a new algorithm for enhancing image resolution from an image sequence. The approach we propose herein uses the integrating resampler for warping. The method is a direct computation, which is fundamentally different from the iterative back-projection approaches proposed in previous work. This paper shows that image-warping techniques may have a strong impact on the quality of image resolution enhancement. By coupling the degradation model of the imaging system directly into the integrating resampler, we can better approximate the warping characteristics of real sensors, which also significantly improve the quality of super-resolution images. Examples of super-resolutions are given for gray-scale images. Evaluations are made visuallyby comparing the resulting images and those using bi-linear resampling and back-projection and quantitativelyusing OCR as a fundamental measure. The paper shows that even when the images are qualitatively similar, quantitative differences appear in machine processing. q 2000 Elsevier Science B.V. All rights reserved. Keywords: Super-resolution; Imaging-consistent restoration/reconstruction; Integrating resampling; Integrating resampler; Quantitative evaluatio n; OCR; Bilinear resampling; Image reconstruction; Image restoration; Image warping 1. Background and introduction The basic premise of super-resolution is to combine images (and in some work scene/data models) to produce an image that has higher “resolution” than the original image. It is not sufficient to just resample the image—size is not the same as resolution. Increasing resolution could be viewed as either increasing the signal-to-noise ratio while keeping the size fixed and/or approximating the image at a larger size with reasonable approximations for frequencies higher than those representable at the original size. In either case, it requires additional information and assumptions. There are, of course, some fundamental limits on what can be gained; lens and photo-sites cause blur, and the discrete sampling causes frequencies above the Nyquist rate to alias. While in general the aliasing is theoretically impossible to separate, the problem occurs mostly along sharp “intensity edges”, so multiple images with slightly different imaging positions can be used to reduce its impact. There are two different groups of research that use the term super-resolution—one is based on multiple images, which will be considered within this paper; the other uses “scene/object” models. Early research on image-based super-resolution dates back to the work by Huang and Tsai [8]. They solved the problem in the frequency domain, but they disregarded sensor blurring. Furthermore, they assumed only inter-frame translations. Gross [7] solved the problem by first merging the low resolution images by interpolation and then obtaining the high-resolution image by deblurring the merged image. Again, only inter-frame translations were considered. Peleg and Keren [11,14] solved the problem by first estimating an initial guess of the high-resolution image, then simulating the imaging process and minimizing the difference between the observed and simulated low-resolution images. Irani and Peleg [9,10] used a back-projection method similar to that used in tomography to minimize the same difference. Bascle et al. [1] used the same back-projection method to minimize the difference between the predicted and actual images except that a simple motion blur model was included. All previous work ignores the impact of image warping techniques. In this paper, we show that warping techniques may have a strong impact on the quality of image resolution enhancement. Image warping has been around almost as long as image processing, allowing users to reshape the image geometry. Image warping requires the underlying image to be “resampled” at non-integer locations; hence, it requires image reconstruction. When the goal of warping is to produce output for human viewing, only moderately accurate image intensities are needed. In these cases, techniques Image and Vision Computing 18 (2000) 761–771 0262-8856/00/$ see front matter q 2000 Elsevier Science B.V. All rights reserved. PII: S0262-8856(99)00044-X www.elsevier.com/locate/imavis * Corresponding author. Tel.: 11-610-7584061. E-mail address:tboult@eecs.lehigh.edu (T.E. Boult). using bi-linear interpolation have been found sufficient. However, as a step for applications such as super-resolution, the precision of the warped intensity values is often important. For these problems, bi-linear image reconstruction may not be sufficient. This paper shows how the integrating resampler can be used to enhance image resolution. This builds upon the ideas of imaging-consistent restoration/reconstruction algorithms proposed in Ref. [2] and the integrating resampler proposed in Ref. [5]. A final contribution of this paper is a quantitative comparison and analysis of our techniques. In this paper, we use a character recognition rate based on a commercial OCR program. We chose this because it is easy to make it quantitative, and we consider this to be a good measure for other applications such as license-plate reading and other pattern recognition problems. In what follows in this paper, the image formation process and the relationship between restoration, reconstruction, and super-resolution is briefly reviewed in Section 2. The integrating resampler—an efficient method for warping using imaging-consistent reconstruction/restoration algorithms—is given in Section 3. This algorithm is well suited for today’s pipelined micro-processors. In addition, the integrating resampler can allow for modifications of the intensity values to better approximate the warping characteristics of real sensors. The super-resolution algorithm where we compare the integrating resampler with bi-linear resampling is presented in Section 4. Quantitative measurement of super-resolution imaging using OCR is shown in Section 5. 2. Image formation and image restoration In this section, we briefly review the image formation process as well as the sensor model proposed in Ref. [2]. Image formation is generally described as a sequence of filtering operations. See Fig. 1. Let f x; y be the intensity distribution of a scene at the front aperture of a lens. That distribution is acted upon by h1x; y; the blurring component of the lens, yielding f1x; y: A geometric distortion functionh2x; y is then applied to produce imagef2u; v: At this point, f2 strikes the image sensor where it undergoes additional blurring by a point spread functionh3u; v to generate image f3u; v: This blurring reflects the limitation of the sensor to accurately resolve each point without the influence of neighboring points. We choose to use a simple model wherein this blurring takes place within one pixel because for CCD and CID cameras, the physical boundaries between photo-sites generally allow only insignificant charge transfer between pixels. Imagef3 undergoes spatial sampling as it hits the discrete photo-sites in a CCD or CID camera. The combination of h3 with sampling is known as areas sampling. It reflects the finite size of the sampling area. If h3 is taken to be an impulse, then we have point sampling. This ideal, often assumed for theoretical considerations, is generally not true in practice. In either case, intensities in the sampled image Is are now defined only for integer values of u and v. The digital imageI u; v is obtained via an analog-todigital converter that quantizes the samples of Is. This completes the image formation process, leaving I u; v as the starting point for subsequent processing, including image reconstruction and restoration. Reconstruction and restoration start with I (and models for one or morehis), and seek to solve for one or more of the fjs. Recoveringf is known as image restoration and is of considerable interest in image processing. Note that recovering a functional form off2 is an intra-pixel restoration. Reconstruction limits itself to the problem of deriving the continuous functionf3. Restoration attempts to estimate the original input functionf2 (or f1). The goal of super-resolution is to recover a discrete approximation, with resolution higher than Is, to either f1 (plain super-resolution) orf (super-resolution with deblurring). Because super-resolution increases the signal-to-noise ratio, it significantly aids in the ill-condition problem of deblurring. 3. Imaging-consistency and the integrating resampler An algorithm is called imaging-consistent if it is the exact solution for some input function, which, according to the imaging model, would have generated the measured input. For image reconstruction, we achieve this by first restoring the image to yieldf2, then blurring it again by the pixel’s PSF. Details can be found in Ref. [2]. Only one-dimensional image models are presented herein; higher dimensions are treated separately. M.-C. Chiang, T.E. Boult / Image and Vision Computing 18 (2000) 761–771 762 Fig. 1. The image formation process and the relationship between restoration, reconstruction, and super-resolution. The first imaging-consistent method to consider is based on a piecewise quadratic model for the image. If we assume a Rect PSF filter (1 inside and 0 outside the pixel), the imaging consistent algorithm is easy to derive. To ensure that the function is continuous and that the method is local, we define the value of the reconstruction at the pixel boundarieski andki11 to be equal toEi andEi11: Any method of approximation could be used to compute Ei, though our examples will only include linear interpolation and cubic convolution. See [3, Section 2.3] for more details. Given the valuesEi at the pixel edges, an additional constraint, that the integral across the pixel must equal Vi, results in exactly three constraints: qi21=2  Ei ; qi1=2  Ei11; Z1=2 2 1=2 qix dx Vi : 1 From Eq. (1), one can det

We propose a virtual try-on method based on generative adversarial networks (GANs). By considering clothing regions, this method enables us to reflect the pattern of clothes better than Conditional Analogy GAN (CAGAN), an existing virtual try-on method based on GANs. Our method first obtains the clothing region on a person by using a human parsing model learned with a large-scale dataset. Next, using the acquired region, the clothing part is removed from a human image. A desired clothing image is added to the blank area. The network learns how to apply new clothing to the area of people’s clothing. Results demonstrate the possibility of reflecting a clothing pattern. Furthermore, an image of the clothes that the person is originally wearing becomes unnecessary during testing. In experiments, we generate images using images gathered from Zaland (a fashion E-commerce site).

To see is to sketch - free-hand sketching naturally builds ties between human and machine vision. In this paper, we present a novel approach for translating an object photo to a sketch, mimicking the human sketching process. This is an extremely challenging task because the photo and sketch domains differ significantly. Furthermore, human sketches exhibit various levels of sophistication and abstraction even when depicting the same object instance in a reference photo. This means that even if photo-sketch pairs are available, they only provide weak supervision signal to learn a translation model. Compared with existing supervised approaches that solve the problem of D(E(photo)) â†’ sketch), where E(Â·) and D(Â·) denote encoder and decoder respectively, we take advantage of the inverse problem (e.g., D(E(sketch) â†’ photo), and combine with the unsupervised learning tasks of within-domain reconstruction, all within a multi-task learning framework. Compared with existing unsupervised approaches based on cycle consistency (i.e., D(E(D(E(photo)))) â†’ photo), we introduce a shortcut consistency enforced at the encoder bottleneck (e.g., D(E(photo)) â†’ photo) to exploit the additional self-supervision. Both qualitative and quantitative results show that the proposed model is superior to a number of state-of-the-art alternatives. We also show that the synthetic sketches can be used to train a better fine-grained sketch-based image retrieval (FG-SBIR) model, effectively alleviating the problem of sketch data scarcity.

In recent years, there have been tremendous advancements in the field of machine learning. These advancements have been made through both academic as well as industrial research. Lately, a fair amount of research has been dedicated to the usage of generative models in the field of computer vision and image classification. These generative models have been popularized through a new framework called Generative Adversarial Networks. Moreover, many modified versions of this framework have been proposed in the last two years. We study the original model proposed by Goodfellow et al. (2014) as well as modifications over the original model and provide a comparative analysis of these models.

Different methods for image denoising have complementary strengths and can be combined to improve image denoising performance, as has been noted by several authors [11, 7]. Mosseri et al. [11] distinguish between internal and external methods depending whether they exploit internal or external statistics [13]. They also propose a rule-based scheme (PatchSNR) to combine these two classes of algorithms. In this paper, we test the underlying assumptions and show that many images might not be easily split into regions where internal methods or external methods are preferable. Instead we propose a learning based approach using a neural network, that automatically combines denoising results from an internal and from an external method. This approach outperforms both other combination methods and state-of-the-art stand-alone image denoising methods, hereby further closing the gap to the theoretically achievable performance limits of denoising [9]. Our denoising results can be replicated with a publicly available toolbox.

In the wide-angle (WA) images embedded in an around-view monitor system, the subject(s) in the peripheral region is normally small and has little information. Furthermore, since the outer region suffers from the non-uniform blur phenomenon and artifact caused by the inherent optical characteristic of WA lenses, its visual quality tends to deteriorate. According to our experiments, conventional image enhancement techniques rarely improve the degraded visual quality of the outer region of WA images. In order to solve the above-mentioned problem, this paper proposes a joint sharpness enhancement (SE) and super-resolution (SR) algorithm which can improve the sharpness and resolution of WA images together. The proposed SE algorithm improves the sharpness of the deteriorated WA images by exploiting self-similarity. Also, the proposed SR algorithm generates super-resolved images by using high-resolution information which is classified according to the extended local binary pattern-based classifier and learned on a pattern basis. Experimental results show that the proposed scheme effectively improves the sharpness and resolution of the input deteriorated WA images. Even in terms of quantitative metrics such as just noticeable blur, structural similarity, and peak signal-to-noise ratio. Finally, the proposed scheme guarantees real-time processing such that it achieves 720p video at 29 Hz on a low cost GPU platform.


Question: A generative adversarial network for photo-realistic single image super-resolution.

This paper presents a pioneering technique, the Neural Algorithm of Artistic Style, developed by researchers at the Werner Reichardt Centre for Integrative Neuroscience and Institute of Theoretical Physics at the University of Tübingen, Germany, in collaboration with the Bernstein Center for Computational Neuroscience, the Graduate School for Neural Information Processing, and the Max Planck Institute for Biological Cybernetics, all in Tübingen, Germany, as well as the Department of Neuroscience at Baylor College of Medicine in Houston, TX, USA. The corresponding author for this work can be reached at leon.gatys@bethgelab.org.

The Neural Algorithm of Artistic Style is a groundbreaking method that enables the transfer of artistic styles from one image to another, effectively transforming the visual aesthetics of an input image into that of a desired style. This algorithm operates by separating and recombining the content and style representations of an image, allowing for the creation of new images that possess the content of one image with the style of another. By doing so, it bridges the gap between classical art and computational techniques, offering a novel approach to generating artistic imagery using deep neural networks. This technique has not only expanded the horizons of digital art but also has implications for various applications in image processing and computer vision, demonstrating the versatility and power of neural networks in manipulating visual content in sophisticated ways.

We consider the problem of data augmentation, i.e., generating artificial samples to extend a given corpus of training data. Specifically, we propose attributed-guided augmentation (AGA) which learns a mapping that allows to synthesize data such that an attribute of a synthesized sample is at a desired value or strength. This is particularly interesting in situations where little data with no attribute annotation is available for learning, but we have access to a large external corpus of heavily annotated samples. While prior works primarily augment in the space of images, we propose to perform augmentation in feature space instead. We implement our approach as a deep encoder-decoder architecture that learns the synthesis function in an end-to-end manner. We demonstrate the utility of our approach on the problems of (1) one-shot object recognition in a transfer-learning setting where we have no prior knowledge of the new classes, as well as (2) object-based one-shot scene recognition. As external data, we leverage 3D depth and pose information from the SUN RGB-D dataset. Our experiments show that attribute-guided augmentation of high-level CNN features considerably improves one-shot recognition performance on both problems.

Generative Adversarial Nets (GANs), which use discriminators to help train a generative model, have been successful particularly in computer vision for generating images. However, there are many restrictions in its applications to natural language tasks–mainly, it is difficult to back-propagate through discrete-value random variables. Yet recent publications have applied GAN with promising results. Sequence GAN (Yu et al. 2017) introduces a solution by modeling the data generator as a reinforcement learning (RL) policy to overcome the generator differentiation problem, with the RL reward signals produced by the discriminator after it judges complete sequences. However, problems with this model persist, as the GAN training objective is inherently unstable, producing a large variation of results that make it difficult to fool the discriminator. Maximum-Likelihood Augmented Discrete GAN (Che at al. 2017) suggests a new low-variance objective for the generator, using a normalized reward signal from the discriminator that corresponds to log-likelihood. Our project explores both proposed implementations: we produce experimental results on both synthetic and real-world discrete datasets to explore the effectiveness of GAN over strong baselines.

Going deeper and wider in neural architectures improves their accuracy, while the limited GPU DRAM places an undesired restriction on the network design domain. Deep Learning (DL) practitioners either need to change to less desired network architectures, or nontrivially dissect a network across multiGPUs. These distract DL practitioners from concentrating on their original machine learning tasks. We present SuperNeurons: a dynamic GPU memory scheduling runtime to enable the network training far beyond the GPU DRAM capacity. SuperNeurons features 3 memory optimizations, Liveness Analysis, Unified Tensor Pool, and Cost-Aware Recomputation; together they effectively reduce the network-wide peak memory usage down to the maximal memory usage among layers. We also address the performance issues in these memory-saving techniques. Given the limited GPU DRAM, SuperNeurons not only provisions the necessary memory for the training, but also dynamically allocates the memory for convolution workspaces to achieve the high performance. Evaluations against Caffe, Torch, MXNet and TensorFlow have demonstrated that SuperNeurons trains at least 3.2432 deeper network than current ones with the leading performance. Particularly, SuperNeurons can train ResNet2500 that has 104 basic network layers on a 12GB K40c.

Image processing and pixel-wise dense prediction have been advanced by harnessing the capabilities of deep learning. One central issue of deep learning is the limited capacity to handle joint upsampling. We present a deep learning building block for joint upsampling, namely guided filtering layer. This layer aims at efficiently generating the high-resolution output given the corresponding low-resolution one and a high-resolution guidance map. The proposed layer is composed of a guided filter, which is reformulated as a fully differentiable block. To this end, we show that a guided filter can be expressed as a group of spatial varying linear transformation matrices. This layer could be integrated with the convolutional neural networks (CNNs) and jointly optimized through end-to-end training. To further take advantage of end-to-end training, we plug in a trainable transformation function that generates task-specific guidance maps. By integrating the CNNs and the proposed layer, we form deep guided filtering networks. The proposed networks are evaluated on five advanced image processing tasks. Experiments on MIT-Adobe FiveK Dataset demonstrate that the proposed approach runs 10-100Ã— faster and achieves the state-of-the-art performance. We also show that the proposed guided filtering layer helps to improve the performance of multiple pixel-wise dense prediction tasks. The code is available at https://github.com/wuhuikai/DeepGuidedFilter.

Modeling the distribution of natural images is a landmark problem in unsupervised learning. This task requires an image model that is at once expressive, tractable and scalable. We present a deep neural network that sequentially predicts the pixels in an image along the two spatial dimensions. Our method models the discrete probability of the raw pixel values and encodes the complete set of dependencies in the image. Architectural novelties include fast twodimensional recurrent layers and an effective use of residual connections in deep recurrent networks. We achieve log-likelihood scores on natural images that are considerably better than the previous state of the art. Our main results also provide benchmarks on the diverse ImageNet dataset. Samples generated from the model appear crisp, varied and globally coherent.

To what extent is the success of deep visualization due to the training? Could we do deep visualization using untrained, random weight networks? To address this issue, we explore new and powerful generative models for three popular deep visualization tasks using untrained, random weight convolutional neural networks. First we invert representations in feature spaces and reconstruct images from white noise inputs. The reconstruction quality is statistically higher than that of the same method applied on well trained networks with the same architecture. Next we synthesize textures using scaled correlations of representations in multiple layers and our results are almost indistinguishable with the original natural texture and the synthesized textures based on the trained network. Third, by recasting the content of an image in the style of various artworks, we create artistic images with high perceptual quality, highly competitive to the prior work of Gatys et al. on pretrained networks. To our knowledge this is the first demonstration of image representations using untrained deep neural networks. Our work provides a new and fascinating tool to study the representation of deep network architecture and sheds light on new understandings on deep visualization.

Although it is typically expected that using a large amount of labeled training data would lead to improve performance in deep learning, it is generally difficult to obtain such DataBase (DB). In competitions such as the Detection and Classification of Acoustic Scenes and Events (DCASE) challenge Task 1, participants are constrained to use a relatively small DB as a rule, which is similar to the aforementioned issue. To improve Acoustic Scene Classification (ASC) performance without employing additional DB, this paper proposes to use Generative Adversarial Networks (GAN) based method for generating additional training DB. Since it is not clear whether every sample generated by GAN would have equal impact in classification performance, this paper proposes to use Support Vector Machine (SVM) hyper plane for each class as reference for selecting samples, which have class discriminative information. Based on the crossvalidated experiments on development DB, the usage of the generated features could improve ASC performance.

This paper introduces a new algorithm for enhancing image resolution from an image sequence. The approach we propose herein uses the integrating resampler for warping. The method is a direct computation, which is fundamentally different from the iterative back-projection approaches proposed in previous work. This paper shows that image-warping techniques may have a strong impact on the quality of image resolution enhancement. By coupling the degradation model of the imaging system directly into the integrating resampler, we can better approximate the warping characteristics of real sensors, which also significantly improve the quality of super-resolution images. Examples of super-resolutions are given for gray-scale images. Evaluations are made visuallyby comparing the resulting images and those using bi-linear resampling and back-projection and quantitativelyusing OCR as a fundamental measure. The paper shows that even when the images are qualitatively similar, quantitative differences appear in machine processing. q 2000 Elsevier Science B.V. All rights reserved. Keywords: Super-resolution; Imaging-consistent restoration/reconstruction; Integrating resampling; Integrating resampler; Quantitative evaluatio n; OCR; Bilinear resampling; Image reconstruction; Image restoration; Image warping 1. Background and introduction The basic premise of super-resolution is to combine images (and in some work scene/data models) to produce an image that has higher “resolution” than the original image. It is not sufficient to just resample the image—size is not the same as resolution. Increasing resolution could be viewed as either increasing the signal-to-noise ratio while keeping the size fixed and/or approximating the image at a larger size with reasonable approximations for frequencies higher than those representable at the original size. In either case, it requires additional information and assumptions. There are, of course, some fundamental limits on what can be gained; lens and photo-sites cause blur, and the discrete sampling causes frequencies above the Nyquist rate to alias. While in general the aliasing is theoretically impossible to separate, the problem occurs mostly along sharp “intensity edges”, so multiple images with slightly different imaging positions can be used to reduce its impact. There are two different groups of research that use the term super-resolution—one is based on multiple images, which will be considered within this paper; the other uses “scene/object” models. Early research on image-based super-resolution dates back to the work by Huang and Tsai [8]. They solved the problem in the frequency domain, but they disregarded sensor blurring. Furthermore, they assumed only inter-frame translations. Gross [7] solved the problem by first merging the low resolution images by interpolation and then obtaining the high-resolution image by deblurring the merged image. Again, only inter-frame translations were considered. Peleg and Keren [11,14] solved the problem by first estimating an initial guess of the high-resolution image, then simulating the imaging process and minimizing the difference between the observed and simulated low-resolution images. Irani and Peleg [9,10] used a back-projection method similar to that used in tomography to minimize the same difference. Bascle et al. [1] used the same back-projection method to minimize the difference between the predicted and actual images except that a simple motion blur model was included. All previous work ignores the impact of image warping techniques. In this paper, we show that warping techniques may have a strong impact on the quality of image resolution enhancement. Image warping has been around almost as long as image processing, allowing users to reshape the image geometry. Image warping requires the underlying image to be “resampled” at non-integer locations; hence, it requires image reconstruction. When the goal of warping is to produce output for human viewing, only moderately accurate image intensities are needed. In these cases, techniques Image and Vision Computing 18 (2000) 761–771 0262-8856/00/$ see front matter q 2000 Elsevier Science B.V. All rights reserved. PII: S0262-8856(99)00044-X www.elsevier.com/locate/imavis * Corresponding author. Tel.: 11-610-7584061. E-mail address:tboult@eecs.lehigh.edu (T.E. Boult). using bi-linear interpolation have been found sufficient. However, as a step for applications such as super-resolution, the precision of the warped intensity values is often important. For these problems, bi-linear image reconstruction may not be sufficient. This paper shows how the integrating resampler can be used to enhance image resolution. This builds upon the ideas of imaging-consistent restoration/reconstruction algorithms proposed in Ref. [2] and the integrating resampler proposed in Ref. [5]. A final contribution of this paper is a quantitative comparison and analysis of our techniques. In this paper, we use a character recognition rate based on a commercial OCR program. We chose this because it is easy to make it quantitative, and we consider this to be a good measure for other applications such as license-plate reading and other pattern recognition problems. In what follows in this paper, the image formation process and the relationship between restoration, reconstruction, and super-resolution is briefly reviewed in Section 2. The integrating resampler—an efficient method for warping using imaging-consistent reconstruction/restoration algorithms—is given in Section 3. This algorithm is well suited for today’s pipelined micro-processors. In addition, the integrating resampler can allow for modifications of the intensity values to better approximate the warping characteristics of real sensors. The super-resolution algorithm where we compare the integrating resampler with bi-linear resampling is presented in Section 4. Quantitative measurement of super-resolution imaging using OCR is shown in Section 5. 2. Image formation and image restoration In this section, we briefly review the image formation process as well as the sensor model proposed in Ref. [2]. Image formation is generally described as a sequence of filtering operations. See Fig. 1. Let f x; y be the intensity distribution of a scene at the front aperture of a lens. That distribution is acted upon by h1x; y; the blurring component of the lens, yielding f1x; y: A geometric distortion functionh2x; y is then applied to produce imagef2u; v: At this point, f2 strikes the image sensor where it undergoes additional blurring by a point spread functionh3u; v to generate image f3u; v: This blurring reflects the limitation of the sensor to accurately resolve each point without the influence of neighboring points. We choose to use a simple model wherein this blurring takes place within one pixel because for CCD and CID cameras, the physical boundaries between photo-sites generally allow only insignificant charge transfer between pixels. Imagef3 undergoes spatial sampling as it hits the discrete photo-sites in a CCD or CID camera. The combination of h3 with sampling is known as areas sampling. It reflects the finite size of the sampling area. If h3 is taken to be an impulse, then we have point sampling. This ideal, often assumed for theoretical considerations, is generally not true in practice. In either case, intensities in the sampled image Is are now defined only for integer values of u and v. The digital imageI u; v is obtained via an analog-todigital converter that quantizes the samples of Is. This completes the image formation process, leaving I u; v as the starting point for subsequent processing, including image reconstruction and restoration. Reconstruction and restoration start with I (and models for one or morehis), and seek to solve for one or more of the fjs. Recoveringf is known as image restoration and is of considerable interest in image processing. Note that recovering a functional form off2 is an intra-pixel restoration. Reconstruction limits itself to the problem of deriving the continuous functionf3. Restoration attempts to estimate the original input functionf2 (or f1). The goal of super-resolution is to recover a discrete approximation, with resolution higher than Is, to either f1 (plain super-resolution) orf (super-resolution with deblurring). Because super-resolution increases the signal-to-noise ratio, it significantly aids in the ill-condition problem of deblurring. 3. Imaging-consistency and the integrating resampler An algorithm is called imaging-consistent if it is the exact solution for some input function, which, according to the imaging model, would have generated the measured input. For image reconstruction, we achieve this by first restoring the image to yieldf2, then blurring it again by the pixel’s PSF. Details can be found in Ref. [2]. Only one-dimensional image models are presented herein; higher dimensions are treated separately. M.-C. Chiang, T.E. Boult / Image and Vision Computing 18 (2000) 761–771 762 Fig. 1. The image formation process and the relationship between restoration, reconstruction, and super-resolution. The first imaging-consistent method to consider is based on a piecewise quadratic model for the image. If we assume a Rect PSF filter (1 inside and 0 outside the pixel), the imaging consistent algorithm is easy to derive. To ensure that the function is continuous and that the method is local, we define the value of the reconstruction at the pixel boundarieski andki11 to be equal toEi andEi11: Any method of approximation could be used to compute Ei, though our examples will only include linear interpolation and cubic convolution. See [3, Section 2.3] for more details. Given the valuesEi at the pixel edges, an additional constraint, that the integral across the pixel must equal Vi, results in exactly three constraints: qi21=2  Ei ; qi1=2  Ei11; Z1=2 2 1=2 qix dx Vi : 1 From Eq. (1), one can det

With advances of recent technologies, augmented reality systems and autonomous vehicles gained a lot of interest from academics and industry. Both these areas rely on scene geometry understanding, which usually requires depth map estimation. However, in case of systems with limited computational resources, such as smartphones or autonomous robots, high resolution dense depth map estimation may be challenging. In this paper, we study the problem of semi-dense depth map interpolation along with low resolution depth map upsampling. We present an end-to-end learnable residual convolutional neural network architecture that achieves fast interpolation of semi-dense depth maps with different sparse depth distributions: uniform, sparse grid and along intensity image gradient. We also propose a loss function combining classical mean squared error with perceptual loss widely used in intensity image super-resolution and style transfer tasks. We show that with some modifications, this architecture can be used for depth map super-resolution. Finally, we evaluate our results on both synthetic and real data, and consider applications for autonomous vehicles and creating AR/MR video games.

Deep residual networks were shown to be able to scale up to thousands of layers and still have improving performance. However, each fraction of a percent of improved accuracy costs nearly doubling the number of layers, and so training very deep residual networks has a problem of diminishing feature reuse, which makes these networks very slow to train. To tackle these problems, in this paper we conduct a detailed experimental study on the architecture of ResNet blocks, based on which we propose a novel architecture where we decrease depth and increase width of residual networks. We call the resulting network structures wide residual networks (WRNs) and show that these are far superior over their commonly used thin and very deep counterparts. For example, we demonstrate that even a simple 16-layer-deep wide residual network outperforms in accuracy and efficiency all previous deep residual networks, including thousand-layerdeep networks, achieving new state-of-the-art results on CIFAR, SVHN, COCO, and significant improvements on ImageNet. Our code and models are available at https: //github.com/szagoruyko/wide-residual-networks.


Question: Double ErrP detection for automated error correction in an ERP-based BCI speller system.

In this study, we investigate the generation of error-related electroencephalogram (EEG) potentials, specifically Error-Related Potentials (ErrPs), during simulated brain-computer interaction (BCI) tasks. BCIs are susceptible to misinterpretations of the user's intentions, which can lead to errors in command execution. A promising strategy to enhance BCI performance involves the detection of ErrPs, which are neural responses that occur shortly after an error, to trigger a corrective mechanism. While prior research has documented the presence of ErrPs in standard choice reaction tasks, the critical question in BCI applications is whether these potentials are also elicited when the BCI system itself commits an error in recognizing the user's command. To address this, we designed an experiment in which five healthy participants engaged in a novel human-robot interaction task, simulating a BCI environment where the interface occasionally misinterprets the subject's intended action. Our findings strongly suggest the presence of ErrPs in response to these simulated BCI errors. The next step is to develop a method to reliably detect these ErrPs in real-time, using a brief window of EEG data immediately following the feedback indicating an error. We have made significant progress in this area, achieving an average classification accuracy of 83.5% for correct trials and 79.2% for error trials, using a classifier trained on data collected up to three months prior. This work paves the way for the integration of automatic error correction mechanisms in BCI systems, potentially leading to significant improvements in their reliability and user experience.

Recently, a novel statistical model has been proposed to estimate population effects and individual variability between subgroups simultaneously, by extending Lasso methods. We will for the first time apply this so-called ℓ(1)-penalized linear regression mixed-effects model for a large scale real world problem: we study a large set of brain computer interface data and through the novel estimator are able to obtain a subject-independent classifier that compares favorably with prior zero-training algorithms. This unifying model inherently compensates shifts in the input space attributed to the individuality of a subject. In particular we are now for the first time able to differentiate within-subject and between-subject variability. Thus a deeper understanding both of the underlying statistical and physiological structures of the data is gained.

The empirical and theoretical development of the P300 event-related brain potential (ERP) is reviewed by considering factors that contribute to its amplitude, latency, and general characteristics. The neuropsychological origins of the P3a and P3b subcomponents are detailed, and how target/standard discrimination difficulty modulates scalp topography is discussed. The neural loci of P3a and P3b generation are outlined, and a cognitive model is proffered: P3a originates from stimulus-driven frontal attention mechanisms during task processing, whereas P3b originates from temporal-parietal activity associated with attention and appears related to subsequent memory processing. Neurotransmitter actions associating P3a to frontal/dopaminergic and P3b to parietal/norepinephrine pathways are highlighted. Neuroinhibition is suggested as an overarching theoretical mechanism for P300, which is elicited when stimulus detection engages memory operations.

This paper compares several approaches to come up with the Verilog HDL model of the thermometer-to-binary encoder with bubble error correction. It has been demonstrated that implementations of different ideas to correct bubble errors yield circuits whose parameters tremendously vary in delay, area and power consumption. The shortest delay is achieved for the design synthesized from the model which mimics a human reading temperature on classic liquid-in-glass thermometer.

In this paper trade-offs among certain computational factors in hash coding are analyzed. The paradigm problem considered is that of testing a series of messages one-by-one for membership in a given set of messages. Two new hash-coding methods are examined and compared with a particular conventional hash-coding method. The computational factors considered are the size of the hash area (space), the time required to identify a message as a nonmember of the given set (reject time), and an allowable error frequency.
The new methods are intended to reduce the amount of space required to contain the hash-coded information from that associated with conventional methods. The reduction in space is accomplished by exploiting the possibility that a small fraction of errors of commission may be tolerable in some applications, in particular, applications in which a large amount of data is involved and a core resident hash area is consequently not feasible using conventional methods.
In such applications, it is envisaged that overall performance could be improved by using a smaller core resident hash area in conjunction with the new methods and, when necessary, by using some secondary and perhaps time-consuming test to “catch” the small fraction of errors associated with the new methods. An example is discussed which illustrates possible areas of application for the new methods.
Analysis of the paradigm problem demonstrates that allowing a small number of test messages to be falsely identified as members of the given set will permit a much smaller hash area to be used without increasing reject time.

In this paper, we build a Chinese text corrector which can correct spelling mistakes precisely in Chinese texts. Our motivation is inspired by the recently proposed seq2seq model which consider the text corrector as a sequence learning problem. To begin with, we propose a biased-decoding method to improve the bilingual evaluation understudy (BLEU) score of our model. Secondly, we adopt a more reasonable OOV token scheme, which enhances the robustness of our correction mechanism. Moreover, to test the performance of our proposed model thoroughly, we establish a corpus which includes 600,000 sentences from news data of Sogou Labs. Experiments show that our corrector model can achieve better corrector results based on the corpus.

Demand for interactive wearable devices is rapidly increasing with the development of smart devices. To accurately utilize wearable devices for remote robot controls, limited data should be analyzed and utilized efficiently. For example, the motions by a wearable device, called Myo device, can be estimated by measuring its orientation, and calculating a Bayesian probability based on these orientation data. Given that Myo device can measure various types of data, the accuracy of its motion estimation can be increased by utilizing these additional types of data. This paper proposes a motion estimation method based on weighted Bayesian probability and concurrently measured data, orientations and electromyograms (EMG). The most probable motion among estimated is treated as a final estimated motion. Thus, recognition accuracy can be improved when compared to the traditional methods that employ only a single type of data. In our experiments, seven subjects perform five predefined motions. When orientation is measured by the traditional methods, the sum of the motion estimation errors is 37.3%; likewise, when only EMG data are used, the error in motion estimation by the proposed method was also 37.3%. The proposed combined method has an error of 25%. Therefore, the proposed method reduces motion estimation errors by 12%.

This study presents a novel myoelectric pattern recognition strategy towards restoration of hand function after incomplete cervical spinal cord Injury (SCI). High density surface electromyogram (EMG) signals comprised of 57 channels were recorded from the forearm of nine subjects with incomplete cervical SCI while they tried to perform six different hand grasp patterns. A series of pattern recognition algorithms with different EMG feature sets and classifiers were implemented to identify the intended tasks of each SCI subject. High average overall accuracies (>; 97%) were achieved in classification of seven different classes (six intended hand grasp patterns plus a hand rest pattern), indicating that substantial motor control information can be extracted from partially paralyzed muscles of SCI subjects. Such information can potentially enable volitional control of assistive devices, thereby facilitating restoration of hand function. Furthermore, it was possible to maintain high levels of classification accuracy with a very limited number of electrodes selected from the high density surface EMG recordings. This demonstrates clinical feasibility and robustness in the concept of using myoelectric pattern recognition techniques toward improved function restoration for individuals with spinal injury.

This paper proposed a multimodal fusion between brain and peripheral signals for emotion detection. The input signals were electroencephalogram, galvanic skin resistance, temperature, blood pressure and respiration, which can reflect the influence of emotion on the central nervous system and autonomic nervous system respectively. The acquisition protocol is based on a subset of pictures which correspond to three specific areas of valance-arousal emotional space (positively excited, negatively excited, and calm). The features extracted from input signals, and to improve the results, correlation dimension as a strong nonlinear feature is used for brain signals. The performance of the Quadratic Discriminant Classifier has been evaluated on different feature sets: peripheral signals, EEG's, and both. In comparison among the results of different feature sets, EEG signals seem to perform better than other physiological signals, and the results confirm the interest of using brain signals as peripherals in emotion assessment. According to the improvement in EEG results compare in each raw of the table, it seems that nonlinear features would lead to better understanding of how emotional activities work.

In natural language, people often misuse a word (called a “confused word”) in place of other words (called “confusing words”). In misspelling corrections, many approaches to finding and correcting misspelling errors are based on a simple notion called a “confusion set.” The confusion set of a confused word consists of confusing words. In this article, we propose a new method of building Chinese character confusion sets.
 Our method is composed of two major phases. In the first phase, we build a list of seed confusion sets for each Chinese character, which is based on measuring similarity in character pinyin or similarity in character shape. In this phase, all confusion sets are constructed manually, and the confusion sets are organized into a graph, called a “seed confusion graph” (SCG), in which vertices denote characters and edges are pairs of characters in the form (confused character, confusing character).
 In the second phase, we extend the SCG by acquiring more pairs of (confused character, confusing character) from a large Chinese corpus. For this, we use several word patterns (or patterns) to generate new confusion pairs and then verify the pairs before adding them into a SCG. Comprehensive experiments show that our method of extending confusion sets is effective. Also, we shall use the confusion sets in Chinese misspelling corrections to show the utility of our method.

The present study investigates automation misuse based on complacency and automation bias in interacting with a decision aid in a process control system. The effect of a preventive training intervention which includes exposing participants to rare automation failures is examined. Complacency is reflected in an inappropriate checking and monitoring of automated functions. In interaction with automated decision aids complacency might result in commission errors, i.e., following automatically generated recommendations even though they are false. Yet, empirical evidence proving this kind of relationship is still lacking. A laboratory experiment (N 1⁄4 24) was conducted using a process control simulation. An automated decision aid provided advice for fault diagnosis and management. Complacency was directly measured by the participants’ information sampling behavior, i.e., the amount of information sampled in order to verify the automated recommendations. Possible commission errors were assessed when the aid provided false recommendations. The results provide clear evidence for complacency, reflected in an insufficient verification of the automation, while commission errors were associated with high levels of complacency. Hence, commission errors seem to be a possible, albeit not an inevitable consequence of complacency. Furthermore, exposing operators to automation failures during training significantly decreased complacency and thus represents a suitable means to reduce this risk, even though it might not avoid it completely. Potential applications of this research include the design of training protocols in order to prevent automation misuse in interaction with automated decision aids. r 2008 Elsevier Ltd. All rights reserved.