Question: Texture generation utilizing convolutional neural networks.

In this work, we introduce an advanced convolutional neural network (CNN) architecture, coined Inception, which establishes a new benchmark for image classification and object detection tasks in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The key innovation of our architecture is its efficient use of computational resources, achieved through a meticulously designed network structure. Specifically, we expanded both the depth and breadth of the network while maintaining a constant computational budget, thus maximizing resource utilization. Architectural choices were guided by the Hebbian principle and the concept of multi-scale processing to enhance performance. A specific instance of our architecture, dubbed GoogLeNet, is a 22-layer deep network whose efficacy is evaluated in the context of image classification and object detection, demonstrating its superiority in these domains.

Convolutional networks are one of the most widely employed architectures in computer vision and machine learning. In order to leverage their ability to learn complex functions, large amounts of data are required for training. Training a large convolutional network to produce state-of-the-art results can take weeks, even when using modern GPUs. Producing labels using a trained network can also be costly when dealing with web-scale datasets. In this work, we present a simple algorithm which accelerates training and inference by a significant factor, and can yield improvements of over an order of magnitude compared to existing state-of-the-art implementations. This is done by computing convolutions as pointwise products in the Fourier domain while reusing the same transformed feature map many times. The algorithm is implemented on a GPU architecture and addresses a number of related challenges.

Demosaicking and denoising are the key first stages of the digital imaging pipeline but they are also a severely ill-posed problem that infers three color values per pixel from a single noisy measurement. Earlier methods rely on hand-crafted filters or priors and still exhibit disturbing visual artifacts in hard cases such as moiré or thin edges. We introduce a new data-driven approach for these challenges: we train a deep neural network on a large corpus of images instead of using hand-tuned filters. While deep learning has shown great success, its naive application using existing training datasets does not give satisfactory results for our problem because these datasets lack hard cases. To create a better training set, we present metrics to identify difficult patches and techniques for mining community photographs for such patches. Our experiments show that this network and training procedure outperform state-of-the-art both on noisy and noise-free data. Furthermore, our algorithm is an order of magnitude faster than the previous best performing techniques.

A new method fabricates custom surface reflectance and spatially varying bidirectional reflectance distribution functions (svBRDFs). Researchers optimize a microgeometry for a range of normal distribution functions and simulate the resulting surface's effective reflectance. Using the simulation's results, they reproduce an input svBRDF's appearance by distributing the microgeometry on the printed material's surface. This method lets people print svBRDFs on planar samples with current 3D printing technology, even with a limited set of printing materials. It extends naturally to printing svBRDFs on arbitrary shapes.

Generative Adversarial Networks (GAN) can achieve promising performance on learning complex data distributions on different types of data. In this paper, we first show a straightforward extension of existing GAN algorithm is not applicable to point clouds, because the constraint required for discriminators is undefined for set data. We propose a two fold modification to GAN algorithm for learning to generate point clouds (PC-GAN). First, we combine ideas from hierarchical Bayesian modeling and implicit generative models by learning a hierarchical and interpretable sampling process. A key component of our method is that we train a posterior inference network for the hidden variables. Second, instead of using only state-of-the-art Wasserstein GAN objective, we propose a sandwiching objective, which results in a tighter Wasserstein distance estimate than the commonly used dual form. Thereby, PC-GAN defines a generic framework that can incorporate many existing GAN algorithms. We validate our claims on ModelNet40 benchmark dataset. Using the distance between generated point clouds and true meshes as metric, we find that PC-GAN trained by the sandwiching objective achieves better results on test data than the existing methods. Moreover, as a byproduct, PCGAN learns versatile latent representations of point clouds, which can achieve competitive performance with other unsupervised learning algorithms on object recognition task. Lastly, we also provide studies on generating unseen classes of objects and transforming image to point cloud, which demonstrates the compelling generalization capability and potentials of PC-GAN.

Discriminative deep learning approaches have shown impressive results for problems where human-labeled ground truth is plentiful, but what about tasks where labels are difficult or impossible to obtain? This paper tackles one such problem: establishing dense visual correspondence across different object instances. For this task, although we do not know what the ground-truth is, we know it should be consistent across instances of that category. We exploit this consistency as a supervisory signal to train a convolutional neural network to predict cross-instance correspondences between pairs of images depicting objects of the same category. For each pair of training images we find an appropriate 3D CAD model and render two synthetic views to link in with the pair, establishing a correspondence flow 4-cycle. We use ground-truth synthetic-to-synthetic correspondences, provided by the rendering engine, to train a ConvNet to predict synthetic-to-real, real-to-real and real-to-synthetic correspondences that are cycle-consistent with the ground-truth. At test time, no CAD models are required. We demonstrate that our end-to-end trained ConvNet supervised by cycle-consistency outperforms state-of-the-art pairwise matching methods in correspondence-related tasks.

We present a new, fast and flexible pipeline for indoor scene synthesis that is based on deep convolutional generative models. Our method operates on a top-down image-based representation, and inserts objects iteratively into the scene by predicting their category, location, orientation and size with separate neural network modules. Our pipeline naturally supports automatic completion of partial scenes, as well as synthesis of complete scenes. Our method is significantly faster than the previous image-based method and generates result that outperforms state-of-the-art generative scene models in terms of faithfulness to training data and perceived visual quality.

Convolutional neural networks (CNNs) have massively impacted visual recognition in 2D images, and are now ubiquitous in state-of-the-art approaches. While CNNs naturally extend to other domains, such as audio and video, where data is also organized in rectangular grids, they do not easily generalize to other types of data such as 3D shape meshes, social network graphs or molecular graphs. To handle such data, we propose a novel graph-convolutional network architecture that builds on a generic formulation that relaxes the 1-to-1 correspondence between filter weights and data elements around the center of the convolution. The main novelty of our architecture is that the shape of the filter is a function of the features in the previous network layer, which is learned as an integral part of the neural network. Experimental evaluations on digit recognition, semi-supervised document classification, and 3D shape correspondence yield state-of-the-art results, significantly improving over previous work for shape correspondence.

Giving robots the ability to classify surface textures requires appropriate sensors and algorithms. Inspired by the biology of human tactile perception, we implement a neurorobotic texture classifier with a recurrent spiking neural network, using a novel semisupervised approach for classifying dynamic stimuli. Input to the network is supplied by accelerometers mounted on a robotic arm. The sensor data are encoded by a heterogeneous population of neurons, modeled to match the spiking activity of mechanoreceptor cells. This activity is convolved by a hidden layer using bandpass filters to extract nonlinear frequency information from the spike trains. The resulting high-dimensional feature representation is then continuously classified using a neurally implemented support vector machine. We demonstrate that our system classifies 18 metal surface textures scanned in two opposite directions at a constant velocity. We also demonstrate that our approach significantly improves upon a baseline model that does not use the described feature extraction. This method can be performed in real-time using neuromorphic hardware, and can be extended to other applications that process dynamic stimuli online.

Generative adversarial networks (GANs) are recently highly successful in generative applications involving images and start being applied to time series data. Here we describe EEG-GAN as a framework to generate electroencephalographic (EEG) brain signals. We introduce a modification to the improved training of Wasserstein GANs to stabilize training and investigate a range of architectural choices critical for time series generation (most notably upand down-sampling). For evaluation we consider and compare different metrics such as Inception score, Frechet inception distance and sliced Wasserstein distance, together showing that our EEG-GAN framework generated naturalistic EEG examples. It thus opens up a range of new generative application scenarios in the neuroscientific and neurological context, such as data augmentation in brain-computer interfacing tasks, EEG super-sampling, or restoration of corrupted data segments. The possibility to generate signals of a certain class and/or with specific properties may also open a new avenue for research into the underlying structure of brain signals.

Generic generation and manipulation of text is challenging and has limited success compared to recent deep generative modeling in visual domain. This paper aims at generating plausible natural language sentences, whose attributes are dynamically controlled by learning disentangled latent representations with designated semantics. We propose a new neural generative model which combines variational auto-encoders and holistic attribute discriminators for effective imposition of semantic structures. With differentiable approximation to discrete text samples, explicit constraints on independent attribute controls, and efficient collaborative learning of generator and discriminators, our model learns highly interpretable representations from even only word annotations, and produces realistic sentences with desired attributes. Quantitative evaluation validates the accuracy of sentence and attribute generation.


Question: Using convolutional neural networks for texture synthesis.

A Non-parametric Sampling Technique for Texture Synthesis
This paper introduces a novel non-parametric technique for synthesizing textures. The synthesis process involves expanding a new image outward, starting from an initial seed, incrementally adding one pixel at a time. Underlying this process is a Markov random field model, where the conditional probability distribution of a pixel, given its previously synthesized neighbors, is estimated. This estimation is achieved by querying the sample image to identify all similar pixel neighborhoods. A key feature of the method is its ability to control the degree of randomness through a single, perceptually intuitive parameter. By prioritizing the preservation of local structure, this approach is capable of generating high-quality textures for a diverse range of synthetic and real-world examples.

We present a novel CNN-based image editing method that allows the user to change the semantic information of an image over a user-specified region. Our method makes this possible by combining the idea of manifold projection with spatial conditional batch normalization (sCBN), a version of conditional batch normalization with userspecifiable spatial weight maps. With sCBN and manifold projection, our method lets the user perform (1) spatial class translation that changes the class of an object over an arbitrary region of user’s choice, and (2) semantic transplantation that transplants semantic information contained in an arbitrary region of the reference image to an arbitrary region in the target image. These two transformations can be used simultaneously, and can realize a complex composite image-editing task like “change the nose of a beagle to that of a bulldog, and open her mouth”. The user can also use our method with intuitive copy-paste-style manipulations. We demonstrate the power of our method on various images. Code will be available at https://github. com/pfnet-research/neural-collage.

Generative Adversarial Nets [8] were recently introduced as a novel way to train generative models. In this work we introduce the conditional version of generative adversarial nets, which can be constructed by simply feeding the data, y, we wish to condition on to both the generator and discriminator. We show that this model can generate MNIST digits conditioned on class labels. We also illustrate how this model could be used to learn a multi-modal model, and provide preliminary examples of an application to image tagging in which we demonstrate how this approach can generate descriptive tags which are not part of training labels.

In this paper we evaluated deep-learning frameworks based on Convolutional Neural Networks for the accurate classification of multispectral remote sensing data. Certain state-of-the-art models have been tested on the publicly available SAT-4 and SAT-6 high resolution satellite multispectral datasets. In particular, the performed benchmark included the AlexNet, AlexNet-small and VGG models which had been trained and applied to both datasets exploiting all the available spectral information. Deep Belief Networks, Autoencoders and other semi-supervised frameworks have been, also, compared. The high level features that were calculated from the tested models managed to classify the different land cover classes with significantly high accuracy rates i.e., above 99.9%. The experimental results demonstrate the great potentials of advanced deep-learning frameworks for the supervised classification of high resolution multispectral remote sensing data.

In this paper, we propose a very simple deep learning network for image classification that is based on very basic data processing components: 1) cascaded principal component analysis (PCA); 2) binary hashing; and 3) blockwise histograms. In the proposed architecture, the PCA is employed to learn multistage filter banks. This is followed by simple binary hashing and block histograms for indexing and pooling. This architecture is thus called the PCA network (PCANet) and can be extremely easily and efficiently designed and learned. For comparison and to provide a better understanding, we also introduce and study two simple variations of PCANet: 1) RandNet and 2) LDANet. They share the same topology as PCANet, but their cascaded filters are either randomly selected or learned from linear discriminant analysis. We have extensively tested these basic networks on many benchmark visual data sets for different tasks, including Labeled Faces in the Wild (LFW) for face verification; the MultiPIE, Extended Yale B, AR, Facial Recognition Technology (FERET) data sets for face recognition; and MNIST for hand-written digit recognition. Surprisingly, for all tasks, such a seemingly naive PCANet model is on par with the state-of-the-art features either prefixed, highly hand-crafted, or carefully learned [by deep neural networks (DNNs)]. Even more surprisingly, the model sets new records for many classification tasks on the Extended Yale B, AR, and FERET data sets and on MNIST variations. Additional experiments on other public data sets also demonstrate the potential of PCANet to serve as a simple but highly competitive baseline for texture classification and object recognition.

The performance of deep convolution neural networks will be further enhanced with the expansion of the training data set. For the image classification tasks, it is necessary to expand the insufficient training image samples through various data augmentation methods. This paper explores the impact of various data augmentation methods on image classification tasks with deep convolution Neural network, in which Alexnet is employed as the pre-training network model and a subset of CIFAR10 and ImageNet (10 categories) are selected as the original data set. The data augmentation methods used in this paper include: GAN/WGAN, Flipping, Cropping, Shifting, PCA jittering, Color jittering, Noise, Rotation, and some combinations. Experimental results show that, under the same condition of multiple increasing, the performance evaluation on small-scale data sets is more obvious, the four individual methods (Cropping, Flipping, WGAN, Rotation) perform generally better than others, and some appropriate combination methods are slightly more effective than the individuals.

1 Texture synthesized from 3D scans of snake-skin and mushroom laminae. ABSTRACT 3D sampling is a robust new strategy for exploring and creating designs. 3D textures are sampled and mixed almost like music to create new multifunctional surfaces and material microstructures (Figure 1). This paper presents several design cases performed using 3D sampling techniques, demonstrating how they can be used to explore and enhance product ideas, variations, and functions in subtle and responsive ways. In each case, design variations are generated and their mechanical behavior is evaluated against performance criteria using computational simulation or empirical testing. This work aims to promote creativity and discovery by introducing irregular geometric structures within trial-and-error feedback loops. Sayjel Vijay Patel SUTD DManD Centre

Parameterized models can produce realistic, manipulable images of human faces— with a surprisingly small number of parameters.

We present Deep Voice 3, a fully-convolutional attention-based neural textto-speech (TTS) system. Deep Voice 3 matches state-of-the-art neural speech synthesis systems in naturalness while training an order of magnitude faster. We scale Deep Voice 3 to dataset sizes unprecedented for TTS, training on more than eight hundred hours of audio from over two thousand speakers. In addition, we identify common error modes of attention-based speech synthesis networks, demonstrate how to mitigate them, and compare several different waveform synthesis methods. We also describe how to scale inference to ten million queries per day on a single GPU server.

In this paper we propose a new and efficient texture feature extraction method: the Segmentation-based Fractal Texture Analysis, or SFTA. The extraction algorithm consists in decomposing the input image into a set of binary images from which the fractal dimensions of the resulting regions are computed in order to describe segmented texture patterns. The decomposition of the input image is achieved by the Two-Threshold Binary Decomposition (TTBD) algorithm, which we also propose in this work. We evaluated SFTA for the tasks of content-based image retrieval (CBIR) and image classification, comparing its performance to that of other widely employed feature extraction methods such as Haralick and Gabor filter banks. SFTA achieved higher precision and accuracy for CBIR and image classification. Additionally, SFTA was at least 3.7 times faster than Gabor and 1.6 times faster than Haralick with respect to feature extraction time.

We propose a novel multiple-input multiple-output convolution neural network (MIMO-Net) for cell segmentation in fluorescence microscopy images. The proposed network trains the network parameters using multiple resolutions of the input image, connects the intermediate layers for better localization and context and generates the output using multi-resolution deconvolution filters. The MIMO-Net allows us to deal with variable intensity cell boundaries and highly variable cell size in the mouse pancreatic tissue by adding extra convolutional layers which bypass the max-pooling operation. The results show that our method outperforms state-of-the-art deep learning based approaches for segmentation.


Question: Texture generation utilizing convolutional neural networks.

This paper introduces an efficient algorithm for texture synthesis that significantly enhances the speed and quality of generated textures, making it a valuable tool for various applications in computer graphics, vision, and image processing. The proposed algorithm is user-friendly, requiring only a sample texture as input to produce high-quality results that are comparable or superior to those from previous methods, yet it operates at a speed that is 100 times faster. This remarkable speedup opens up new possibilities for texture synthesis in scenarios where it was previously deemed impractical. Specifically, we have successfully applied this algorithm to constrained synthesis for image editing and the generation of temporal textures. The algorithm's foundation lies in Markov Random Field texture models, which it leverages for texture generation through a deterministic searching process. To further expedite the synthesis process, we employ a technique known as tree-structured vector quantization. This enhancement not only speeds up the texture generation but also ensures that the synthesized textures maintain a high level of realism and detail.

With the advent of perceptual loss functions, new possibilities in super-resolution have emerged, and we currently have models that successfully generate near-photorealistic high-resolution images from their low-resolution observations. Up to now, however, such approaches have been exclusively limited to single image super-resolution. The application of perceptual loss functions on video processing still entails several challenges, mostly related to the lack of temporal consistency of the generated images, i.e., flickering artifacts. In this work, we present a novel adversarial recurrent network for video upscaling that is able to produce realistic textures in a temporally consistent way. The proposed architecture naturally leverages information from previous frames due to its recurrent architecture, i.e. the input to the generator is composed of the low-resolution image and, additionally, the warped output of the network at the previous step. We also propose an additional loss function to further reinforce temporal consistency in the generated sequences. The experimental validation of our algorithm shows the effectiveness of our approach which obtains competitive samples in terms of perceptual quality with improved temporal consistency.

The success of deep learning often derives from well-chosen operational building blocks. In this work, we revise the temporal convolution operation in CNNs to better adapt it to text processing. Instead of concatenating word representations, we appeal to tensor algebra and use low-rank n-gram tensors to directly exploit interactions between words already at the convolution stage. Moreover, we extend the n-gram convolution to non-consecutive words to recognize patterns with intervening words. Through a combination of lowrank tensors, and pattern weighting, we can efficiently evaluate the resulting convolution operation via dynamic programming. We test the resulting architecture on standard sentiment classification and news categorization tasks. Our model achieves state-of-the-art performance both in terms of accuracy and training speed. For instance, we obtain 51.2% accuracy on the fine-grained sentiment classification task.1

In the last two years, convolutional neural networks (CNNs) have achieved an impressive suite of results on standard recognition datasets and tasks. CNN-based features seem poised to quickly replace engineered representations, such as SIFT and HOG. However, compared to SIFT and HOG, we understand much less about the nature of the features learned by large CNNs. In this paper, we experimentally probe several aspects of CNN feature learning in an attempt to help practitioners gain useful, evidence-backed intuitions about how to apply CNNs to computer vision problems.

Learned 3D representations of human faces are useful for computer vision problems such as 3D face tracking and reconstruction from images, as well as graphics applications such as character generation and animation. Traditional models learn a latent representation of a face using linear subspaces or higher-order tensor generalizations. Due to this linearity, they can not capture extreme deformations and nonlinear expressions. To address this, we introduce a versatile model that learns a non-linear representation of a face using spectral convolutions on a mesh surface. We introduce mesh sampling operations that enable a hierarchical mesh representation that captures non-linear variations in shape and expression at multiple scales within the model. In a variational setting, our model samples diverse realistic 3D faces from a multivariate Gaussian distribution. Our training data consists of 20,466 meshes of extreme expressions captured over 12 different subjects. Despite limited training data, our trained model outperforms state-of-the-art face models with 50% lower reconstruction error, while using 75% fewer parameters. We also show that, replacing the expression space of an existing state-ofthe-art face model with our autoencoder, achieves a lower reconstruction error. Our data, model and code are available at http://github.com/ anuragranj/coma.

Generative Adversarial Nets [8] were recently introduced as a novel way to train generative models. In this work we introduce the conditional version of generative adversarial nets, which can be constructed by simply feeding the data, y, we wish to condition on to both the generator and discriminator. We show that this model can generate MNIST digits conditioned on class labels. We also illustrate how this model could be used to learn a multi-modal model, and provide preliminary examples of an application to image tagging in which we demonstrate how this approach can generate descriptive tags which are not part of training labels.

Convolutional neural networks (CNNs) have massively impacted visual recognition in 2D images, and are now ubiquitous in state-of-the-art approaches. While CNNs naturally extend to other domains, such as audio and video, where data is also organized in rectangular grids, they do not easily generalize to other types of data such as 3D shape meshes, social network graphs or molecular graphs. To handle such data, we propose a novel graph-convolutional network architecture that builds on a generic formulation that relaxes the 1-to-1 correspondence between filter weights and data elements around the center of the convolution. The main novelty of our architecture is that the shape of the filter is a function of the features in the previous network layer, which is learned as an integral part of the neural network. Experimental evaluations on digit recognition, semi-supervised document classification, and 3D shape correspondence yield state-of-the-art results, significantly improving over previous work for shape correspondence.

We present Deep Voice 3, a fully-convolutional attention-based neural textto-speech (TTS) system. Deep Voice 3 matches state-of-the-art neural speech synthesis systems in naturalness while training an order of magnitude faster. We scale Deep Voice 3 to dataset sizes unprecedented for TTS, training on more than eight hundred hours of audio from over two thousand speakers. In addition, we identify common error modes of attention-based speech synthesis networks, demonstrate how to mitigate them, and compare several different waveform synthesis methods. We also describe how to scale inference to ten million queries per day on a single GPU server.

We propose multi-view and volumetric convolutional neural networks (ConvNets) for 3D shape recognition, which combines surface normal and height fields to capture local geometry and physical size of an object. This strategy helps distinguishing between objects with similar geometries but different sizes. This is especially useful for enhancing volumetric ConvNets and classifying 3D scans with insufficient surface details. Experimental results on CAD and real-world scan datasets showed that our technique outperforms previous approaches.

Standard 3D convolution operations require much larger amounts of memory and computation cost than 2D convolution operations. The fact has hindered the development of deep neural nets in many 3D vision tasks. In this paper, we investigate the possibility of applying depthwise separable convolutions in 3D scenario and introduce the use of 3D depthwise convolution. A 3D depthwise convolution splits a single standard 3D convolution into two separate steps, which would drastically reduce the number of parameters in 3D convolutions with more than one order of magnitude. We experiment with 3D depthwise convolution on popular CNN architectures and also compare it with a similar structure called pseudo-3D convolution. The results demonstrate that, with 3D depthwise convolutions, 3D vision tasks like classification and reconstruction can be carried out with more light-weighted neural networks while still delivering comparable performances.

We present a new texture classification scheme which is invariant to surface-rotation. Many texture classification approaches have been presented in the past that are image-rotation invariant, However, image rotation is not necessarily the same as surface rotation. We have therefore developed a classifier that uses invariants that are derived from surface properties rather than image properties. Previously we developed a scheme that used surface gradient (normal) fields estimated using photometric stereo. In this paper we augment these data with albedo information and an also employ an additional feature set: the radial spectrum. We used 30 real textures to test the new classifier. A classification accuracy of 91% was achieved when albedo and gradient 1D polar and radial features were combined. The best performance was also achieved by using 2D albedo and gradient spectra. The classification accuracy is 99%.


Question: Using convolutional neural networks for texture synthesis.

Advancements in Example-based Texture Generation using Deep Learning

The field of example-based texture synthesis has seen remarkable advancements, particularly with the integration of deep learning techniques. These methodologies allow for the creation of expansive textures, tailored to meet specific requirements, from a given sample. Our comprehensive review serves three primary purposes: (1) to provide an accessible introduction for newcomers to the field, ensuring clarity and understanding of the subject matter, (2) to conduct a thorough analysis and comparison of various synthesis techniques, and (3) to outline potential avenues for future research, inspiring and guiding those interested in texture synthesis. We encompass not only foundational algorithms but also explore their extensions and practical applications in texture synthesis, emphasizing the role of deep learning in enhancing synthesis outcomes.

Convolutional Neural Networks define an exceptionally powerful class of models, but are still limited by the lack of ability to be spatially invariant to the input data in a computationally and parameter efficient manner. In this work we introduce a new learnable module, the Spatial Transformer, which explicitly allows the spatial manipulation of data within the network. This differentiable module can be inserted into existing convolutional architectures, giving neural networks the ability to actively spatially transform feature maps, conditional on the feature map itself, without any extra training supervision or modification to the optimisation process. We show that the use of spatial transformers results in models which learn invariance to translation, scale, rotation and more generic warping, resulting in state-of-the-art performance on several benchmarks, and for a number of classes of transformations.

We present a new texture classification scheme which is invariant to surface-rotation. Many texture classification approaches have been presented in the past that are image-rotation invariant, However, image rotation is not necessarily the same as surface rotation. We have therefore developed a classifier that uses invariants that are derived from surface properties rather than image properties. Previously we developed a scheme that used surface gradient (normal) fields estimated using photometric stereo. In this paper we augment these data with albedo information and an also employ an additional feature set: the radial spectrum. We used 30 real textures to test the new classifier. A classification accuracy of 91% was achieved when albedo and gradient 1D polar and radial features were combined. The best performance was also achieved by using 2D albedo and gradient spectra. The classification accuracy is 99%.

Recognizing arbitrary multi-character text in unconstrained natural photographs is a hard problem. In this paper, we address an equally hard sub-problem in this domain viz. recognizing arbitrary multi-digit numbers from Street View imagery. Traditional approaches to solve this problem typically separate out the localization, segmentation, and recognition steps. In this paper we propose a unified approach that integrates these three steps via the use of a deep convolutional neural network that operates directly on the image pixels. We employ the DistBelief (Dean et al., 2012) implementation of deep neural networks in order to train large, distributed neural networks on high quality images. We find that the performance of this approach increases with the depth of the convolutional network, with the best performance occurring in the deepest architecture we trained, with eleven hidden layers. We evaluate this approach on the publicly available SVHN dataset and achieve over 96% accuracy in recognizing complete street numbers. We show that on a per-digit recognition task, we improve upon the state-of-the-art and achieve 97.84% accuracy. We also evaluate this approach on an even more challenging dataset generated from Street View imagery containing several tens of millions of street number annotations and achieve over 90% accuracy. Our evaluations further indicate that at specific operating thresholds, the performance of the proposed system is comparable to that of human operators. To date, our system has helped us extract close to 100 million physical street numbers from Street View imagery worldwide.

Improvements in color constancy have arisen from the use of convolutional neural networks (CNNs). However, the patch-based CNNs that exist for this problem are faced with the issue of estimation ambiguity, where a patch may contain insufficient information to establish a unique or even a limited possible range of illumination colors. Image patches with estimation ambiguity not only appear with great frequency in photographs, but also significantly degrade the quality of network training and inference. To overcome this problem, we present a fully convolutional network architecture in which patches throughout an image can carry different confidence weights according to the value they provide for color constancy estimation. These confidence weights are learned and applied within a novel pooling layer where the local estimates are merged into a global solution. With this formulation, the network is able to determine what to learn and how to pool automatically from color constancy datasets without additional supervision. The proposed network also allows for end-to-end training, and achieves higher efficiency and accuracy. On standard benchmarks, our network outperforms the previous state-of-the-art while achieving 120x greater efficiency.

Convolutional neural networks (CNNs) have recently exhibited an excellent performance in hyperspectral image classification tasks. However, the straightforward CNNbased network architecture still finds obstacles when effectively exploiting the relationships between hyperspectral imaging (HSI) features in the spectral–spatial domain, which is a key factor to deal with the high level of complexity present in remotely sensed HSI data. Despite the fact that deeper architectures try to mitigate these limitations, they also find challenges with the convergence of the network parameters, which eventually limit the classification performance under highly demanding scenarios. In this paper, we propose a new CNN architecture based on spectral–spatial capsule networks in order to achieve a highly accurate classification of HSIs while significantly reducing the network design complexity. Specifically, based on Hinton’s capsule networks, we develop a CNN model extension that redefines the concept of capsule units to become spectral– spatial units specialized in classifying remotely sensed HSI data. The proposed model is composed by several building blocks, called spectral–spatial capsules, which are able to learn HSI spectral–spatial features considering their corresponding spatial positions in the scene, their associated spectral signatures, and also their possible transformations. Our experiments, conducted Manuscript received June 9, 2018; revised August 22, 2018; accepted September 19, 2018. This work was supported in part by the Ministerio de Educación (Resolución de 26 de diciembre de 2014 y de 19 de noviembre de 2015, de la Secretaría de Estado de Educación, Formación Profesional y Universidades, por la que se convocan ayudas para la formación de profesorado universitario, de los subprogramas de Formación y de Movilidad incluidos en el Programa Estatal de Promoción del Talento y su Empleabilidad, and en el marco del Plan Estatal de Investigación Científica y Técnica y de Innovación 2013–2016), in part by the Junta de Extremadura (decreto 297/2014, ayudas para la realización de actividades de investigación y desarrollo tecnológico, and de divulgación y de transferencia de conocimiento por los Grupos de Investigación de Extremadura) under Grant GR15005, in part by the Generalitat Valenciana under Contract APOSTD/2017/007, in part by the Spanish Ministry of Economy under Project ESP2016-79503-C2-2P and Project TIN2015-63646-C5-5-R, in part by the National Natural Science Foundation of China under Grant 61771496, in part by the National Key Research and Development Program of China under Grant 2017YFB0502900, and in part by the Guangdong Provincial Natural Science Foundation under Grant 2016A030313254. (Corresponding author: Jun Li.) M. E. Paoletti, J. M. Haut, J. Plaza, and A. Plaza are with the Hyperspectral Computing Laboratory, Department of Technology of Computers and Communications, Escuela Politécnica, University of Extremadura, 10003 Cáceres, Spain (e-mail: mpaoletti@unex.es; juanmariohaut@unex.es; jplaza@unex.es; aplaza@unex.es). R. Fernandez-Beltran and F. Pla are with the Institute of New Imaging Technologies, Universitat Jaume I, 12071 Castellón de la Plana, Spain (e-mail: rufernan@uji.es; pla@uji.es). J. Li is with the Guangdong Provincial Key Laboratory of Urbanization and Geosimulation, Center of Integrated Geographic Information Analysis, School of Geography and Planning, Sun Yat-sen University, Guangzhou 510275, China (e-mail: lijun48@mail.sysu.edu.cn). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TGRS.2018.2871782 using five well-known HSI data sets and several state-of-theart classification methods, reveal that our HSI classification approach based on spectral–spatial capsules is able to provide competitive advantages in terms of both classification accuracy and computational time.

1 Texture synthesized from 3D scans of snake-skin and mushroom laminae. ABSTRACT 3D sampling is a robust new strategy for exploring and creating designs. 3D textures are sampled and mixed almost like music to create new multifunctional surfaces and material microstructures (Figure 1). This paper presents several design cases performed using 3D sampling techniques, demonstrating how they can be used to explore and enhance product ideas, variations, and functions in subtle and responsive ways. In each case, design variations are generated and their mechanical behavior is evaluated against performance criteria using computational simulation or empirical testing. This work aims to promote creativity and discovery by introducing irregular geometric structures within trial-and-error feedback loops. Sayjel Vijay Patel SUTD DManD Centre

Inspired by the recent advances of deep learning and computer graphics techniques in generating sophisticated artistic styles and architectural designs, we envision a general design generation methodology that uses various specification and generative methods and tools in an integral design cycle.

Most of the existing datasets for scene text recognition merely consist of a few thousand training samples with a very limited vocabulary, which cannot meet the requirement of the state-of-the-art deep learning based text recognition methods. Meanwhile, although the synthetic datasets (e.g., SynthText90k) usually contain millions of samples, they cannot fit the data distribution of the small target datasets in natural scenes completely. To address these problems, we propose a word data generating method called SynthText-Transfer, which is capable of emulating the distribution of the target dataset. SynthText-Transfer uses a style transfer method to generate samples with arbitray text content, which preserve the texture of the reference sample in the target dataset. The generated images are not only visibly similar with real images, but also capable of improving the accuracy of the state-of-the-art text recognition methods, especially for the English and Chinese dataset with a large alphabet (in which many characters only appear in few samples, making it hard to learn for sequence models). Moreover, the proposed method is fast and flexible, with a competitive speed among common style transfer methods.

Our goal is to classify 3D models directly using convolutional neural network. Most of existing approaches rely on a set of human-engineered features. We use 3D convolutional neural network to let the network learn the features over 3D space to minimize classification error. We trained and tested over ShapeNet dataset with data augmentation by applying random transformations. We made various visual analysis to find out what the network has learned. We extended our work to extract additional information such as pose of the 3D model.

Convolutional autoencoders have emerged as popular methods for unsupervised defect segmentation on image data. Most commonly, this task is performed by thresholding a per-pixel reconstruction error based on an `-distance. This procedure, however, leads to large residuals whenever the reconstruction includes slight localization inaccuracies around edges. It also fails to reveal defective regions that have been visually altered when intensity values stay roughly consistent. We show that these problems prevent these approaches from being applied to complex real-world scenarios and that they cannot be easily avoided by employing more elaborate architectures such as variational or feature matching autoencoders. We propose to use a perceptual loss function based on structural similarity that examines inter-dependencies between local image regions, taking into account luminance, contrast, and structural information, instead of simply comparing single pixel values. It achieves significant performance gains on a challenging real-world dataset of nanofibrous materials and a novel dataset of two woven fabrics over state-of-the-art approaches for unsupervised defect segmentation that use per-pixel reconstruction


Question: Combining learning and reasoning services for explainable information integration.

This paper presents a groundbreaking technique for elucidating the decision-making process of complex machine learning models, particularly focusing on Deep Neural Networks (DNNs). DNNs have revolutionized various domains, including image classification, natural language understanding, and human action detection, by achieving state-of-the-art performance. However, their intricate, multilayered architecture often results in a lack of transparency, which hinders their interpretability and practical applicability. To address this challenge, we propose a novel method to decompose the classification decisions made by DNNs into contributions from individual input elements. Although our primary application is in image classification, the methodology is versatile and can be extended to a wide range of data types, learning tasks, and network structures. The core of our approach is the deep Taylor decomposition, which efficiently leverages the network's architecture to trace back the decision-making process from the output to the input layer. We validate the efficacy of our technique through empirical evaluations on benchmark datasets such as MNIST and ILSVRC, showcasing its potential to enhance the explainability of nonlinear classification decisions.

Computer systems that can directly and accurately answer peoples' questions over a broad domain of human knowledge have been envisioned by scientists and writers since the advent of computers themselves. Open domain question answering holds tremendous promise for facilitating informed decision making over vast volumes of natural language content. Applications in business intelligence, healthcare, customer support, enterprise knowledge management, social computing, science and government would all benefit from deep language processing. The DeepQA project (www.ibm.com/deepqa) is aimed at illustrating how the advancement and integration of Natural Language Processing (NLP), Information Retrieval (IR), Machine Learning (ML), massively parallel computation and Knowledge Representation and Reasoning (KR&R) can greatly advance open-domain automatic Question Answering. An exciting proof-point in this challenge is to develop a computer system that can successfully compete against top human players at the Jeopardy! quiz show (www.jeopardy.com). Attaining champion-level performance Jeopardy! requires a computer to rapidly answer rich open-domain questions, and to predict its own performance on any given category/question. The system must deliver high degrees of precision and confidence over a very broad range of knowledge and natural language content and with a 3-second response time. To do this DeepQA generates, evidences and evaluates many competing hypotheses. A key to success is automatically learning and combining accurate confidences across an array of complex algorithms and over different dimensions of evidence. Accurate confidences are needed to know when to "buzz in" against your competitors and how much to bet. Critical for winning at Jeopardy!, High precision and accurate confidence computations are just as critical for providing real value in business settings where helping users focus on the right content sooner and with greater confidence can make all the difference. The need for speed and high precision demands a massively parallel compute platform capable of generating, evaluating and combing 1000's of hypotheses and their associated evidence. In this talk I will introduce the audience to the Jeopardy! Challenge and describe our technical approach and our progress on this grand-challenge problem.

Rules and ontologies play a key role in the layered architecture of the Semantic Web, as they are used to ascribe meaning to, and to reason about, data on the Web. While the Ontology Layer of the Semantic Web is quite developed, and the Web Ontology Language (OWL) is a W3C recommendation since a couple of years already, the rules layer is far less developed and an active area of research; a number of initiatives and proposals have been made so far, but no standard as been released yet. Many implementations of rule engines are around which deal with Semantic Web data in one or another way. This article gives a comprehensive, although not exhaustive, overview of such systems, describes their supported languages, and sets them in relation with theoretical approaches for combining rules and ontologies as foreseen in the Semantic Web architecture. In the course of this, we identify desired properties and common features of rule languages and evaluate existing systems against their support. Furthermore, we review technical problems underlying the integration of rules and ontologies, and classify representative proposals for theoretical integration approaches into different categories.

Recommender systems represent user preferences for the purpose of suggesting items to purchase or examine. They have become fundamental applications in electronic commerce and information access, providing suggestions that effectively prune large information spaces so that users are directed toward those items that best meet their needs and preferences. A variety of techniques have been proposed for performing recommendation, including content-based, collaborative, knowledge-based and other techniques. To improve performance, these methods have sometimes been combined in hybrid recommenders. This paper surveys the landscape of actual and possible hybrid recommenders, and introduces a novel hybrid, EntreeC, a system that combines knowledge-based recommendation and collaborative filtering to recommend restaurants. Further, we show that semantic ratings obtained from the knowledge-based part of the system enhance the effectiveness of collaborative filtering.

The goal of giving a well-defined meaning to information is currently shared by endeavors such as the Semantic Web as well as by current trends within Knowledge Management. They all depend on the large-scale formalization of knowledge and on the availability of formal metadata about information resources. However, the question how to provide the necessary formal metadata in an effective and efficient way is still not solved to a satisfactory extent. Certainly, the most effective way to provide such metadata as well as formalized knowledge is to let humans encode them directly into the system, but this is neither efficient nor feasible. Furthermore, as current social studies show, individual knowledge is often less powerful than the collective knowledge of a certain community.As a potential way out of the knowledge acquisition bottleneck, we present a novel methodology that acquires collective knowledge from the World Wide Web using the GoogleTM API. In particular, we present PANKOW, a concrete instantiation of this methodology which is evaluated in two experiments: one with the aim of classifying novel instances with regard to an existing ontology and one with the aim of learning sub-/superconcept relations.

Usage of background knowledge about events and their relations to other concepts in the application domain, can improve the quality of event processing. In this paper, we describe a system for knowledge-based event detection of complex stock market events based on available background knowledge about stock market companies. Our system profits from data fusion of live event stream and background knowledge about companies which is stored in a knowledge base. Users of our system can express their queries in a rule language which provides functionalities to specify semantic queries about companies in the SPARQL query language for querying the external knowledge base and combine it with event data stream. Background makes it possible to detect stock market events based on companies attributes and not only based on syntactic processing of stock price and volume.

Existing knowledge-based question answering systems often rely on small annotated training data. While shallow methods like relation extraction are robust to data scarcity, they are less expressive than the deep meaning representation methods like semantic parsing, thereby failing at answering questions involving multiple constraints. Here we alleviate this problem by empowering a relation extraction method with additional evidence from Wikipedia. We first present a neural network based relation extractor to retrieve the candidate answers from Freebase, and then infer over Wikipedia to validate these answers. Experiments on the WebQuestions question answering dataset show that our method achieves an F1 of 53.3%, a substantial improvement over the state-of-the-art.

Diagnosis, or the method to connect causes to its effects, is an important reasoning task for obtaining insight on cities and reaching the concept of sustainable and smarter cities that is envisioned nowadays. This paper, focusing on transportation and its road traffic, presents how road traffic congestions can be detected and diagnosed in quasi real-time. We adapt pure Artificial Intelligence diagnosis techniques to fully exploit knowledge which is captured through relevant semantics-augmented stream and static data from various domains. Our prototype of semantic-aware diagnosis of road traffic congestions, experimented in Dublin Ireland, works efficiently with large, heterogeneous information sources and delivers value-added services to citizens and city managers in quasi real-time.

Recommender Systems are commonly used in web applications to support users in finding items of their interest. We here propose to use Recommender Systems in Internet of Things scenarios to support human decision making in the physical world. For instance, users' choices (visit actions) for points of interests (POIs) while they explore a sensor enabled city can be tracked and used to generate recommendations for not yet visited POIs. In this PhD research, we propose a novel learning approach for generating an explainable human behaviour model and relevant recommendations in sensor enabled areas. Moreover, we propose techniques for simulating user behaviour and analyse the collective dynamics of a population of users.

Expert finding is an information retrieval task that is concerned with the search for the most knowledgeable people with respect to a specific topic, and the search is based on documents that describe people’s activities. The task involves taking a user query as input and returning a list of people who are sorted by their level of expertise with respect to the user query. Despite recent interest in the area, the current state-of-the-art techniques lack in principled approaches for optimally combining different sources of evidence. This article proposes two frameworks for combining multiple estimators of expertise. These estimators are derived from textual contents, from graph-structure of the citation patterns for the community of experts, and from profile information about the experts. More specifically, this article explores the use of supervised learning to rank methods, as well as rank aggregation approaches, for combing all of the estimators of expertise. Several supervised learning algorithms, which are representative of the pointwise, pairwise and listwise approaches, were tested, and various state-of-the-art data fusion techniques were also explored for the rank aggregation framework. Experiments that were performed on a dataset of academic publications from the Computer Science domain attest the adequacy of the proposed approaches.

Here we propose an OWL encoded context ontology (CONON) for modeling context in pervasive computing environments, and for supporting logic-based context reasoning. CONON provides an upper context ontology that captures general concepts about basic context, and also provides extensibility for adding domain-specific ontology in a hierarchical manner. Based on this context ontology, we have studied the use of logic reasoning to check the consistency of context information, and to reason over low-level, explicit context to derive high-level, implicit context. By giving a performance study for our prototype, we quantitatively evaluate the feasibility of logic based context reasoning for nontime-critical applications in pervasive computing environments, where we always have to deal carefully with the limitation of computational resources.


Question: Combining learning and reasoning services for explainable information integration.

Introducing the Bayesian Case Model (BCM): A Unified Framework for Case-Based Reasoning and Prototype-Based Classification and Clustering
The Bayesian Case Model (BCM) is proposed as a comprehensive framework that integrates the strengths of case-based reasoning (CBR) and prototype-based classification and clustering within a Bayesian generative model. BCM enriches the CBR paradigm by anchoring it in a Bayesian framework, enabling the learning of prototypes—representative observations that epitomize the core characteristics of data clusters—through joint inference on cluster assignments, prototype identification, and feature selection. Notably, BCM promotes model sparsity by discerning subspaces, the subsets of features that are critical for defining prototypes. This dual focus on prototypes and subspaces not only enhances interpretability by pinpointing key features but also maintains high classification accuracy. Empirical evidence from human subject studies confirms that BCM-generated explanations significantly boost users' comprehension compared to those from existing methodologies.

We present a new Convolutional Neural Network (CNN) model for text classification that jointly exploits labels on documents and their constituent sentences. Specifically, we consider scenarios in which annotators explicitly mark sentences (or snippets) that support their overall document categorization, i.e., they provide rationales. Our model exploits such supervision via a hierarchical approach in which each document is represented by a linear combination of the vector representations of its component sentences. We propose a sentence-level convolutional model that estimates the probability that a given sentence is a rationale, and we then scale the contribution of each sentence to the aggregate document representation in proportion to these estimates. Experiments on five classification datasets that have document labels and associated rationales demonstrate that our approach consistently outperforms strong baselines. Moreover, our model naturally provides explanations for its predictions.

Process mining aims at discovering, monitoring, and improving business processes by extracting knowledge from event logs. In this respect, process mining can be applied only if there are proper event logs that are compatible with accepted standards, such as extensible event stream (XES). Unfortunately, in many real world set-ups, such event logs are not explicitly given, but instead are implicitly represented in legacy information systems. In this work, we exploit a framework and associated methodology for the extraction of XES event logs from relational data sources that we have recently introduced. Our approach is based on describing logs by means of suitable annotations of a conceptual model of the available data, and builds on the ontology-based data access (OBDA) paradigm for the actual log extraction. Making use of a realworld case study in the services domain, we compare our novel approach with a more traditional extract-transform-load based one, and are able to illustrate its added value. We also present a set of tools that we have developed and that support the OBDA-based log extraction framework. The tools are integrated as plugins of the ProM process mining suite.

Autonomous robots that are to assist humans in their daily lives must recognize and understand the meaning of objects in their environment. However, the open nature of the world means robots must be able to learn and extend their knowledge about previously unknown objects on-line. In this work we investigate the problem of unknown object hypotheses generation, and employ a semantic web-mining framework along with deep-learning-based object detectors. This allows us to make use of both visual and semantic features in combined hypotheses generation. Experiments on data from mobile robots in real world application deployments show that this combination improves performance over the use of either method in isolation.

In recent years patents have become increasingly important for businesses to protect their intellectual capital and as a valuable source of information. Patent information is, however, not employed to its full potential and the interpretation of structured and unstructured patent information in large volumes remains a challenge. We address this by proposing an integrated interdisciplinary approach that uses natural language processing and machine learning techniques to formalize multilingual patent information in an ontology. The ontology further contains patent and domain specific knowledge, which allows for aligning patents with technological fields of interest and other business-related artifacts. Our empirical evaluation shows that for categorizing patents according to well-known technological fields of interest, the approach achieves high accuracy with selected feature sets compared to related work focussing on monolingual patents. We further show that combining OWL RL reasoning with SPARQL querying over the patent knowledge base allows for answering complex business queries and illustrate this with real-world use cases from the automotive domain.

Information extraction is a form of shallow text processing which locates a specified set of relevant items in natural language documents. Such systems can be useful, but require domain-specific knowledge and rules, and are time-consuming and difficult to build by hand, making infomation extraction a good testbed for the application of machine learning techniques to natural language processing. This paper presents a system, RAPIER, that takes pairs of documents and filled templates and induces pattern-match rules that directly extract fillers for the slots in the template. The learning algorithm incorporates techniques from several inductive logic programming systems and learns unbounded patterns that include constraints on the words and part-of-speech tags surrounding the filler. Encouraging results are presented on learning to extract information from computer job postings from the newsgroup misc. jobs. offered.

This paper describes an approach for generating rich and compelling explanations in recommender systems, based on opinions mined from user-generated reviews. The explanations highlight the features of a recommended item that matter most to the user and also relate them to other recommendation alternatives and the user's past activities to provide a context.

In SW (SW), Ontology is an important layer architecture that is develop with the help of web ontology language (OWL) and makes the web semantic. OWL is designs and creates metadata that required for developing SW applications. It is an essential part of SW. In this paper we design an education and non-education domain ontology framework which is developed by using protégé an ontology editor tool developed by Stanford University. We also show the output for both ontologies by using pellet reasoner based on the proposed ontology frameworks. We analyze from our result that OWL improves the inference compare to Resource Description Framework (RDF) and Resource Description Framework Schema (RDFS). OWL also provides more relevant information compare to RDF and RDFS and OWL precision and recall is better compare to RDF and RDFS. In our paper we used Pellet reasoner that retrieve relevant information from the education and non education ontology and protégé 4.3 alpha tool for the construction of education and non education domain ontology.

1College of Computer Science, Sichuan Normal University, Chengdu, Sichuan, China 2Developmental Neurocognition Lab, Department of Psychological Sciences, Birkbeck, University of London, London, UK 3College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, China Correspondence Juan Yang, College of Computer Science, Sichuan Normal University, Chengdu, Sichuan, China. Email: jkxy_yjuan@sicnu.edu.cn Funding information National Natural Science Foundation of China, Grant/Award Number: (61402309); UK Economic and Social Research Council, Grant/Award Number: RES-062-23-2721 Summary Mapping problems are typical research topics related to natural language learning, and they include not only classification mappings but also nonclassification mappings, such as verbs and their past tenses. Connectionist computational models are one of the most popular approaches for simulating those mapping problems; however, their lack of explanatory ability has prevented them from being further used to understand the language learning process. Therefore, the work of extracting rational rules from a connectionist model is as important as simulating the mapping behaviors. Unfortunately, there is no available technique that can be directly applied in those computational models to simulate nonclassification problems. In this paper, an autoencoder-based connectionist computational model is proposed to derive a rule extraction method that can construct “If-Then” rational relations with high fidelity for nonclassification mapping problems. To demonstrate its generalizability, this computational model is extended to a modified version to address a multi-label classification mapping problem related to cognitive style prediction. Experiments prove this computational model's simulation ability and its explanatory ability on nonclassification problems by comparing its fidelity performances with those of the classical connectionist computational model (multilayer perceptron artificial neural network), and its similar ability on a multi-label classification problem (Felder-Silverman learning style classification) by comparing its prediction accuracy with those of other rule induction techniques.

Machine learning models are becoming more and more powerful and accurate, but their good predictions usually come with a high complexity. Depending on the situation, such a lack of interpretability can be an important and blocking issue. This is especially the case when trust is needed on the user side in order to take a decision based on the model prediction. For instance, when an insurance company uses a machine learning algorithm in order to detect fraudsters: the company would trust the model to be based on meaningful variables before actually taking action and investigating on a particular individual. In this thesis, several explanation methods are described and compared on multiple datasets (text data, numerical), on classification and regression problems.

We study question-answering over semi-structured data. We introduce a new way to apply the technique of semantic parsing by applying machine learning only to provide annotations that the system infers to be missing; all the other parsing logic is in the form of manually authored rules. In effect, the machine learning is used to provide non-syntactic matches, a step that is ill-suited to manual rules. The advantage of this approach is in its debuggability and in its transparency to the end-user. We demonstrate the effectiveness of the approach by achieving state-of-the-art performance of 40.42% on a standard benchmark dataset over tables from Wikipedia.


Question: Combining learning and reasoning services for explainable information integration.

In recent years, deep learning has emerged as a transformative force in various domains, reshaping the landscape of modern society. Its applications span from web searches and content filtering on social media platforms to personalized recommendations on e-commerce sites, all the while becoming increasingly integrated into consumer electronics such as cameras and smartphones. The ubiquity of deep learning in these applications is a testament to its capability in processing raw, natural data in a manner that conventional machine-learning techniques could not.

Historically, building a pattern-recognition or machine-learning system involved meticulous engineering and domain expertise to craft a feature extractor that could transform raw data into a feature vector suitable for pattern detection or classification. Deep learning, however, automates this process through a series of transformations that create multiple levels of representation. Each level, starting from the raw input, is processed by simple yet non-linear modules that incrementally abstract the data, allowing for the learning of complex functions.

For classification tasks, these layers of representation highlight critical features of the input while minimizing irrelevant variations. An illustrative example is the processing of images: the first layer might detect edges and their orientations, the second layer recognizes motifs by identifying specific edge arrangements, and subsequent layers assemble these motifs into larger, more complex structures. This hierarchical learning is not engineered by humans but is learned from data through a general learning procedure.

Deep learning's prowess lies in its ability to uncover intricate structures in high-dimensional data, making it a versatile tool across scientific, commercial, and governmental sectors. It has set new benchmarks in image and speech recognition, excelled in predicting drug molecule activity, analyzing particle accelerator data, reconstructing brain circuits, and forecasting the impact of mutations on gene expression and disease. Remarkably, it has also shown great promise in natural language understanding, specifically in tasks like topic classification, sentiment analysis, question answering, and language translation.

The future of deep learning looks bright, as it requires minimal human intervention and can capitalize on growing computational power and data availability. Ongoing advancements in learning algorithms and architectures for deep neural networks are expected to further accelerate its progress, leading to more widespread adoption and success in various fields.

Semantic Space is a pervasive computing infrastructure that exploits semantic Web technologies to support explicit representation, expressive querying, and flexible reasoning of contexts in smart spaces.

We present a rule-based system for computer-aided circuit analysis. The set of rules, called EL, is written in a rule language called ARS. Rules are implemented by ARS as pattern-directed invocation demons monitoring an associative data base. Deductions are performed in an antecedent manner, giving EL's analysis a catch-as-catch-can flavour suggestive of the behavior of expert circuit analyzers. We call this style of circuit analysis propagation of constraints. The system threads deduced fact~ with justifications which mention the antecedent facts attd the rule used. These justifications may be examined by the user to gain in right into the operation of the set of rules as they apply to a problem. ~q~e same justifications are used by the system to determine the currently active dat~-base context Jor reasoning in hypothetical situations. They are also used by the system in the at~alysis of failures to reduce the search space. This leads to effective control of combinatorial search which we call dependency-directed backtracking.

Story understanding systems need to be able to perform commonsense reasoning, specifically regarding characters’ goals and their associated actions. Some efforts have been made to form large-scale commonsense knowledge bases, but integrating that knowledge into story understanding systems remains a challenge. We have implemented the Aspire system, an application of large-scale commonsense knowledge to story understanding. Aspire extends Genesis, a rule-based story understanding system, with tens of thousands of goalrelated assertions from the commonsense semantic network ConceptNet. Aspire uses ConceptNet’s knowledge to infer plausible implicit character goals and story causal connections at a scale unprecedented in the space of story understanding. Genesis’s rule-based inference enables precise story analysis, while ConceptNet’s relatively inexact but widely applicable knowledge provides a significant breadth of coverage difficult to achieve solely using rules. Genesis uses Aspire’s inferences to answer questions about stories, and these answers were found to be plausible in a small study. Though we focus on Genesis and ConceptNet, demonstrating the value of supplementing precise reasoning systems with large-scale, scruffy commonsense knowledge is our primary contribution.

Here we propose an OWL encoded context ontology (CONON) for modeling context in pervasive computing environments, and for supporting logic-based context reasoning. CONON provides an upper context ontology that captures general concepts about basic context, and also provides extensibility for adding domain-specific ontology in a hierarchical manner. Based on this context ontology, we have studied the use of logic reasoning to check the consistency of context information, and to reason over low-level, explicit context to derive high-level, implicit context. By giving a performance study for our prototype, we quantitatively evaluate the feasibility of logic based context reasoning for nontime-critical applications in pervasive computing environments, where we always have to deal carefully with the limitation of computational resources.

We tackle the problem of entity linking for large collections of online pages; Our system, ZenCrowd, identifies entities from natural language text using state of the art techniques and automatically connects them to the Linked Open Data cloud. We show how one can take advantage of human intelligence to improve the quality of the links by dynamically generating micro-tasks on an online crowdsourcing platform. We develop a probabilistic framework to make sensible decisions about candidate links and to identify unreliable human workers. We evaluate ZenCrowd in a real deployment and show how a combination of both probabilistic reasoning and crowdsourcing techniques can significantly improve the quality of the links, while limiting the amount of work performed by the crowd.

Memory-Based Learning (MBL) is based on the storage of all available training data, and similarity-based reasoning for handling new cases. By interpreting tasks such as POS tagging and shallow parsing as classification tasks, the advantages of MBL (implicit smoothing of sparse data, automatic integration and relevance weighting of information sources, handling exceptional data) contribute to state-of-the-art accuracy. However, Hidden Markov Models (HMM) typically achieve higher accuracy than MBL (and other Machine Learning approaches) for tasks such as POS tagging and chunking. In this paper, we investigate how the advantages of MBL, such as its potential to integrate various sources of information, come to play when we compare our approach to HMMs on two Information Extraction (IE) datasets: the well-known Seminar Announcement data set and a new German Curriculum Vitae data set. 1 Memory-Based Language Processing Memory-Based Learning (MBL) is a supervised classification-based learning method. A vector of feature values (an instance) is associated with a class by a classifier that lazily extrapolates from the most similar set (nearest neighbors) selected from all stored training examples. This is in contrast to eager learning methods like decision tree learning [26], rule induction [9], or Inductive Logic Programming [7], which abstract a generalized structure from the training set beforehand (forgetting the examples themselves), and use that to derive a classification for a new instance. In MBL, a distance metric on the feature space defines what are the nearest neighbors of an instance. Metrics with feature weights based on information-theory or other relevance statistics allow us to use rich representations of instances and their context, and to balance the influences of diverse information sources in computing distance. Natural Language Processing (NLP) tasks typically concern the mapping of an input representation (e.g., a series of words) into an output representation (e.g., the POS tags corresponding to each word in the input). Most NLP tasks can therefore easily be interpreted as sequences of classification 34 Zavrel, Daelemans tasks: e.g., given a word and some representation of its context, decide what tag to assign to each word in its context. By creating a separate classification instance (a “moving window” approach) for each word and its context, shallow syntactic or semantic structures can be produced for whole sentences or texts. In this paper, we argue that more semantic and complex input-output mappings, such as Information Extraction, can also effectively be modeled by such a Memory-based classification-oriented framework, and that this approach has a number of very interesting advantages over rivalling methods, most notably that each classification decision can be made dependent on a very rich and diverse set of features. The properties of MBL as a lazy, similarity-based learning method seem make a good fit to the properties of typical disambiguation problems in NLP: • Similar input representations lead to similar output. E.g., words occurring in a similar context in general have the same POS tag. Similarity-based reasoning is the core of MBL. • Many sub-generalizations and exceptions. By keeping in memory all training instances, exceptions included, an MBL approach can capture generalization from exceptional or low-frequency cases according to [12]. • Need for integration of diverse types of information. E.g., in Information Extraction, lexical features, spelling features, syntactic as well as phrasal context features, global text structure, and layout features can potentially be very relevant. • Automatic smoothing in very rich event spaces. Supervised learning of NLP tasks regularly runs into problems of sparse data; not enough training data is available to extract reliable parameters for complex models. MBL incorporates an implicit robust form of smoothing by similarity [33]. In the remainder of this Section, we will show how a memory-, similarity-, and classification-based approach can be applied to shallow syntactic parsing, and can lead to state-of-the-art accuracy. Most of the tasks discussed here can also easily be modeled using Hidden Markov Models (HMM), and often with surprising accuracy. We will discuss the strengths of the HMMs and draw a comparison between the classification-based MBL method and the sequence-optimizing HMM approach (Section 1.2). 1.1 Memory-Based Shallow Parsing Shallow parsing is an important component of most text analysis systems in Text Mining applications such as information extraction, summary generation, and question answering. It includes discovering the main constituents of sentences (NPs, VPs, PPs) and their heads, and determining syntactic relationships like subject, object, adjunct relations between verbs and heads of other constituents. This is an important first step to understanding the who, what, when, and where of sentences in a text. Feature-Rich Memory-Based Classification for Information Extraction 35 In our approach to memory-based shallow parsing, we carve up the syntactic analysis process into a number of classification tasks with input vectors representing a focus item and a dynamically selected surrounding context. These classification tasks can be segmentation tasks (e.g., decide whether a focus word or tag is the start or end of an NP) or disambiguation tasks (e.g., decide whether a chunk is the subject NP, the object NP or neither). Output of some memory-based modules is used as input by other memory-based modules (e.g., a tagger feeds a chunker and the latter feeds a syntactic relation assignment module). Similar ideas about cascading of processing steps have also been explored in other approaches to text analysis: e.g., finite state partial parsing [1,18], statistical decision tree parsing [23], and maximum entropy parsing [30]. The approach briefly described here is explained and evaluated in more detail in [10,11,6] . Chunking The phrase chunking task can be defined as a classification task by generalizing the approach of [28], who proposed to convert NP-chunking to tagging each word with I for a word inside an NP, O for outside an NP, and B for between two NPs). The decision on these so called IOB tags for a word can be made by looking at the Part-of-Speech tag and the identity of the focus word and its local context. For the more general task of chunking other non-recursive phrases, we simply extend the tag set with IOB tags for each type of phrase. To illustrate this encoding with the extended IOB tag set, we can tag the sentence: But/CC [NP the/DT dollar/NN NP] [ADVP later/RB ADVP] [VP rebounded/VBD VP] ,/, [VP finishing/VBG VP] [ADJP slightly/RB higher/R ADJP] [Prep against/IN Prep] [NP the/DT yen/NNS NP] [ADJP although/IN ADJP] [ADJP slightly/RB lower/JJR ADJP] [Prep against/IN Prep] [NP the/DT mark/NN NP] ./. as: But/CCO the/DTI−NP dollar/NNI−NP later/RBI−ADV P rebounded/VBDI−V P ,/,O finishing/VBGI−V P slightly/RBI−ADV P higher/RBRI−ADV P against/INI−Prep the/DTI−NP yen/NNSI−NP although/INI−ADJP slightly/RBB−ADJP lower/JJRI−ADJP against/INI−Prep the/DTI−NP mark/NNI−NP ./.O Table 1 (from [6]) shows the accuracy of this memory-based chunking approach when training and testing on Wall Street Journal material. We report Precision, Recall, and Fβ=1 scores, a weighted harmonic mean of Recall and Precision (Fβ = (β+1)∗P∗R β2∗P+R) ). 1 An online demonstration of the Memory-Based Shallow Parser can be found at http://ilk.kub.nl . 36 Zavrel, Daelemans type precision recall Fβ=1 NPchunks 92.5 92.2 92.3 VPchunks 91.9 91.7 91.8 ADJPchunks 68.4 65.0 66.7 ADVPchunks 78.0 77.9 77.9 Prepchunks 95.5 96.7 96.1 PPchunks 91.9 92.2 92.0 ADVFUNCs 78.0 69.5 73.5 Table 1. Results of chunking–labeling experiments. Reproduced from [6]. Grammatical Relation Finding After POS tagging, phrase chunking and labeling, the last step of the shallow parsing consists of resolving the (types of) attachment between labeled phrases. This is done by using a classifier to assign a grammatical relation (GR) between pairs of words in a sentence. In our approach, one of these words is always a verb, since this yields the most important GRs. The other word (focus) is the head of the phrase which is annotated with this grammatical relation in the treebank (e.g., a noun as head of an NP). An instance for such a pair of words is constructed by extracting a set of feature values from the sentence. The instance contains information about the verb and the focus: a feature for the word form and a feature for the POS of both. It also has similar features for the local context of the focus. Experiments on the training data suggest an optimal context width of two words to the left and one to the right. In addition to the lexical and the local context information, superficial information about clause structure was included: the distance from the verb to the focus, counted in words. A negative distance means that the focus is to the left of the verb. Other features contain the number of other verbs between the verb and the focus, and the number of intervening commas. These features were chosen by manual “feature engineering”. Table 2 shows some of the feature-value instances corresponding to the following sentence (POS tags after the slash, chunks denoted with square and curly brackets, and adverbial functions after the dash): [ADVP Not /RB surprisingly /RB ADVP] ,/, [NP Peter /NNP Miller /NNP NP] ,/, [NP who /WP NP] [VP organized /VBD VP] [NP the /DT conference /NN NP] {PP-LOC [Prep in /IN Prep] [NP New /NNP York /NNP NP] PP-LOC} ,/, [VP does /VBZ not /RB want /VB to /TO come /VB VP] {PP-DIR [Prep to /IN Prep] [NP Paris /NNP NP] PP-DIR} [Prep without /IN Prep] [VP bringing /VBG VP] [NP his /PRP$ wife /NN NP]. Table 3 shows the results of the experiments. In the first row, only POS tag features are used. Other rows show the results when adding several types of chunk informati

In distributed geospatial applications with heterogeneous databases, an ontology-driven approach to data integration relies on the alignment of the concepts of a global ontology that describe the domain, with the concepts of the ontologies that describe the data in the distributed databases. Once the alignment between the global ontology and each distributed ontology is established, agreements that encode a variety of mappings between concepts are derived. In this way, users can potentially query hundreds of geospatial databases using a single query. Using our approach, querying can be easily extended to new data sources and, therefore, to new regions. In this paper, we describe the AgreementMaker, a tool that displays the ontologies, supports several mapping layers visually, presents automatically generated mappings, and finally produces the agreements. r 2007 Elsevier Ltd. All rights reserved.

Building neural networks to query a knowledge base (a table) with natural language is an emerging research topic in deep learning. An executor for table querying typically requires multiple steps of execution because queries may have complicated structures. In previous studies, researchers have developed either fully distributed executors or symbolic executors for table querying. A distributed executor can be trained in an end-to-end fashion, but is weak in terms of execution efficiency and explicit interpretability. A symbolic executor is efficient in execution, but is very difficult to train especially at initial stages. In this paper, we propose to couple distributed and symbolic execution for natural language queries, where the symbolic executor is pretrained with the distributed executor’s intermediate execution results in a step-by-step fashion. Experiments show that our approach significantly outperforms both distributed and symbolic executors, exhibiting high accuracy, high learning efficiency, high execution efficiency, and high interpretability.

We propose a simple approach to combining first-order logic and probabilistic graphical models in a single representation. A Markov logic network (MLN) is a first-order knowledge base with a weight attached to each formula (or clause). Together with a set of constants representing objects in the domain, it specifies a ground Markov network containing one feature for each possible grounding of a first-order formula in the KB, with the corresponding weight. Inference in MLNs is performed by MCMC over the minimal subset of the ground network required for answering the query. Weights are efficiently learned from relational databases by iteratively optimizing a pseudo-likelihood measure. Optionally, additional clauses are learned using inductive logic programming techniques. Experiments with a real-world database and knowledge base in a university domain illustrate the promise of this approach.

A new paradigm of the evolving computational intelligence systems (ECIS) is introduced in a generic framework of the knowledge and data integration (KDI). This generalization of the recent advances in the development of evolving fuzzy and neuro-fuzzy models and the more analytical angle of consideration through the prism of knowledge evolution as opposed to the usually used datacentred approach marks the novelty of the present paper. ECIS constitutes a suitable paradigm for adaptive modeling of continuous dynamic processes and tracing the evolution of knowledge. The elements of evolution, such as inheritance and structure development are related to the knowledge and data pattern dynamics and are considered in the context of an individual system/model. Another novelty of this paper consists of the comparison at a conceptual level between the concept of models and knowledge captured by these models evolution and the well known paradigm of evolutionary computation. Although ECIS differs from the concept of evolutionary (genetic) computing, both paradigms heavily borrow from the same source – nature and human evolution. As the origin of knowledge, humans are the best model of an evolving intelligent system. Instead of considering the evolution of population of spices or genes as the evolutionary computation algorithms does the ECIS concentrate on the evolution of a single intelligent system. The aim is to develop the intelligence/knowledge of this system through an evolution using inheritance and modification, upgrade and reduction. This approach is also suitable for the integration of new data and existing models into new models that can be incrementally adapted to future incoming data. This powerful new concept has been recently introduced by the authors in a series of parallel works and is still under intensive development. It forms the conceptual basis for the development of the truly intelligent systems. Another specific of this paper includes bringing together the two working examples of ECIS, namely ECOS and EFS. The ideas are supported by illustrative examples (a synthetic non-linear function for the ECOS case and a benchmark problem of house price modelling from UCI repository for the case of EFS).