Question: Neural program synthesis guided by execution.

This paper introduces a framework for Neural Program Meta-Induction, focusing on enhancing data and computational efficiency in program induction by capitalizing on insights from related tasks. Contrary to prevailing methods that require extensive input/output (I/O) examples to learn underlying mappings, we propose strategies for cross-task knowledge transfer to bolster program induction in data-limited contexts. Our methodology comprises two primary approaches: portfolio adaptation and meta program induction. Portfolio adaptation entails pretraining a collection of induction models on a diverse set of related tasks, with the most effective model then refined for the novel task through transfer learning. In contrast, meta program induction employs a k-shot learning technique to enable a model to generalize to new tasks without further training. To assess the efficacy of our proposals, we developed a benchmark dataset consisting of programs written in the Karel programming language. Through comprehensive experimental evaluations on this benchmark, we establish that our approaches markedly outperform the baseline induction method, which lacks knowledge transfer. We further scrutinize the comparative performance of both methods and pinpoint conditions under which they excel. Notably, meta induction surpasses all existing methods under conditions of extreme data scarcity, i.e., when fewer than ten examples are available. As the number of I/O examples grows (i.e., exceeding a thousand), portfolio adapted program induction emerges as the superior approach. For intermediate data volumes, we illustrate that the amalgamation of adapted meta program induction yields the most robust performance.

We propose a quantum generalisation of a classical neural network. The classical neurons are firstly rendered reversible by adding ancillary bits. Then they are generalised to being quantum reversible, i.e. unitary. (The classical networks we generalise are called feedforward, and have step-function activation functions.) The quantum network can be trained efficiently using gradient descent on a cost function to perform quantum generalisations of classical tasks. We demonstrate numerically that it can: (i) compress quantum states onto a minimal number of qubits, creating a quantum autoencoder, and (ii) discover quantum communication protocols such as teleportation. Our general recipe is theoretical and implementation-independent. The quantum neuron module can naturally be implemented photonically.

Most of computer science focuses on automatically solving given computational problems. I focus on automatically inventing or discovering problems in a way inspired by the playful behavior of animals and humans, to train a more and more general problem solver from scratch in an unsupervised fashion. Consider the infinite set of all computable descriptions of tasks with possibly computable solutions. Given a general problem-solving architecture, at any given time, the novel algorithmic framework PowerPlay (Schmidhuber, 2011) searches the space of possible pairs of new tasks and modifications of the current problem solver, until it finds a more powerful problem solver that provably solves all previously learned tasks plus the new one, while the unmodified predecessor does not. Newly invented tasks may require to achieve a wow-effect by making previously learned skills more efficient such that they require less time and space. New skills may (partially) re-use previously learned skills. The greedy search of typical PowerPlay variants uses time-optimal program search to order candidate pairs of tasks and solver modifications by their conditional computational (time and space) complexity, given the stored experience so far. The new task and its corresponding task-solving skill are those first found and validated. This biases the search toward pairs that can be described compactly and validated quickly. The computational costs of validating new tasks need not grow with task repertoire size. Standard problem solver architectures of personal computers or neural networks tend to generalize by solving numerous tasks outside the self-invented training set; PowerPlay's ongoing search for novelty keeps breaking the generalization abilities of its present solver. This is related to Gödel's sequence of increasingly powerful formal theories based on adding formerly unprovable statements to the axioms without affecting previously provable theorems. The continually increasing repertoire of problem-solving procedures can be exploited by a parallel search for solutions to additional externally posed tasks. PowerPlay may be viewed as a greedy but practical implementation of basic principles of creativity (Schmidhuber, 2006a, 2010). A first experimental analysis can be found in separate papers (Srivastava et al., 2012a,b, 2013).

The increasing use of deep neural networks for safety-critical applications, such as autonomous driving and flight control, raises concerns about their safety and reliability. Formal verification can address these concerns by guaranteeing that a deep learning system operates as intended, but the state-of-the-art is limited to small systems. In this work-in-progress report we give an overview of our work on mitigating this difficulty, by pursuing two complementary directions: devising scalable verification techniques, and identifying design choices that result in deep learning systems that are more amenable to verification. ACM Reference Format: Lindsey Kuper, Guy Katz, Justin Gottschlich, Kyle Julian, Clark Barrett, and Mykel J. Kochenderfer. 2018. Toward Scalable Verification for SafetyCritical Deep Networks. In Proceedings of SysML Conference (SysML). ACM, New York, NY, USA, 3 pages.

Our goal is to deploy a high-accuracy system starting with zero training examples. We consider an on-the-job setting, where as inputs arrive, we use real-time crowdsourcing to resolve uncertainty where needed and output our prediction when confident. As the model improves over time, the reliance on crowdsourcing queries decreases. We cast our setting as a stochastic game based on Bayesian decision theory, which allows us to balance latency, cost, and accuracy objectives in a principled way. Computing the optimal policy is intractable, so we develop an approximation based on Monte Carlo Tree Search. We tested our approach on three datasets—named-entity recognition, sentiment classification, and image classification. On the NER task we obtained more than an order of magnitude reduction in cost compared to full human annotation, while boosting performance relative to the expert provided labels. We also achieve a 8% F1 improvement over having a single human label the whole set, and a 28% F1 improvement over online learning. “Poor is the pupil who does not surpass his master.” – Leonardo da Vinci

Many relevant tasks require an agent to reach a certain state, or to manipulate objects into a desired configuration. For example, we might want a robot to align and assemble a gear onto an axle or insert and turn a key in a lock. These tasks present considerable difficulties for reinforcement learning approaches, since the natural reward function for such goal-oriented tasks is sparse and prohibitive amounts of exploration are required to reach the goal and receive a learning signal. Past approaches tackle these problems by manually designing a task-specific reward shaping function to help guide the learning. Instead, we propose a method to learn these tasks without requiring any prior task knowledge other than obtaining a single state in which the task is achieved. The robot is trained in “reverse”, gradually learning to reach the goal from a set of starting positions increasingly far from the goal. Our method automatically generates a curriculum of starting positions that adapts to the agent’s performance, leading to efficient training on such tasks. We demonstrate our approach on difficult simulated fine-grained manipulation problems, not solvable by state-of-the-art reinforcement learning methods.

Recent neural network approaches to summarization are largely either sentence-extractive, choosing a set of sentences as the summary, or abstractive, generating the summary from a seq2seq model. In this work, we present a neural model for single-document summarization based on joint extraction and compression. Following recent successful extractive models, we frame the summarization problem as a series of local decisions. Our model chooses sentences from the document and then decides which of a set of compression options to apply to each selected sentence. We compute this set of options using discrete compression rules based on syntactic constituency parses; however, our approach is modular and can flexibly use any available source of compressions. For learning, we construct oracle extractive-compressive summaries that reflect uncertainty over our model’s decision sequence, then learn both of our components jointly with this supervision. Experimental results on the CNN/Daily Mail and New York Times datasets show that our model achieves the state-of-the-art performance on content selection evaluated by ROUGE. Moreover, human and manual evaluation show that our model’s output generally remains grammatical.

The GUINNESS is a tool flow for the deep neural network toward FPGA implementation [3,4,5] based on the GUI (Graphical User Interface) including both the binarized deep neural network training on GPUs and the inference on an FPGA. It generates the trained the Binarized deep neural network [2] on the desktop PC, then, it generates the bitstream by using standard the FPGA CAD tool flow. All the operation is done on the GUI, thus, the designer is not necessary to write any scripts to descript the neural network structure, training behaviour, only specify the values for hyper parameters. After finished the training, it automatically generates C++ codes to synthesis the bitstream using the Xilinx SDSoC system design tool flow. Thus, our tool flow is suitable for the software programmers who are not familiar with the FPGA design.

We propose SEARNN, a novel training algorithm for recurrent neural networks (RNNs) inspired by the “learning to search” (L2S) approach to structured prediction. RNNs have been widely successful in structured prediction applications such as machine translation or parsing, and are commonly trained using maximum likelihood estimation (MLE). Unfortunately, this training loss is not always an appropriate surrogate for the test error: by only maximizing the ground truth probability, it fails to exploit the wealth of information offered by structured losses. Further, it introduces discrepancies between training and predicting (such as exposure bias) that may hurt test performance. Instead, SEARNN leverages test-alike search space exploration to introduce global-local losses that are closer to the test error. We demonstrate improved performance over MLE on three different tasks: OCR, spelling correction and text chunking. Finally, we propose a subsampling strategy to enable SEARNN to scale to large vocabulary sizes.

We present the first user study on neural interactive translation prediction. Our neural translation aid was built into existing online translation software. Depending on the prefix already typed by the user the system suggests a list of continuations. We assessed the impact of the system on human translators in a user study with both professional and non-professional translators. The analysis was done with respect to translation speed and translation accuracy as assessed by human judges. We find that our neural translation aid enables relatively fast translations and does not compromise on quality.

An important challenge in neuroevolution is to evolve complex neural networks with multiple modes of behavior. Indirect encodings can potentially answer this challenge. Yet in practice, indirect encodings do not yield effective multimodal controllers. Thus, this paper introduces novel multimodal extensions to HyperNEAT, a popular indirect encoding. A previous multimodal HyperNEAT approach called situational policy geometry assumes that multiple brains benefit from being embedded within an explicit geometric space. However, experiments here illustrate that this assumption unnecessarily constrains evolution, resulting in lower performance. Specifically, this paper introduces HyperNEAT extensions for evolving many brains without assuming geometric relationships between them. The resulting MultiBrain HyperNEAT can exploit human-specified task divisions to decide when each brain controls the agent, or can automatically discover when brains should be used, by means of preference neurons. A further extension called module mutation allows evolution to discover the number of brains, enabling multimodal behavior with even less expert knowledge. Experiments in several multimodal domains highlight that multi-brain approaches are more effective than HyperNEAT without multimodal extensions, and show that brains without a geometric relation to each other outperform situational policy geometry. The conclusion is that Multi-Brain HyperNEAT provides several promising techniques for evolving complex multimodal behavior.


Question: Neural program synthesis guided by execution strategies.

Introducing the Neural Programmer-Interpreter (NPI): A novel recurrent and compositional neural network architecture designed to learn and execute programs autonomously. The NPI architecture comprises three learnable components: a task-agnostic recurrent core, a persistent key-value program memory, and specialized encoders tailored to diverse perceptual environments with unique affordances. This framework enables a single NPI to operate effectively across different environments by learning to compose lower-level programs into higher-level abstractions, thereby reducing sample complexity and enhancing generalization compared to conventional sequence-to-sequence LSTMs. The program memory facilitates efficient learning of new tasks by leveraging existing programs, while the NPI can utilize the environment, such as a scratch pad with read-write capabilities, to store intermediate computation results, alleviating the memory load on recurrent hidden units. Our training approach for the NPI involves using fully-supervised execution traces, where each program is accompanied by sequences of calls to immediate subprograms conditioned on the input. This method allows the NPI to learn from a limited number of rich examples rather than a large dataset of weaker labels. We showcase the versatility of our model by training it to execute various compositional programs, including arithmetic operations like addition, sorting algorithms, and the canonicalization of 3D models. Notably, a single NPI is capable of executing these programs along with their 21 associated subprograms, demonstrating its adaptability and learning capacity.

Humans learn complex latent structures from their environments (e.g., natural language, mathematics, music, social hierarchies). In cognitive science and cognitive neuroscience, models that infer higher-order structures from sensory or first-order representations have been proposed to account for the complexity and flexibility of human behavior. But how do the structures that these models invoke arise in neural systems in the first place? To answer this question, we explain how a system can learn latent representational structures (i.e., predicates) from experience with wholly unstructured data. During the process of predicate learning, an artificial neural network exploits the naturally occurring dynamic properties of distributed computing across neuronal assemblies in order to learn predicates, but also to combine them compositionally, two computational aspects which appear to be necessary for human behavior as per formal theories in multiple domains. We describe how predicates can be combined generatively using neural oscillations to achieve human-like extrapolation and compositionality in an artificial neural network. The ability to learn predicates from experience, to represent structures compositionally, and to extrapolate to unseen data offers an inroads to understanding and modeling the most complex human behaviors.

We describe SLING, a framework for parsing natural language into semantic frames. SLING supports general transition-based, neural-network parsing with bidirectional LSTM input encoding and a Transition Based Recurrent Unit (TBRU) for output decoding. The parsing model is trained end-to-end using only the text tokens as input. The transition system has been designed to output frame graphs directly without any intervening symbolic representation. The SLING framework includes an efficient and scalable frame store implementation as well as a neural network JIT compiler for fast inference during parsing. SLING is implemented in C++ and it is available for download on GitHub.

We present an approach for the verification of feed-forward neural networks in which all nodes have a piece-wise linear activation function. Such networks are often used in deep learning and have been shown to be hard to verify for modern satisfiability modulo theory (SMT) and integer linear programming (ILP) solvers. The starting point of our approach is the addition of a global linear approximation of the overall network behavior to the verification problem that helps with SMT-like reasoning over the network behavior. We present a specialized verification algorithm that employs this approximation in a search process in which it infers additional node phases for the non-linear nodes in the network from partial node phase assignments, similar to unit propagation in classical SAT solving. We also show how to infer additional conflict clauses and safe node fixtures from the results of the analysis steps performed during the search. The resulting approach is evaluated on collision avoidance and handwritten digit recognition case studies.

Deep learning methods have recently achieved great empirical success on machine translation, dialogue response generation, summarization, and other text generation tasks. At a high level, the technique has been to train end-to-end neural network models consisting of an encoder model to produce a hidden representation of the source text, followed by a decoder model to generate the target. While such models have significantly fewer pieces than earlier systems, significant tuning is still required to achieve good performance. For text generation models in particular, the decoder can behave in undesired ways, such as by generating truncated or repetitive outputs, outputting bland and generic responses, or in some cases producing ungrammatical gibberish. This paper is intended as a practical guide for resolving such undesired behavior in text generation models, with the aim of helping enable real-world applications.

We propose a method combining relational-logic representations with neural network learning. A general lifted architecture, possibly reflecting some background domain knowledge, is described through relational rules which may be handcrafted or learned. The relational rule-set serves as a template for unfolding possibly deep neural networks whose structures also reflect the structures of given training or testing relational examples. Different networks corresponding to different examples share their weights, which co-evolve during training by stochastic gradient descent algorithm. The framework allows for hierarchical relational modeling constructs and learning of latent relational concepts through shared hidden layers weights corresponding to the rules. Discovery of notable relational concepts and experiments on 78 relational learning benchmarks demonstrate favorable performance of the method.

This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time. The approach is demonstrated for text (where the data are discrete) and online handwriting (where the data are real-valued). It is then extended to handwriting synthesis by allowing the network to condition its predictions on a text sequence. The resulting system is able to generate highly realistic cursive handwriting in a wide variety of styles.

Dynamic test generation is a form of dynamic program analysis that attempts to compute test inputs to drive a program along a specific program path. Directed Automated Random Testing, or DART for short, blends dynamic test generation with model checking techniques with the goal of systematically executing all feasible program paths of a program while detecting various types of errors using run-time checking tools (like Purify, for instance). Unfortunately, systematically executing all feasible program paths does not scale to large, realistic programs.This paper addresses this major limitation and proposes to perform dynamic test generation compositionally, by adapting known techniques for interprocedural static analysis. Specifically, we introduce a new algorithm, dubbed SMART for Systematic Modular Automated Random Testing, that extends DART by testing functions in isolation, encoding test results as function summaries expressed using input preconditions and output postconditions, and then re-using those summaries when testing higher-level functions. We show that, for a fixed reasoning capability, our compositional approach to dynamic test generation (SMART) is both sound and complete compared to monolithic dynamic test generation (DART). In other words, SMART can perform dynamic test generation compositionally without any reduction in program path coverage. We also show that, given a bound on the maximum number of feasible paths in individual program functions, the number of program executions explored by SMART is linear in that bound, while the number of program executions explored by DART can be exponential in that bound. We present examples of C programs and preliminary experimental results that illustrate and validate empirically these properties.

Being able to model long-term dependencies in sequential data, such as text, has been among the long-standing challenges of recurrent neural networks (RNNs). This issue is strictly related to the absence of explicit planning in current RNN architectures, more explicitly, the network is trained to predict only the next token given previous ones. In this paper, we introduce a simple way of biasing the RNNs towards planning behavior. Particularly, we introduce an additional neural network which is trained to generate the sequence in reverse order, and we require closeness between the states of the forward RNN and backward RNN that predict the same token. At each step, the states of the forward RNN are required to match the future information contained in the backward states. We hypothesize that the approach eases modeling of long-term dependencies thus helping in generating more globally consistent samples. The model trained with conditional generation achieved 4% relative improvement (CER of 7.3 compared to a baseline of 7.6).

We present a memory augmented neural network for natural language understanding: Neural Semantic Encoders. NSE is equipped with a novel memory update rule and has a variable sized encoding memory that evolves over time and maintains the understanding of input sequences through read, compose and write operations. NSE can also access multiple and shared memories. In this paper, we demonstrated the effectiveness and the flexibility of NSE on five different natural language tasks: natural language inference, question answering, sentence classification, document sentiment analysis and machine translation where NSE achieved state-of-the-art performance when evaluated on publically available benchmarks. For example, our shared-memory model showed an encouraging result on neural machine translation, improving an attention-based baseline by approximately 1.0 BLEU.

We present a generative model to map natural language questions into SQL queries. Existing neural network based approaches typically generate a SQL query wordby-word, however, a large portion of the generated results is incorrect or not executable due to the mismatch between question words and table contents. Our approach addresses this problem by considering the structure of table and the syntax of SQL language. The quality of the generated SQL query is significantly improved through (1) learning to replicate content from column names, cells or SQL keywords; and (2) improving the generation of WHERE clause by leveraging the column-cell relation. Experiments are conducted on WikiSQL, a recently released dataset with the largest questionSQL pairs. Our approach significantly improves the state-of-the-art execution accuracy from 69.0% to 74.4%.


Question: Neural program synthesis guided by execution strategies.

This paper explores the enhancement of neural programming architectures through the integration of recursion, aiming to improve their generalization capabilities and interpretability. Traditionally, neural networks designed to learn programs from data have struggled with generalizability, particularly when confronted with inputs of higher complexity. To tackle this challenge, we propose the incorporation of recursion as a fundamental abstraction in neural architectures. As a case study, we apply this recursive enhancement to the Neural Programmer-Interpreter framework, testing its efficacy on a set of tasks that include grade-school addition, bubble sort, topological sort, and quicksort. Our results indicate a significant improvement in both generalizability and interpretability, even with limited training data. By breaking down the problem into more manageable sub-problems, recursion effectively narrows the scope of each neural network component, facilitating a more comprehensive understanding and analysis of the system's behavior. This approach paves the way for proving guarantees about the overall system, a crucial step towards robustly learning the semantics of programs. Our findings suggest that the inclusion of a concept like recursion is essential for neural architectures to effectively learn and generalize program semantics.

Ladder logic diagram (LLD) as the interfacing programming language of programmable logic controllers (PLCs) is utilized in modern discrete event control systems. However, LLD is hard to debug and maintain in practice. This is due to many factors such as non-structured nature of LLD, the LLD programmers' background, and the huge sizes of real world LLD. In this paper, we introduce a recurrent neural network (RNN) based technique for PLC program diagnosis. A manufacturing control system example has been presented to illustrate the applicability of the proposed algorithm. This method could be very advantageous in reducing the complexity in PLC control programs diagnosis because of the ease of use of the RNN compared to debugging the LLD code.

Procedural representations provide powerful means for generating complex geometric structures. They are also notoriously difficult to control. In this article, we present an algorithm for controlling grammar-based procedural models. Given a grammar and a high-level specification of the desired production, the algorithm computes a production from the grammar that conforms to the specification. This production is generated by optimizing over the space of possible productions from the grammar. The algorithm supports specifications of many forms, including geometric shapes and analytical objectives. We demonstrate the algorithm on procedural models of trees, cities, buildings, and Mondrian paintings.

Being able to model long-term dependencies in sequential data, such as text, has been among the long-standing challenges of recurrent neural networks (RNNs). This issue is strictly related to the absence of explicit planning in current RNN architectures, more explicitly, the network is trained to predict only the next token given previous ones. In this paper, we introduce a simple way of biasing the RNNs towards planning behavior. Particularly, we introduce an additional neural network which is trained to generate the sequence in reverse order, and we require closeness between the states of the forward RNN and backward RNN that predict the same token. At each step, the states of the forward RNN are required to match the future information contained in the backward states. We hypothesize that the approach eases modeling of long-term dependencies thus helping in generating more globally consistent samples. The model trained with conditional generation achieved 4% relative improvement (CER of 7.3 compared to a baseline of 7.6).

Recent research in machine learning has focused on supervised induction for simple classification and reinforcement learning for simple reactive behaviors. In the process, the field has become disconnected from AI’s original goal of creating complete intelligent agents. In this paper, I review recent work on machine learning for planning, language, vision, and other topics that runs counter to this trend and thus holds interest for the broader AI research community. I also suggest some steps to encourage further research along these lines.

In order for reinforcement learning systems to learn quickly in vast action spaces such as the space of all possible pieces of text or the space of all images, leveraging human intuition and creativity is key. However, a human-designed action space is likely to be initially imperfect and limited; furthermore, humans may improve at creating useful actions with practice or new information. Therefore, we propose a framework in which a human adds actions to a reinforcement learning system over time to boost performance. In this setting, however, it is key that we use human effort as efficiently as possible, and one significant danger is that humans waste effort adding actions at places (states) that aren’t very important. Therefore, we propose Expected Local Improvement (ELI), an automated method which selects states at which to query humans for a new action. We evaluate ELI on a variety of simulated domains adapted from the literature, including domains with over a million actions and domains where the simulated experts change over time. We find ELI demonstrates excellent empirical performance, even in settings where the synthetic “experts” are quite poor.

We present the first user study on neural interactive translation prediction. Our neural translation aid was built into existing online translation software. Depending on the prefix already typed by the user the system suggests a list of continuations. We assessed the impact of the system on human translators in a user study with both professional and non-professional translators. The analysis was done with respect to translation speed and translation accuracy as assessed by human judges. We find that our neural translation aid enables relatively fast translations and does not compromise on quality.

A critical component to enabling intelligent reasoning in partially observable environments is memory. Despite this importance, Deep Reinforcement Learning (DRL) agents have so far used relatively simple memory architectures, with the main methods to overcome partial observability being either a temporal convolution over the past k frames or an LSTM layer. More recent work (Oh et al., 2016) has went beyond these architectures by using memory networks which can allow more sophisticated addressing schemes over the past k frames. But even these architectures are unsatisfactory due to the reason that they are limited to only remembering information from the last k frames. In this paper, we develop a memory system with an adaptable write operator that is customized to the sorts of 3D environments that DRL agents typically interact with. This architecture, called the Neural Map, uses a spatially structured 2D memory image to learn to store arbitrary information about the environment over long time lags. We demonstrate empirically that the Neural Map surpasses previous DRL memories on a set of challenging 2D and 3D maze environments and show that it is capable of generalizing to environments that were not seen during training.

We describe DyNet, a toolkit for implementing neural network models based on dynamic declaration of network structure. In the static declaration strategy that is used in toolkits like Theano, CNTK, and TensorFlow, the user first defines a computation graph (a symbolic representation of the computation), and then examples are fed into an engine that executes this computation and computes its derivatives. In DyNet’s dynamic declaration strategy, computation graph construction is mostly transparent, being implicitly constructed by executing procedural code that computes the network outputs, and the user is free to use different network structures for each input. Dynamic declaration thus facilitates the implementation of more complicated network architectures, and DyNet is specifically designed to allow users to implement their models in a way that is idiomatic in their preferred programming language (C++ or Python). One challenge with dynamic declaration is that because the symbolic computation graph is defined anew for every training example, its construction must have low overhead. To achieve this, DyNet has an optimized C++ backend and lightweight graph representation. Experiments show that DyNet’s speeds are faster than or comparable with static declaration toolkits, and significantly faster than Chainer, another dynamic declaration toolkit. DyNet is released opensource under the Apache 2.0 license, and available at http://github.com/clab/dynet. Carnegie Mellon University, Pittsburgh, PA, USA Nara Institute of Science and Technology, Ikoma, Japan DeepMind, London, UK Bar Ilan University, Ramat Gan, Israel Allen Institute for Artificial Intelligence, Seattle, WA, USA University of Notre Dame, Notre Dame, IN, USA IBM T.J. Watson Research Center, Yorktown Heights, NY, USA University of Melbourne, Melbourne, Australia Johns Hopkins University, Baltimore, MD, USA Google, New York, NY, USA Google, Mountain View, CA, USA University of Washington, Seattle, USA Microsoft Research, Redmond, WA, USA University of Edinburgh, Edinburgh, UK 1 ar X iv :1 70 1. 03 98 0v 1 [ st at .M L ] 1 5 Ja n 20 17

We present a memory augmented neural network for natural language understanding: Neural Semantic Encoders. NSE is equipped with a novel memory update rule and has a variable sized encoding memory that evolves over time and maintains the understanding of input sequences through read, compose and write operations. NSE can also access multiple and shared memories. In this paper, we demonstrated the effectiveness and the flexibility of NSE on five different natural language tasks: natural language inference, question answering, sentence classification, document sentiment analysis and machine translation where NSE achieved state-of-the-art performance when evaluated on publically available benchmarks. For example, our shared-memory model showed an encouraging result on neural machine translation, improving an attention-based baseline by approximately 1.0 BLEU.

Advances in deep generative networks have led to impressive results in recent years. Nevertheless, such models can often waste their capacity on the minutiae of datasets, presumably due to weak inductive biases in their decoders. This is where graphics engines may come in handy since they abstract away low-level details and represent images as high-level programs. Current methods that combine deep learning and renderers are limited by hand-crafted likelihood or distance functions, a need for large amounts of supervision, or difficulties in scaling their inference algorithms to richer datasets. To mitigate these issues, we present SPIRAL, an adversarially trained agent that generates a program which is executed by a graphics engine to interpret and sample images. The goal of this agent is to fool a discriminator network that distinguishes between real and rendered data, trained with a distributed reinforcement learning setup without any supervision. A surprising finding is that using the discriminator’s output as a reward signal is the key to allow the agent to make meaningful progress at matching the desired output rendering. To the best of our knowledge, this is the first demonstration of an end-to-end, unsupervised and adversarial inverse graphics agent on challenging real world (MNIST, OMNIGLOT, CELEBA) and synthetic 3D datasets. A video of the agent can be found at https://youtu.be/iSyvwAwa7vk.


Question: A neural program synthesis method using execution guidance.

This paper introduces an enhanced neural network architecture, dubbed Neural Turing Machines (NTMs), which integrates external memory resources to augment computational capabilities. Unlike traditional neural networks, NTMs interact with these memory components through attention mechanisms, enabling a more sophisticated handling of data. Drawing parallels with classical computing models like Turing Machines and Von Neumann architectures, NTMs distinguish themselves by being fully differentiable and trainable via gradient descent, thus facilitating efficient learning. Early experiments reveal that NTMs are capable of learning and executing basic algorithms, including copying, sorting, and associative recall, solely from input-output examples, showcasing their potential for algorithmic inference and problem-solving.

**Question**: A Direct Search Method to solve Economic Dispatch Problem with Valve-Point Effect

**Documents**: A Novel Hybrid Genetic Algorithm-Particle Swarm Optimization Approach for Recurrent Network Design

**Similar Documents**: This paper presents a pioneering method, combining Genetic Algorithm (GA) and Particle Swarm Optimization (PSO), termed Hybrid GA-PSO, tailored for the design of recurrent neural networks. The proposed approach integrates the strengths of both GA and PSO to create an enhanced evolutionary learning algorithm. Within Hybrid GA-PSO, new generations are formed through a dual process: traditional GA operations of crossover and mutation, and the influence of PSO. The elite strategy is a core component, where the best-performing individuals, known as elites, undergo a maturation process akin to natural selection, facilitated by PSO. These refined elites and their offspring form the subsequent generation. The Hybrid GA-PSO method is applied to the design of recurrent neural and fuzzy networks, demonstrating its efficacy in solving temporal sequence production and dynamic plant control problems, respectively. Comparative studies with standalone GA and PSO highlight the superior performance of Hybrid GA-PSO in recurrent network design tasks.

Search engines play an important role in our everyday lives by assisting us in finding the information we need. When we input a complex query, however, results are often far from satisfactory. In this work, we introduce a query reformulation system based on a neural network that rewrites a query to maximize the number of relevant documents returned. We train this neural network with reinforcement learning. The actions correspond to selecting terms to build a reformulated query, and the reward is the document recall. We evaluate our approach on three datasets against strong baselines and show a relative improvement of 5-20% in terms of recall. Furthermore, we present a simple method to estimate a conservative upperbound performance of a model in a particular environment and verify that there is still large room for improvements.

Andor, D., Alberti, C., Weiss, D., Severyn, A., Presta, A., Ganchev, K., Petrov, S., and Collins, M. (2016). Globally normalized transition-based neural networks. In Association for Computational Linguistics. Ballesteros, M., Goldberg, Y., Dyer, C., and Smith, N. A. (2016). Training with exploration improves a greedy stack-LSTM parser. Proceedings of the conference on empirical methods in natural language processing. Chen, D. and Manning, C. D. (2014). A fast and accurate dependency parser using neural networks. In Proceedings of the conference on empirical methods in natural language processing, pages 740–750. Cheng, H., Fang, H., He, X., Gao, J., and Deng, L. (2016). Bi-directional attention with agreement for dependency parsing. arXiv preprint arXiv:1608.02076. Hashimoto, K., Xiong, C., Tsuruoka, Y., and Socher, R. (2016). A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01587. Kiperwasser, E. and Goldberg, Y. (2016). Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4:313–327. Kuncoro, A., Ballesteros, M., Kong, L., Dyer, C., Neubig, G., and Smith, N. A. (2016). What do recurrent neural network grammars learn about syntax? CoRR, abs/1611.05774. McDonald, R. T. and Pereira, F. C. (2006). Online learning of approximate dependency parsing algorithms. In EACL. Nivre, J., Hall, J., and Nilsson, J. (2006). Maltparser: A data-driven parser-generator for dependency parsing. In Proceedings of LREC, volume 6, pages 2216–2219.

We present the first class of mathematically rigorous, general, fully self-referential, self-improving, optimally efficient problem solvers. Inspired by Kurt Gödel’s celebrated self-referential formulas (1931), such a problem solver rewrites any part of its own code as soon as it has found a proof that the rewrite is useful, where the problem-dependent utility function and the hardware and the entire initial code are described by axioms encoded in an initial proof searcher which is also part of the initial code. The searcher systematically and efficiently tests computable proof techniques (programs whose outputs are proofs) until it finds a provably useful, computable self-rewrite. We show that such a self-rewrite is globally optimal—no local maxima!—since the code first had to prove that it is not useful to continue the proof search for alternative self-rewrites. Unlike previous non-self-referential methods based on hardwired proof searchers, ours not only boasts an optimal order of complexity but can optimally reduce any slowdowns hidden by the O()-notation, provided the utility of such speed-ups is provable at all.

Abstmct-Designing robots that learn by themselves to perform complex real-world tasks is a still-open challenge for the field of Robotics and Artificial Intelligence. In this paper we present the robot learning problem as a lifelong problem, in which a robot faces a collection of tasks over its entire lifetime. Such a scenario provides the opportunity to gather general-purpose knowledge that transfers across tasks. We illustrate a particular learning mechanism, explanation-based neural network learning, that transfers knowledge between related tasks via neural network action models. The learning approach is illustrated using a mobile robot, equipped with visual, ultrasonic and laser sensors. In less than 10 minutes operation time, the robot is able to learn to navigate to a marked target object in a natural office environment.

People learning new concepts can often generalize successfully from just a single example, yet machine learning algorithms typically require tens or hundreds of examples to perform with similar accuracy. People can also use learned concepts in richer ways than conventional algorithms—for action, imagination, and explanation. We present a computational model that captures these human learning abilities for a large class of simple visual concepts: handwritten characters from the world’s alphabets. The model represents concepts as simple programs that best explain observed examples under a Bayesian criterion. On a challenging one-shot classification task, the model achieves human-level performance while outperforming recent deep learning approaches. We also present several “visual Turing tests” probing the model’s creative generalization abilities, which in many cases are indistinguishable from human behavior.

Deep neural networks (DNNs) applications are now increasingly pervasive and powerful. However, fashion designers are lagging behind in leveraging this increasingly common technology. DNNs are not yet a standard part of fashion de sign practice, either clothes patterns or prototyping tools. In this paper, we present DeepWear, a method using deep convolutional generative adversarial networks for clothes design. The DNNs learn the feature of specific brand clothes and generate images then patterns instructed from the images are made, and an author creates clothes based on that. We evaluated this system by evaluating the credibility of the actual sold clothes on market with our clothes. As the result, we found it is possible to make clothes look like actual products from the generated images. Our findings have implications for collaborative design between machine and human intelligence.

The GUINNESS is a tool flow for the deep neural network toward FPGA implementation [3,4,5] based on the GUI (Graphical User Interface) including both the binarized deep neural network training on GPUs and the inference on an FPGA. It generates the trained the Binarized deep neural network [2] on the desktop PC, then, it generates the bitstream by using standard the FPGA CAD tool flow. All the operation is done on the GUI, thus, the designer is not necessary to write any scripts to descript the neural network structure, training behaviour, only specify the values for hyper parameters. After finished the training, it automatically generates C++ codes to synthesis the bitstream using the Xilinx SDSoC system design tool flow. Thus, our tool flow is suitable for the software programmers who are not familiar with the FPGA design.

We present a memory augmented neural network for natural language understanding: Neural Semantic Encoders. NSE is equipped with a novel memory update rule and has a variable sized encoding memory that evolves over time and maintains the understanding of input sequences through read, compose and write operations. NSE can also access multiple and shared memories. In this paper, we demonstrated the effectiveness and the flexibility of NSE on five different natural language tasks: natural language inference, question answering, sentence classification, document sentiment analysis and machine translation where NSE achieved state-of-the-art performance when evaluated on publically available benchmarks. For example, our shared-memory model showed an encouraging result on neural machine translation, improving an attention-based baseline by approximately 1.0 BLEU.

The authors show that a multiple-input, single-output, single-hidden-layer feedforward network with (known) hardwired connections from input to hidden layer, monotone squashing at the hidden layer and no squashing at the output embeds as a special case a so-called Fourier network, which yields a Fourier series approximation properties of Fourier series representations. In particular, approximation to any desired accuracy of any square integrable function can be achieved by such a network, using sufficiently many hidden units. In this sense, such networks do not make avoidable mistakes.<<ETX>>

This paper presents HeNet, a hierarchical ensemble neural network, applied to classify hardware-generated control flow traces for malware detection. Deep learning-based malware detection has so far focused on analyzing executable files and runtime API calls. Static code analysis approaches face challenges due to obfuscated code and adversarial perturbations. Behavioral data collected during execution is more difficult to obfuscate but recent research has shown successful attacks against API call based malware classifiers. We investigate control flow based characterization of a program execution to build robust deep learning malware classifiers. HeNet consists of a low-level behavior model and a toplevel ensemble model. The low-level model is a per-application behavior model, trained via transfer learning on a time-series of images generated from control flow trace of an execution. We use Intel R © Processor Trace enabled processor for low overhead execution tracing and design a lightweight image conversion and segmentation of the control flow trace. The top-level ensemble model aggregates the behavior classification of all the trace segments and detects an attack. The use of hardware trace adds portability to our system and the use of deep learning eliminates the manual effort of feature engineering. We evaluate HeNet against real-world exploitations of PDF readers. HeNet achieves 100% accuracy and 0% false positive on test set, and higher classification accuracy compared to classical machine learning algorithms.


Question: A neural program synthesis method using execution guidance.

This paper introduces SPIRAL, a novel approach that combines adversarial learning with reinforcement to synthesize programs capable of generating images. Despite recent advancements in deep generative networks, these models often struggle with overfitting to dataset minutiae due to a lack of strong inductive biases in their decoders. Graphics engines, which represent images as high-level programs and abstract low-level details, offer a potential solution to this issue. However, existing methods that integrate deep learning with rendering engines are constrained by manual likelihood or distance functions, a reliance on extensive supervision, or scalability challenges in their inference algorithms. To address these limitations, SPIRAL employs an adversarially trained agent to produce a program that a graphics engine executes to interpret and sample images. The agent's objective is to deceive a discriminator network, which can distinguish between actual and rendered data, through a distributed reinforcement learning framework without requiring supervision. A noteworthy observation is that utilizing the discriminator's output as a reward signal is essential for the agent to make substantial progress in aligning the output rendering with the desired outcome. As far as we know, this is the first instance of an end-to-end, unsupervised, and adversarial inverse graphics agent demonstrated on complex real-world (MNIST, OMNIGLOT, CELEBA) and synthetic 3D datasets. A demonstration video of the agent's performance can be viewed at https://youtu.be/iSyvwAwa7vk.

• Neural network language model and distributed representation for words (Vector representation) • Capture syntactic and remantic regularities in language • Outperform state-of-the-art

Deep neural networks have emerged as a widely used and effective means for tackling complex, real-world problems. However, a major obstacle in applying them to safety-critical systems is the great difficulty in providing formal guarantees about their behavior. We present a novel, scalable, and efficient technique for verifying properties of deep neural networks (or providing counter-examples). The technique is based on the simplex method, extended to handle the non-convex Rectified Linear Unit (ReLU ) activation function, which is a crucial ingredient in many modern neural networks. The verification procedure tackles neural networks as a whole, without making any simplifying assumptions. We evaluated our technique on a prototype deep neural network implementation of the next-generation airborne collision avoidance system for unmanned aircraft (ACAS Xu). Results show that our technique can successfully prove properties of networks that are an order of magnitude larger than the largest networks verified using existing methods.

We present the first class of mathematically rigorous, general, fully self-referential, self-improving, optimally efficient problem solvers. Inspired by Kurt Gödel’s celebrated self-referential formulas (1931), such a problem solver rewrites any part of its own code as soon as it has found a proof that the rewrite is useful, where the problem-dependent utility function and the hardware and the entire initial code are described by axioms encoded in an initial proof searcher which is also part of the initial code. The searcher systematically and efficiently tests computable proof techniques (programs whose outputs are proofs) until it finds a provably useful, computable self-rewrite. We show that such a self-rewrite is globally optimal—no local maxima!—since the code first had to prove that it is not useful to continue the proof search for alternative self-rewrites. Unlike previous non-self-referential methods based on hardwired proof searchers, ours not only boasts an optimal order of complexity but can optimally reduce any slowdowns hidden by the O()-notation, provided the utility of such speed-ups is provable at all.

Pex automatically produces a small test suite with high code coverage for a .NET program. To this end, Pex performs a systematic program analysis (using dynamic symbolic execution, similar to pathbounded model-checking) to determine test inputs for Parameterized Unit Tests. Pex learns the program behavior by monitoring execution traces. Pex uses a constraint solver to produce new test inputs which exercise different program behavior. The result is an automatically generated small test suite which often achieves high code coverage. In one case study, we applied Pex to a core component of the .NET runtime which had already been extensively tested over several years. Pex found errors, including a serious issue.

Summary form only given. A novel method called dynamic node creation (DNC) that attacks issues of training large networks and of testing networks with different numbers of hidden layer units is presented. DNC sequentially adds nodes one at a time to the hidden layer(s) of the network until the desired approximation accuracy is achieved. Simulation results for parity, symmetry, binary addition, and the encoder problem are presented. The procedure was capable of finding known minimal topologies in many cases, and was always within three nodes of the minimum. Computational expense for finding the solutions was comparable to training normal backpropagation (BP) networks with the same final topologies. Starting out with fewer nodes than needed to solve the problem actually seems to help find a solution. The method yielded a solution for every problem tried. BP applied to the same large networks with randomized initial weights was unable, after repeated attempts, to replicate some minimum solutions found by DNC.<<ETX>>

What are the computational tasks that an executive controller for visual attention must solve? This question is posed in the context of the Selective Tuning model of attention. The range of required computations go beyond top-down bias signals or region-of-interest determinations, and must deal with overt and covert fixations, process timing and synchronization, information routing, memory, matching control to task, spatial localization, priming, and coordination of bottom-up with top-down information. During task execution, results must be monitored to ensure the expected results. This description includes the kinds of elements that are common in the control of any kind of complex machine or system. We seek a mechanistic integration of the above, in other words, algorithms that accomplish control. Such algorithms operate on representations, transforming a representation of one kind into another, which then forms the input to yet another algorithm. Cognitive Programs (CPs) are hypothesized to capture exactly such representational transformations via stepwise sequences of operations. CPs, an updated and modernized offspring of Ullman's Visual Routines, impose an algorithmic structure to the set of attentional functions and play a role in the overall shaping of attentional modulation of the visual system so that it provides its best performance. This requires that we consider the visual system as a dynamic, yet general-purpose processor tuned to the task and input of the moment. This differs dramatically from the almost universal cognitive and computational views, which regard vision as a passively observing module to which simple questions about percepts can be posed, regardless of task. Differing from Visual Routines, CPs explicitly involve the critical elements of Visual Task Executive (vTE), Visual Attention Executive (vAE), and Visual Working Memory (vWM). Cognitive Programs provide the software that directs the actions of the Selective Tuning model of visual attention.

Compilers and computer-aided design tools are essential for fine-grained control of nanoscale quantum-mechanical systems. A proposed four-phase design flow assists with computations by transforming a quantum algorithm from a high-level language program into precisely scheduled physical actions.

Prediction without justification has limited applicability. As a remedy, we learn to extract pieces of input text as justifications – rationales – that are tailored to be short and coherent, yet sufficient for making the same prediction. Our approach combines two modular components, generator and encoder, which are trained to operate well together. The generator specifies a distribution over text fragments as candidate rationales and these are passed through the encoder for prediction. Rationales are never given during training. Instead, the model is regularized by desiderata for rationales. We evaluate the approach on multi-aspect sentiment analysis against manually annotated test cases. Our approach outperforms attention-based baseline by a significant margin. We also successfully illustrate the method on the question retrieval task.1

We present a technique for automatically generating features for data-driven program analyses. Recently data-driven approaches for building a program analysis have been developed, which mine existing codebases and automatically learn heuristics for finding a cost-effective abstraction for a given analysis task. Such approaches reduce the burden of the analysis designers, but they do not remove it completely; they still leave the nontrivial task of designing so called features to the hands of the designers. Our technique aims at automating this feature design process. The idea is to use programs as features after reducing and abstracting them. Our technique goes through selected program-query pairs in codebases, and it reduces and abstracts the program in each pair to a few lines of code, while ensuring that the analysis behaves similarly for the original and the new programs with respect to the query. Each reduced program serves as a boolean feature for program-query pairs. This feature evaluates to true for a given program-query pair when (as a program) it is included in the program part of the pair. We have implemented our approach for three real-world static analyses. The experimental results show that these analyses with automatically-generated features are cost-effective and consistently perform well on a wide range of programs.

Knowledge bases are employed in a variety of applications from natural language processing to semantic web search; alas, in practice their usefulness is hurt by their incompleteness. Embedding models attain state-of-the-art accuracy in knowledge base completion, but their predictions are notoriously hard to interpret. In this paper, we adapt “pedagogical approaches” (from the literature on neural networks) so as to interpret embedding models by extracting weighted Horn rules from them. We show how pedagogical approaches have to be adapted to take upon the large-scale relational aspects of knowledge bases and show experimentally their strengths and weaknesses.


Question: A system for identifying plants utilizing their leaf characteristics.

This paper explores the utilization of the Gray Level Co-occurrence Matrix (GLCM) technique for the purpose of image texture feature extraction, particularly focusing on its application in motion estimation within images. The GLCM method is renowned for its ability to capture second-order statistical texture features, which are essential for various image analysis tasks. In this study, four key features—Angular Second Moment, Correlation, Inverse Difference Moment, and Entropy—are derived from images using a Xilinx Field-Programmable Gate Array (FPGA). The findings indicate that these texture features exhibit a high level of discrimination accuracy, consume minimal computational resources, and are thus well-suited for real-time pattern recognition tasks, including the efficient identification and categorization of plant species based on their leaf characteristics. This approach not only streamlines the process of feature extraction but also enhances the precision of pattern recognition systems, making it a promising method for plant identification through leaf analysis.

This paper describes an approach for recognizing instances of a 3D object in a single camera image and for determining their 3D poses. A hierarchical model is generated solely based on the geometry information of a 3D CAD model of the object. The approach does not rely on texture or reflectance information of the object's surface, making it useful for a wide range of industrial and robotic applications, e.g., bin-picking. A hierarchical view-based approach that addresses typical problems of previous methods is applied: It handles true perspective, is robust to noise, occlusions, and clutter to an extent that is sufficient for many practical applications, and is invariant to contrast changes. For the generation of this hierarchical model, a new model image generation technique by which scale-space effects can be taken into account is presented. The necessary object views are derived using a similarity-based aspect graph. The high robustness of an exhaustive search is combined with an efficient hierarchical search. The 3D pose is refined by using a least-squares adjustment that minimizes geometric distances in the image, yielding a position accuracy of up to 0.12 percent with respect to the object distance, and an orientation accuracy of up to 0.35 degree in our tests. The recognition time is largely independent of the complexity of the object, but depends mainly on the range of poses within which the object may appear in front of the camera. For efficiency reasons, the approach allows the restriction of the pose range depending on the application. Typical runtimes are in the range of a few hundred ms.

The paper presents an automated system for classification of fruits. A dataset containing five different fruits was constructed using an ordinary camera. All the fruits were analyzed on the basis of their color (RGB space), shape and texture and then classified using different classifiers to find the classifier that gives the best accuracy. Gray Level Cooccurance Matrix(GLCM) is used to calculate texture features .Best accuracy was achieved by support vector machine (svm). All the processing was carried out in Matlab. Keywords—computer vision, pattern recognition, support vector machine, texture features.

This study was carried out with the purpose of testing the ability of deep learning machine vision to identify microscopic objects and geometries found in chemical crystal structures. A database of 6994 images taken with a light microscope showing microscopic crystal details of selected chemical compounds along with 180 images of an unknown chemical was created to train and test, respectively the deep learning models. The models used were GoogLeNet (22 layers deep network) and VGG-16 (16 layers deep network), based on the Caffe framework (University of California, Berkeley, CA) of the DIGITS platform (NVIDIA Corporation, Santa Clara, CA). The two models were successfully trained with the images, having validation accuracy values of 97.38% and 99.65% respectively. Finally, both models were able to correctly identify the unknown chemical sample with a high probability score of 93.34% (GoogLeNet) and 99.41% (VGG-16). The positive results found in this study can be further applied to other unknown sample identification tasks using light microscopy coupled with deep learning machine vision.

This paper presents a comprehensive test of the principal tasks in document image analysis (DIA), starting with binarization, text line segmentation, and isolated character/glyph recognition, and continuing on to word recognition and transliteration for a new and challenging collection of palm leaf manuscripts from Southeast Asia. This research presents and is performed on a complete dataset collection of Southeast Asian palm leaf manuscripts. It contains three different scripts: Khmer script from Cambodia, and Balinese script and Sundanese script from Indonesia. The binarization task is evaluated on many methods up to the latest in some binarization competitions. The seam carving method is evaluated for the text line segmentation task, compared to a recently new text line segmentation method for palm leaf manuscripts. For the isolated character/glyph recognition task, the evaluation is reported from the handcrafted feature extraction method, the neural network with unsupervised learning feature, and the Convolutional Neural Network (CNN) based method. Finally, the Recurrent Neural Network-Long Short-Term Memory (RNN-LSTM) based method is used to analyze the word recognition and transliteration task for the palm leaf manuscripts. The results from all experiments provide the latest findings and a quantitative benchmark for palm leaf manuscripts analysis for researchers in the DIA community.

Correct identification of prescription pills based on their visual appearance is a key step required to assure patient safety and facilitate more effective patient care. With the availability of high-quality cameras and computational power on smartphones, it is possible and helpful to identify unknown prescription pills using smartphones. Towards this goal, in 2016, the U.S. National Library of Medicine (NLM) of the National Institutes of Health (NIH) announced a nationwide competition, calling for the creation of a mobile vision system that can recognize pills automatically from a mobile phone picture under unconstrained real-world settings. In this paper, we present the design and evaluation of such mobile pill image recognition system called MobileDeepPill. The development of MobileDeepPill involves three key innovations: a triplet loss function which attains invariances to real-world noisiness that deteriorates the quality of pill images taken by mobile phones; a multi-CNNs model that collectively captures the shape, color and imprints characteristics of the pills; and a Knowledge Distillation-based deep model compression framework that significantly reduces the size of the multi-CNNs model without deteriorating its recognition performance. Our deep learning-based pill image recognition algorithm wins the First Prize (champion) of the NIH NLM Pill Image Recognition Challenge. Given its promising performance, we believe MobileDeepPill helps NIH tackle a critical problem with significant societal impact and will benefit millions of healthcare personnel and the general public.

Figures are very important non-textual information contained in scientific documents. Current digital libraries do not provide users tools to retrieve documents based on the information available within the figures. We propose an architecture for retrieving documents by integrating figures and other information. The initial step in enabling integrated document search is to categorize figures into a set of pre-defined types. We propose several categories of figures based on their functionalities in scholarly articles. We have developed a machine-learning-based approach for automatic categorization of figures. Both global features, such as texture, and part features, such as lines, are utilized in the architecture for discriminating among figure categories. The proposed approach has been evaluated on a testbed document set collected from the CiteSeer scientific literature digital library. Experimental evaluation has demonstrated that our algorithms can produce acceptable results for realworld use. Our tools will be integrated into a scientific document digital library.

Pattern recognition is a tough problem for computers, although humans are wired for it. Pattern recognition is becoming increasingly important in the age of automation and information handling and retrieval. This paper reviews possible application areas of Pattern recognition. Author covers various sub-disciplines of pattern recognition based on learning methods, such as supervised, unsupervised, semi-supervised learning and key research areas such as grammar induction. Novel solutions to these possible problems could be well deployed for character recognition, speech analysis, man and machine diagnostics, person identification, industrial inspection and so on. The paper concludes with brief discussion on open issues that need to be addressed by future researchers.

Recently, a new ICT approach to agriculture called “Smart Agriculture” has been received great attention to support farmers' decision-making for good final yield on various kinds of field conditions. For this purpose, this paper presents two image sensing methods that enable an automatic observation to capture flowers and seedpods of soybeans in real fields. The developed image sensing methods are considered as sensors in an agricultural cyber-physical system in which big data on the growth status of agricultural plants and environmental information (e.g., weather, temperature, humidity, solar radiation, soil condition, etc.) are analyzed to mine useful rules for appropriate cultivation. The proposed image sensing methods are constructed by combining several image processing and machine learning techniques. The flower detection is realized based on a coarse-to-fine approach where candidate areas of flowers are first detected by SLIC and hue information, and the acceptance of flowers is decided by CNN. In the seedpod detection, candidates of seedpod regions are first detected by the Viola-Jones object detection method, and we also use CNN to make a final decision on the acceptance of detected seedpods. The performance of the proposed image sensing methods is evaluated for a data set of soybean images that were taken from a crowd of soybeans in real agricultural fields in Hokkaido, Japan.

Food image recognition is a key enabler for many smart home applications such as smart kitchen and smart personal nutrition log. In order to improve living experience and life quality, smart home systems collect valuable insights of users’ preferences, nutrition intake and health conditions via accurate and robust food image recognition. In addition, efficiency is also a major concern since many smart home applications are deployed on mobile devices where high-end GPUs are not available. In this paper, we investigate compact and efficient food image recognition methods, namely low-level and mid-level approaches. Considering the real application scenario where only limited and noisy data are available, we first proposed a superpixel based Linear Distance Coding (LDC) framework where distinctive low-level food image features are extracted to improve performance. On a challenging small food image dataset where only 12 training images are available per category, our framework has shown superior performance in both accuracy and robustness. In addition, to better model deformable food part distribution, we extend LDC’s feature-to-class distance idea and propose a mid-level superpixel food parts-to-class distance mining framework. The proposed framework show superior performance on a benchmark food image datasets compared to other low-level and mid-level approaches in the literature.

A new concept for the detection of small objects from modular optoelectronic multispectral scanner (MOMS-02) high spatial resolution panchromatic satellite imagery is presented. We combine supervised shape classification with unsupervised image segmentation in an iterative procedure which allows a target-oriented search for specific object shapes.


Question: A plant recognition system utilizing leaf characteristics for identification.

An Automated Approach for Plant Species Identification through Leaf Image Analysis and Neural Network Classifiers

This research paper outlines an innovative system designed for the automatic identification of plant species by analyzing leaf images. The study focuses on three distinct plant types, employing two contrasting shape modeling techniques for leaf feature extraction. The first technique, based on the Moments-Invariant (M-I) model, considers the first four normalized central moments, exploring their efficacy both independently and in combined 2-D and 3-D feature spaces. The second technique utilizes the CentroidRadii (C-R) model, which involves the detection of leaf boundaries through an edge detection algorithm. Following this, 36 radii at 10-degree angular intervals are used to construct the feature vector. To enhance the system's accuracy, a hybrid set of features, integrating both the M-I and C-R models, is created and assessed to determine if this amalgamation can lead to superior performance. Neural network classifiers are employed to differentiate between the various plant species based on the generated feature vectors. The dataset comprises 180 leaf images, equally distributed among the three plant categories. The system achieves recognition accuracies within the impressive range of 90% to 100%, results that are on par with the highest figures documented in the existing scientific literature. Keywords: plant identification; moment invariants; centroid-radii model; neural network; computer vision.

Soft computing in the field of agriculture science is being employed with computer vision techniques in order to detect the diseases in crops to increase the overall yield. A Modified Rotation Kernel Transformation(MRKT) based directional feature extraction scheme is presents to resolve the issues occurring due to shape, color or other deceptive features during plant disease recognition. The MRKT based scheme is used to calculate the directional features and histograms for plant parts like leaf and fruit of Mango crop. These histograms and the directional feature set with use of artificial neural network lead to a better recognition technique of Anthracnose disease which occurs in form of black spots on fruits and leaves of mango plant. The results obtained using proposed MRKT directional feature set have shows the better results with accuracy up to 98%.

The objective of the current study was to analyze the seasonal effect on differentiating tree species in an urban environment using multi-temporal hyperspectral data, Light Detection And Ranging (LiDAR) data, and a tree species database collected from the field. Two Airborne Imaging Spectrometer for Applications (AISA) hyperspectral images were collected, covering the Summer and Fall seasons. In order to make both datasets spatially and spectrally compatible, several preprocessing steps, including band reduction and a spatial degradation, were performed. An object-oriented classification was performed on both images using training data collected randomly from the tree species database. The seven dominant tree species (Gleditsia triacanthos, Acer saccharum, Tilia Americana, Quercus palustris, Pinus strobus and Picea glauca) were used in the classification. The results from this analysis did not show any major difference in overall accuracy between the two seasons. Overall accuracy was approximately 57% for the Summer dataset and 56% for the Fall dataset. However, the Fall dataset provided more consistent results for all tree species while the Summer dataset had a few higher individual class accuracies. Further, adding LiDAR into the classification improved the results by 19% for both fall and summer. This is mainly due to the removal of shadow effect and the addition of elevation data to separate low and high vegetation.

Over the last 20 years, several different techniques have been proposed for computer recognition of human faces. The purpose of this paper is to compare two simple but general strategies on a common database (frontal images of faces of 47 people: 26 males and 21 females, four images per person). We have developed and implemented two new algorithms; the first one is based on the computation of a set of geometrical features, such as nose width and length, mouth position, and chin shape, and the second one is based on almost-grey-level template matching. The results obtained on the testing sets (about 90% correct recognition using geometrical features and perfect recognition using template matching) favor our implementation of the template-matching approach.

In this research, an algorithm for disease spot segmentation using image processing techniques in plant leaf is implemented. This is the first and important phase for automatic detection and classification of plant diseases. Disease spots are different in color but not in intensity, in comparison with plant leaf color. So we color transform of RGB image can be used for better segmentation of disease spots. In this paper a comparison of the effect of CIELAB, HSI and YCbCr color space in the process of disease spot detection is done. Median filter is used for image smoothing. Finally threshold can be calculated by applying Otsu method on color component to detect the disease spot. An algorithm which is independent of background noise, plant type and disease spot color was developed and experiments were carried out on different “Monocot” and “Dicot” family plant leaves with both, noise free (white) and noisy background.

There is a large collection of Handwritten English paper documents of Historical and Scientific importance. But paper documents are not recognised directly by computer. Hence the closest way of indexing these documents is by storing their document digital image. Hence a large database of document images can replace the paper documents. But the document and data corresponding to each image cannot be directly recognised by the computer. This paper applies the technique of word spotting using Modified Character Shape Code to Handwritten English document images for quick and efficient query search of words on a database of document images. It is different from other Word Spotting techniques as it implements two level of selection for word segments to match search query. First based on word size and then based on character shape code of query. It makes the process faster and more efficient and reduces the need of multiple pre-processing.

In this paper, a new approach in human identification is investigated. For this purpose, a standard 12-lead electrocardiogram (ECG) recorded during rest is used. Selected features extracted from the ECG are used to identify a person in a predetermined group. Multivariate analysis is used for the identification task. Experiments show that it is possible to identify a person by features extracted from one lead only. Hence, only three electrodes have to be attached on the person to be identified. This makes the method applicable without too much effort.

A novel approach is proposed in this paper for automatic forest fire detection from video. Based on 3D point cloud of the collected sample fire pixels, Gaussian mixture model is built and helps segment some possible flame regions in single image. Then the new specific flame pattern is defined for forest, and three types of fire colors are labeled accordingly. With 11 static features including color distributions, texture parameters and shape roundness, the static SVM classifier is trained and filters the segmented results. Using defined overlapping degree and varying degree, the remained candidate regions are matched among consecutive frames. Subsequently the variations of color, texture, roundness, area, contour are computed, then the average and the mean square deviation of them are obtained. Together with the flickering frequency from temporal wavelet based Fourier descriptors analysis of flame contour, 27 dynamic features are used to train the dynamic SVM classifier, which is applied for final decision. Our approach has been tested with dozens of video clips, and it can detect forest fire while recognize the fire like objects, such as red house, bright light and flying flag. Except for the acceptable accuracy, our detection algorithm performs in real time, which proves its value for computer vision based forest fire surveillance.

Texture is one of the important characteristics used in identifying objects or regions of interest in an image, whether the image be a photomicrograph, an aerial photograph, or a satellite image. This paper describes some easily computable textural features based on graytone spatial dependancies, and illustrates their application in categoryidentification tasks of three different kinds of image data: photomicrographs of five kinds of sandstones, 1:20 000 panchromatic aerial photographs of eight land-use categories, and Earth Resources Technology Satellite (ERTS) multispecial imagery containing seven land-use categories. We use two kinds of decision rules: one for which the decision regions are convex polyhedra (a piecewise linear decision rule), and one for which the decision regions are rectangular parallelpipeds (a min-max decision rule). In each experiment the data set was divided into two parts, a training set and a test set. Test set identification accuracy is 89 percent for the photomicrographs, 82 percent for the aerial photographic imagery, and 83 percent for the satellite imagery. These results indicate that the easily computable textural features probably have a general applicability for a wide variety of image-classification applications.

The available parking management systems require human efforts for recording entries of coming and leaving vehicles from parking in sheets. For huge parking it is difficult to keep track of the information. Use of Radio Frequency Identification known as RFID technology reduces human efforts and Optical Character Recognition known as OCR, enabled system will provide an automated system for parking management. And also it will provide the control over the access of parking space by the use of boom barriers. For huge parking it will be an effective system. Parking can be a commercial one or can be a Very Important Person (VIP) parking. Depending on the usage system can be used. Vehicle cannot be parked in parking if it is not registered. OCR acts as a solution for vehicles which are not registered. To keep track of the happenings in parking cameras are used. Places where automated systems are required, the proposed system is an answer. System combines the two strong technologies RFID and OCR.

— Invented in 1976, Number Plates Recognition (NPR) has since found wide commercial applications, making its research prospects challenging and scientifically interesting. A complete NPR system functions by vz steps, license plate; localization, sizing and orientation, normalization, character recognitions and geometric analysis. This paper is a review of NPR preliminary stages; it explains number plate localization, sizing and orientations as well as normalizations sections of the Number Plates Detection and Recognition-Tanzania Case study. MATLAB R2012b is employed in these processes. The input incorporated includes front and rear photographic images of vehicles, for proximity and simulation purposes the ample angle of image is 90 degree +-15. The captured image is converted to gray scale, binarized and edge detection algorithms are used to enhance edges. The output of this stage provides the input feature extraction, segmentation and


Question: A system for identifying plants utilizing their distinctive leaf characteristics.

This document outlines an innovative system for the automatic inspection and feature extraction of tobacco leaves, specifically focusing on flue-cured varieties. Utilizing advanced machine vision techniques, the system aims to automate the process of analyzing and extracting key features from tobacco leaves, including color, size, shape, and surface texture. The integration of machine vision in this context offers a significant step forward in addressing the complexities of tobacco leaf feature extraction and analysis. The findings from the conducted experiments confirm the system's effectiveness in accurately extracting features from tobacco leaves, making it a promising solution for the automated classification of these leaves in various applications.

Paper documents such as passports, visas and banknotes are frequently checked by inspection of security elements. In particular, optically variable devices such as holograms are important, but difficult to inspect. Augmented Reality can provide all relevant information on standard mobile devices. However, hologram verification on mobiles still takes long and provides lower accuracy than inspection by human individuals using appropriate reference information. We aim to address these drawbacks by automatic matching combined with a special parametrization of an efficient goal-oriented user interface which supports constrained navigation. We first evaluate a series of similarity measures for matching hologram patches to provide a sound basis for automatic decisions. Then a re-parametrized user interface is proposed based on observations of typical user behavior during document capture. These measures help to reduce capture time to approximately 15 s with better decisions regarding the evaluated samples than what can be achieved by untrained users.

Human Vision system is the most complex and accurate system. In order to extract better features about Synthetic Aperture Radar (SAR) targets, a SAR automatic target recognition (ATR) algorithm based on human visual cortical system is proposed. This algorithm contains three stages: (1) Image preprocessing (we use a Kuan filter to do the enhancement and an adaptive Intersecting Cortical Model (ICM) to do the segmentation) (2) Feature extraction using a sparse autoencoder. (3) Classification using a softmax regression classifier. Experiment result of MSTAR public data shows a better performance of recognition.

It usually takes a fusion of image processing and machine learning algorithms in order to build a fully-functioning computer vision system for hand gesture recognition. Fortunately, the complexity of developing such a system could be alleviated by treating the system as a collection of multiple sub-systems working together, in such a way that they can be dealt with in isolation. Machine learning need to feed on thousands of exemplars (e.g. images, features) to automatically establish some recognisable patterns for all possible classes (e.g. hand gestures) that applies to the problem domain. A good number of exemplars helps, but it is also important to note that the efficacy of these exemplars depends on the variability of illumination conditions, hand postures, angles of rotation, scaling and on the number of volunteers from whom the hand gesture images were taken. These exemplars are usually subjected to image processing first, to reduce the presence of noise and extract the important features from the images. These features serve as inputs to the machine learning system. Different sub-systems are integrated together to form a complete computer vision system for gesture recognition. The main contribution of this work is on the production of the exemplars. We discuss how a dataset of standard American Sign Language (ASL) hand gestures containing 2425 images from 5 individuals, with variations in lighting conditions and hand postures is generated with the aid of image processing techniques. A minor contribution is given in the form of a specific feature extraction method called moment invariants, for which the computation method and the values are furnished with the dataset.

In most of the food industries, farms and fruit markets the quality sorting of the fruits is done manually; only a few automated sorting systems are developed for this. Each fruit changes its color in its life span; including stages not ripe, semi ripped, completely ripped or at the end rotten. Hence, we can predict the quality of the fruits by processing its color images and then applying pattern recognition techniques on those images. By quality of fruit we mean size, ripeness, sweetness, longetivity, diseased/ rotten etc. Some advance technologies which are used for classification of fruits are Infrared Imaging, Magnetic Resonance Imaging, Thermal Sensing, Electronic Nose sensing etc; but these are costlier as compared to color image processing and pattern recognition techniques. This paper summarizes and compares various techniques of Pattern Recognition which are used effectively for fruit sorting. All the fruits have their unique ways of ripening stages; hence the pattern recognition techniques used for one fruit does not necessarily match for the other fruit. This paper reviews the high quality research work done for the quality classification of Apples, Dates, Mangoes, Cranberry, Jatropha, Tomatoes, and some other fruits using pattern recognition techniques. KeywordsClassification & predication, Pattern Recognition, Image Processing, Fruit Quality __________________________________________________*****_________________________________________________

Running races, such as marathons, are broadly covered by professional as well as amateur photographers. This leads to a constantly growing number of photos covering a race, making the process of identifying a particular runner in such datasets difficult. Today, such identification is often done manually. In running races, each competitor has an identification number, called the Racing Bib Number (RBN), used to identify that competitor during the race. RBNs are usually printed on a paper or cardboard tag and pined onto the competitor’s T-shirt during the race. We introduce an automatic system that receives a set of natural images taken in running sports events and outputs the participants’ RBN.

A general-purpose, fully interconnected neural-net chip was used to perform computationally intensive tasks for handwritten digit recognition. The chip has nearly 3000 programmable connections, which can be set for template matching. The templates can be reprogrammed as needed during the recognition sequence. The recognition process proceeds in four major steps. First, the image is captured using a TV camera and a digital framegrab. This image is converted, using a digital computer, to either black or white pixels and scaled to fill a 16*16-pixel frame. Next, using the neural-net chip, the image is skeletonized, i.e. the image is thinned to a backbone one pixel wide. Then, the chip is programmed, and a feature map is created by template-matching stored primitive patterns on the chip with regions on the skeletonized image. Finally, recognition, based on the feature map, is achieved using any one of a variety of statistical and heuristic techniques on a digital computer. Best scores range between 90% and 99% correct classification, depending on the quality of the original handwritten digits.<<ETX>>

This paper mainly discussed the process to classify Anthracnose and Downey Mildew, watermelon leaf diseases using neural network analysis. A few of infected leaf samples were collected and they were captured using a digital camera with specific calibration procedure under controlled environment. The classification on the watermelon's leaf diseases is based on color feature extraction from RGB color model where the RGB pixel color indices have been extracted from the identified Regions of Interest (ROI). The proposed automated classification model involved the process of diseases classification using Statistical Package for the Social Sciences (SPSS) and Neural Network Pattern Recognition Toolbox in MATLAB. Determinations in this work have shown that the type of leaf diseases achieved 75.9% of accuracy based on its RGB mean color component.

In informational age there are many information crucial to a individual. To protect the data from the unwanted use we proposed the biometric system of vein recognition as today's world demand for a easy and accurate biometric system to protect our data. The biometric system is accurate and easy to use as every individual has a unique vein pattern. In this paper we have studied various method used by different people to achieve the goal to secure world and give the reader a brief prospective of this technology.

The mammalian visual system is endowed with a nearly infinite capacity for the recognition of patterns and objects. To have acquired this capability the visual system must have solved what is a fundamentally combinatorial problem. Any given image consists of a collection of features, consisting of local contrast borders of luminance and wavelength, distributed across the visual field. For one to detect and recognize an object within a scene, the features comprising the object must be identified and segregated from those comprising other objects. This problem is inherently difficult to solve because of the combinatorial nature of visual images. To appreciate this point, consider a simple local feature such as a small vertically oriented line segment placed within a fixed location of the visual field. When combined with other line segments, this feature can form a nearly infinite number of geometrical objects. Any one of these objects may coexist with an equally large number of other