Question: FaCT++ Description Logic Reasoner: A System Overview

This paper introduces Racer, a foundational inference engine designed for the Semantic Web. Racer serves as a pivotal component, providing inference capabilities that are leveraged by various network clients. These clients include OilEd, an ontology editor; RICE, a visualization tool; and Protege 2, an ontology development environment. Racer's inference server offers two Application Programming Interfaces (APIs), which facilitate its integration with these clients. Notably, Racer supports the DIG protocol over HTTP, alongside a TCP-based protocol that includes advanced query functionalities. The engine is equipped to handle web ontology languages such as DAML+OIL, RDF, and OWL, making it a versatile tool for semantic web applications.

Information on the Internet is fragmented and presented in different data sources, which makes automatic knowledge harvesting and understanding formidable for machines, and even for humans. Knowledge graphs have become prevalent in both of industry and academic circles these years, to be one of the most efficient and effective knowledge integration approaches. Techniques for knowledge graph construction can mine information from either structured, semi-structured, or even unstructured data sources, and finally integrate the information into knowledge, represented in a graph. Furthermore, knowledge graph is able to organize information in an easy-to-maintain, easy-to-understand and easy-to-use manner. In this paper, we give a summarization of techniques for constructing knowledge graphs. We review the existing knowledge graph systems developed by both academia and industry. We discuss in detail about the process of building knowledge graphs, and survey state-of-the-art techniques for automatic knowledge graph checking and expansion via logical inferring and reasoning. We also review the issues of graph data management by introducing the knowledge data models and graph databases, especially from a NoSQL point of view. Finally, we overview current knowledge graph systems and discuss the future research directions.

In this paper, we describe an end-to-end system that automatically extracts RDF triples describing entity relations and properties from unstructured text. This system is based on a pipeline of text processing modules that includes a semantic parser and a coreference solver. By using coreference chains, we group entity actions and properties described in different sentences and convert them into entity triples. We applied our system to over 114,000 Wikipedia articles and we could extract more than 1,000,000 triples. Using an ontology-mapping system that we bootstrapped using existing DBpedia triples, we mapped 189,000 extracted triples onto the DBpedia namespace. These extracted entities are available online in the N-Triple format1.

The ontology matching process with background knowledge is more suitable to match heterogeneous ontologies, since background knowledge is used as a mediator or a reference to identify relation between two concepts being matched. This method is called indirect matching and the system is called indirect matching system. This paper reviews the motivation that described the urgency of ontology matching, the various background knowledge and their strengths, also indirect matching process. At the end we provide the comparison of indirect matching system. Based on the comparison, mapping repair function were added to improve the quality of mapping. The purpose of this paper is to help in guiding new practitioners get a general idea on the ontology matching field and to determine possible research lines.

Software development in highly variable domains constrained by tight regulations and with many business concepts involved results in hard to deliver and maintain applications, due to the complexity of dealing with the large number of concepts provided by the different parties and system involved in the process. One way to tackle these problems is thru combining software product lines and model-driven software development supported by ontologies. Software product lines and model-driven approaches would promote reuse on the software artifacts and, if supported by an ontological layer, those artifacts would be domain-validated. We intend to create a new conceptual framework for software development with domain validated models in highly variable domains. To define such a framework we will propose a model that relates several dimensions and areas of software development thru time and abstraction levels. This model would guarantee to the software house traceability of components, domain validated artifacts, easy to maintain and reusable components, due to the relations and mappings we propose to establish in the conceptual framework, between the software artifacts and the ontology.

Data integration is central in Web application development because these applications typically deal with a variety of information formats. Ontology-driven applications face the additional challenge of integrating these multiple formats with the information stored in ontologies. A number of mappings are required to reconcile the variety of formats to produce a coherent overall system. To address these mappings we have developed a number of open source tools that support transformations between some of the common formats encountered when developing an ontology-driven Web application. The Semantic Web Rule Language (SWRL) is a central building block in these tools. We describe these tools and illustrate their use in the development of a prototype Web-based application.

The DBpedia project extracts structured information from Wikipedia editions in 97 different languages and combines this information into a large multi-lingual knowledge base covering many specific domains and general world knowledge. The knowledge base contains textual descriptions (titles and abstracts) of concepts in up to 97 languages. It also contains structured knowledge that has been extracted from the infobox systems of Wikipedias in 15 different languages and is mapped onto a single consistent ontology by a community effort. The knowledge base can be queried using a structured query language and all its data sets are freely available for download. In this paper, we describe the general DBpedia knowledge base and extended data sets that specifically aim at supporting computational linguistics tasks. These task include Entity Linking, Word Sense Disambiguation, Question Answering, Slot Filling and Relationship Extraction. These use cases are outlined, pointing at added value that the structured data of DBpedia provides.

The convergence of semantic Web with service oriented computing is manifested by Semantic Web Services (SWS) technology. It addresses the major challenge of automated, interoperable and meaningful coordination of Web Services to be carried out by intelligent software agents. In this chapter, we briefly discuss prominent SWS description frameworks, that are the standard SAWSDL, OWL-S and WSML. This is complemented by main critics of Semantic Web Services, and selected references to further readings on the subject.

Citations are important in academic dissemination. To help researchers check the completeness of citations while authoring a paper, we introduce a citation recommendation system called RefSeer. Researchers can use it to find related works to cited while authoring papers. It can also be used by reviewers to check the completeness of a paper's references. RefSeer presents both topic based global recommendation and also citation-context based local recommendation. By evaluating the quality of recommendation, we show that such recommendation system can recommend citations with good precision and recall. We also show that our recommendation system is very efficient and scalable.

MC& CYC project is the building, over the coming decade, of a large knowledge base (or KB) of real world facts and heuristics and-as a part of the KB itself-methods for efficiently reasoning over the KB. As the title of this article suggests, our hypothesis is that the two major limitations to building large intelligent programs might be overcome by using such a system. We briefly illustrate how common sense reasoning and analogy can widen the knowledge acquisition bottleneck The next section (“How CYC Works”) illustrates how those same two abilities can solve problems of the type that stymie current expert systems. We then report how the project is being conducted currently: its strategic philosophy, its tactical methodology, and a case study of how we are currently putting that into practice. We conclude with a discussion of the project’s feasibility and timetable.

In this paper, we present an approach based on Ontology Learning and Natural Language Processing for automatic construction of expressive Ontologies, specifically in OWL DL with ALC expressivity, from a natural language text. The viability of our approach is demonstrated through the generation of descriptions of complex axioms from concepts defined by users and glossaries found at Wikipedia. We evaluated our approach in an experiment with entry sentences enriched with hierarchy axioms, disjunction, conjunction, negation, as well as existential and universal quantification to impose restriction of properties. The obtained results prove that our model is an effective solution for knowledge representation and automatic construction of expressive Ontologies. Thereby, it assists professionals involved in processes for obtain, construct and model knowledge domain.


Question: FaCT++ description logic reasoner: an overview of the system.

This paper delves into the rationale and methodology behind the creation of ontologies, which are explicit, formal definitions of the terms and relationships within a specific domain. Originally the purview of Artificial Intelligence (AI) research, ontology development has become increasingly accessible to domain experts, transforming from a niche academic endeavor to a widely adopted practice. Today, ontologies are a common feature on the World-Wide Web, serving various purposes. They can be found in the form of large taxonomies that categorize websites, such as those seen on Yahoo!, or in detailed product classifications and feature listings, as utilized by e-commerce giants like Amazon.com. The World-Wide Web Consortium (W3C) has been instrumental in advancing the use of ontologies on the web through the development of the Resource Description Framework (RDF), a language designed to encode knowledge on web pages in a format that can be understood by electronic agents searching for information. The Defense Advanced Research Projects Agency (DARPA) has further contributed to this field by extending RDF with more sophisticated constructs in the form of DAML (DARPA Agent Markup Language), enhancing agent interaction on the web. The proliferation of ontologies extends to various disciplines, where standardized ontologies are being developed to facilitate information sharing and annotation among domain experts. In the medical field, for instance, extensive, standardized vocabularies such as SNOMED and the Unified Medical Language System's semantic network have been created. Moreover, comprehensive, general-purpose ontologies are emerging, such as the UNSPSC ontology developed through a collaboration between the United Nations Development Program and Dun & Bradstreet, which provides a structured terminology for products and services. So, why is there a need to develop an ontology? The answer lies in its ability to define a shared vocabulary among researchers and practitioners working within a specific domain, encompassing machine-interpretable definitions of fundamental concepts and their interrelations. The motivations for developing an ontology are multifaceted, encompassing:

The modern enterprise software stack—a collection of applications supporting bookkeeping, analytics, planning, and forecasting for enterprise data—is in danger of collapsing under its own weight. The task of building and maintaining enterprise software is tedious and laborious; applications are cumbersome for end-users; and adapting to new computing hardware and infrastructures is difficult. We believe that much of the complexity in today’s architecture is accidental, rather than inherent. This tutorial provides an overview of the LogicBlox platform, a ambitious redesign of the enterprise software stack centered around a unified declarative programming model, based on an extended version of Datalog. 1 The Enterprise Hairball Modern enterprise applications involve an enormously complex technology stack composed of many disparate systems programmed in a hodgepodge of programming languages. We refer to this stack, depicted in Figure 1, as “the enterprise hairball.” First, there is an online transaction processing (OLTP) layer that performs bookkeeping of the core business data for an enterprise. Such data could include the current product catalog, recent sales figures, current outstanding invoices, customer account balances, and so forth. This OLTP layer typically includes a relational DBMS—programmed in a combination of a query language (SQL), a stored procedure language (like PL/SQL or TSQL), and a batch programming language like Pro*C—an application server, such as Oracle WebLogic [35], IBM WebSphere [36], or SAP NetWeaver [26]—programmed in an object-oriented language like Java, C#, or ABAP—and a web browser front-end, programmed using HTML and Javascript. In order to track the performance of the enterprise over time, a second business intelligence (BI) layer typically holds five to ten years of historical information that was originally recorded in the OLTP layer and performs read-only analyses on this information. This layer typically includes another DBMS (or, more commonly, a BI variant like Teradata [33] or IBM Netezza [25]) along with a BI application server such as Microstrategy [23], SAP BusinessObjects [5], or IBM Cognos [7], programmed using a vendor-specific declarative language. Fig. 1: Enterprise software components and technology stack example. Data is moved from the transaction layer to the BI layer via so-called extracttransform-load (ETL) tools that come with their own tool-specific programming language. Finally, in order to plan the future actions of an enterprise, there is a planning layer, which supports a class of read-write use cases for which the BI layer is unsuitable. This layer typically includes a planning application server, like Oracle Hyperion [13] or IBM Cognos TM1 [34], that are programmed using a vendor-specific declarative language or a standard language like MDX [21], and spreadsheets like Microsoft Excel that are programmed in a vendor-specific formula language (e.g. A1 = B17 D12) and optionally a scripting language like VBA. In order to enhance or automate decisions made in the planning layer, statistical predictive models are often prototyped using modeling tools like SAS, Matlab, SPSS, or R, and then rewritten for production in C++ or Java so they can be embedded in the OLTP or OLAP layers. In summary, enterprise software developed according to the hairball model must be programmed using a dozen or so different programming languages, running in almost as many system-scale components. Some of the languages are imperative (both object-oriented and not) and some are declarative (every possible flavor). Most of the languages used are vendor-specific and tied to the component in which they run (e.g. ABAP, Excel, R, OPL, etc.). Even the languages that are based on open standards are not easily ported from one component to another because of significant variations in performance or because of vendor specific extensions (e.g., the same SQL query on Oracle will perform very differently on DB2). Recent innovations in infrastructure technology for supporting much larger numbers of users (Web applications) and big data (predictive analytics), including NoSQL [28], NewSQL [27] and analytic databases, have introduced even more components into the technology stack described above, and have not helped reduce its overall complexity. This complexity makes enterprise software hard to build, hard to implement, and hard to change. Simple changes—like extending a product identifier by 3 characters or an employee identifier by 1 digit—often necessitate modifications to most of the different components in the stack, requiring thousands of days of person effort and costing many millions of dollars. An extreme example of the problems caused by this accidental complexity occurred in the lead-up to the year 2000, where the simple problem of adding two digits to the year field (the Y2K problem) cost humanity over $300 billion dollars [24]. Moreover, as a result of the time required for such changes, individuals within an enterprise often resort to ad hoc, error-prone methods to perform the calculations needed in order to make timely business decisions. For example, they often roll their own extensions using spreadsheets, where they incompletely or incorrectly implement such calculations based on copies of the data that is not kept in sync with the OLTP database and thus may no longer accurately reflect the state of their enterprise.

Aspect extraction aims to extract fine-grained opinion targets from opinion texts. Recent work has shown that the syntactical approach performs well. In this paper, we show that Logic Programming, particularly Answer Set Programming (ASP), can be used to elegantly and efficiently implement the key components of syntax based aspect extraction. Specifically, the well known double propagation (DP) method is implemented using 8 ASP rules that naturally model all key ideas in the DP method. Our experiment on a widely used data set also shows that the ASP implementation is much faster than a Java-based implementation. Syntactical approach has its limitation too. To further improve the performance of syntactical approach, we identify a set of general words from Word Net that have little chance to be an aspect and prune them when extracting aspects. The concept of general words and their pruning are concisely captured by 10 new ASP rules, and a natural extension of the 8 rules for the original DP method. Experimental results show a major improvement in precision with almost no drop in recall compared with those reported in the existing work on a typical benchmark data set. Logic Programming provides a convenient and effective tool to encode and thus test knowledge needed to improve the aspect extraction methods so that the researchers can focus on the identification and discovery of new knowledge to improve aspect extraction.

Probabilistic soft logic (PSL) is a framework for collective, probabilistic reasoning in relational domains. PSL uses first order logic rules as a template language for graphical models over random variables with soft truth values from the interval [0, 1]. Inference in this setting is a continuous optimization task, which can be solved efficiently. This paper provides an overview of the PSL language and its techniques for inference and weight learning. An implementation of PSL is available at http://psl.umiacs.umd.edu/.

The Semantic Network, a component of the Unified Medical Language System® (UMLS), describes core biomedical knowledge consisting of semantic types and relationships. It is a well established, semi-formal ontology in widespread use for over a decade. We expected to “publish” this ontology on the Semantic Web, using OWL, with relatively little effort. However, we ran into a number of problems concerning alternative interpretations of the SN notation and the inability to express some of the interpretations in OWL. We detail these problems, as a cautionary tale to others planning to publish pre-existing ontologies on the Semantic Web, as a list of issues to consider when describing formally concepts in any ontology, and as a collection of criteria for evaluating alternative representations, which could form part of a methodology of ontology development.

We propose a simple approach to combining first-order logic and probabilistic graphical models in a single representation. A Markov logic network (MLN) is a first-order knowledge base with a weight attached to each formula (or clause). Together with a set of constants representing objects in the domain, it specifies a ground Markov network containing one feature for each possible grounding of a first-order formula in the KB, with the corresponding weight. Inference in MLNs is performed by MCMC over the minimal subset of the ground network required for answering the query. Weights are efficiently learned from relational databases by iteratively optimizing a pseudo-likelihood measure. Optionally, additional clauses are learned using inductive logic programming techniques. Experiments with a real-world database and knowledge base in a university domain illustrate the promise of this approach.

Adaptation addresses the complexity and overload caused by an increasing amount of available resources by the identification and presentation of relevant resources. Drawbacks of existing approaches to this end include the limited degree of customization, difficulties in the acquisition of model information and the lack of control and transparency of the system's adaptive behavior. This paper proposes an approach which addresses these drawbacks to perform meaningful and effective adaptation by the application of semantic Web technologies. This comprises the use of an ontology and adaptation rules for knowledge representation and inference engines for reasoning. The focus of this paper lies in the presentation of adaptation rules

This paper reports on the results of an independent evaluation of the techniques presented in the VLDB 2007 paper “Scalable Semantic Web Data Management Using Vertical Partitioning”, authored by D. Abadi, A. Marcus, S. R. Madden, and K. Hollenbach [1]. We revisit the proposed benchmark and examine both the data and query space coverage. The benchmark is extended to cover a larger portion of the query space in a canonical way. Repeatability of the experiments is assessed using the code base obtained from the authors. Inspired by the proposed vertically-partitioned storage solution for RDF data and the performance figures using a column-store, we conduct a complementary analysis of state-of-the-art RDF storage solutions. To this end, we employ MonetDB/SQL, a fully-functional open source column-store, and a well-known – for its performance – commercial row-store DBMS. We implement two relational RDF storage solutions – triple-store and vertically-partitioned – in both systems. This allows us to expand the scope of [1] with the performance characterization along both dimensions – triple-store vs. vertically-partitioned and row-store vs. column-store – individually, before analyzing their combined effects. A detailed report of the experimental test-bed, as well as an in-depth analysis of the parameters involved, clarify the scope of the solution originally presented and position the results in a broader context by covering more systems.

Companies require data from multiple sources to develop new information systems, such as social networking, e-commerce and location-based services. Systems rely on complex, multi-stakeholder data supply-chains to deliver value. These data supply-chains have complex privacy requirements: privacy policies affecting multiple stakeholders (e.g. user, developer, company, government) regulate the collection, use and sharing of data over multiple jurisdictions (e.g. California, United States, Europe). Increasingly, regulators expect companies to ensure consistency between company privacy policies and company data practices. To address this problem, we propose a methodology to map policy requirements in natural language to a formal representation in Description Logic. Using the formal representation, we reason about conflicting requirements within a single policy and among multiple policies in a data supply chain. Further, we enable tracing data flows within the supply-chain. We derive our methodology from an exploratory case study of Facebook platform policy. We demonstrate the feasibility of our approach in an evaluation involving Facebook, Zynga and AOL-Advertising policies. Our results identify three conflicts that exist between Facebook and Zynga policies, and one conflict within the AOL Advertising policy.

We present an implementation of a probabilistic first-order logic called TensorLog, in which classes of logical queries are compiled into differentiable functions in a neuralnetwork infrastructure such as Tensorflow or Theano. This leads to a close integration of probabilistic logical reasoning with deep-learning infrastructure: in particular, it enables high-performance deep learning frameworks to be used for tuning the parameters of a probabilistic logic. Experimental results show that TensorLog scales to problems involving hundreds of thousands of knowledge-base triples and tens of thousands of examples.

This paper presents a new and very rich class of (concurrent) programming languages, based on the notion of computing with partial information, and the concomitant notions of consistency and entailment.1 In this framework, computation emerges from the interaction of concurrently executing agents that communicate by placing, checking and instantiating constraints on shared variables. Such a view of computation is interesting in the context of programming languages because of the ability to represent and manipulate partial information about the domain of discourse, in the context of concurrency because of the use of constraints for communication and control, and in the context of AI because of the availability of simple yet powerful mechanisms for controlling inference, and the promise that very rich representational/programming languages, sharing the same set of abstract properties, may be possible.
To reflect this view of computation, [Sar89] develops the cc family of languages. We present here one member of the family, cc(↓, →) (pronounced “cc with Ask and Choose”) which provides the basic operations of blocking Ask and atomic Tell and an algebra of behaviors closed under prefixing, indeterministic choice, interleaving, and hiding, and provides a mutual recursion operator. cc(↓, →) is (intentionally!) very similar to Milner's CCS, but for the radically different underlying concept of communication, which, in fact, provides a general—and elegant—alternative approach to “value-passing” in CCS. At the same time it adequately captures the framework of committed choice concurrent logic programming languages. We present the rules of behavior for cc agents, motivate a notion of “visible action” for them, and develop the notion of c-trees and reactive congruence analogous to Milner's synchronization trees and observational congruence. We also present an equational characterization of reactive congruence for Finitary cc(↓, →).


Question: FaCT++ Description Logic Reasoner: A System Overview

In the realm of knowledge representation and reasoning, the FaCT++ Description Logic Reasoner has emerged as a significant tool. This system, designed to work with the intricacies of the Semantic Web, offers a powerful framework for understanding and processing complex information structures. Unlike general scientific publications that might cover the Semantic Web in a broader context, such as those found in Scientific American, FaCT++ focuses specifically on the practical application of Description Logics. It enables the inference of new knowledge from existing data, facilitating a deeper understanding of information encoded in semantic formats. This capability is crucial for applications ranging from automated reasoning in databases to the enhancement of search algorithms, making FaCT++ an indispensable component in the development of intelligent systems that can reason about and interact with the vast amount of data available on the Semantic Web. By leveraging Description Logics, FaCT++ contributes to the advancement of the Semantic Web's goal of making the web more machine-readable and accessible to automated processes.

We present a question answering system (CASIA@V2) over Linked Data (DBpedia), which translates natural language questions into structured queries automatically. Existing systems usually adopt a pipeline framework, which contains four major steps: 1) Decomposing the question and detecting candidate phrases; 2) mapping the detected phrases into semantic items of Linked Data; 3) grouping the mapped semantic items into semantic triples; and 4) generating the rightful SPARQL query. We present a jointly learning framework using Markov Logic Network(MLN) for phrase detection, phrases mapping to semantic items and semantic items grouping. We formulate the knowledge for resolving the ambiguities in three steps of QALD as first-order logic clauses in a MLN. We evaluate our approach on QALD-4 test dataset and achieve an F-measure score of 0.36, an average precision of 0.32 and an average recall of 0.40 over 50 questions.

The Object Constraint Language (OCL) started as a complement of the UML notation with the goal to overcome the limitations of UML (and in general, any graphical notation) in terms of precisely specifying detailed aspects of a system design. Since then, OCL has become a key component of any model-driven engineering (MDE) technique as the default language for expressing all kinds of (meta)model query, manipulation and specification requirements. Among many other applications, OCL is frequently used to express model transformations (as part of the source and target patterns of transformation rules), well-formedness rules (as part of the definition of new domain-specific languages), or code-generation templates (as a way to express the generation patterns and rules). This chapter pretends to provide a comprehensive view of this language, its many applications and available tool support as well as the latest research developments and open challenges around it.

In recent years patents have become increasingly important for businesses to protect their intellectual capital and as a valuable source of information. Patent information is, however, not employed to its full potential and the interpretation of structured and unstructured patent information in large volumes remains a challenge. We address this by proposing an integrated interdisciplinary approach that uses natural language processing and machine learning techniques to formalize multilingual patent information in an ontology. The ontology further contains patent and domain specific knowledge, which allows for aligning patents with technological fields of interest and other business-related artifacts. Our empirical evaluation shows that for categorizing patents according to well-known technological fields of interest, the approach achieves high accuracy with selected feature sets compared to related work focussing on monolingual patents. We further show that combining OWL RL reasoning with SPARQL querying over the patent knowledge base allows for answering complex business queries and illustrate this with real-world use cases from the automotive domain.

Triple stores are the backbone of increasingly many Data Web applications. It is thus evident that the performance of those stores is mission critical for individual projects as well as for data integration on the Data Web in general. Consequently, it is of central importance during the implementation of any of these applications to have a clear picture of the weaknesses and strengths of current triple store implementations. In this paper, we propose a generic SPARQL benchmark creation procedure, which we apply to the DBpedia knowledge base. Previous approaches often compared relational and triple stores and, thus, settled on measuring performance against a relational database which had been converted to RDF by using SQL-like queries. In contrast to those approaches, our benchmark is based on queries that were actually issued by humans and applications against existing RDF data not resembling a relational schema. Our generic procedure for benchmark creation is based on query-log mining, clustering and SPARQL feature analysis. We argue that a pure SPARQL benchmark is more useful to compare existing triple stores and provide results for the popular triple store implementations Virtuoso, Sesame, Jena-TDB, and BigOWLIM. The subsequent comparison of our results with other benchmark results indicates that the performance of triple stores is by far less homogeneous than suggested by previous benchmarks.

Mixed integer linear programming (MILP) is a powerful representation often used to formulate decision-making problems under uncertainty. However, it lacks a natural mechanism to reason about objects, classes of objects, and relations. First-order logic (FOL), on the other hand, excels at reasoning about classes of objects, but lacks a rich representation of uncertainty. While representing propositional logic in MILP has been extensively explored, no theory exists yet for fully combining FOL with MILP. We propose a new representation, called first-order programming or FOP, which subsumes both FOL and MILP. We establish formal methods for reasoning about first order programs, including a sound and complete lifted inference procedure for integer first order programs. Since FOP can offer exponential savings in representation and proof size compared to FOL, and since representations and proofs are never significantly longer in FOP than in FOL, we anticipate that inference in FOP will be more tractable than inference in FOL for corresponding problems.

Efficient management of RDF data is an important factor in realizing the Semantic Web vision. Performance and scalability issues are becoming increasingly pressing as Semantic Web technology is applied to real-world applications. In this paper, we examine the reasons why current data management solutions for RDF data scale poorly, and explore the fundamental scalability limitations of these approaches. We review the state of the art for improving performance for RDF databases and consider a recent suggestion, “property tables.” We then discuss practically and empirically why this solution has undesirable features. As an improvement, we propose an alternative solution: vertically partitioning the RDF data. We compare the performance of vertical partitioning with prior art on queries generated by a Web-based RDF browser over a large-scale (more than 50 million triples) catalog of library data. Our results show that a vertical partitioned schema achieves similar performance to the property table technique while being much simpler to design. Further, if a column-oriented DBMS (a database architected specially for the vertically partitioned case) is used instead of a row-oriented DBMS, another order of magnitude performance improvement is observed, with query times dropping from minutes to several seconds.

We presenta novel generati ve model for natural languagetree structuresin whichsemantic(lexical dependenc y) andsyntacticstructuresare scoredwith separatemodels.Thisfactorizationprovidesconceptual simplicity, straightforwardopportunitiesfor separatelyimproving the componentmodels,anda level of performancealreadycloseto thatof similar, non-factoredmodels.Most importantly, unlikeothermodernparsing models,thefactoredmodeladmitsanextremelyeffectiveA parsingalgorithm,which makesefficient,exactinferencefeasible.

Large-scale knowledge graphs such as those in the Linked Data cloud are typically represented as subject-predicate-object triples. However, many facts about the world involve more than two entities. While n-ary relations can be converted to triples in a number of ways, unfortunately, the structurally different choices made in different knowledge sources significantly impede our ability to connect them. They also make it impossible to query the data concisely and without prior knowledge of each individual source. We present FrameBase, a wide-coverage knowledge-base schema that uses linguistic frames to seamlessly represent and query n-ary relations from other knowledge bases, at different levels of granularity connected by logical entailment. It also opens possibilities to draw on natural language processing techniques for querying and data mining.

This paper gives an overview about the development of the field of Knowledge Engineering over the last 15 years. We discuss the paradigm shift from a transfer view to a modeling view and describe two approaches which considerably shaped research in Knowledge Engineering: Role-limiting Methods and Generic Tasks. To illustrate various concepts and methods which evolved in the last years we describe three modeling frameworks: CommonKADS, MIKE, and PROTÉGÉ-II. This description is supplemented by discussing some important methodological developments in more detail: specification languages for knowledge-based systems, problem-solving methods, and ontologies. We conclude with outlining the relationship of Knowledge Engineering to Software Engineering, Information Integration and Knowledge Management.

In the process of knowledge discovery and representation in large datasets using formal concept analysis, complexity plays a major role in identifying all the formal concepts and constructing the concept lattice(digraph of the concepts). For identifying the formal concepts and constructing the digraph from the identified concepts in very large datasets, various distributed algorithms are available in the literature. However, the existing distributed algorithms are not very well suitable for concept generation because it is an iterative process. The existing algorithms are implemented using distributed frameworks like MapReduce and Open MP, these frameworks are not appropriate for iterative applications. Hence, in this paper we proposed efficient distributed algorithms for both formal concept generation and concept lattice digraph construction in large formal contexts using Apache Spark. Various performance metrics are considered for the evaluation of the proposed work, the results of the evaluation proves that the proposed algorithms are efficient for concept generation and lattice graph construction in comparison with the existing algorithms.


Question: EvoNN: a customizable evolutionary neural network featuring heterogeneous activation functions.

This paper explores the evolution of artificial neural networks (ANNs) through the lens of neuroevolution, focusing on how architectural design and learning algorithms can be optimized for specific tasks. ANNs have found extensive application in various domains, including pattern classification and robotic control. The selection of an appropriate architecture and learning algorithm is crucial for the effective deployment of neural networks. Evolutionary search methods offer an automated solution to these design challenges by harnessing the principles of evolution to optimize neural network configurations. The paper reviews the latest advancements in neuroevolution techniques, which have been significantly influenced by recent insights from neuroscience and evolutionary biology. It highlights the development of sophisticated methods for evolving ANNs, particularly emphasizing the creation of learning architectures. The synthesis of these learning architectures is a key area of recent research, demonstrating the potential for evolving ANNs that can learn and adapt in complex environments. This comprehensive overview provides a detailed look at the state-of-the-art in neuroevolution, showcasing the latest techniques for evolving neural networks with customized architectures and heterogeneous activation functions, akin to the EvoNN framework.

The recognition of a character begins with analyzing its form and extracting the features that will be exploited for the identification. Primitives can be described as a tool to distinguish an object of one class from another object of another class. It is necessary to define the significant primitives. The size of vector primitives can be large if a large number of primitives are extracted including redundant and irrelevant features. As a result, the performance of the recognition system becomes poor, and as the number of features increases, so does the computing time. Feature selection, therefore, is required to ensure the selection of a subset of features that gives accurate recognition. In our work we propose a feature selection approach based genetic algorithm to improve the discrimination capacity of the Multilayer Perceptron Neural Networks (MLP).

General Matrix to Matrix multiplication (GEMM) is the cornerstone for a wide gamut of applications in high performance computing (HPC), scientific computing (SC) and more recently, deep learning. In this work, we present a customizable matrix multiplication framework for the Intel HARPv2 CPU+FPGA platform that includes support for both traditional single precision floating point and reduced precision workloads. Our framework supports arbitrary size GEMMs and consists of two parts: (1) a simple application programming interface (API) for easy configuration and integration into existing software and (2) a highly customizable hardware template. The API provides both compile and runtime options for controlling key aspects of the hardware template including dynamic precision switching; interleaving and block size control; and fused deep learning specific operations. The framework currently supports single precision floating point (FP32), 16, 8, 4 and 2 bit Integer and Fixed Point (INT16, INT8, INT4, INT2) and more exotic data types for deep learning workloads: INT16xTernary, INT8xTernary, BinaryxBinary.
 We compare our implementation to the latest NVIDIA Pascal GPU and evaluate the performance benefits provided by optimizations built into the hardware template. Using three neural networks (AlexNet, VGGNet and ResNet) we illustrate that reduced precision representations such as binary achieve the best performance, and that the HARPv2 enables fine-grained partitioning of computations over both the Xeon and FPGA. We observe up to 50x improvement in execution time compared to single precision floating point, and that runtime configuration options can improve the efficiency of certain layers in AlexNet up to 4x, achieving an overall 1.3x improvement over the entire network.

Hessian-Free optimization (HF) is an approach for unconstrained minimization of real-valued smooth objective functions. Like standard Newton’s method, it uses local quadratic approximations to generate update proposals. It belongs to the broad class of approximate Newton methods that are practical for problems of very high dimensionality, such as the training objectives of large neural networks. Different algorithms that use many of the same key principles have appeared in the literatures of various communities under different names such as Newton-CG, CG-Steihaug, Newton-Lanczos, and Truncated Newton [27, 28], but applications to machine learning and especially neural networks, have been limited or non-existent until recently. With the work of Martens [22] and later Martens and Sutskever [23] it has been demonstrated that such an approach, if carefully designed and implemented, can work very well for optimizing nonconvex functions such as the training objective for deep neural networks and recurrent neural networks (RNNs), given sensible random initializations. This was significant because gradient descent methods have been observed to be very slow and sometimes completely ineffective [16, 4, 17] on these problems, unless special non-random initializations schemes like layer-wise pre-training [16, 15, 3] are used. HF, which is a general optimization-based approach, can be used in conjunction with or as an alternative to existing pre-training methods and is more widely applicable, since it relies on fewer assumptions about the specific structure of the network. In this report we will first describe the basic HF approach, and then examine well-known general purpose performance-improving techniques as well as others that are specific to HF (versus other Truncated-Newton type approaches) or to neural networks. We will also provide practical tips for creating efficient and correct implementations, and discuss the pitfalls which may arise when designing and using an HF-based approach in a particular application.

An artificial neural network ensemble is a learning paradigm where several artificial neural networks are jointly used to solve a problem. In this paper, an automatic pathological diagnosis procedure named Neural Ensemble-based Detection (NED) is proposed, which utilizes an artificial neural network ensemble to identify lung cancer cells in the images of the specimens of needle biopsies obtained from the bodies of the subjects to be diagnosed. The ensemble is built on a two-level ensemble architecture. The first-level ensemble is used to judge whether a cell is normal with high confidence where each individual network has only two outputs respectively normal cell or cancer cell. The predictions of those individual networks are combined by a novel method presented in this paper, i.e. full voting which judges a cell to be normal only when all the individual networks judge it is normal. The second-level ensemble is used to deal with the cells that are judged as cancer cells by the first-level ensemble, where each individual network has five outputs respectively adenocarcinoma, squamous cell carcinoma, small cell carcinoma, large cell carcinoma, and normal, among which the former four are different types of lung cancer cells. The predictions of those individual networks are combined by a prevailing method, i.e. plurality voting. Through adopting those techniques, NED achieves not only a high rate of overall identification, but also a low rate of false negative identification, i.e. a low rate of judging cancer cells to be normal ones, which is important in saving lives due to reducing missing diagnoses of cancer patients.

Artificial neural networks’ (ANNs’) generalization powers have in recent years received admiration of finance researchers and practitioners. Their usage in such areas as bankruptcy prediction, debt-risk assessment, and security-market applications has yielded promising results. With such intensive research and proven ability of the ANN in the area of security-market application and the growing importance of the role of equity securities in Singapore, it has motivated the conceptual development of this work in using the ANN in stock selection. With their proven generalization ability, neural networks are able to infer the characteristics of performing stocks from the historical patterns. The performance of stocks is reflective of the profitability and quality of management of the underlying company. Such information is reflected in financial and technical variables. As such, the ANN is used as a tool to uncover the intricate relationships between the performance of stocks and the related financial and technical variables. Historical data, such as financial variables (inputs) and performance of the stock (output) is used in this ANN application. Experimental results obtained thus far have been very encouraging. IDEA GROUP PUBLISHING This paper appears in the publication, Artificial Neural Networks in Finance and Manufacturing edited by Joarder Kamruzzaman, Rezaul Begg, and Ruhul Sarker© 2006, Idea Group Inc. 701 E. Chocolate Avenue, Suite 200, Hershey PA 17033-1240, USA Tel: 717/533-8845; Fax 717/533-8661; URL-http://www.idea-group.com ITB13014 Improving Returns on Stock Investment through Neural Network Selection 153 Copyright © 2006, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. Introduction With the growing importance in the role of equities to both international and local investors, the selection of attractive stocks is of utmost importance to ensure a good return. Therefore, a reliable tool in the selection process can be of great assistance to these investors. An effective and efficient tool/system gives the investor the competitive edge over others as he/she can identify the performing stocks with minimum effort. In assisting the investors in their decision-making process, both the academics and practitioners have devised trading strategies, rules, and concepts based on fundamental and technical analysis. Innovative investors opt to employ information technology to improve the efficiency in the process. This is done through transforming trading strategies into computer-known language so as to exploit the logical processing power of the computer. This greatly reduces the time and effort in short-listing the list of attractive stocks. In the age where information technology is dominant, such computerized rule-based expert systems have severe limitations that will affect their effectiveness and efficiency. In particular, their inability in handling nonlinear relationships between financial variables and stock prices has been a major shortcoming. However, with the significant advancement in the field of ANNs, these limitations have found a solution. In this work, the generalization ability of the ANN is being harnessed in creating an effective and efficient tool for stock selection. Results of the research in this field have so far been very encouraging. Application of Neural Network in Stock Investment One of the earliest studies was by Halquist and Schmoll (1989), who used a neural network model to predict trends in the S&P 500 index. They found that the model was able to predict the trends 61% of the time. This was followed by Trippi and DeSieno (1992) and Grudnitski and Osburn (1993). Trippi and DeSieno (1992) devised an S&P 500 trading system that consisted of several trained neural networks and a set of rules for combining the network results to generate a composite recommended trading strategy. The trading system was used to predict S&P 500 index futures and the results showed that this system significantly outperformed the passive buy-and-hold strategy. Grudnitski and Osburn (1993) used a neural network to predict the monthly price changes and trading return in the S&P 500 index futures. The results showed that the neural network was able to predict correctly 75% of the time and gave a positive return above risk. Another work on predicting S&P 500 index futures was by Tsaih, Hsu, and Lai (1998). Similar to Trippi and DeSieno (1992), Tsaih et al. (1998) also integrated a rule-based system technique with a neural network to produce a trading system. However, in the Tsaih et al. (1998) study, they used reasoning neural networks instead of the backpropagation method used by Trippi and Desieno (1992). Empirical results in the daily prediction of price changes in the S&P 500 index futures showed that this hybrid artificialintelligence (AI) approach outperformed the passive buy-and-hold investment strategy. 11 more pages are available in the full version of this document, which may be purchased using the "Add to Cart" button on the publisher's webpage: www.igi-global.com/chapter/improving-returns-stockinvestment-through/5354

In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarizes relevant work, much of it from the previous millennium. Shallow and Deep Learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.

This issue of International Journal of Intelligent Systems includes extended versions of selected papers from the 4th International Conference on Soft Computing, held in Iizuka, Japan, September 30]October 5, 1996. The topic of the special issue is ‘‘Hybrid Intelligent Adaptive Systems.’’ Research on hybrid systems is one of the key issues of developing intelligent systems and it can apply a wide range of tools, including artificial neural networks, fuzzy logic, knowledge-based systems, genetic algorithms, evolutionary computation, and chaos models. The papers in this issue have been carefully reviewed and modified to give the readers a comprehensive overview of theoretical aspects, design, and implementation issues of hybrid intelligent adaptive systems. In the first paper by Kasabov and Kozma, a general framework of developing hybrid, intelligent, and adaptive systems is given. This work develops multimodular, fuzzy neural network systems and applies it to phoneme-based speech recognition. The second paper by Miyata, Furuhashi, and Uchikawa proposes fuzzy abductive inference with degrees of manifestations. This method infers irredundant combinations of candidates with degrees of belief for the manifestations. It is also demonstrated that the results of the inference method are applicable to medical diagnosis and system fault detection. Cho adopts ideas of artificial life in his work to develop evolutionary neural networks. The introduced modular neural network can evole its structure autonomously using a structural genetic code. The effectiveness of the method is demonstrated on the example of handwritten digit recognition. Feuring and Lippe study theoretical aspects of fuzzy neural networks. They propose a training algorithm for fuzzy neural networks that satisfies a certain goodness criterion. The second part of the special issue contains articles related to time series analysis and systems control. Yamazaki, Kang, and Ochiai introduce a hierarchical neural network system for adaptive, intelligent control, based on the analogy with the human thinking process. The optimum parameter space of the neural network system is found by a self-controllable algorithm, which can lead to either equilibrium or to nonequilibrium, chaotic behavior. The results of this study are applied, e.g., to laser beam analysis, semiconductor design, and design of magnetic devices. The work by Kozma, Kasabov, Kim, and Cohen presents a chaotic neuro-fuzzy method for time series analysis and process control. The

Artificial Neural Networks (ANNs) have been applied to predict many complex problems. In this paper ANNs are applied to horse racing prediction. We employed Back-Propagation, Back-Propagation with Momentum, QuasiNewton, Levenberg-Marquardt and Conjugate Gradient Descent learning algorithms for real horse racing data and the performances of five supervised NN algorithms were analyzed. Data collected from AQUEDUCT Race Track in NY, include 100 actual races from 1 January to 29 January 2010. The experimental results demonstrate that NNs are appropriate methods in horse racing prediction context. Results show that BP algorithm performs slightly better than other algorithms but it needs a longer training time and more parameter selection. Furthermore, LM algorithm is the fastest. Key-Words: Artificial Neural Networks, Time Series Analysis, Horse Racing Prediction, Learning Algorithms, BackPropagation

We propose a quantum generalisation of a classical neural network. The classical neurons are firstly rendered reversible by adding ancillary bits. Then they are generalised to being quantum reversible, i.e. unitary. (The classical networks we generalise are called feedforward, and have step-function activation functions.) The quantum network can be trained efficiently using gradient descent on a cost function to perform quantum generalisations of classical tasks. We demonstrate numerically that it can: (i) compress quantum states onto a minimal number of qubits, creating a quantum autoencoder, and (ii) discover quantum communication protocols such as teleportation. Our general recipe is theoretical and implementation-independent. The quantum neuron module can naturally be implemented photonically.

Statistical process control (SPC) is a method for improving the quality o f products. Control charting plays a most important role in SPC. SPC control charts arc used for monitoring and detecting unnatural process behaviour. Unnatural patterns in control charts indicate unnatural causes for variations. Control chart pattern recognition is therefore important in SPC. Past research shows that although certain types o f charts, such as the CUSUM chart, might have powerful detection ability, they lack robustness and do not function automatically. In recent years, neural network techniques have been applied to automatic pattern recognition. Spiking Neural Networks (SNNs) belong to the third generation o f artificial neural networks, with spiking neurons as processing elements. In SNNs, time is an important feature for information representation and processing. This thesis proposes the application o f SNN techniques to control chart pattern recognition. It is designed to present an analysis o f the existing learning algorithms o f SNN for pattern recognition and to explain how and why spiking neurons have more computational power in comparison to the previous generation o f neural networks. This thesis focuses on the architecture and the learning procedure o f the network. Four new learning algorithms are presented with their specific architecture: Spiking Learning Vector Quantisation (S-LVQ), Enhanced-Spiking Learning Vector Quantisation (NS-LVQ), S-LVQ with Bees and NS-LVQ with Bees. The latter two algorithms employ a new intelligent swarm-based optimisation called the Bees Algorithm to optimise the LVQ pattern recognition networks. Overall, the aim o f the research is to develop a simple architecture for the proposed network as well as to develop a network that is efficient for application to control chart pattern recognition. Experiments show that the proposed architecture and the learning procedure give high pattern recognition accuracies.


Question: EvoNN: a customizable evolutionary neural network incorporating heterogeneous activation functions.

This paper delves into the pioneering technique of Genetic Programming (GP), a method for automatically programming computers using the principles of natural selection. Inspired by the evolutionary mechanisms found in nature, GP leverages genetic algorithms to create and optimize computer programs. The work, authored by John R. Koza, introduces a paradigm shift in how we think about programming, moving away from traditional, human-designed algorithms towards a system where programs evolve through a process of selection, crossover, and mutation. The goal of GP is to generate solutions to complex problems that can adapt and evolve over time, much like organisms in the natural world. This approach has been applied across various domains, from control systems to artificial intelligence, demonstrating its versatility and potential for innovation. GP represents a significant step towards the development of autonomous systems that can learn and evolve without human intervention, akin to the complex adaptive systems found in nature. By harnessing the power of natural selection, GP opens up new possibilities for the creation of intelligent, self-evolving software that can tackle problems in ways that traditional programming cannot.

The Achilles Heel of stochastic optimization algorithms is getting trapped on local optima. Novelty Search avoids this problem by encouraging a search in all interesting directions. That occurs by replacing a performance objective with a reward for novel behaviors, as defined by a human-crafted, and often simple, behavioral distance function. While Novelty Search is a major conceptual breakthrough and outperforms traditional stochastic optimization on certain problems, it is not clear how to apply it to challenging, high-dimensional problems where specifying a useful behavioral distance function is difficult. For example, in the space of images, how do you encourage novelty to produce hawks and heroes instead of endless pixel static? Here we propose a new algorithm, the Innovation Engine, that builds on Novelty Search by replacing the human-crafted behavioral distance with a Deep Neural Network (DNN) that can recognize interesting differences between phenotypes. The key insight is that DNNs can recognize similarities and differences between phenotypes at an abstract level, wherein novelty means interesting novelty. For example, a novelty pressure in image space does not explore in the low-level pixel space, but instead creates a pressure to create new types of images (e.g. churches, mosques, obelisks, etc.). Here we describe the long-term vision for the Innovation Engine algorithm, which involves many technical challenges that remain to be solved. We then implement a simplified version of the algorithm that enables us to explore some of the algorithm's key motivations. Our initial results, in the domain of images, suggest that Innovation Engines could ultimately automate the production of endless streams of interesting solutions in any domain: e.g. producing intelligent software, robot controllers, optimized physical components, and art.

NEVESIM is a software package for event-driven simulation of networks of spiking neurons with a fast simulation core in C++, and a scripting user interface in the Python programming language. It supports simulation of heterogeneous networks with different types of neurons and synapses, and can be easily extended by the user with new neuron and synapse types. To enable heterogeneous networks and extensibility, NEVESIM is designed to decouple the simulation logic of communicating events (spikes) between the neurons at a network level from the implementation of the internal dynamics of individual neurons. In this paper we will present the simulation framework of NEVESIM, its concepts and features, as well as some aspects of the object-oriented design approaches and simulation strategies that were utilized to efficiently implement the concepts and functionalities of the framework. We will also give an overview of the Python user interface, its basic commands and constructs, and also discuss the benefits of integrating NEVESIM with Python. One of the valuable capabilities of the simulator is to simulate exactly and efficiently networks of stochastic spiking neurons from the recently developed theoretical framework of neural sampling. This functionality was implemented as an extension on top of the basic NEVESIM framework. Altogether, the intended purpose of the NEVESIM framework is to provide a basis for further extensions that support simulation of various neural network models incorporating different neuron and synapse types that can potentially also use different simulation strategies.

General Matrix to Matrix multiplication (GEMM) is the cornerstone for a wide gamut of applications in high performance computing (HPC), scientific computing (SC) and more recently, deep learning. In this work, we present a customizable matrix multiplication framework for the Intel HARPv2 CPU+FPGA platform that includes support for both traditional single precision floating point and reduced precision workloads. Our framework supports arbitrary size GEMMs and consists of two parts: (1) a simple application programming interface (API) for easy configuration and integration into existing software and (2) a highly customizable hardware template. The API provides both compile and runtime options for controlling key aspects of the hardware template including dynamic precision switching; interleaving and block size control; and fused deep learning specific operations. The framework currently supports single precision floating point (FP32), 16, 8, 4 and 2 bit Integer and Fixed Point (INT16, INT8, INT4, INT2) and more exotic data types for deep learning workloads: INT16xTernary, INT8xTernary, BinaryxBinary.
 We compare our implementation to the latest NVIDIA Pascal GPU and evaluate the performance benefits provided by optimizations built into the hardware template. Using three neural networks (AlexNet, VGGNet and ResNet) we illustrate that reduced precision representations such as binary achieve the best performance, and that the HARPv2 enables fine-grained partitioning of computations over both the Xeon and FPGA. We observe up to 50x improvement in execution time compared to single precision floating point, and that runtime configuration options can improve the efficiency of certain layers in AlexNet up to 4x, achieving an overall 1.3x improvement over the entire network.

Pattern recognition is one of the traditional uses of neural networks. When trained with gradient-based learning methods, these networks can learn the classification of input data by example. An introduction to classifiers and gradient-based learning is given. It is shown how several perceptrons can be combined and trained gradient-based. Furthermore, an overview of convolutional neural networks, as well as a real-world example, are discussed.

In this paper, a new representation of neural tensor networks is presented. Recently, state-of-the-art neural tensor networks have been introduced to complete RDF knowledge bases. However, mathematical model representation of these networks is still a challenging problem, due to tensor parameters. To solve this problem, it is proposed that these networks can be represented as two-layer perceptron network. To complete the network topology, the traditional gradient based learning rule is then developed. It should be mentioned that for tensor networks there have been developed some learning rules which are complex in nature due to the complexity of the objective function used. Indeed, this paper is aimed to show that the tensor network can be viewed and represented by the two-layer feedforward neural network in its traditional form. The simulation results presented in the paper easily verify this claim.

The Self-Organizing Incremental Neural Network (SOINN) is an unsupervised classifier that is capable of online incremental learning. Studies have been performed not only for improving the SOINN, but also for applying it to various problems. Furthermore, using the SOINN, more intelligent functions are achieved, such as association, reasoning, and so on. In this paper, we show how to use the SOINN software and to apply it to the above problems.

-We give conditions ensuring that multilayer jeedJorward networks with as Jew as a single hidden layer and an appropriately smooth hidden layer activation fimction are capable o f arbitrarily accurate approximation to an arbitrao' function and its derivatives. In fact, these networks can approximate functions that are not dtifferentiable in the classical sense, but possess only a generalized derivative, as is the case jor certain piecewise dtlf[erentiable Junctions. The conditions imposed on the hidden layer activation function are relatively mild: the conditions imposed on the domain o f the fimction to be approximated have practical intplications. Our approximation results provide a previously missing theoretical justification jor the use of multilayer /~'ed/brward networks in applications requiring simultaneous approximation o f a function and its derivatives. Keywords--Approximation, Derivatives, Sobolev space, Feedforward networks. 1. I N T R O D U C T I O N The capability of sufficiently complex multilayer feedforward networks to approximate an unknown mapping./~. R' ----~ R arbitrarily well has been recently investigated by Cybenko (1989), Funahashi (1989), Hecht-Nielsen (1989), Hornik, Stinchcombe, and White (1989) (HSW) (all for sigmoid hidden layer activation functions) and Stinchcombe and White (1989) (SW) (non-sigmoid hidden layer activation functions). In applications, it may be desirable to approximate not only the unknown mapping, but also its unknown derivatives. This is the case in Jordan's (1989) recent investigation of robot learning of smooth movement . Jordan states: The Jacobian matrix 8z/Ox . . . is the matrix that relates small changes in the controller output to small changes in the task space results and cannot be assumed to be available a priori, or provided by the environment. However, all of the derivatives in the matrix are forward derivatives. They are easily obtained by differentiation if a forward model is available. The forward model itself must be learned, but this can be achieved directly by system idenAcknowledgements: We arc indebted to Angelo Melino for pressing us on the issue addressed here and to the referees for numerous helpful suggestions. White's participation was supported by NSF Grant SES-8806990. Requests for reprints should be sent to Halbert White, Department of Economics, D-008, University of California, San Diego. La Jolla, CA 92093. tification. Once the model is accurate over a particular domain, its derivatives provide a learning operator that allows the system to convert errors in task space into errors in articulatory space and thereby change the controller. Thus, learning an adequate approximation to the Jacobian matrix of an unknown mapping is a key component of Jordan 's approach to robot learning of smooth movement . Despite the success of Jordan 's experiments, there is no existing theoretical guarantee that multilayer feedforward networks generally have the capability to approximate an unknown mapping and its derivatives simultaneously. For example, a network with hard limiting hidden layer activations approximates unknown mappings with a piecewise-constant function, the first derivatives of which exist and are zero almost everywhere. Obviously, the derivatives of such a network output function cannot approximate the derivatives of an arbitrary function. Intuition suggests that networks having smooth hidden layer activation functions ought to have output function derivatives that will approximate the derivatives of an unknown mapping. However , the justification for this intuition is not obvious. Consider the class of single hidden layer feedforward networks having network output functions belonging to the set 2. "(G) -~ {g : x"-~ R[ g(x) : /:,G(_i';,,):

The optimization of deep neural networks can be more challenging than traditional convex optimization problems due to the highly non-convex nature of the loss function, e.g. it can involve pathological landscapes such as saddle-surfaces that can be difficult to escape for algorithms based on simple gradient descent. In this paper, we attack the problem of optimization of highly non-convex neural networks by starting with a smoothed – or mollified – objective function which becomes more complex as the training proceeds. Our proposition is inspired by the recent studies in continuation methods: similar to curriculum methods, we begin learning an easier (possibly convex) objective function and let it evolve during the training, until it eventually goes back to being the original, difficult to optimize, objective function. The complexity of the mollified networks is controlled by a single hyperparameter which is annealed during the training. We show improvements on various difficult optimization tasks and establish a relationship between recent works on continuation methods for neural networks and mollifiers.

1 A neural network is a massively parallel distributed processor that has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects (Haykin 1998): 1. Knowledge is acquired by the network through a learning process. 2. Interconnection strengths known as synaptic weights are used to store the knowledge. Basically, learning is a process by which the free parameters (i.e., synaptic weights and bias levels) of a neural network are adapted through a continuing process of stimulation by the environment in which the network is embedded. The type of learning is determined by the manner in which the parameter changes take place. In a general sense, the learning process may be classified as follows:

OBJECTIVE
Brain-computer interfaces (BCI) enable direct communication with a computer, using neural activity as the control signal. This neural signal is generally chosen from a variety of well-studied electroencephalogram (EEG) signals. For a given BCI paradigm, feature extractors and classifiers are tailored to the distinct characteristics of its expected EEG control signal, limiting its application to that specific signal. Convolutional neural networks (CNNs), which have been used in computer vision and speech recognition to perform automatic feature extraction and classification, have successfully been applied to EEG-based BCIs; however, they have mainly been applied to single BCI paradigms and thus it remains unclear how these architectures generalize to other paradigms. Here, we ask if we can design a single CNN architecture to accurately classify EEG signals from different BCI paradigms, while simultaneously being as compact as possible.

APPROACH
In this work we introduce EEGNet, a compact convolutional neural network for EEG-based BCIs. We introduce the use of depthwise and separable convolutions to construct an EEG-specific model which encapsulates well-known EEG feature extraction concepts for BCI. We compare EEGNet, both for within-subject and cross-subject classification, to current state-of-the-art approaches across four BCI paradigms: P300 visual-evoked potentials, error-related negativity responses (ERN), movement-related cortical potentials (MRCP), and sensory motor rhythms (SMR).

MAIN RESULTS
We show that EEGNet generalizes across paradigms better than, and achieves comparably high performance to, the reference algorithms when only limited training data is available across all tested paradigms. In addition, we demonstrate three different approaches to visualize the contents of a trained EEGNet model to enable interpretation of the learned features.

SIGNIFICANCE
Our results suggest that EEGNet is robust enough to learn a wide variety of interpretable features over a range of BCI tasks. Our models can be found at: https://github.com/vlawhern/arl-eegmodels.


Question: EvoNN: a customizable evolutionary neural network featuring heterogeneous activation functions.

This comprehensive review explores the diverse landscape of neural transfer functions, highlighting their pivotal role in shaping the complexity and performance of neural networks. While sigmoidal functions have historically dominated the field, there's no inherent guarantee that they always yield the most effective decision boundaries. An extensive taxonomy of activation and output functions is outlined, alongside an in-depth analysis of the merits of both local and non-local neural transfer functions. The review delves into lesser-known transfer function variants and innovative combinations of activation/output functions. Among these, universal transfer functions stand out due to their parametrizable nature, allowing a seamless transition from localized to delocalized characteristics. Additional neural transfer functions covered include those based on non-Euclidean distance measures, bicentral functions derived from the product or linear combinations of sigmoid pairs, and modifications that enable the practical rotation of localized decision boundaries in high-dimensional spaces. The review also touches upon nonlinear input preprocessing techniques, presenting an alternative method to alter decision boundary shapes, thus enriching the design space of neural network architectures.

A new heuristic approach for minimizing possibly nonlinear and non differentiable continuous space functions is presented. By means of an extensive testbed, which includes the De Jong functions, it will be demonstrated that the new method converges faster and with more certainty than Adaptive Simulated Annealing as well as the Annealed Nelder&Mead approach, both of which have a reputation for being very powerful. The new method requires few control variables, is robust, easy to use and lends itself very well to parallel computation. ________________________________________ 1)International Computer Science Institute, 1947 Center Street, Berkeley, CA 94704-1198, Suite 600, Fax: 510-643-7684. E-mail: storn@icsi.berkeley.edu. On leave from Siemens AG, ZFE T SN 2, OttoHahn-Ring 6, D-81739 Muenchen, Germany. Fax: 01149-636-44577, Email:rainer.storn@zfe.siemens.de. 2)836 Owl Circle, Vacaville, CA 95687, kprice@solano.community.net.

The back propagation neural network (BPNN) can easily fall into the local minimum point in time series forecasting. A hybrid approach that combines the adaptive differential evolution (ADE) algorithm with BPNN, called ADE–BPNN, is designed to improve the forecasting accuracy of BPNN. ADE is first applied to search for the global initial connection weights and thresholds of BPNN. Then, BPNN is employed to thoroughly search for the optimal weights and thresholds. Two comparative real-life series data sets are used to verify the feasibility and effectiveness of the hybrid method. The proposed ADE–BPNN can effectively improve forecasting accuracy relative to basic BPNN, autoregressive integrated moving average model (ARIMA), and other hybrid models. 2014 Elsevier Ltd. All rights reserved.

Convolutional neural networks (CNNs) have gained great success in various computer vision applications. However, training a CNN model is computation-intensive and time-consuming. Hence training is mainly processed on large clusters of high-performance processors like server CPUs and GPUs. In this paper, we propose an FPGA-based processor design to accelerate the training process of CNNs. We first analyze the operations in all types of CNN layers in the training process. A uniform computation engine design is proposed to efficiently carry out all kinds of operations based on the analysis. Then a scalable accelerator framework is presented that exploits the parallelism further by unrolling the loops in two levels. The proposed accelerator design is demonstrated by implementing a processor on the Xilinx ZU19EG FPGA working at 200 MHz. The evaluation results on a group of CNN models show that our processor is 5.7 to 10.7-fold faster than the software implementations on the Intel Core i5-4440 CPU(@3.10GHz).

This tutorial is a worked-out version of a 5-hour course originally held at AIS in September/October 2002. It has two distinct components. First, it contains a mathematically-oriented crash course on traditional training methods for recurrent neural networks, covering back-propagation through time (BPTT), real-time recurrent learning (RTRL), and extended Kalman filtering approaches (EKF). This material is covered in Sections 2 – 5. The remaining sections 1 and 6 – 9 are much more gentle, more detailed, and illustrated with simple examples. They are intended to be useful as a stand-alone tutorial for the echo state network (ESN) approach to recurrent neural network training.

The attempts to segment medical ultrasound images have had limited success than the attempts to segment images from other medical imaging modalities. In this project, we attempt to segment medical ultrasound images using convolutional neural networks (CNNs) with a group of noisy activation functions which have recently been demonstrated to improve the performance of neural networks. We report on the segmentation results using a U-Net-like CNN with noisy rectified linear unit (NReLU) functions, noisy hard sigmoid (NHSigmoid) functions, and noisy hard tanh (NHTanh) function on a small data set.

Evolutionary Learning proceeds by evolving a population of classifiers, from which it generally returns (with some notable exceptions) the single best-of-run classifier as final result. In the meanwhile, Ensemble Learning, one of the most efficient approaches in supervised Machine Learning for the last decade, proceeds by building a population of diverse classifiers. Ensemble Learning with Evolutionary Computation thus receives increasing attention. The Evolutionary Ensemble Learning (EEL) approach presented in this paper features two contributions. First, a new fitness function, inspired by co-evolution and enforcing the classifier diversity, is presented. Further, a new selection criterion based on the classification margin is proposed. This criterion is used to extract the classifier ensemble from the final population only (Off-EEL) or incrementally along evolution (On-EEL). Experiments on a set of benchmark problems show that Off-EEL outperforms single-hypothesis evolutionary learning and state-of-art Boosting and generates smaller classifier ensembles.

Multilayer perceptron (MLP) (trained with back propagation learning algorithm) takes large computational time. The complexity of the network increases as the number of layers and number of nodes in layers increases. Further, it is also very difficult to decide the number of nodes in a layer and the number of layers in the network required for solving a problem a priori. In this paper an improved particle swarm optimization (IPSO) is used to train the functional link artificial neural network (FLANN) for classification and we name it ISO-FLANN. In contrast to MLP, FLANN has less architectural complexity, easier to train, and more insight may be gained in the classification problem. Further, we rely on global classification ata mining unctional link artificial neural networks ulti-layer perception article swarm optimization mproved particle swarm optimization VM capabilities of IPSO to explore the entire weight space, which is plagued by a host of local optima. Using the functionally expanded features; FLANN overcomes the non-linear nature of problems. We believe that the combined efforts of FLANN and IPSO (IPSO + FLANN = ISO − FLANN) by harnessing their best attributes can give rise to a robust classifier. An extensive simulation study is presented to show the effectiveness of proposed classifier. Results are compared with MLP, support vector machine(SVM) with radial basis function (RBF) kernel, FLANN with gradiend descent learning and fuzzy swarm net (FSN). SN

In biomimetic engineering, we may take inspiration from the products of biological evolution: we may instantiate biologically realistic neural architectures and algorithms in robots, or we may construct robots with morphologies that are found in nature. Alternatively, we may take inspiration from the process of evolution: we may evolve populations of robots in simulation and then manufacture physical versions of the most interesting or more capable robots that evolve. If we follow this latter approach and evolve both the neural and morphological subsystems of machines, we can perform controlled experiments that provide unique insight into how bodies and brains can work together to produce adaptive behavior, regardless of whether such bodies and brains are instantiated in a biological or technological substrate. In this paper, we review selected projects that use such methods to investigate the synergies and tradeoffs between neural architecture, morphology, action, and adaptive behavior.

Neuromorphic algorithms, which are comprised of highly complex, large-scale networks of artificial neurons, are increasingly used for a variety of recognition, classification, search and vision tasks. However, their computational and energy requirements can be quite high, and hence their energy-efficient implementation is of great interest.
 We propose a new approach to design energy-efficient hardware implementations of large-scale neural networks (NNs) using approximate computing. Our work is motivated by the observations that (i) NNs are used in applications where less-than-perfect results are acceptable, and often inevitable, and (ii) they are highly resilient to inexactness in many (but not all) of their constituent computations. We make two key contributions. First, we propose a method to transform any given NN into an Approximate Neural Network (AxNN). This is performed by (i) adapting the backpropagation technique, which is commonly used to train these networks, to quantify the impact of approximating each neuron to the overall network quality (e.g., classification accuracy), and (ii) selectively approximating those neurons that impact network quality the least. Further, we make the key observation that training is a naturally error-healing process that can be used to mitigate the impact of approximations to neurons. Therefore, we incrementally retrain the network with the approximations in-place, reclaiming a significant portion of the quality ceded by approximations. As a second contribution, we propose a programmable and quality-configurable neuromorphic processing engine (qcNPE), which utilizes arrays of specialized processing elements that execute neuron computations with dynamically configurable accuracies and can be used to execute AxNNs from diverse applications. We evaluated the proposed approach by constructing AXNNs for 6 recognition applications (ranging in complexity from 12-47,818 neurons and 160-3,155,968 connections) and executing them on two different platforms--qcNPE implementation containing 272 processing elements in 45nm technology and a commodity Intel Xeon server. Our results demonstrate 1.14X-1.92X energy benefits for virtually no loss (< 0.5%) in output quality, and even higher improvements (upto 2.3X) when some loss (upto 7.5%) in output quality is acceptable.

Many recent deep learning platforms rely on thirdparty libraries (such as cuBLAS) to utilize the computing power of modern hardware accelerators (such as GPUs). However, we observe that they may achieve suboptimal performance because the library functions are not used appropriately. In this paper, we target at optimizing the operations of multiplying a matrix with the transpose of another matrix (referred to as NT operation hereafter), which contribute half of the training time of fully connected deep neural networks. Rather than directly calling the library function, we propose a supervised learning based algorithm selection approach named MTNN, which uses a gradient boosted decision tree to select one from two alternative NT implementations intelligently: (1) calling the cuBLAS library function; (2) calling our proposed algorithm TNN that uses an efficient out-of-place matrix transpose. We evaluate the performance of MTNN on two modern GPUs: NVIDIA GTX 1080 and NVIDIA Titan X Pascal. MTNN can achieve 96% of prediction accuracy with very low computational overhead, which results in an average of 54% performance improvement on a range of NT operations. To further evaluate the impact of MTNN on the training process of deep neural networks, we have integrated MTNN into a popular deep learning platform Caffe. Our experimental results show that the revised Caffe can outperform the original one by an average of 28%. Both MTNN and the revised Caffe are open-source.


Question: EvoNN: a customizable evolutionary neural network featuring heterogeneous activation functions.

This paper presents a novel hybrid methodology, ADE-BPNN, that integrates the Adaptive Differential Evolution (ADE) algorithm with the Back Propagation Neural Network (BPNN) to enhance time series forecasting accuracy. BPNN, while powerful, is prone to getting trapped in local minima, which can significantly impact its forecasting capabilities. The ADE-BPNN approach is designed to address this limitation by leveraging ADE to determine the global initial connection weights and thresholds of the BPNN. Subsequently, the BPNN is utilized to conduct a detailed search for the optimal weights and thresholds. The efficacy of this hybrid method is demonstrated through its application to two real-life time series datasets. The results indicate that ADE-BPNN significantly outperforms not only the basic BPNN but also the Autoregressive Integrated Moving Average (ARIMA) model and other hybrid models in terms of forecasting accuracy. This research underscores the potential of combining evolutionary algorithms with neural networks to overcome the shortcomings of traditional forecasting methods and improve overall prediction performance.

In this paper, an algorithm based on the concepts of genetic algorithms that uses an estimation of a probability distribution of promising solutions in order to generate new candidate solutions is proposed. To estimate the distribution, techniques for modeling multivariate data by Bayesian networks are used. The proposed algorithm identifies, reproduces and mixes building blocks up to a specified order. It is independent of the ordering of the variables in the strings representing the solutions. Moreover, prior information about the problem can be incorporated into the algorithm. However, prior information is not essential. Preliminary experiments show that the BOA outperforms the simple genetic algorithm even on decomposable functions with tight building blocks as a problem size grows.

This paper presents a method to employ particle swarms optim izers in a cooperative configuration. This is achieved by splitting the input vector into several sub-vectors, each w hich is optimized cooperatively in its own swarm. The applic ation of this technique to neural network training is investigate d, with promising results.

Multilayer perceptron (MLP) (trained with back propagation learning algorithm) takes large computational time. The complexity of the network increases as the number of layers and number of nodes in layers increases. Further, it is also very difficult to decide the number of nodes in a layer and the number of layers in the network required for solving a problem a priori. In this paper an improved particle swarm optimization (IPSO) is used to train the functional link artificial neural network (FLANN) for classification and we name it ISO-FLANN. In contrast to MLP, FLANN has less architectural complexity, easier to train, and more insight may be gained in the classification problem. Further, we rely on global classification ata mining unctional link artificial neural networks ulti-layer perception article swarm optimization mproved particle swarm optimization VM capabilities of IPSO to explore the entire weight space, which is plagued by a host of local optima. Using the functionally expanded features; FLANN overcomes the non-linear nature of problems. We believe that the combined efforts of FLANN and IPSO (IPSO + FLANN = ISO − FLANN) by harnessing their best attributes can give rise to a robust classifier. An extensive simulation study is presented to show the effectiveness of proposed classifier. Results are compared with MLP, support vector machine(SVM) with radial basis function (RBF) kernel, FLANN with gradiend descent learning and fuzzy swarm net (FSN). SN

Many modern neural simulators focus on the simulation of networks of spiking neurons on parallel hardware. Another important framework in computational neuroscience, rate-coded neural networks, is mostly difficult or impossible to implement using these simulators. We present here the ANNarchy (Artificial Neural Networks architect) neural simulator, which allows to easily define and simulate rate-coded and spiking networks, as well as combinations of both. The interface in Python has been designed to be close to the PyNN interface, while the definition of neuron and synapse models can be specified using an equation-oriented mathematical description similar to the Brian neural simulator. This information is used to generate C++ code that will efficiently perform the simulation on the chosen parallel hardware (multi-core system or graphical processing unit). Several numerical methods are available to transform ordinary differential equations into an efficient C++code. We compare the parallel performance of the simulator to existing solutions.

Abstr,,ct. In this paper we demonstrate that finite linear combinations of compositions of a fixed, univariate function and a set ofaffine functionals can uniformly approximate any continuous function of n real variables with support in the unit hypercube; only mild conditions are imposed on the univariate function. Our results settle an open question about representability in the class of single bidden layer neural networks. In particular, we show that arbitrary decision regions can be arbitrarily well approximated by continuous feedforward neural networks with only a single internal, hidden layer and any continuous sigmoidal nonlinearity. The paper discusses approximation properties of other possible types of nonlinearities that might be implemented by artificial neural networks.

Quantum neural networks (QNNs) have been explored as one of the best approach for improving the computational efficiency of neural networks. Because of the powerful and fantastic performance of quantum computation, some researchers have begun considering the implications of quantum computation on the field of artificial neural networks (ANNs).The purpose of this paper is to introduce an application of QNNs in construction of cryptography system in which two networks exchange their outputs (in qubits) and the key to be synchronized between two communicating parties. This system is based on multilayer qubit QNNs trained with back-propagation algorithm.

Techniques for automatically designing deep neural network architectures such as reinforcement learning based approaches have recently shown promising results. However, their success is based on vast computational resources (e.g. hundreds of GPUs), making them difficult to be widely used. A noticeable limitation is that they still design and train each network from scratch during the exploration of the architecture space, which is highly inefficient. In this paper, we propose a new framework toward efficient architecture search by exploring the architecture space based on the current network and reusing its weights. We employ a reinforcement learning agent as the meta-controller, whose action is to grow the network depth or layer width with function-preserving transformations. As such, the previously validated networks can be reused for further exploration, thus saves a large amount of computational cost. We apply our method to explore the architecture space of the plain convolutional neural networks (no skip-connections, branching etc.) on image benchmark datasets (CIFAR-10, SVHN) with restricted computational resources (5 GPUs). Our method can design highly competitive networks that outperform existing networks using the same design scheme. On CIFAR-10, our model without skip-connections achieves 4.23% test error rate, exceeding a vast majority of modern architectures and approaching DenseNet. Furthermore, by applying our method to explore the DenseNet architecture space, we are able to achieve more accurate networks with fewer parameters.

In this paper we describe parallel implementation of ANN training procedure based on block mode back-propagation learning algorithm. Two different approaches to parallelization were implemented. The first is data parallelization using POSIX threads, it is suitable for multi-core computers. The second is node parallelization using high performance SIMD architecture of GPU with CUDA, suitable for CUDA enabled computers. We compare the speed-up of both approaches by learning typically-sized network on the real-world phoneme-state classification task, showing nearly 10 times reduction when using CUDA version, while the 8-core server with multi-thread version gives only 4 times reduction. In both cases we compared to an already BLAS optimized implementation. The training tool will be released as Open-Source software under project name TNet.

As a widely used non-linear activation, Rectified Linear Unit (ReLU) separates noise and signal in a feature map by learning a threshold or bias. However, we argue that the classification of noise and signal not only depends on the magnitude of responses, but also the context of how the feature responses would be used to detect more abstract patterns in higher layers. In order to output multiple response maps with magnitude in different ranges for a particular visual pattern, existing networks employing ReLU and its variants have to learn a large number of redundant filters. In this paper, we propose a multi-bias non-linear activation (MBA) layer to explore the information hidden in the magnitudes of responses. It is placed after the convolution layer to decouple the responses to a convolution kernel into multiple maps by multi-thresholding magnitudes, thus generating more patterns in the feature space at a low computational cost. It provides great flexibility of selecting responses to different visual patterns in different magnitude ranges to form rich representations in higher layers. Such a simple and yet effective scheme achieves the stateof-the-art performance on several benchmarks.

In this work we study the problem of network morphism, an effective learning scheme to morph a well-trained neural network to a new one with the network function completely preserved. Different from existing work where basic morphing types on the layer level were addressed, we target at the central problem of network morphism at a higher level, i.e., how a convolutional layer can be morphed into an arbitrary module of a neural network. To simplify the representation of a network, we abstract a module as a graph with blobs as vertices and convolutional layers as edges, based on which the morphing process is able to be formulated as a graph transformation problem. Two atomic morphing operations are introduced to compose the graphs, based on which modules are classified into two families, i.e., simple morphable modules and complex modules. We present practical morphing solutions for both of these two families, and prove that any reasonable module can be morphed from a single convolutional layer. Extensive experiments have been conducted based on the state-of-the-art ResNet on benchmark datasets, and the effectiveness of the proposed solution has been verified.