Question: Jack the Reader - A framework for machine reading comprehension.

Introducing SQuAD: A Large-Scale Dataset for Machine Comprehension
This paper debuts the Stanford Question Answering Dataset (SQuAD), a significant contribution to the field of machine comprehension. Comprising over 100,000 questions, SQuAD is distinguished by its reliance on crowd-sourced questions based on Wikipedia articles, with each question's answer being a specific segment of text from the relevant article. An in-depth analysis of the dataset reveals the various reasoning skills required to answer these questions, with a focus on dependency and constituency trees. The paper presents a robust logistic regression model that attains an F1 score of 51.0%, marking a substantial leap from a rudimentary baseline (20%). Despite this achievement, human performance (86.8%) remains notably superior, underscoring the dataset's suitability as a benchmark for future research endeavors. SQuAD is freely accessible at https://stanford-qa.com, providing researchers with a valuable resource for advancing machine comprehension capabilities.

We present our winning solution for the WSDM Cup 2017 triple scoring task. We devise an ensemble of four base scorers, so as to leverage the power of both text and knowledge bases for that task. Then we further refine the outputs of the ensemble by trigger word detection, achieving even better predictive accuracy. The code is available at https://github.com/wsdm-cup-2017/bokchoy.

As part of the information fusion task we wish to automatically fuse information derived from the text extraction process with data from a structured knowledge base. This process will involve resolving, aggregating, integrating and abstracting information via the methodologies of Knowledge Representation and Reasoning into a single comprehensive description of an individual or event. This report surveys the key principles underlying research in the field of Knowledge Representation and Reasoning. It represents an initial step in deciding upon a Knowledge Representation and Reasoning system for our information fusion task. APPROVED FOR PUBLIC RELEASE

Most current chatbot engines are designed to reply to user utterances based on existing utterance-response (or Q-R)1 pairs. In this paper, we present DocChat, a novel information retrieval approach for chatbot engines that can leverage unstructured documents, instead of Q-R pairs, to respond to utterances. A learning to rank model with features designed at different levels of granularity is proposed to measure the relevance between utterances and responses directly. We evaluate our proposed approach in both English and Chinese: (i) For English, we evaluate DocChat on WikiQA and QASent, two answer sentence selection tasks, and compare it with state-of-the-art methods. Reasonable improvements and good adaptability are observed. (ii) For Chinese, we compare DocChat with XiaoIce2, a famous chitchat engine in China, and side-by-side evaluation shows that DocChat is a perfect complement for chatbot engines using Q-R pairs as main source of responses.

Machine reading using differentiable reasoning models has recently shown remarkable progress. In this context, End-to-End trainable Memory Networks (MemN2N) have demonstrated promising performance on simple natural language based reasoning tasks such as factual reasoning and basic deduction. However, other tasks, namely multi-fact questionanswering, positional reasoning or dialog related tasks, remain challenging particularly due to the necessity of more complex interactions between the memory and controller modules composing this family of models. In this paper, we introduce a novel end-to-end memory access regulation mechanism inspired by the current progress on the connection short-cutting principle in the field of computer vision. Concretely, we develop a Gated End-toEnd trainable Memory Network architecture (GMemN2N). From the machine learning perspective, this new capability is learned in an end-to-end fashion without the use of any additional supervision signal which is, as far as our knowledge goes, the first of its kind. Our experiments show significant improvements on the most challenging tasks in the 20 bAbI dataset, without the use of any domain knowledge. Then, we show improvements on the Dialog bAbI tasks including the real human-bot conversion-based Dialog State Tracking Challenge (DSTC-2) dataset. On these two datasets, our model sets the new state of the art. ∗work done as an Intern at XRCE

This paper describes how language is grounded by a comprehension system called Lucia within a robotic agent called Rosie that can manipulate objects and navigate indoors. The whole system is built within the Soar cognitive architecture and uses Embodied Construction Grammar (ECG) as a formalism for describing linguistic knowledge. Grounding is performed using knowledge from the grammar itself, from the linguistic context, from the agent's perception, and from an ontology of long-term knowledge about object categories and properties and actions the agent can perform. The paper also describes a benchmark corpus of 200 sentences in this domain, along with test versions of the world model and ontology, and gold-standard meanings for each of the sentences. The benchmark is contained in the supplemental materials.

In the 1980s, plot units were proposed as a conceptual knowledge structure for representing and summarizing narrative stories. Our research explores whether current NLP technology can be used to automatically produce plot unit representations for narrative text. We create a system called AESOP that exploits a variety of existing resources to identify affect states and applies “projection rules” to map the affect states onto the characters in a story. We also use corpus-based techniques to generate a new type of affect knowledge base: verbs that impart positive or negative states onto their patients (e.g., being eaten is an undesirable state, but being fed is a desirable state). We harvest these “patient polarity verbs” from a Web corpus using two techniques: co-occurrence with Evil/Kind Agent patterns, and bootstrapping over conjunctions of verbs. We evaluate the plot unit representations produced by our system on a small collection of Aesop’s fables.

Domain dependence of NLP systems is one of the major obstacles to their application in large-scale text analysis, also restricting the applicability of FrameNet semantic role labeling (SRL) systems. Yet, current FrameNet SRL systems are still only evaluated on a single in-domain test set. For the first time, we study the domain dependence of FrameNet SRL on a wide range of benchmark sets. We create a novel test set for FrameNet SRL based on user-generated web text and find that the major bottleneck for out-of-domain FrameNet SRL is the frame identification step. To address this problem, we develop a simple, yet efficient system based on distributed word representations. Our system closely approaches the state-of-the-art in-domain while outperforming the best available frame identification system out-of-domain. We publish our system and test data for research purposes.1

Our goal is to answer elementary-level science questions using knowledge extracted automatically from science textbooks, expressed in a subset of first-order logic. Given the incomplete and noisy nature of these automatically extracted rules, Markov Logic Networks (MLNs) seem a natural model to use, but the exact way of leveraging MLNs is by no means obvious. We investigate three ways of applying MLNs to our task. In the first, we simply use the extracted science rules directly as MLN clauses. Unlike typical MLN applications, our domain has long and complex rules, leading to an unmanageable number of groundings. We exploit the structure present in hard constraints to improve tractability, but the formulation remains ineffective. In the second approach, we instead interpret science rules as describing prototypical entities, thus mapping rules directly to grounded MLN assertions, whose constants are then clustered using existing entity resolution methods. This drastically simplifies the network, but still suffers from brittleness. Finally, our third approach, called Praline, uses MLNs to align the lexical elements as well as define and control how inference should be performed in this task. Our experiments, demonstrating a 15% accuracy boost and a 10x reduction in runtime, suggest that the flexibility and different inference semantics of Praline are a better fit for the natural language question answering task.

We describe theWHY2-ATLAS intelligent tutoring system for qualitative physics that interacts with students via natural language dialogue. We focus on the issue of analyzing and responding to multi-sentential explanations. We explore approaches for achieving a deeper understanding of these explanations and dialogue management approaches and strategies for providing appropriate feedback on them.

Comprehension is the goal of reading. However, students often encounter reading difficulties due to the lack of background knowledge and proper reading strategy. Unfortunately, print text provides very limited assistance to one’s reading comprehension through its static knowledge representations such as symbols, charts, and graphs. Integrating digital materials and reading strategy into paper-based reading activities may bring opportunities for learners to make meaning of the print material. In this study, QR codes were adopted in association with mobile technology to deliver supplementary materials and questions to support students’ reading. QR codes were printed on paper prints to provide direct access to digital materials and scaffolded questions. Smartphones were used to scan the printed QR codes to fetch predesigned digital resources and scaffolded questions over the Internet. A quasi-experiment was conducted to evaluate the effectiveness of direct access to the digital materials prepared by the instructor using QR codes and that of scaffolded questioning in improving students’ reading comprehension. The results suggested that direct access to digital resources using QR codes does not significantly influence students’ reading comprehension; however, the reading strategy of scaffolded questioning significantly improves students’ understanding about the text. The survey showed that most students agreed that the integrated print-and-digital-materialbased learning system benefits English reading comprehension but may not be as efficient as expected. The implications of the findings shed light on future improvement of the system. 2011 Elsevier Ltd. All rights reserved.


Question: Jack the Reader - A framework for machine reading.

This paper introduces the Unstructured Information Management Architecture (UIMA), a middleware architecture developed by IBM Research to address the challenges of processing vast volumes of unstructured information, particularly multilingual natural language text. UIMA was conceived in response to a growing demand for text and voice analytics in various commercial applications, including life sciences, e-commerce, technical support, advanced search, and intelligence sectors. With over 200 researchers spread across six global labs, IBM Research has a significant focus on Unstructured Information Management (UIM) technologies, particularly in Natural Language Processing (NLP). These researchers are involved in a wide range of activities, from natural language dialog and information retrieval to document classification, machine translation, bioinformatics, and open-domain question answering.

UIMA aims to enhance the organization's capability to quickly discover and combine different technologies and approaches, thereby accelerating scientific progress. It provides a robust software framework and a common architecture that facilitates the reuse and integration of research results, particularly in the context of NLP. This is crucial for accelerating the transfer of research outcomes into IBM's product platforms.

At the core of UIMA are powerful search capabilities and a data-driven framework designed for the development, composition, and distributed deployment of analysis engines. This framework enables the analysis of large volumes of unstructured information while consulting structured information sources to uncover, organize, and present knowledge relevant to the application's end-user. Structured information is defined as data whose intended meaning is unambiguous and explicitly represented in its structure or format, akin to a relational database table. Conversely, unstructured information is data whose intended meaning is only loosely implied by its form, such as natural language documents, voice, audio, image, and video.

UIMA's design is particularly beneficial in a large research organization like IBM, where the efficient leveraging and integration of UIM assets is essential. The architecture addresses challenges such as organizational structure, skills alignment, development inefficiencies, and speed to product. By providing a common framework and engineering discipline, UIMA facilitates technology reuse and integration, enables quicker scientific experimentation, and speeds up the path to product.

The primary objectives of UIMA are to accelerate scientific advances by enabling the rapid combination of UIM technologies and to expedite the transfer of these technologies to product by providing an architecture that promotes reuse and supports flexible deployment options. This includes the development of middleware frameworks that offer a component-based infrastructure suitable for various deployment environments, ranging from lightweight and embeddable implementations to highly scalable solutions that can exploit clusters of machines for high throughput and high availability service-based offerings.

Reiter and Dale’s book is about natural language generation (NLG), and describes methods for building systems of this kind. It offers a comprehensive overview of the field of NLG, from a perspective that has rarely been described elsewhere. It is very valuable for people with little or even no knowledge about NLG particulars who intend to investigate applications where NLG technology might play a certain role, as well as for people who want to obtain an application-oriented overview of the field. In this sense, all communities intended to be served by the book—students, academics in related areas, and software developers—are well addressed. In addition, some sections provide material that is probably new even to researchers with considerable experience in the field, such as the discussions on corpus analysis and domain modeling. In the introduction, the field of NLG is briefly characterized from researchand application-oriented perspectives and is illustrated by screen shots produced by several systems. Then conditions for effective uses of this technology are elaborated and contrasted with conditions where other techniques are more appropriate, and methods for determining the intended functionality of a system to be built are discussed. The main sections of the book are devoted to the prototypical architecture of applicationoriented NLG systems and their major processing phases: document planning, microplanning, and surface realization. Each of these three phases is illustrated by a number of detailed examples, demonstrating the successive refinements of utterance specifications in the course of processing. In the final section, the embedding of natural language processing technology is discussed, featuring typography, combined uses with graphics and hypertext, and integration with speech. The methods are illustrated by a large number of examples—the book contains more than 120 figures on its 248 pages. At the end of each section, a number of useful references for further reading are related to the section topics. In the appendix, a table summarizing the 35 systems referred to in the book is given. In general, the presentations in the book are beneficial for their high degree of detail in documenting representations. Running examples are illustrated by intermediate representations at various stages of processing. They demonstrate very well the increasing degrees of precision in which utterances are specified and the associated commitments about possible text portions that express these specifications. Particularly valuable are the comparisons of alternative representations, ranging from skeletal propositions over lexicalized case frames to canned text, including a comparison of the consequences for the distribution of work over the system components involved and discussions of the associated pros and cons. These chapters provide a number of valuable hints, given the currently low degree of standardization in the field. Further important aspects addressed, which have been widely ignored in the literature so far,

We report on research in multi-document summarization and on evaluation of summarization in the framework of cross-lingual information retrieval. This work was carried out during a summer workshop on Language Engineering held at Johns Hopkins University by a team of nine researchers from seven universities. The goals of the research were as follows: (1) to develop a toolkit for evaluation of single-document and multi-document summarizers, (2) to develop a modular multi-document summarizer, called MEAD, that works in both English and Chinese, and (3) to perform a meta-evaluation of four automatic summarizers, including MEAD, using several types of evaluation measures: some currently used by summarization researchers and a couple of novel techniques. Central to the experiments in this workshop was the cross-lingual experimental setup based on a large-scale Chinese and English parallel corpus. An extensive set of human judgments were specifically prepared by the Linguistic Data Consortium for our research. These human judgments include a) which documents are relevant to a certain query and b) which sentences in the relevant documents are most relevant to the query and which therefore constitute a good summary of the cluster. These judgments were used to construct variable-length multiand single document summaries as model summaries. Since one of the novel evaluation metrics that we used, Relevance Correlation, is based on the premise that good summaries preserve query relevance both within a language and across languages, we made use of a cross-lingual Information Retrieval (IR) engine. We evaluated the quality of the automatic summaries using co-selection and content-based evaluation, two established techniques. A relatively new metric, relative utility, was also extensively tested. Part of the new scientific contribution is the measurement of relevance correlation, which we introduced and systematically examined in this workshop. Relevance correlation measures the quality of summaries in comparison to the entire documents as a function of how much document relevance drops if summaries are indexed instead of documents. Our results show that this measure is sensible, in that it correlates with more established evaluation measures. Another contribution is the cross-lingual setup which allows us to automatically translate English queries into Chinese, perform Chinese IR with or without summarization. This allows us to calculate relevance correlation for English and for Chinese in parallel (i.e., for the same queries) and to take direct cross-lingual comparisons of evaluations. Additionally, an alternative way of constructing Chinese model summaries from English ones was implemented which relies on the sentence alignment of English and Chinese documents. The results of our large-scale meta-evaluation are numerous, but some of the highlights are the following: (1) All evaluation measures rank human summaries first, which is an appropriate and expected property of such measures, (2) Both relevance correlation and the content-based measures place leading sentence extracts ahead of the more sophisticated summarizers, (3) Relative utility ranks our system, MEAD, as the best summarizer for shorter summaries, although for longer summaries, lead-based summaries outperform MEAD, (4) Co-selection measurements show overall low agreement amongst humans (above chance), whereas relative utility reports higher numbers on the same data (but does not normalize for chance). The deliverable resources and software include: (1) a turn-key extractive multi-document summarizer, MEAD, which allows users to add their own features based on single sentences or pairs of sentences, (2) a large corpus of summaries produced by several automatic methods, including baseline and random summaries, (3) a collection of manual summaries produced by the Linguistic Data Consortium (LDC), (4) a battery of evaluation routines, (5) a collection of IR queries in English and Chinese and the corresponding relevance judgments from the Hong Kong news collection, (6) SMART relevance outputs for both full documents and summaries, (7) XML tools for processing of documents and summaries. JHU 2001 Summer workshop final report Evaluation of Text Summarization

Figures are very important non-textual information contained in scientific documents. Current digital libraries do not provide users tools to retrieve documents based on the information available within the figures. We propose an architecture for retrieving documents by integrating figures and other information. The initial step in enabling integrated document search is to categorize figures into a set of pre-defined types. We propose several categories of figures based on their functionalities in scholarly articles. We have developed a machine-learning-based approach for automatic categorization of figures. Both global features, such as texture, and part features, such as lines, are utilized in the architecture for discriminating among figure categories. The proposed approach has been evaluated on a testbed document set collected from the CiteSeer scientific literature digital library. Experimental evaluation has demonstrated that our algorithms can produce acceptable results for realworld use. Our tools will be integrated into a scientific document digital library.

Inquiry is fundamental to communication, and machines cannot effectively collaborate with humans unless they can ask questions. In this thesis work, we explore how can we teach machines to ask clarification questions when faced with uncertainty, a goal of increasing importance in today’s automated society. We do a preliminary study using data from StackExchange, a plentiful online resource where people routinely ask clarifying questions to posts so that they can better offer assistance to the original poster. We build neural network models inspired by the idea of the expected value of perfect information: a good question is one whose expected answer is going to be most useful. To build generalizable systems, we propose two future research directions: a template-based model and a sequence-to-sequence based neural generative model.

Answering questions correctly from standardized eighth-grade science tests is itself a test of machine intelligence.

We consider the ROC story cloze task (Mostafazadeh et al., 2016) and present several findings. We develop a model that uses hierarchical recurrent networks with attention to encode the sentences in the story and score candidate endings. By discarding the large training set and only training on the validation set, we achieve an accuracy of 74.7%. Even when we discard the story plots (sentences before the ending) and only train to choose the better of two endings, we can still reach 72.5%. We then analyze this “ending-only” task setting. We estimate human accuracy to be 78% and find several types of clues that lead to this high accuracy, including those related to sentiment, negation, and general ending likelihood regardless of the story context.

We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com.

This paper presents a deep semantic similarity model (DSSM), a special type of deep neural networks designed for text analysis, for recommending target documents to be of interest to a user based on a source document that she is reading. We observe, identify, and detect naturally occurring signals of interestingness in click transitions on the Web between source and target documents, which we collect from commercial Web browser logs. The DSSM is trained on millions of Web transitions, and maps source-target document pairs to feature vectors in a latent space in such a way that the distance between source documents and their corresponding interesting targets in that space is minimized. The effectiveness of the DSSM is demonstrated using two interestingness tasks: automatic highlighting and contextual entity search. The results on large-scale, real-world datasets show that the semantics of documents are important for modeling interestingness and that the DSSM leads to significant quality improvement on both tasks, outperforming not only the classic document models that do not use semantics but also state-of-the-art topic models.

Worldwide, around 10% of the population has dyslexia, a specific learning disorder. Most of previous eye tracking experiments with people with and without dyslexia have found differences between populations suggesting that eye movements reflect the difficulties of individuals with dyslexia. In this paper, we present the first statistical model to predict readers with and without dyslexia using eye tracking measures. The model is trained and evaluated in a 10-fold cross experiment with a dataset composed of 1,135 readings of people with and without dyslexia that were recorded with an eye tracker. Our model, based on a Support Vector Machine binary classifier, reaches 80.18% accuracy using the most informative features. To the best of our knowledge, this is the first time that eye tracking measures are used to predict automatically readers with dyslexia using machine learning.

Text extraction is a crucial stage of analyzing Journal papers. Journal papers generally are in PDF format which is semi structured data. Journal papers are presented into different sections like Introduction, Methodology, Experimental setup, Result and analysis etc. so that it is easy to access information from any section as per the reader's interest. The main importance on section extraction is to find a representative subset of the data, which contains the information of the entire set. Various approaches to extract sections from research papers include stastical methods, NLP, Machine Learning etc. In this paper we present review of various extraction techniques from a PDF document.


Question: Jack the Reader - A framework for machine reading comprehension.

Freebase: An Open and Collaborative Repository of Structured Human Knowledge
Freebase is an innovative, scalable database designed to store structured general human knowledge, drawing inspiration from Semantic Web research and collaborative data projects like Wikipedia. This database is graph-shaped, enabling efficient storage and retrieval of interconnected information. Freebase offers unrestricted read and write access through an HTTP-based graph-query API, facilitating its use in research, the curation of structured data, and the development of applications. The database is openly accessible, and all content within Freebase is licensed under permissive terms, such as Creative Commons and GFDL, ensuring wide availability and reuse.

**Question**: An Overview of the Microsoft Bot Framework

**Documents**: The Microsoft Bot Framework: A Comprehensive Guide for Developers
The Microsoft Bot Framework is a comprehensive set of tools and services designed to help developers create, connect, and deploy bots that interact with users in a conversational manner. The framework includes the Bot Builder SDK, which offers a rich set of tools for building bots that can be integrated into various applications and services. It also includes the Bot Connector Service, which enables bots to communicate with users across multiple channels, such as Skype, Slack, and Facebook Messenger. Additionally, the Bot Framework provides the Bot State Emulator, a tool for testing and debugging bots, and the Bot Analytics service, which allows developers to monitor bot performance and user engagement. The framework supports multiple programming languages, including C# and Node.js, and provides a range of pre-built connectors for integrating bots with popular services and platforms.

**Similar Documents**: The Microsoft Bot Framework: A Complete Toolkit for Bot Development
The Microsoft Bot Framework is an all-encompassing suite of tools and services aimed at assisting developers in creating, connecting, and deploying bots capable of engaging in natural, conversational interactions with users. Central to the framework is the Bot Builder SDK, which offers a diverse array of tools for crafting bots that can be seamlessly integrated into various applications and services. The framework also features the Bot Connector Service, facilitating bot communication with users across a multitude of channels, encompassing platforms like Skype, Slack, and Facebook Messenger. To aid in bot testing and debugging, the Bot Framework includes the Bot State Emulator, while the Bot Analytics service enables developers to track bot performance and gauge user engagement. Supporting a variety of programming languages, such as C# and Node.js, the framework also offers a plethora of pre-built connectors to facilitate bot integration with widely-used services and platforms.

Recently proposed Story Cloze Test [Mostafazadeh et al., 2016] is a commonsense machine comprehension application to deal with natural language understanding problem. This dataset contains a lot of story tests which require commonsense inference ability. Unfortunately, the training data is almost unsupervised where each context document followed with only one positive sentence that can be inferred from the context. However, in the testing period, we must make inference from two candidate sentences. To tackle this problem, we employ the generative adversarial networks (GANs) to generate fake sentence. We proposed a Conditional GANs (CGANs) in which the generator is conditioned by the context. Our experiments show the advantage of the CGANs in discriminating sentence and achieve state-of-the-art results in commonsense story reading comprehension task compared with previous feature engineering and deep learning methods.

How do people acquire skill at comprehending what they read? That is the simple question to which we shall try to make a tentative answer. To begin, we have to acknowledge some complexities about the concept of reading comprehension and what it means to develop it.

Most tasks in natural language processing can be cast into question answering (QA) problems over language input. We introduce the dynamic memory network (DMN), a unified neural network framework which processes input sequences and questions, forms semantic and episodic memories, and generates relevant answers. Questions trigger an iterative attention process which allows the model to condition its attention on the result of previous iterations. These results are then reasoned over in a hierarchical recurrent sequence model to generate answers. The DMN can be trained end-to-end and obtains state of the art results on several types of tasks and datasets: question answering (Facebook’s bAbI dataset), sequence modeling for part of speech tagging (WSJ-PTB), coreference resolution (Quizbowl dataset) and text classification for sentiment analysis (Stanford Sentiment Treebank). The model relies exclusively on trained word vector representations and requires no string matching or manually engineered features.

Many applications in text processing require significant human effort for either labeling large document collections (when learning statistical models) or extrapolating rules from them (when using knowledge engineering). In this work, we describe away to reduce this effort, while retaining the methods' accuracy, by constructing a hybrid classifier that utilizes human reasoning over automatically discovered text patterns to complement machine learning. Using a standard sentiment-classification dataset and real customer feedback data, we demonstrate that the resulting technique results in significant reduction of the human effort required to obtain a given classification accuracy. Moreover, the hybrid text classifier also results in a significant boost in accuracy over machine-learning based classifiers when a comparable amount of labeled data is used.

Researchers and scientists increasingly find themselves in the position of having to quickly understand large amounts of technical material. Our goal is to effectively serve this need by using bibliometric text mining and summarization techniques to generate summaries of scientific literature. We show how we can use citations to produce automatically generated, readily consumable, technical extractive summaries. We first propose C-LexRank, a model for summarizing single scientific articles based on citations, which employs community detection and extracts salient information-rich sentences. Next, we further extend our experiments to summarize a set of papers, which cover the same scientific topic. We generate extractive summaries of a set of Question Answering (QA) and Dependency Parsing (DP) papers, their abstracts, and their citation sentences and show that citations have unique information amenable to creating a summary.

Most research on multi-document summarization explores methods that generate summaries based on queries regardless of the users’ preferences. We note that, different users can generate somewhat different summaries on the basis of the same source data and query. This paper presents our study on how to exploit the information regards how users summarized their texts. Models of different users can be used either separately, or in an ensemble-like fashion. Machine learning methods are explored in the construction of the individual models. However, we explore yet another hypothesis. We believe that the sentences selected into the summary should be coherent and supplement each other in their meaning. One method to model this relationship between sentences is by detecting actor–object relationship (AOR). The sentences that satisfy this relationship have their importance value enhanced. This paper combines ensemble summarizing system and AOR to generate summaries. We have evaluated this method on DUC 2006 and DUC 2007 using ROUGE measure. Experimental results show the supervised method that exploits the ensemble summarizing system combined with AOR outperforms previous models when considering performance in query-based multidocument summarization tasks. Keyword User-based summarization · Actor–object relationship · Multi-document summarization · Ensemble summarizing system · Training data construction Communicated by V. Loia. M. Valizadeh (B) · P. Brazdil LIAAD INESC Tec, University of Porto, Porto, Portugal e-mail: valizadehmr@gmail.com P. Brazdil FEP, University of Porto, Porto, Portugal e-mail: pbrazdil@inescporto.pt

This paper proposes quiz-style information presentation for interactive systems as a means to improve user understanding in educational tasks. Since the nature of quizzes can highly motivate users to stay voluntarily engaged in the interaction and keep their attention on receiving information, it is expected that information presented as quizzes can be better understood by users. To verify the effectiveness of the approach, we implemented read-out and quiz systems and performed comparison experiments using human subjects. In the task of memorizing biographical facts, the results showed that user understanding for the quiz system was significantly better than that for the read-out system, and that the subjects were more willing to use the quiz system despite the long duration of the quizzes. This indicates that quiz-style information presentation promotes engagement in the interaction with the system, leading to the improved user understanding.

Suppose we want to build a system that answers a natural language question by representing its semantics as a logical forxm and computing the answer given a structured database of facts. The core part of such a system is the semantic parser that maps questions to logical forms. Semantic parsers are typically trained from examples of questions annotated with their target logical forms, but this type of annotation is expensive.Our goal is to instead learn a semantic parser from question–answer pairs, where the logical form is modeled as a latent variable. We develop a new semantic formalism, dependency-based compositional semantics (DCS) and define a log-linear distribution over DCS logical forms. The model parameters are estimated using a simple procedure that alternates between beam search and numerical optimization. On two standard semantic parsing benchmarks, we show that our system obtains comparable accuracies to even state-of-the-art systems that do require annotated logical forms.

In the 1980s, plot units were proposed as a conceptual knowledge structure for representing and summarizing narrative stories. Our research explores whether current NLP technology can be used to automatically produce plot unit representations for narrative text. We create a system called AESOP that exploits a variety of existing resources to identify affect states and applies “projection rules” to map the affect states onto the characters in a story. We also use corpus-based techniques to generate a new type of affect knowledge base: verbs that impart positive or negative states onto their patients (e.g., being eaten is an undesirable state, but being fed is a desirable state). We harvest these “patient polarity verbs” from a Web corpus using two techniques: co-occurrence with Evil/Kind Agent patterns, and bootstrapping over conjunctions of verbs. We evaluate the plot unit representations produced by our system on a small collection of Aesop’s fables.

Automatic question generation (QG) is a useful yet challenging task in NLP. Recent neural network-based approaches represent the state-of-the-art in this task, but they are not without shortcomings. Firstly, these models lack the ability to handle rare words and the word repetition problem. Moreover, all previous works optimize the cross-entropy loss, which can induce inconsistencies between training (objective) and testing (evaluation measure). In this paper, we present a novel deep reinforcement learning based framework for automatic question generation. The generator of the framework is a sequence-to-sequence model, enhanced with the copy mechanism to handle the rare-words problem and the coverage mechanism to solve the word repetition problem. The evaluator model of the framework evaluates and assigns a reward to each predicted question. The overall model is trained by learning the parameters of the generator network which maximizes the reward. Our framework allows us to directly optimize any task-specific score including evaluation measures such as BLEU, GLEU, ROUGE-L, etc., suitable for sequence to sequence tasks such as QG. Our comprehensive evaluation shows that our approach significantly outperforms state-of-the-art systems on the widely-used SQuAD benchmark in both automatic and human evaluation.


Question: Jack the Reader - A framework for machine reading comprehension.

Introducing TriviaQA: A Novel Large-Scale Reading Comprehension Dataset

In this paper, we unveil TriviaQA, a reading comprehension dataset that presents a significant challenge to existing machine learning models. TriviaQA comprises over 650,000 question-answer-evidence triples, designed to test the limits of current reading comprehension systems. The dataset is unique in its composition, featuring 95,000 question-answer pairs crafted by trivia enthusiasts, along with independently gathered evidence documents that offer high-quality, albeit distant, supervision for answering these questions. Each question is accompanied by an average of six evidence documents, providing rich context and requiring sophisticated analysis.

TriviaQA distinguishes itself from other large-scale datasets in several key aspects. Firstly, the questions are notably complex, often requiring the understanding and integration of multiple concepts. Secondly, there is a marked disparity between the syntactic and lexical structure of the questions and the corresponding answer-evidence sentences, necessitating robust semantic parsing capabilities. Thirdly, the dataset frequently demands cross-sentence reasoning to accurately identify the answers, highlighting the need for models that can effectively navigate and synthesize information across different parts of the text.

To gauge the performance of existing approaches, we present two baseline algorithms. The first is a feature-based classifier, a traditional machine learning technique that relies on handcrafted features to make predictions. The second is a state-of-the-art neural network, which has demonstrated impressive results on the SQuAD reading comprehension dataset. Despite their strengths, neither of these approaches comes close to human performance on TriviaQA, achieving accuracies of 23% and 40%, respectively, compared to 80% for humans. These results underscore the complexity of TriviaQA and the considerable gap that remains to be bridged in the quest for human-level reading comprehension in machines.

In conclusion, TriviaQA offers a rigorous testbed for researchers and developers working on reading comprehension systems. Its unique characteristics and challenging nature provide ample opportunities for the advancement of machine learning techniques, particularly in the areas of natural language processing and understanding. Future studies that address the specific challenges posed by TriviaQA are likely to contribute significantly to the field, potentially leading to breakthroughs in machine reading and comprehension.

OntoNotes is a five year multi-site collaboration between BBN Technologies, Information Sciences Institute of University of Southern California, University of Colorado, University of Pennsylvania and Brandeis University. The goal of the OntoNotes project is to provide linguistic data annotated with a skeletal representation of the literal meaning of sentences including syntactic parse, predicate-argument structure, coreference, and word senses linked to an ontology, allowing a new generation of language understanding technologies to be developed with new functional capabilities. In its third year of existence, the OntoNotes project has generated a large amount of high quality data covering various layers of linguistic annotation. This is probably the first time that data of such quality has been available in large quantities covering multiple genres (newswire, broadcast news, broadcast conversation and weblogs) and languages (English, Chinese and Arabic). The guiding principle has been to find a “sweet spot” in the space of inter-tagger agreement, productivity, and depth of representation. The most effective use of this resource for research requires simultaneous access to multiple layers of annotation. This has been made possible by representing the corpus with a relational database to accommodate the dense connectedness of the data and ensure consistency across layers. In order to facilitate ease of understanding and manipulability, the database has also been supplemented with a object-oriented Python API. The tutorial consists of two parts. In the first part we will familiarize the user with this new resource, describe the various layers of annotations in some detail and discuss the linguistic principles and sometimes practical considerations behind the important design decisions that shapes the corpus. We will also describe the salient differences between the three languages at each layer of annotation and how linguistic peculiarities of different languages were handled in the data. In the second part, we will describe the data formats of each of the layers and talk about various design decisions that went into the creation of the architecture of the database and the individual tables comprising it, along with issues that came up during the representation process and compromises that were made without sacrificing some primary objectives one of which being the independent existence of each layer that is necessary to allow multi-site collaboration. We will explain how the database schema attempts to interconnect all the layers. Then we will go into the details of the Python API that allows easy access to each of the layers and show that by making the objects closely resemble database tables, the API allows for their flexible integration. This will be followed by a hands-on working session.

What just happened in artificial intelligence and how it is being misunderstood.

Given a question and a set of answer candidates, answer triggering determines whether the candidate set contains any correct answers. If yes, it then outputs a correct one. In contrast to existing pipeline methods which first consider individual candidate answers separately and then make a prediction based on a threshold, we propose an end-to-end deep neural network framework, which is trained by a novel group-level objective function that directly optimizes the answer triggering performance. Our objective function penalizes three potential types of error and allows training the framework in an end-to-end manner. Experimental results on the WIKIQA benchmark show that our framework outperforms the state of the arts by a 6.6% absolute gain under F1 measure1.

Question generation is an emerging research area of artificial intelligence in education. Question authoring tools are important in educational technologies, e.g., intelligent tutoring systems, as well as in dialogue systems. Approaches to generate factual questions, i.e., questions that have concrete answers, mainly make use of the syntactical and semantic information in a declarative sentence, which is then transformed into questions. Recently, some research has been conducted to investigate Chinese factual question generation with some limited success. Reported performance is poor due to unavoidable errors (e.g., sentence parsing, name entity recognition, and rule-based question transformation errors) and the complexity of long Chinese sentences. This article introduces a novel Chinese question generation system based on three stages, sentence simplification, question generation and ranking, to address the challenge of automatically generating factual questions in Chinese. The proposed approach and system have been evaluated on sentences from the New Practical Chinese Reader corpus. Experimental results show that ranking improves more than 20 percentage of questions rated as acceptable by annotators, from 65 percent of all questions to 87 percent of the top ranked 25 percent questions.

What would it take to teach a computer to play a game entirely through language and sketching? In this paper, we present an implemented program through which an instructor can teach the rules of simple board games using such input. We describe the architecture, information flow, and vocabulary of instructional events and walk through an annotated example. In our approach, the instructional and communication events guide abductive reasoning for language interpretation and help to integrate information from sketching and language. Having a general target representation enables the learning process to be viewed more as translation and problem solving than as induction. Lastly, learning by demonstration complements and extends instruction, resulting in concrete, operational rules.

Models that learn to represent textual and knowledge base relations in the same continuous latent space are able to perform joint inferences among the two kinds of relations and obtain high accuracy on knowledge base completion (Riedel et al., 2013). In this paper we propose a model that captures the compositional structure of textual relations, and jointly optimizes entity, knowledge base, and textual relation representations. The proposed model significantly improves performance over a model that does not share parameters among textual relations with common sub-structure.

We presentan unsupervisedapproachto recognizingdiscourserelationsof CONTRAST, EXPLANATION-EVIDENCE, CONDITION andELABORATION that hold betweenarbitraryspansof texts. We show that discourserelation classifierstrained on examples that are automaticallyextractedfrom massi ve amountsof text can be usedto distinguishbetweensomeof theserelationswith accuraciesashigh as 93%, even whenthe relationsarenot explicitly markedby cuephrases.

Most tasks in natural language processing can be cast into question answering (QA) problems over language input. We introduce the dynamic memory network (DMN), a unified neural network framework which processes input sequences and questions, forms semantic and episodic memories, and generates relevant answers. Questions trigger an iterative attention process which allows the model to condition its attention on the result of previous iterations. These results are then reasoned over in a hierarchical recurrent sequence model to generate answers. The DMN can be trained end-to-end and obtains state of the art results on several types of tasks and datasets: question answering (Facebook’s bAbI dataset), sequence modeling for part of speech tagging (WSJ-PTB), coreference resolution (Quizbowl dataset) and text classification for sentiment analysis (Stanford Sentiment Treebank). The model relies exclusively on trained word vector representations and requires no string matching or manually engineered features.

This paper describes the PASCAL Network of Excellence first Recognising Textual Entailment (RTE-1) Challenge benchmark. The RTE task is defined as recognizing, given two text fragments, whether the meaning of one text can be inferred (entailed) from the other. This application-independent task is suggested as capturing major inferences about the variability of semantic expression which are commonly needed across multiple applications. The Challenge has raised noticeable attention in the research community, attracting 17 submissions from diverse groups, suggesting the generic relevance of the task.

Recently proposed Story Cloze Test [Mostafazadeh et al., 2016] is a commonsense machine comprehension application to deal with natural language understanding problem. This dataset contains a lot of story tests which require commonsense inference ability. Unfortunately, the training data is almost unsupervised where each context document followed with only one positive sentence that can be inferred from the context. However, in the testing period, we must make inference from two candidate sentences. To tackle this problem, we employ the generative adversarial networks (GANs) to generate fake sentence. We proposed a Conditional GANs (CGANs) in which the generator is conditioned by the context. Our experiments show the advantage of the CGANs in discriminating sentence and achieve state-of-the-art results in commonsense story reading comprehension task compared with previous feature engineering and deep learning methods.


Question: A fresh perspective on predictive state techniques for learning dynamical systems.

This paper presents an innovative approach to modeling sequence data through the extension of Hidden Markov Models (HMMs) into a nonparametric framework, thereby overcoming the limitations of discrete latent states and predominantly Gaussian or discrete observations. Traditional HMMs have been widely used but have often relied on local search heuristics for learning, with few exceptions like the spectral methods. Here, we introduce a novel kernel spectral algorithm for nonparametric HMMs that avoids local minima, marking a significant advancement in learning algorithms for HMMs. Our method allows for the modeling of structured and non-Gaussian continuous distributions, expanding the applicability of HMMs to a broader range of data types. We validate the effectiveness of our embedded HMMs through applications in diverse domains, such as robot vision, slot car inertial sensor data analysis, and audio event classification. In these scenarios, our approach outperforms state-of-the-art methods, demonstrating its superiority in handling complex and varied data sequences.

Temporal diierence (TD) methods constitute a class of methods for learning predictions in multi-step prediction problems, parameterized by a recency factor. Currently the most important application of these methods is to temporal credit assignment in reinforcement learning. Well known reinforcement learning algorithms, such as AHC or Q-learning, may be viewed as instances of TD learning. This paper examines the issues of the eecient and general implementation of TD() for arbitrary , for use with reinforcement learning algorithms optimizing the discounted sum of rewards. The traditional approach, based on eligibility traces, is argued to suuer from both ineeciency and lack of generality. The TTD (Truncated Temporal Diierences) procedure is proposed as an alternative, that indeed only approximates TD(), but requires very little computation per action and can be used with arbitrary function representation methods. The idea from which it is derived is fairly simple and not new, but probably unexplored so far. Encouraging experimental results are presented, suggesting that using > 0 with the TTD procedure allows one to obtain a signiicant learning speedup at essentially the same cost as usual TD(0) learning.

This paper advocates a new ML-based programming framework, called Programming by Feedback (PF), which involves a sequence of interactions between the active computer and the user. The latter only provides preference judgments on pairs of solutions supplied by the active computer. The active computer involves two components: the learning component estimates the user’s utility function and accounts for the user’s (possibly limited) competence; the optimization component explores the search space and returns the most appropriate candidate solution. A proof of principle of the approach is proposed, showing that PF requires a handful of interactions in order to solve some discrete and continuous benchmark problems.

This study presents a novel Adaptive resonance theory-Counterpropagation neural network (ART-CPN) for solving forecasting problems. The network is based on the ART concept and the CPN learning algorithm for constructing the neural network. The vigilance parameter is used to automatically generate the nodes of the cluster layer for the CPN learning process. This process improves the initial weight problem and the adaptive nodes of the cluster layer (Kohonen layer). ART-CPN involves real-time learning and is capable of developing a more stable and plastic prediction model of input patterns by self-organization. The advantages of ART-CPN include the ability to cluster, learn and construct the network model for forecasting problems. The network was applied to solve the real forecasting problems. The learning algorithm revealed better learning efficiency and good prediction performance. q 2004 Elsevier Ltd. All rights reserved.

In this paper we study how to learn stochastic, multimodal transition dynamics in reinforcement learning (RL) tasks. We focus on evaluating transition function estimation, while we defer planning over this model to future work. Stochasticity is a fundamental property of many task environments. However, discriminative function approximators have difficulty estimating multimodal stochasticity. In contrast, deep generative models do capture complex high-dimensional outcome distributions. First we discuss why, amongst such models, conditional variational inference (VI) is theoretically most appealing for model-based RL. Subsequently, we compare different VI models on their ability to learn complex stochasticity on simulated functions, as well as on a typical RL gridworld with multimodal dynamics. Results show VI successfully predicts multimodal outcomes, but also robustly ignores these for deterministic parts of the transition dynamics. In summary, we show a robust method to learn multimodal transitions using function approximation, which is a key preliminary for model-based RL in stochastic domains.

This paper extends previous work with Dyna a class of architectures for intelligent systems based on approximating dynamic program ming methods Dyna architectures integrate trial and error reinforcement learning and execution time planning into a single process operating alternately on the world and on a learned model of the world In this paper I present and show results for two Dyna archi tectures The Dyna PI architecture is based on dynamic programming s policy iteration method and can be related to existing AI ideas such as evaluation functions and uni versal plans reactive systems Using a nav igation task results are shown for a simple Dyna PI system that simultaneously learns by trial and error learns a world model and plans optimal routes using the evolving world model The Dyna Q architecture is based on Watkins s Q learning a new kind of rein forcement learning Dyna Q uses a less famil iar set of data structures than does Dyna PI but is arguably simpler to implement and use We show that Dyna Q architectures are easy to adapt for use in changing environments Introduction to Dyna How should a robot decide what to do The traditional answer in AI has been that it should deduce its best action in light of its current goals and world model i e that it should plan However it is now widely recognized that planning s usefulness is limited by its computational complexity and by its dependence on an accurate world model An alternative approach is to do the planning in advance and compile its result into a set of rapid reactions or situation action rules which are then used for real time decision making Yet a third approach is to learn a good set of reactions by trial and error this has the advantage of eliminating the dependence on a world model In this paper I brie y introduce Dyna a class of simple architectures integrating and permitting tradeo s among these three approaches Dyna architectures use machine learning algo rithms to approximate the conventional optimal con trol technique known as dynamic programming DP Bellman Ross DP itself is not a learn ing method but rather a computational method for determining optimal behavior given a complete model of the task to be solved It is very similar to state space search but di ers in that it is more incremental and never considers actual action sequences explicitly only single actions at a time This makes DP more amenable to incremental planning at execution time and also makes it more suitable for stochastic or in completely modeled environments as it need not con sider the extremely large number of sequences possi ble in an uncertain environment Learned world mod els are likely to be stochastic and uncertain making DP approaches particularly promising for learning sys tems Dyna architectures are those that learn a world model online while using approximations to DP to learn and plan optimal behavior Intuitively Dyna is based on the old idea that planning is like trial and error learning from hypothet ical experience Craik Dennett The theory of Dyna is based on the theory of DP e g Ross and on DP s relationship to reinforcement learning Watkins Barto Sutton Watkins to temporal di erence learning Sutton and to AI methods for planning and search Korf Werbos has previously argued for the general idea of building AI systems that approx imate dynamic programming and Whitehead and others Sutton Barto Sutton Pinette Rumelhart et al have presented results for the speci c idea of augmenting a reinforcement learning system with a world model used for planning Dyna PI Dyna by Approximating Policy Iteration I call the rst Dyna architecture Dyna PI because it is based on approximating a DP method known as pol icy iteration Howard The Dyna PI architec ture consists of four components interacting as shown in Figure The policy is simply the function formed by the current set of reactions it receives as input a description of the current state of the world and pro duces as output an action to be sent to the world The world represents the task to be solved prototypi cally it is the robot s external environment The world receives actions from the policy and produces a next state output and a reward output The overall task is de ned as maximizing the long term average reward per time step cf Russell The architecture also includes an explicit world model The world model is intended to mimic the one step input output behavior of the real world Finally the Dyna PI architecture in cludes an evaluation function that rapidly maps states to values much as the policy rapidly maps states to actions The evaluation function the policy and the world model are each updated by separate learning processes WORLD Action Reward (scalar) Heuristic Reward (scalar) State EVALUATION FUNCTION

In this paper, we study the challenging problem of predicting the dynamics of objects in static images. Given a query object in an image, our goal is to provide a physical understanding of the object in terms of the forces acting upon it and its long term motion as response to those forces. Direct and explicit estimation of the forces and the motion of objects from a single image is extremely challenging. We define intermediate physical abstractions called Newtonian scenarios and introduce Newtonian Neural Network (N3) that learns to map a single image to a state in a Newtonian scenario. Our evaluations show that our method can reliably predict dynamics of a query object from a single image. In addition, our approach can provide physical reasoning that supports the predicted dynamics in terms of velocity and force vectors. To spur research in this direction we compiled Visual Newtonian Dynamics (VIND) dataset that includes more than 6000 videos aligned with Newtonian scenarios represented using game engines, and more than 4500 still images with their ground truth dynamics.

This paper introduces a robust learning scheme that can dynamically predict the stability of the reconnection of subnetworks to a main grid. As the future electrical power systems tend towards smarter and greener technology, the deploymen t of self sufficient networks, or microgrids, becomes more likel y. Microgrids may operate on their own or synchronized with the main grid, thus control methods need to take into account islandi ng and reconnecting said networks. The ability to optimally and safely reconnect a portion of the grid is not well understood and, as of now, limited to raw synchronization between interconnection points. A support vector machine (SVM) leveraging real-time data from phasor measurement units (PMUs) is proposed to predict in real time whether the reconnection of a sub-network to the main grid would lead to stability or instability. A dynam ics simulator fed with pre-acquired system parameters is used t o create training data for the SVM in various operating states. The classifier was tested on a variety of cases and operating poin ts to ensure diversity. Accuracies of approximately 90% were obs erved throughout most conditions when making dynamic predictions of a given network. Keywords—Synchrophasor, machine learning, microgrid, islanding, reconnection I. I NTRODUCTION As we make strides towards a smarter power system, it is important to explore new techniques and innovations to fully capture the potential of such a dynamic entity. Many large blackout events, such as the blackout of 2003, could have been prevented with smarter controls and better monito ring [1]. Phasor measurement units, or PMUs, are one such breakthrough that will allow progress to be made in both monitoring and implementing control to the system [2]. PMUs allow for direct measurement of bus voltages and angles at high sample rates which makes dynamic state estimation more feasible [3], [4]. With the use of PMUs, it is possible to improve upon current state estimation [5] and potentially o pen up new ways to control the grid. The addition of control techniques and dynamic monitoring will be important as we begin to integrate newer solutions, such as microgrids, int o the power network. With these advanced monitoring devices, microgrids become more feasible due to the potential for rea ltime monitoring schemes. The integration of microgrids bri ng many benefits such as the ability to operate while islanded as well as interconnected with the main grid; they provide a smooth integration for renewable energy sources that matc h C. Lassetter, E. Cotilla-Sanchez, and J. Kim are with the Sch ool of Electrical Engineering & Computer Science, Oregon State University, C orvallis, OR, 97331 USA e-mail:{lassettc,ecs,kimjinsu }@oregonstate.edu local demand. Unfortunately the implementation of microgr ids is still challenging due to lacking experience with the beha vior of control schemes during off-nominal operation. Currently, microgrids are being phased in slowly due in part to the difficulty of operating subnetworks independent ly as well as determining when they can be reconnected to the main grid. Works in the literature have focused on the potential of reconnecting microgrids to the main grid, in particular aiming at synchronizing the buses at points of interconnect with respects to their voltages, frequencies , and angles [6]. Effort has been directed at creating control sch emes to minimize power flow at the point of common coupling (PCC) using direct machine control, load shedding, as well a s energy storage, to aid in smooth reconnection [7], [8]. Upon reconnection of an islanded sub-network to the main grid, instability can cause damage on both ends. It is important to track instabilities on both the microgrid and main grid upon reconnection to accurately depict the outcome of reconnect ion. Most works focus on the synchronization of two networks before reconnection [9], [10]. In some cases we may need to look at larger microgrids or subnetworks in which multipl e PCCs exist. In such scenarios, it becomes much more difficult to implement a control scheme that satisfies goodreconnection tolerances in regards to minimizing bus frequency, angle, a nd voltage differences at each PCC. In addition to the possibil ity of multiple PCCs, it is possible that direct manipulation of the system becomes limited, compromised, or unsupported with respect to synchronization. In order to address these short comings, we implement an algorithm that dynamically tracks and makes predictions based on the system states, providing real-time stability information of potential reconnectio ns. An algorithm that tracks potential reconnection and island ing times needs to be robust to prevent incorrect operation during these scenarios. PMUs make use of GPS synchronization [11] which can create an attack platform for adversarie s by changing or shifting the time synchronization. Incorrec t or compromised usage could lead to incorrect predictions that would degrade system stability due to hidden failures that remain dormant until triggered by contingencies [12]. We propose an algorithm that can make accurate predictions in face of potentially compromised measurements. Due to the complexity of the power grid, it is difficult to come up with a verbatim standard depicting the potential sta bility after reconnection of a subnetwork. With advances in the artificial intelligence community, we can make use of machin e learning algorithms in order to explore vast combinations o f sensor inputs, states, and control actions. This can be done in a

This manuscript surveys reinforcement learning from the perspective of optimization and control with a focus on continuous control applications. It surveys the general formulation, terminology, and typical experimental implementations of reinforcement learning and reviews competing solution paradigms. In order to compare the relative merits of various techniques, this survey presents a case study of the Linear Quadratic Regulator (LQR) with unknown dynamics, perhaps the simplest and best-studied problem in optimal control. The manuscript describes how merging techniques from learning theory and control can provide non-asymptotic characterizations of LQR performance and shows that these characterizations tend to match experimental behavior. In turn, when revisiting more complex applications, many of the observed phenomena in LQR persist. In particular, theory and experiment demonstrate the role and importance of models and the cost of generality in reinforcement learning algorithms. This survey concludes with a discussion of some of the challenges in designing learning systems that safely and reliably interact with complex and uncertain environments and how tools from reinforcement learning and control might be combined to approach these challenges.

A key challenge in scaling up robot learning to many skills and environments is removing the need for human supervision, so that robots can collect their own data and improve their own performance without being limited by the cost of requesting human feedback. Model-based reinforcement learning holds the promise of enabling an agent to learn to predict the effects of its actions, which could provide flexible predictive models for a wide range of tasks and environments, without detailed human supervision. We develop a method for combining deep action-conditioned video prediction models with model-predictive control that uses entirely unlabeled training data. Our approach does not require a calibrated camera, an instrumented training set-up, nor precise sensing and actuation. Our results show that our method enables a real robot to perform nonprehensile manipulation — pushing objects — and can handle novel objects not seen during training.

Predicting when a person might be frustrated can provide an intelligent system with important information about when to initiate interaction. For example, an automated Learning Companion or Intelligent Tutoring System might use this information to intervene, providing support to the learner who is likely to otherwise quit, while leaving engaged learners free to discover things without interruption. This paper presents the first automated method that assesses, using multiple channels of affect-related information, whether a learner is about to click on a button saying ‘‘I’m frustrated.’’ The new method was tested on data gathered from 24 participants using an automated Learning Companion. Their indication of frustration was automatically predicted from the collected data with 79% accuracy ðchance 1⁄4 58%Þ. The new assessment method is based on Gaussian process classification and Bayesian inference. Its performance suggests that non-verbal channels carrying affective cues can help provide important information to a system for formulating a more intelligent response. r 2007 Elsevier Ltd. All rights reserved.


Question: A fresh perspective on predictive state techniques for the learning of dynamical systems.

This paper introduces a groundbreaking technique for resolving a key challenge in artificial intelligence: planning to maximize future reward under uncertainty in environments with partial observability. We propose and validate a novel algorithm capable of accurately learning a model of the environment directly from sequences of action-observation pairs, effectively bridging the gap between observations and actions. Our method involves planning in the learned model to derive a policy that closely approximates optimality in the original environment. We detail an efficient and statistically consistent spectral algorithm designed for learning the parameters of a Predictive State Representation (PSR). To illustrate the efficacy of our approach, we apply it to a complex, high-dimensional mobile robot planning task that relies on visual input. We conduct approximate point-based planning in the PSR learned from the task. Our analysis reveals that the algorithm adeptly learns a state space that succinctly encapsulates the critical aspects of the environment. This compact representation facilitates precise predictions using a minimal set of parameters, enabling successful and efficient planning strategies.

The premise of this article is that learning procedures used to train artificial neural networks are inherently statistical techniques. It follows that statistical theory can provide considerable insight into the properties, advantages, and disadvantages of different network learning methods. We review concepts and analytical results from the literatures of mathematical statistics, econometrics, systems identification, and optimization theory relevant to the analysis of learning in artificial neural networks. Because of the considerable variety of available learning procedures and necessary limitations of space, we cannot provide a comprehensive treatment. Our focus is primarily on learning procedures for feedforward networks. However, many of the concepts and issues arising in this framework are also quite broadly relevant to other network learning paradigms. In addition to providing useful insights, the material reviewed here suggests some potentially useful new training methods for artificial neural networks.

This paper proposes a framework for modeling and controlling systems described by interdependent physical laws, logic rules, and operating constraints, denoted as Mixed Logical Dynamical (MLD) systems. These are described by linear dynamic equations subject to linear inequalities involving real and integer variables. MLD systems include constrained linear systems, finite state machines, some classes of discrete event systems, and nonlinear systems which can be approximated by piecewise linear functions. A predictive control scheme is proposed which is able to stabilize MLD systems on desired reference trajectories while fulfilling operating constraints, and possibly take into account previous qualitative knowledge in the form of heuristic rules. Due to the presence of integer variables, the resulting on-line optimization procedures are solved through Mixed Integer Quadratic Programming (MIQP), for which efficient solvers have been recently developed. Some examples and a simulation case study on a complex gas supply system are reported.

A learner’s modifiable components are called its policy. An algorithm that modifies the policy is a learning algorithm. If the learning algorithm has modifiable components represented as part of the policy, then we speak of a self-modifying policy (SMP). SMPs can modify the way they modify themselves etc. They are of interest in situations where the initial learning algorithm itself can be improved by experience — this is what we call “learning to learn”. How can we force some (stochastic) SMP to trigger better and better self-modifications? The success-story algorithm (SSA) addresses this question in a lifelong reinforcement learning context. During the learner’s life-time, SSA is occasionally called at times computed according to SMP itself. SSA uses backtracking to undo those SMP-generated SMP-modifications that have not been empirically observed to trigger lifelong reward accelerations (measured up until the current SSA call — this evaluates the long-term effects of SMP-modifications setting the stage for later SMP-modifications). SMP-modifications that survive SSA represent a lifelong success history. Until the next SSA call, they build the basis for additional SMP-modifications. Solely by self-modifications our SMP/SSA-based learners solve a complex task in a partially observable environment (POE) whose state space is far bigger than most reported in the POE literature.

We extend the herding algorithm to continuous spaces by using the kernel trick. The resulting “kernel herding” algorithm is an infinite memory deterministic process that learns to approximate a PDF with a collection of samples. We show that kernel herding decreases the error of expectations of functions in the Hilbert space at a rateO(1/T )which is much faster than the usual O(1/ √ T ) for iid random samples. We illustrate kernel herding by approximating Bayesian predictive distributions.

In recent years significant progress has been made in successfully training recurrent neural networks (RNNs) on sequence learning problems involving long range temporal dependencies. The progress has been made on three fronts: (a) Algorithmic improvements involving sophisticated optimization techniques, (b) network design involving complex hidden layer nodes and specialized recurrent layer connections and (c) weight initialization methods. In this paper, we focus on recently proposed weight initialization with identity matrix for the recurrent weights in a RNN. This initialization is specifically proposed for hidden nodes with Rectified Linear Unit (ReLU) non linearity. We offer a simple dynamical systems perspective on weight initialization process, which allows us to propose a modified weight initialization strategy. We show that this initialization technique leads to successfully training RNNs composed of ReLUs. We demonstrate that our proposal produces comparable or better solution for three toy problems involving long range temporal structure: the addition problem, the multiplication problem and the MNIST classification problem using sequence of pixels. In addition, we present results for a benchmark action recognition problem.

This paper proposes a recurrent fuzzy neural network (RFNN) structure for identifying and controlling nonlinear dynamic systems. The RFNN is inherently a recurrent multilayered connectionist network for realizing fuzzy inference using dynamic fuzzy rules. Temporal relations are embedded in the network by adding feedback connections in the second layer of the fuzzy neural network (FNN). The RFNN expands the basic ability of the FNN to cope with temporal problems. In addition, results for the FNNfuzzy inference engine, universal approximation, and convergence analysis are extended to the RFNN. For the control problem, we present the direct and indirect adaptive control approaches using the RFNN. Based on the Lyapunov stability approach, rigorous proofs are presented to guarantee the convergence of the RFNN by choosing appropriate learning rates. Finally, the RFNN is applied in several simulations (time series prediction, identification, and control of nonlinear systems). The results confirm the effectiveness of the RFNN.

A nonlinear model predictive control (NMPC) scheme is developed in this paper based on a self-organizing recurrent radial basis function (SR-RBF) neural network, whose structure and parameters are adjusted concurrently in the training process. The proposed SR-RBF neural network is represented in a general nonlinear form for predicting the future dynamic behaviors of nonlinear systems. To improve the modeling accuracy, a spiking-based growing and pruning algorithm and an adaptive learning algorithm are developed to tune the structure and parameters of the SR-RBF neural network, respectively. Meanwhile, for the control problem, an improved gradient method is utilized for the solution of the optimization problem in NMPC. The stability of the resulting control system is proved based on the Lyapunov stability theory. Finally, the proposed SR-RBF neural network-based NMPC (SR-RBF-NMPC) is used to control the dissolved oxygen (DO) concentration in a wastewater treatment process (WWTP). Comparisons with other existing methods demonstrate that the SR-RBF-NMPC can achieve a considerably better model fitting for WWTP and a better control performance for DO concentration.

A core challenge for an agent learning to interact with the world is to predict how its actions affect objects in its environment. Many existing methods for learning the dynamics of physical interactions require labeled object information. However, to scale real-world interaction learning to a variety of scenes and objects, acquiring labeled data becomes increasingly impractical. To learn about physical object motion without labels, we develop an action-conditioned video prediction model that explicitly models pixel motion, by predicting a distribution over pixel motion from previous frames. Because our model explicitly predicts motion, it is partially invariant to object appearance, enabling it to generalize to previously unseen objects. To explore video prediction for real-world interactive agents, we also introduce a dataset of 50, 000 robot interactions involving pushing motions, including a test set with novel objects. In this dataset, accurate prediction of videos conditioned on the robot’s future actions amounts to learning a “visual imagination” of different futures based on different courses of action. Our experiments show that our proposed method not only produces more accurate video predictions, but also more accurately predicts object motion, when compared to prior methods.

We discsus Hinton’s (1989) relative payoff procedure (RPP), a static reinforcement learning algorithm whose foundation is not stochastic gradient ascent. We show circumstances under which applying the RPP is guaranteed to increase the mean return, even though it can make large changes in the values of the parameters. The proof is based on a mapping between the RPP and a form of the expectation-maximisation procedure of Dempster, Laird & Rubin (1976).

An algorithm, the bootstrap filter, is proposed for implementing recursive Bayesian filters. The required density of the state vector is represented as a set of random samples, which are updated and propagated by the algorithm. The method is not restricted by assumptions of linearity or Gaussian noise: it may be applied to any state transition or measurement model. A simulation example of the bearings only tracking problem is presented. This simulation includes schemes for improving the efficiency of the basic algorithm. For this example, the performance of the bootstrap filter is greatly superior to the standard extended Kalman filter.