Question: SarcasmBot: An open-source module for generating sarcasm in chatbots.

GUS, a Frame-Driven Dialog System: An Evolutionary Perspective
GUS, the Genial Understander System, is the pioneering effort within a series of experimental computer systems, designed to encapsulate the progress and insights gained from our ongoing research into language understanding. These systems serve as milestones, reflecting the cumulative knowledge and the synergistic alignment of our diverse research endeavors, and guiding future directions with enhanced focus. GUS is engineered to engage in a cooperative English dialogue, guided by a specific goal within a narrowly defined domain of discourse. Initially, GUS's domain was constrained to the role of a travel agent, facilitating conversations with clients planning a straightforward return trip to a single Californian city. This domain restriction is strategic, allowing the system to achieve a semblance of realism without the overwhelming task of encompassing the vastness of human knowledge or the English language in its entirety. It also provides users with a tangible goal, narrowing the scope of their interaction and expectations from GUS. The restricted nature of GUS ensures the conversation remains within its competence, guiding it effectively.

The significance of GUS lies not in its superficial behavior, which, while realistic, does not offer unique services or information that cannot be accessed through other means. Its intrigue stems from its endeavor to model the intricate phenomena of natural dialogue and its underlying organizational principles. Natural dialogues are characterized by their spontaneity and unpredictability, which GUS attempts to mimic through a modular design that reduces sensitivity to the variability of ordinary conversation. This design is particularly evident in GUS's ability to handle mixed initiative, indirect answers, anaphora resolution, fragmented sentence comprehension, and discourse interpretation within known conversational frameworks.

Mixed initiative, a hallmark of human conversations, is where both participants intermittently assume control. GUS, unlike many interactive computer systems, does not dominate the conversation but instead allows for a natural flow by ceding control when necessary. This is demonstrated in the dialogue where the client provides additional information beyond the query (e.g., travel date) and in instances where the client inverts the conversation's direction with a question instead of a direct response.

The challenge of indirect answers, where responses are not explicit but require inference, is another area GUS navigates. For instance, when GUS asks about preferred departure time, the client's response indirectly constrains flight selection, illustrating GUS's ability to interpret and act upon such indirect information.

In summary, GUS represents a significant step forward in the simulation of natural human dialogue, offering insights into mixed initiative, indirect answers

Automated grammatical error correction for language learners has attracted a lot of attention in recent years, especially after a number of shared tasks that have encouraged research in the area. Treating the problem as a translation task from 'incorrect' into 'correct' English using statistical machine translation has emerged as a state-of-the-art approach but it requires vast amounts of corrected parallel data to produce useful results. Because manual annotation of incorrect text is laborious and expensive, we can generate artificial error-annotated data by injecting errors deliberately into correct text and thus produce larger amounts of parallel data with much less effort. In this work, we review previous work on artificial error generation and investigate new approaches using random and probabilistic methods for constrained and general error correction. Our methods use error statistics from a reference corpus of learner writing to generate errors in native text that look realistic and plausible in context. We investigate a number of aspects that can play a part in the error generation process, such as the origin of the native texts, the amount of context used to find suitable insertion points, the type of information encoded by the error patterns and the output error distribution. In addition, we explore the use of linguistic information for characterising errors and train systems using different combinations of real and artificial data. Results of our experiments show that the use of artificial errors can improve system performance when they are used in combination with real learner errors, in line with previous research. These improvements are observed for both constrained and general correction, for which probabilistic methods produce the best results. We also demonstrate that systems trained on a combination of real and artificial errors can beat other highly-engineered systems and be more robust, showing that performance can be improved by focusing on the data rather than tuning system parameters. Part of our work is also devoted to the proposal of the I-measure, a new evaluation scheme that scores corrections in terms of improvement on the original text and solves known issues with existing evaluation measures. To my family and L.A. In memory of Kika.

While much work has recently focused on the analysis of social media in order to get a feel for what people think about current topics of interest, there are, however, still many challenges to be faced. Text mining systems originally designed for more regular kinds of texts such as news articles may need to be adapted to deal with facebook posts, tweets etc. In this paper, we discuss a variety of issues related to opinion mining from social media, and the challenges they impose on a Natural Language Processing (NLP) system, along with two example applications we have developed in very different domains. In contrast with the majority of opinion mining work which uses machine learning techniques, we have developed a modular rule-based approach which performs shallow linguistic analysis and builds on a number of linguistic subcomponents to generate the final opinion polarity and score.

An interactive Question Answering (QA) system frequently encounters non-sentential (incomplete) questions. These non-sentential questions may not make sense to the system when a user asks them without the context of conversation. The system thus needs to take into account the conversation context to process the incomplete question. In this work, we present a recurrent neural network (RNN) based encoder decoder network that can generate a complete (intended) question, given an incomplete question and conversation context. RNN encoder decoder networks have been show to work well when trained on a parallel corpus with millions of sentences, however it is extremely hard to obtain conversation data of this magnitude. We therefore propose to decompose the original problem into two separate simplified problems where each problem focuses on an abstraction. Specifically, we train a semantic sequence model to learn semantic patterns, and a syntactic sequence model to learn linguistic patterns. We further combine syntactic and semantic sequence models to generate an ensemble model. Our model achieves a BLEU score of 30.15 as compared to 18.54 using a standard RNN encoder decoder model.

Text production is a key component of many NLP applications. In data-driven approaches, it is used for instance, to generate dialogue turns from dialogue moves [Wen et al., 2015, Wen et al., 2016, Novikova et al., 2017], to verbalise the content of Knowledge bases [Gardent et al., 2017a, Gardent et al., 2017b] or to generate natural English sentences from rich linguistic representations, such as dependency trees [Belz et al., 2011, Mille et al., 2018] or Abstract Meaning Representations [May and Priyadarshi, 2017, Konstas et al., 2017, Song et al., ]. In text-driven methods on the other hand, text production is at work in sentence compression [Knight and Marcu, 2000, Cohn and Lapata, 2008, Filippova and Strube, 2008, Pitler, 2010, Filippova et al., 2015, Toutanova et al., 2016]; sentence fusion [McKeown et al., 2010, Filippova, 2010, Thadani and McKeown, 2013]; paraphrasing [Dras, 1999, Barzilay and McKeown, 2001, Bannard and Callison-Burch, 2005, Wubben et al., 2010, Narayan et al., 2016, Dong et al., 2017, Mallinson et al., 2017]; sentence (or text) simplification [Siddharthan et al., 2004, Zhu et al., 2010, Woodsend and Lapata, 2011, Wubben et al., 2012, Narayan and Gardent, 2014, Xu et al., 2015, Narayan and Gardent, 2016, Zhang and Lapata, 2017, Narayan et al., 2017], text summarisation [Wan, 2010, Nenkova and McKeown, 2011, Woodsend and Lapata, 2010, Rush et al., 2015, Cheng and Lapata, 2016, Nallapati et al., 2016, Chen et al., 2016, Tan and Wan, 2017, See et al., 2017, Nallapati et al., 2017, Paulus et al., 2017, Yasunaga et al., 2017, Narayan et al., 2018a, Narayan et al., 2018b, Pasunuru and Bansal, 2018, Celikyilmaz et al., 2018] and end-to-end dialogue systems [Li et al., 2017]. Following the success of encoder-decoder models in modeling sequence-rewriting tasks such as machine translation [Sutskever et al., 2011, Bahdanau et al., 2014], deep learning models have successfully been applied to the various text production tasks. For instance, [Rush et al., 2015] utilize a local attention-based model for abstractive summarisation, [Shang et al., 2015] propose an encoder-decoder model for response generation, [See et al., 2017] uses a hybrid of encoder-decoder model and pointer network [Vinyals et al., 2015] for story highlight generation, [Dong et al., 2017] exploits an encoder-decoder model for question rephrasing and [Konstas et al., 2017] for AMR generation. In this tutorial, we will cover the fundamentals and the state-of-the-art research on neural models for text production. Each text production task raises a slightly different communication goal (e.g, how to take the dialogue context into account when producing a dialogue turn; how to detect and merge relevant information when summarising a text; or how to produce a well-formed text that correctly capture the information contained in some input data in the case of data-to-text generation). We will outline the constraints specific to each subtasks and examine how the existing neural models account for them.

When heckled, professional comedians frequently lament that “everyone’s a comedian!” It’s easy to see why: professional comedians don’t possess different kinds of brains from others or engage in radically different kinds of behaviors from others, and moreover, the success of their acts is predicated on others’ shared ability to understand and reason about comic situations they describe. The difference between comedians and their audience is a matter not of kind, but of degree, a difference that is reflected in the vocational emphasis they place on humor. Researchers in the field of computational creativity find themselves in a similar situation. As a subdiscipline of artificial intelligence, computational creativity explores theories and practices that give rise to a phenomenon, creativity, that all intelligent systems, human or machine, can legitimately lay claim to. Who is to say that a given AI system is not creative, insofar as it solves nontrivial problems or generates useful outputs that are not hard wired into its programming? As with comedians’ being funny, the difference between studying computational creativity and studying artificial intelligence is one of emphasis rather than one of kind: the field of computational creativity, as typified by a long-running series of workshops at AI-related conferences, places a vocational emphasis on creativity and attempts to draw together the commonalities of what Articles

The main task we address in our research is classification of text using fine-grained attitude labels. The developed @AM system relies on the compositionality principle and a novel approach based on the rules elaborated for semantically distinct verb classes. The evaluation of our method on 1000 sentences, that describe personal experiences, showed promising results: average accuracy on the finegrained level (14 labels) was 62%, on the middle level (7 labels) – 71%, and on the top level (3 labels) – 88%.

Domain dependence of NLP systems is one of the major obstacles to their application in large-scale text analysis, also restricting the applicability of FrameNet semantic role labeling (SRL) systems. Yet, current FrameNet SRL systems are still only evaluated on a single in-domain test set. For the first time, we study the domain dependence of FrameNet SRL on a wide range of benchmark sets. We create a novel test set for FrameNet SRL based on user-generated web text and find that the major bottleneck for out-of-domain FrameNet SRL is the frame identification step. To address this problem, we develop a simple, yet efficient system based on distributed word representations. Our system closely approaches the state-of-the-art in-domain while outperforming the best available frame identification system out-of-domain. We publish our system and test data for research purposes.1

Existing target-specific sentiment recognition methods consider only a single target per tweet, and have been shown to miss nearly half of the actual targets mentioned. We present a corpus of UK election tweets, with an average of 3.09 entities per tweet and more than one type of sentiment in half of the tweets. This requires a method for multi-target specific sentiment recognition, which we develop by using the context around a target as well as syntactic dependencies involving the target. We present results of our method on both a benchmark corpus of single targets and the multi-target election corpus, showing state-of-the art performance in both corpora and outperforming previous approaches to multi-target sentiment task as well as deep learning models for singletarget sentiment.

This Doctoral Consortium paper discusses how NLP can be applied in the domain of BPM in order to automatically generate business process models from existing documentation within the organization. The main idea is that from the syntactic and grammatical structure of a sentence, the components of a business process model can be derived (i.e. activities, resources, tasks, patterns). The result would be a business process model depicted using BPMN - a dedicated business process modeling technique

The utility of additional semantic information for the task of next utterance selection in an automated dialogue system is the focus of study in this paper. In particular, we show that additional information available in the form of dialogue acts –when used along with context given in the form of dialogue history– improves the performance irrespective of the underlying model being generative or discriminative. In order to show the model agnostic behavior of dialogue acts, we experiment with several well-known models such as sequence-to-sequence encoder-decoder model, hierarchical encoder-decoder model, and Siamese-based models with and without hierarchy; and show that in all models, incorporating dialogue acts improves the performance by a significant margin. We, furthermore, propose a novel way of encoding dialogue act information, and use it along with hierarchical encoder to build a model that can use the sequential dialogue act information in a natural way. Our proposed model achieves an MRR of about 84.8% for the task of next utterance selection on a newly introduced DailyDialog dataset, and outperform the baseline models. We also provide a detailed analysis of results including key insights that explain the improvement in MRR because of dialogue act information.


Question: SarcasmBot: An open-source module for generating sarcasm in chatbot conversations.

This paper delves into the practical application and scalability of logistic regression, a staple in statistical data analysis, within the realm of data mining, particularly for large, sparse datasets with binary outcomes. Despite its widespread use in traditional statistical contexts, logistic regression's presence in data mining is notably sparse. This scarcity can be attributed to two prevailing misconceptions. Firstly, there's a perception that linear classification models like logistic regression are inherently inferior to more sophisticated, nonlinear algorithms. Secondly, there's a genuine concern regarding logistic regression's ability to handle the vast volumes of data characteristic of modern data mining environments. Through a comprehensive empirical study, this paper aims to challenge the first notion by demonstrating logistic regression's surprising effectiveness. Additionally, it explores and evaluates various strategies to scale logistic regression to accommodate datasets featuring millions of attributes and records. The findings, derived from a significant life sciences dataset, reveal that logistic regression not only holds its own against a range of contemporary classification methods but also excels in both statistical accuracy and computational efficiency. This research underscores the enduring relevance and potential of logistic regression in the data mining field, highlighting its capacity to perform admirably even in the face of large, complex data landscapes.

The increasing use of social networks and online sites where people can express their opinions has created a growing interest in Opinion Mining. One of the main tasks of Opinion Mining is to determine whether an opinion is positive or negative. Therefore, the role of the feelings expressed on the web has become crucial, mainly due to the concern of businesses and government to automatically identify the semantic orientation of the views of customers or citizens. This is also a concern, in the area of health to identify psychological disorders. This research focuses on the development of a web application called SWePT (Web Service for Polarity detection in Spanish Texts), which implements the Sequential Minimal Optimization (SMO) algorithm, extracting its features from an affective lexicon in Mexican Spanish. For this purpose, a corpus and an affective lexicon in Mexican Spanish were created. The experiments using three (positive, neutral, negative) and five categories (very positive, positive, neutral, negative, and very negative) allow us to demonstrate the effectiveness of the presented method. SWePT has also been implemented in the Emotion-bracelet interface, which shows the opinion of a user graphically.

Automated scoring of written and spoken responses is an NLP application that can significantly impact lives especially when deployed as part of high-stakes tests such as the GRE® and the TOEFL®. Ethical considerations require that automated scoring algorithms treat all testtakers fairly. The educational measurement community has done significant research on fairness in assessments and automated scoring systems must incorporate their recommendations. The best way to do that is by making available automated, non-proprietary tools to NLP researchers that directly incorporate these recommendations and generate the analyses needed to help identify and resolve biases in their scoring systems. In this paper, we attempt to provide such a solution.

Designing the dialogue policy of a spoken dialogue system involves many nontrivial choices. This paper presents a reinforcement learning approach for automatically optimizing a dialogue policy, which addresses the technical challenges in applying reinforcement learning to a working dialogue system with human users. We report on the design, construction and empirical evaluation of NJFun, an experimental spoken dialogue system that provides users with access to information about fun things to do in New Jersey. Our results show that by optimizing its performance via reinforcement learning, NJFun measurably improves system performance.

Social interactions take place in environments that influence people’s behaviours and perceptions. Nowadays, the users of Online Social Network (OSN) generate a massive amount of content based on social interactions. However, OSNs wide popularity and ease of access created a perfect scenario to practice malicious activities, compromising their reliability. To detect automatic information broadcast in OSN, we developed a wavelet-based model that classifies users as being human, legitimate robot, or malicious robot, as a result of spectral patterns obtained from users’ textual content. We create the feature vector from the Discrete Wavelet Transform along with a weighting scheme called Lexicon-based Coefficient Attenuation. In particular, we induce a classification model using the Random Forest algorithm over two real Twitter datasets. The corresponding results show the developed model achieved an average accuracy of 94.47% considering two different scenarios: single theme and miscellaneous one.

SEMAINE has created a large audiovisual database as a part of an iterative approach to building Sensitive Artificial Listener (SAL) agents that can engage a person in a sustained, emotionally colored conversation. Data used to build the agents came from interactions between users and an "operator” simulating a SAL agent, in different configurations: Solid SAL (designed so that operators displayed an appropriate nonverbal behavior) and Semi-automatic SAL (designed so that users' experience approximated interacting with a machine). We then recorded user interactions with the developed system, Automatic SAL, comparing the most communicatively competent version to versions with reduced nonverbal skills. High quality recording was provided by five high-resolution, high-framerate cameras, and four microphones, recorded synchronously. Recordings total 150 participants, for a total of 959 conversations with individual SAL characters, lasting approximately 5 minutes each. Solid SAL recordings are transcribed and extensively annotated: 6-8 raters per clip traced five affective dimensions and 27 associated categories. Other scenarios are labeled on the same pattern, but less fully. Additional information includes FACS annotation on selected extracts, identification of laughs, nods, and shakes, and measures of user engagement with the automatic system. The material is available through a web-accessible database.

Title of dissertation: STRUCTURED APPROACHES FOR EXPLORING INTERPERSONAL RELATIONSHIPS IN NATURAL LANGUAGE TEXT Snigdha Chaturvedi, Doctor of Philosophy, 2016 Dissertation directed by: Dr. Hal Daumé III Department of Computer Science Human relationships have long been studied by scientists from domains like sociology, psychology, literature, etc. for understanding people’s desires, goals, actions and expected behaviors. In this dissertation we study inter-personal relationships as expressed in natural language text. Modeling inter-personal relationships from text finds application in general natural language understanding, as well as real-world domains such as social networks, discussion forums, intelligent virtual agents, etc. We propose that the study of relationships should incorporate not only linguistic cues in text, but also the contexts in which these cues appear. Our investigations, backed by empirical evaluation, support this thesis, and demonstrate that the task benefits from using structured models that incorporate both types of information. We present such structured models to address the task of modeling the nature of relationships between any two given characters from a narrative. To begin with, we assume that relationships are of two types: cooperative and non-cooperative. We first describe an approach to jointly infer relationships between all characters in the narrative, and demonstrate how the task of characterizing the relationship between two characters can benefit from including information about their relationships with other characters in the narrative. We next formulate the relationshipmodeling problem as a sequence prediction task to acknowledge the evolving nature of human relationships, and demonstrate the need to model the history of a relationship in predicting its evolution. Thereafter, we present a data-driven method to automatically discover various types of relationships such as familial, romantic, hostile, etc. Like before, we address the task of modeling evolving relationships but don’t restrict ourselves to two types of relationships. We also demonstrate the need to incorporate not only local historical but also global context while solving this problem. Lastly, we demonstrate a practical application of modeling inter-personal relationships in the domain of online educational discussion forums. Such forums offer opportunities for its users to interact and form deeper relationships. With this view, we address the task of identifying initiation of such deeper relationships between a student and the instructor. Specifically, we analyze contents of the forums to automatically suggest threads to the instructors that require their intervention. By highlighting scenarios that need direct instructor-student interactions, we alleviate the need for the instructor to manually peruse all threads of the forum and also assist students who have limited avenues for communicating with instructors. We do this by incorporating the discourse structure of the thread through latent variables that abstractly represent contents of individual posts and model the flow of information in the thread. Such latent structured models that incorporate the linguistic cues without losing their context can be helpful in other related natural language understanding tasks as well. We demonstrate this by using the model for a very different task: identifying if a stated desire has been fulfilled by the end of a story.

We propose the first unsupervised approach to the problem of modeling dialogue acts in an open domain. Trained on a corpus of noisy Twitter conversations, our method discovers dialogue acts by clustering raw utterances. Because it accounts for the sequential behaviour of these acts, the learned model can provide insight into the shape of communication in a new medium. We address the challenge of evaluating the emergent model with a qualitative visualization and an intrinsic conversation ordering task. This work is inspired by a corpus of 1.3 million Twitter conversations, which will be made publicly available. This huge amount of data, available only because Twitter blurs the line between chatting and publishing, highlights the need to be able to adapt quickly to a new medium.

extraordinarily difficult and poses challenges for both integration and grounding. Here, I propose a unifying idea that meets both challenges: language games. A language game is a sequence of verbal interactions between two agents situated in a specific environment. Language games both integrate the various activities required for dialogue and ground unknown words or phrases in a specific context, which helps constrain possible meanings. Over the past five years, I have been working with a team to develop and test language games on progressively more sophisticated systems, from relatively simple camera-based systems to humanoid robots. The results of our work show that language games are a useful way to both understand and design human–robot interaction.

We introduce Slant, a system that integrates more than a decade of research into computational creativity, and specifically story generation, by connecting subsystems that deal with plot, figuration, and the narrative discourse using a blackboard. The process of integrating these systems highlights differences in the representation of story and has led to a better understanding of how story can be usefully abstracted. The plot generator MEXICA and a component of Curveship are used with little modification in Slant, while the figuration subsystem Fig-S and the template generator GRIOT-Gen, inspired by GRIOT, are also components. The development of the new subsystem Verso, which deals with genre, shows how different genres can be computationally modeled and applied to in-development stories to generate results that are surprising in terms of their connections and valuable in terms of their relationship to cultural questions. Example stories are discussed, as is the potential of the system to allow for broader collaboration, the empirical testing of how subsystems interrelate, and possible contributions in literary and artistic contexts.

The problem of understanding the stream of messages exchanged on social media such as Facebook and Twitter is becoming a major challenge for automated systems. The tremendous amount of data exchanged on these platforms as well as the specific form of language adopted by social media users constitute a new challenging context for existing argument mining techniques. In this paper, we describe a resource of natural language arguments called DART (Dataset of Arguments and their Relations on Twitter) where the complete argument mining pipeline over Twitter messages is considered: (i) we identify which tweets can be considered as arguments and which cannot, and (ii) we identify what is the relation, i.e., support or attack, linking such tweets to each other.


Question: SarcasmBot: An open-source module for generating sarcasm in chatbots.

Introducing TnT: An Advanced Statistical Part-of-Speech Tagger
Trigrams'n'Tags (TnT) represents a sophisticated and efficient statistical part-of-speech tagging system. Contrary to assertions found in some literature, we assert that a tagger grounded in Markov models can achieve performance on par with or even surpassing that of contemporary methodologies, such as the Maximum Entropy framework. Recent evaluations have substantiated this claim, demonstrating that TnT outperforms competitors on tested corpora. In this paper, we detail the foundational model of TnT, elucidating the smoothing techniques and strategies employed for managing unknown words. Additionally, we provide empirical results from evaluations conducted on two distinct corpora, showcasing the robustness and accuracy of our statistical tagger.

We describe our Alexa prize system (called ‘Alana’) which consists of an ensemble of bots, combining rule-based and machine learning systems, and using a contextual ranking mechanism to choose system responses. This paper reports on the version of the system developed and evaluated in the semi-finals of the competition (i.e. up to 15 August 2017), but not on subsequent enhancements. The ranker for this system was trained on real user feedback received during the competition, where we address the problem of how to train on the noisy and sparse feedback obtained during the competition. In order to avoid initial problems of inappropriate and boring utterances coming from big datasets such as Reddit and Twitter, we later focussed on ‘clean’ data sources such as news and facts. We report on experiments with different ranking functions and versions of our NewsBot. We find that a multiturn news strategy is beneficial, and that a ranker trained on the ratings feedback from users is also effective. Our system continuously improved using the data gathered over the course over the competition (1 July – 15 August) . Our final user score (averaged user rating over the whole semi-finals period) was 3.12, and we achieved 3.3 for the averaged user rating over the last week of the semi-finals (8-15 August 2017). We were also able to achieve long dialogues (average 10.7 turns) during the competition period. In subsequent weeks, after the end of the semi-final competition, we have achieved our highest scores of 3.52 (daily average, 18th October), 3.45 (weekly average on 23 and 24 October), and average dialogue lengths of 14.6 turns (1 October), and median dialogue length of 2.25 minutes (average for 7 days on 10th October).

We present a dialogue generation model that directly captures the variability in possible responses to a given input, which reduces the ‘boring output’ issue of deterministic dialogue models. Experiments show that our model generates more diverse outputs than baseline models, and also generates more consistently acceptable output than sampling from a deterministic encoder-decoder model.

We propose an interactive multimodal framework for language learning. Instead of being passively exposed to large amounts of natural text, our learners (implemented as feed-forward neural networks) engage in cooperative referential games starting from a tabula rasa setup, and thus develop their own language from the need to communicate in order to succeed at the game. Preliminary experiments provide promising results, but also suggest that it is important to ensure that agents trained in this way do not develop an adhoc communication code only effective for the game they are playing.

UPDATED—11 December 2017. Artificial Intelligent Chatbots are used in industries such as Banking systems, Customer services, Education. There are many intelligent tutoring systems currently in practice, but few of them are known to assist high school students for learning their general knowledge subjects. This project paper proposes an artificial intelligent chat-bot tool developed to assist high school students for learning their general knowledge subjects. The proposed intelligent chat-bot tool is a web based intelligent tutoring system which can be accessed by large number of students globally, the users are not required to pay licensing fees. The intelligent chat-bot tool will be available 24 hours a day at schools or public libraries or any other location which has internet access. The intelligent chatbot tool will make use of Natural Language Processing techniques to answer the queries by high school students. The intelligent chat-bot tool will be trained on a knowledge base comprising of general knowledge questions and answers. The intelligent chat-bot tool will have ability to participate in small talks with its learners.

The machine learning approach is represented with two methods: the maximum entropy method and support vector machine. Text representation for the maximum entropy method includes the information about the proportion of positive and negative words and collocations, the quantity of interrogation and exclamation marks, emoticons, obscene language. For the support vector machine binary vectors with cosine normalization are built on texts.

As the Web evolves towards a service-oriented architecture, application program interfaces (APIs) are becoming an increasingly important way to provide access to data, services, and devices. We study the problem of natural language interface to APIs (NL2APIs), with a focus on web APIs for web services. Such NL2APIs have many potential benefits, for example, facilitating the integration of web services into virtual assistants.
 We propose the first end-to-end framework to build an NL2API for a given web API. A key challenge is to collect training data, i.e., NL command-API call pairs, from which an NL2API can learn the semantic mapping from ambiguous, informal NL commands to formal API calls. We propose a novel approach to collect training data for NL2API via crowdsourcing, where crowd workers are employed to generate diversified NL commands. We optimize the crowdsourcing process to further reduce the cost. More specifically, we propose a novel hierarchical probabilistic model for the crowdsourcing process, which guides us to allocate budget to those API calls that have a high value for training NL2APIs. We apply our framework to real-world APIs, and show that it can collect high-quality training data at a low cost, and build NL2APIs with good performance from scratch. We also show that our modeling of the crowdsourcing process can improve its effectiveness, such that the training data collected via our approach leads to better performance of NL2APIs than a strong baseline.

This work documents the recent rise in popularity of messaging bots: chatterbot-like agents with simple, textual interfaces that allow users to access information, make use of services, or provide entertainment through online messaging platforms. Conversational interfaces have been often studied in their many facets, including natural language processing, artificial intelligence, human-computer interaction, and usability. In this work we analyze the recent trends in chatterbots and provide a survey of major messaging platforms, reviewing their support for bots and their distinguishing features. We then argue for what we call "Botplication", a bot interface paradigm that makes use of context, history, and structured conversation elements for input and output in order to provide a conversational user experience while overcoming the limitations of text-only interfaces.

This article introduces a sentiment analysis approach that adopts the way humans read, interpret, and extract sentiment from text. Our motivation builds on the assumption that human interpretation should lead to the most accurate assessment of sentiment in text. We call this automated process Human Reading for Sentiment (HRS). Previous research in sentiment analysis has produced many frameworks that can fit one or more of the HRS aspects; however, none of these methods has addressed them all in one approach. HRS provides a meta-framework for developing new sentiment analysis methods or improving existing ones. The proposed framework provides a theoretical lens for zooming in and evaluating aspects of any sentiment analysis method to identify gaps for improvements towards matching the human reading process. Key steps in HRS include the automation of humans low-level and high-level cognitive text processing. This methodology paves the way towards the integration of psychology with computational linguistics and machine learning to employ models of pragmatics and discourse analysis for sentiment analysis. HRS is tested with two state-of-the-art methods; one is based on feature engineering, and the other is based on deep learning. HRS highlighted the gaps in both methods and showed improvements for both.

Natural language generation lies at the core of generative dialogue systems and conversational agents. We describe an ensemble neural language generator, and present several novel methods for data representation and augmentation that yield improved results in our model. We test the model on three datasets in the restaurant, TV and laptop domains, and report both objective and subjective evaluations of our best model. Using a range of automatic metrics, as well as human evaluators, we show that our approach achieves better results than state-of-the-art models on the same datasets.

We study automatic question generation for sentences from text passages in reading comprehension. We introduce an attention-based sequence learning model for the task and investigate the effect of encoding sentencevs. paragraph-level information. In contrast to all previous work, our model does not rely on hand-crafted rules or a sophisticated NLP pipeline; it is instead trainable end-to-end via sequenceto-sequence learning. Automatic evaluation results show that our system significantly outperforms the state-of-the-art rule-based system. In human evaluations, questions generated by our system are also rated as being more natural (i.e., grammaticality, fluency) and as more difficult to answer (in terms of syntactic and lexical divergence from the original text and reasoning needed to answer).


Question: SarcasmBot: An open-source module for generating sarcasm in chatbots.

This paper explores novel training methodologies for Hidden Markov Models (HMMs), offering an alternative to traditional approaches such as maximum-entropy models and conditional random fields (CRFs). The proposed algorithms employ Viterbi decoding on training data, coupled with straightforward additive updates, to optimize model performance. The theoretical underpinnings of these algorithms are grounded in an adapted version of the perceptron algorithm's convergence proof for classification tasks. To substantiate the efficacy of these new algorithms, we present empirical findings from part-of-speech tagging and base noun phrase chunking tasks. In both cases, our results outperform those obtained using a maximum-entropy tagger, highlighting the potential of these methods in enhancing tagging accuracy and efficiency.

With people increasingly using emoticons in written text on the Web in order to express, stress, or disambiguate their sentiment, it is crucial for automated sentiment analysis tools to correctly account for such graphical cues for sentiment. We analyze how emoticons typically convey sentiment and we subsequently propose and evaluate a novel method for exploiting this with a manually created emoticon sentiment lexicon in a lexicon-based polarity classification method. We evaluate our approach on 2,080 Dutch tweets and forum messages, which all contain emoticons. We validate our findings on 10,069 English reviews of apps, some of which contain emoticons. We find that accounting for the sentiment conveyed by emoticons on a paragraph level – and, to a lesser extent, on a sentence level – significantly improves polarity classification performance. Whenever emoticons are used, their associated sentiment tends to dominate the sentiment conveyed by textual cues and forms a good proxy for the polarity of text.

The natural language generation (NLG) component of a spoken dialogue system (SDS) usually needs a substantial amount of handcrafting or a well-labeled dataset to be trained on. These limitations add significantly to development costs and make cross-domain, multi-lingual dialogue systems intractable. Moreover, human languages are context-aware. The most natural response should be directly learned from data rather than depending on predefined syntaxes or rules. This paper presents a statistical language generator based on a joint recurrent and convolutional neural network structure which can be trained on dialogue act-utterance pairs without any semantic alignments or predefined grammar trees. Objective metrics suggest that this new model outperforms previous methods under the same experimental conditions. Results of an evaluation by human judges indicate that it produces not only high quality but linguistically varied utterances which are preferred compared to n-gram and rule-based systems.

We present the SimSensei system, a fully automatic virtual agent that conducts interviews to assess indicators of psychological distress. We emphasize on the perception part of the system, a multimodal framework which captures and analyzes user state for both behavioral understanding and interactional purposes.

We propose the first unsupervised approach to the problem of modeling dialogue acts in an open domain. Trained on a corpus of noisy Twitter conversations, our method discovers dialogue acts by clustering raw utterances. Because it accounts for the sequential behaviour of these acts, the learned model can provide insight into the shape of communication in a new medium. We address the challenge of evaluating the emergent model with a qualitative visualization and an intrinsic conversation ordering task. This work is inspired by a corpus of 1.3 million Twitter conversations, which will be made publicly available. This huge amount of data, available only because Twitter blurs the line between chatting and publishing, highlights the need to be able to adapt quickly to a new medium.

This paper presents a new approach to phrase-level sentiment analysis that first determines whether an expression is neutral or polar and then disambiguates the polarity of the polar expressions. With this approach, the system is able to automatically identify thecontextual polarityfor a large subset of sentiment expressions, achieving results that are significantly better than baseline.

Natural language generation (NLG) is a critical component of spoken dialogue and it has a significant impact both on usability and perceived quality. Most NLG systems in common use employ rules and heuristics and tend to generate rigid and stylised responses without the natural variation of human language. They are also not easily scaled to systems covering multiple domains and languages. This paper presents a statistical language generator based on a semantically controlled Long Short-term Memory (LSTM) structure. The LSTM generator can learn from unaligned data by jointly optimising sentence planning and surface realisation using a simple cross entropy training criterion, and language variation can be easily achieved by sampling from output candidates. With fewer heuristics, an objective evaluation in two differing test domains showed the proposed method improved performance compared to previous methods. Human judges scored the LSTM system higher on informativeness and naturalness and overall preferred it to the other systems.

The way we speak can reveal much about what we intend to communicate, but the words we use often only indirectly relate to the meanings we wish to convey. Verbal irony is a commonly studied form of indirect speech in which a speaker produces an explicit evaluative utterance that implicates an unstated, opposing evaluation. Producing and understanding ironic language, as well as many other types of indirect speech, requires the ability to recognize mental states in others, sometimes described as a capacity for metarepresentation. This article aims to connect common elements between the major theoretical approaches to verbal irony to recent psycholinguistic, developmental, and neuropsychological research demonstrating the necessity for metarepresentation in the effective use of verbal irony in social interaction. Here I will argue that verbal irony is one emergent, strategic possibility given the interface between people’s ability to infer mental states, and use language. Rather than think of ironic communication as a specialized cognitive ability, I will claim that it arises from the same set of abilities that underlie a wide range of inferential communicative behaviors. Language interaction involves a complex interplay of many cognitive abilities. Theorists across many disciplines struggle with questions of how to carve up these abilities, and whether they can be carved up in the first place (e.g., Christiansen and Chater 2008). Among the most difficult questions to address are those that involve how people recognize intentions in others’ behavior, and how that is achieved through language in the context of many other sources of information. The way we speak can reveal much about what we intend to communicate, but the words we use are often quite different from the meanings we wish to convey. People often speak indirectly for a variety of strategic reasons, and these strategies rely intrinsically on social cognition. One well-researched example of this is the phenomenon of verbal irony – a type of indirect speech in which a speaker produces an explicit evaluative utterance that implicates an unstated, opposing evaluation. As described below, this trope has been traditionally defined in rather vague terms, but has generated a great deal of research as a phenomenon in need of special theoretical explanation. Here I will argue that verbal irony is one emergent, strategic possibility given the interface between people’s ability to infer the mental states of others, and use language. Rather than think of ironic communication as a unique skill or specialized cognitive ability, I suggest that it arises from the same set of abilities that underlie a wide range of inferential communicative behaviors. The study of verbal irony production and comprehension has been a multidisciplinary effort lacking extensive interdisciplinary exchange. There have been various taxonomies and definitions of what constitutes irony, both verbal and situational – a struggling enterprise to say the least (Colston and Gibbs 2007). The traditional notion of verbal irony dates back at least to the Roman rhetorician Quintilian, who described irony as a kind of allegory expressing the opposite of what one means, often for mocking Language and Linguistics Compass 6/11 (2012): 673–685, 10.1002/lnc3.364 a 2012 The Author Language and Linguistics Compass a 2012 Blackwell Publishing Ltd communicative effects. Some of the assumptions of this early categorization have carried over into modern psycholinguistic research without proper consideration of underlying communicative functions (Kreuz 2000). Many scholars have linked verbal irony to situational irony – particularly in how both forms seem to involve, at some level, a contradiction between what is expected, and what occurs. While verbal irony often points to unexpected or unfortunate outcomes in people’s actions (Kumon-Nakamura, Glucksberg, and Brown 1995; Pexman 2008), it does not do so necessarily. At present, most contemporary researchers of figurative language and inferential communication have settled on some version of the basic claim that verbal irony is a class of indirect language use where explicit sentence meanings are conceptually contradictory to a network of related implied propositions. Many other tropes, however, are not obviously distinct (e.g., parody, pretense, double entendre, litotes, etc). One common thread between many theoretical and empirical approaches to verbal irony is the distinction language users must make between layers of meaning. This could be the difference between actual and attributed beliefs (Kreuz and Glucksberg 1989; Sperber and Wilson 1986, 1995), real and imagined discourse acts (Clark and Gerrig 1984), relevant or inappropriate to a particular context (Attardo 2000), what is negated and what is implicated (Giora 1995), or failed expectations and attitudes toward those failures (Kumon-Nakamura, Glucksberg, and Brown 1995). Speakers implicitly highlight the contrasts between these different levels of meaning, and in doing so communicate an attitude toward an attributed belief (Sperber and Wilson 1986, 1995). This attitudinal dissociation is typically considered a hallmark of ironic communication, and necessarily requires an implicit understanding of others’ mental states. Consider the following exchange: John: I got another virus on my laptop. Mary: Aren’t you glad you didn’t switch to Mac? In this example John and Mary are discussing John’s computer problems. John reports that he got a virus on his computer – not his first. Mary asks an ironic rhetorical question expressing her negative opinion of Windows PCs. The literal question is in particular opposition to the implied evaluation – the essence of irony. In this example, the irony is largely contained in a single lexical opposition – ‘‘glad’’ instead of, for example, ‘‘sad,’’ but is also communicated by the rhetorical aspect of the question. The question form is one format speakers use to express ironic intentions, but ironic forms can manifest themselves in various ways linguistically, and in a variety of media including television, music, and visual art (Bryant 2011; El Refaie 2005; Ettema and Glasser 1994; Nuolijarvi and Tiittula 2010; Scott 2003). Figurative language devices such as verbal irony can be quite powerful communicative tools. The sentences we speak are often produced in a manner that will allow our audience to derive certain unstated meanings, and the way we put them is often particularly economical because of our heavy reliance on others’ inferential abilities. These meanings are not just efficient, but unique to indirect communication (Gibbs 2000a). The combination of affect directed toward some attributed belief, along with a direct utterance, makes ironic meanings difficult, if not impossible, to precisely capture. But this is due to inferential processes associated with mindreading and linguistic communication, not some distinct process of irony understanding. Similarly, it is difficult to capture, in linguistic terms, many kinds of inferences regarding others’ intentions that people generate in social contexts. In our example, Mary, of course, communicates much more than a negative opinion of PCs. For instance, she implicitly reminds John of previous interactions, some being debates, when their differences of opinion about computer platforms were discussed (i.e., 674 Gregory A. Bryant a 2012 The Author Language and Linguistics Compass 6/11 (2012): 673–685, 10.1002/lnc3.364 Language and Linguistics Compass a 2012 Blackwell Publishing Ltd she echoes previous opinions expressed, perhaps even through literal quotations) (Kreuz and Glucksberg 1989; Sperber and Wilson 1986, 1995). In doing so she reiterates her claim that John’s decision was a mistake, and that he would regret it. By using an ironic rhetorical question, Mary efficiently conveys a complex set of implied meanings that are relevant to John. The irony is not necessarily obvious to an outsider, as is the case with most occurrences of irony in everyday conversations (Gibbs 2000b), but John understands Mary immediately, and Mary knew he would. In the process, Mary potentially reduces John’s perception that she is being critical, and makes him laugh (while actually criticizing him) (Dews and Winner 1995). Alternatively, Mary might be accentuating her criticism by highlighting a contrast between John’s current situation and his earlier stated preference (Colston 1997). Speakers attempt to fulfill particular communicative goals by using indirect language such as verbal irony, and different forms can result in quite different emotional reactions – we should expect it to be used strategically and variably (Bryant 2011; Bryant and Fox Tree 2005; Leggitt and Gibbs 2000). Scholars have suggested many functions for irony in discourse, including speakers’ attempts to be humorous, create solidarity, appear clever, increase memorability, save face, be polite, or alter the valence of an attack or praise (Gibbs 1994; Roberts and Kreuz 1994; Toplak and Katz 2000). The pragmatic uses of ironic speech are far-reaching – ‘‘what can’t irony do?’’ might be a better question. One reason that irony can fulfill so many discourse goals is that includable tokens in the category need only satisfy a couple conditions; thus, an enormous variety of speech acts can qualify. Burgers, van Mulken, and Schellens (2011) recently proposed a procedure for identifying ironies in discourse, and for a basic definition, they looked at the commonalities between different contemporary theories. In their view, verbal irony involves an expressed evaluative utterance that implies an opposing evaluative appraisal. This implied meaning does not need to be the opposite of the stated meaning, but just differ in a scalar manner. For instance, an understatement might still be negative, but less negative than the intended meaning (e.g., when asked about a personal experience with a

Past works on personality detection has shown that psycho-linguistic features, frequency based analysis at lexical level , emotive words and other lexical clues such as number of first person or sec ond person words carry major role to identify personality associated with the te xt. In this work, we propose a new architecture for the same task using common sen se knowledge with associated sentiment polarity and affective labels. To extract the common sense knowledge with sentiment polarity scores and affect iv labels we used Senticnet which is one of the most useful resources for opinion mining and sentiment analysis. In particular, we combined comm n sense knowledge based features with phycho-linguistic features and freque ncy based features and later the features were employed in supervised classifier . We designed five SMO based supervised classifiers for five personality t raits. We observe that the use of common sense knowledge with affective and sentim ent information enhances the accuracy of the existing frameworks which use o nly psycho-linguistic features and frequency based analysis at lexical le ve .

Sentiment in social media is increasingly considered as an important resource for customer segmentation, market understanding, and tackling other socio-economic issues. However, sentiment in social media is difficult to measure since user-generated content is usually short and informal. Although many traditional sentiment analysis methods have been proposed, identifying slang sentiment words remains untackled. One of the reasons is that slang sentiment words are not available in existing dictionaries or sentiment lexicons. To this end, we propose to build the first sentiment dictionary of slang words to aid sentiment analysis of social media content. It is laborious and time-consuming to collect and label the sentiment polarity of a comprehensive list of slang words. We present an approach to leverage web resources to construct an extensive Slang Sentiment word Dictionary (SlangSD) that is easy to maintain and extend. SlangSD is publicly available1 for research purposes. We empirically show the advantages of using SlangSD, the newly-built slang sentiment word dictionary for sentiment classification, and provide examples demonstrating its ease of use with an existing sentiment system.

Similes are easily obtained from web-driven and casebased reasoning approaches. Still, generating thoughtful figurative descriptions with meaningful relation to narrative context and author style has not yet been fully explored. In this paper, the author prepares the foundation for a computational model which can achieve this level of aesthetic complexity. This paper also introduces and evaluates a possible architecture for generating and ranking figurative comparisons on par with humans: the


Question: Aggregated motion features for first-person video analysis.

This paper presents an enhanced approach for action recognition in videos, focusing on the refinement of dense trajectories as a key representation. Recent studies have demonstrated the efficacy of dense trajectories in achieving state-of-the-art performance on various action recognition datasets. Building upon this foundation, we introduce an improvement that accounts for camera motion, thereby enhancing trajectory accuracy. Our method involves the utilization of feature point matching between frames, employing both SURF descriptors and dense optical flow to capture complementary information. These matched points are subsequently utilized in a robust homography estimation process facilitated by RANSAC. Acknowledging that human motion often deviates from camera motion, potentially leading to inconsistent matches, we incorporate a human detector to filter out such mismatches. With the camera motion estimation in hand, we selectively discard trajectories that align closely with this motion, thus isolating the human motion component. Additionally, we apply this motion estimation to adjust the optical flow, effectively neutralizing the influence of camera movement on motion-based descriptors like HOF and MBH. Our experimental findings, obtained on four prominent action datasets (Hollywood2, HMDB51, Olympic Sports, and UCF50), showcase a substantial performance boost over existing state-of-the-art methods.

**Question**: A Direct Search Method to solve Economic Dispatch Problem with Valve-Point Effect

**Documents**: An enhanced particle swarm optimization for economic dispatch with valve-point effect
The economic dispatch problem is a well-known non-convex optimization problem due to the non-smooth cost function caused by the valve-point effect. In this paper, an enhanced particle swarm optimization (EPSO) algorithm is proposed to solve the economic dispatch problem with valve-point effect. The proposed EPSO algorithm is a modified PSO algorithm with an adaptive inertia weight and a new local search mechanism. The adaptive inertia weight is used to balance the exploration and exploitation abilities of the algorithm. The new local search mechanism is used to improve the search efficiency of the algorithm. The proposed EPSO algorithm is tested on three standard test systems and compared with several state-of-the-art algorithms. The results show that the proposed EPSO algorithm can effectively solve the economic dispatch problem with valve-point effect and obtain better solutions than other algorithms.

**Similar Documents**: This paper introduces an advanced particle swarm optimization (APSO) technique tailored for tackling the economic dispatch problem, particularly in scenarios complicated by the valve-point effect. The valve-point effect contributes to the non-convexity of the cost function, making the economic dispatch problem a challenging optimization task. The proposed APSO method represents an evolution of the traditional

We present a novel algorithm for automatically applying constrainable, L1-optimal camera paths to generate stabilized videos by removing undesired motions. Our goal is to compute camera paths that are composed of constant, linear and parabolic segments mimicking the camera motions employed by professional cinematographers. To this end, our algorithm is based on a linear programming framework to minimize the first, second, and third derivatives of the resulting camera path. Our method allows for video stabilization beyond the conventional filtering of camera paths that only suppresses high frequency jitter. We incorporate additional constraints on the path of the camera directly in our algorithm, allowing for stabilized and retargeted videos. Our approach accomplishes this without the need of user interaction or costly 3D reconstruction of the scene, and works as a post-process for videos from any camera or from an online source.

Human activity recognition is an important area of computer vision research. Its applications include surveillance systems, patient monitoring systems, and a variety of systems that involve interactions between persons and electronic devices such as human-computer interfaces. Most of these applications require an automated recognition of high-level activities, composed of multiple simple (or atomic) actions of persons. This article provides a detailed overview of various state-of-the-art research papers on human activity recognition. We discuss both the methodologies developed for simple human actions and those for high-level activities. An approach-based taxonomy is chosen that compares the advantages and limitations of each approach.
 Recognition methodologies for an analysis of the simple actions of a single person are first presented in the article. Space-time volume approaches and sequential approaches that represent and recognize activities directly from input images are discussed. Next, hierarchical recognition methodologies for high-level activities are presented and compared. Statistical approaches, syntactic approaches, and description-based approaches for hierarchical recognition are discussed in the article. In addition, we further discuss the papers on the recognition of human-object interactions and group activities. Public datasets designed for the evaluation of the recognition methodologies are illustrated in our article as well, comparing the methodologies' performances. This review will provide the impetus for future research in more productive areas.

Not all frames are equal – selecting a subset of discriminative frames from a video can improve performance at detecting and recognizing human interactions. In this paper we present models for categorizing a video into one of a number of predefined interactions or for detecting these interactions in a long video sequence. The models represent the interaction by a set of key temporal moments and the spatial structures they entail. For instance: two people approaching each other, then extending their hands before engaging in a “handshaking” interaction. Learning the model parameters requires only weak supervision in the form of an overall label for the interaction. Experimental results on the UT-Interaction and VIRAT datasets verify the efficacy of these structured models for human interactions.

Activity recognition fits within the bigger framework of context awareness. In this paper, we report on our efforts to recognize user activity from accelerometer data. Activity recognition is formulated as a classification problem. Performance of base-level classifiers and meta-level classifiers is compared. Plurality Voting is found to perform consistently well across different settings.

Recent advances on human motion analysis have made the extraction of human skeleton structure feasible, even from single depth images. This structure has been proven quite informative for discriminating actions in a recognition scenario. In this context, we propose a local skeleton descriptor that encodes the relative position of joint quadruples. Such a coding implies a similarity normalisation transform that leads to a compact (6D) view-invariant skeletal feature, referred to as skeletal quad. Further, the use of a Fisher kernel representation is suggested to describe the skeletal quads contained in a (sub)action. A Gaussian mixture model is learnt from training data, so that the generation of any set of quads is encoded by its Fisher vector. Finally, a multi-level representation of Fisher vectors leads to an action description that roughly carries the order of sub-action within each action sequence. Efficient classification is here achieved by linear SVMs. The proposed action representation is tested on widely used datasets, MSRAction3D and HDM05. The experimental evaluation shows that the proposed method outperforms state-of-the-art algorithms that rely only on joints, while it competes with methods that combine joints with extra cues.

Where does one attend when viewing dynamic scenes? Research into the factors influencing gaze location during static scene viewing have reported that low-level visual features contribute very little to gaze location especially when opposed by high-level factors such as viewing task. However, the inclusion of transient features such as motion in dynamic scenes may result in a greater influence of visual features on gaze allocation and coordination of gaze across viewers. In the present study, we investigated the contribution of low- to mid-level visual features to gaze location during free-viewing of a large dataset of videos ranging in content and length. Signal detection analysis on visual features and Gaussian Mixture Models for clustering gaze was used to identify the contribution of visual features to gaze location. The results show that mid-level visual features including corners and orientations can distinguish between actual gaze locations and a randomly sampled baseline. However, temporal features such as flicker, motion, and their respective contrasts were the most predictive of gaze location. Additionally, moments in which all viewers’ gaze tightly clustered in the same location could be predicted by motion. Motion and mid-level visual features may influence gaze allocation in dynamic scenes, but it is currently unclear whether this influence is involuntary or due to correlations with higher order factors such as scene semantics.

In this paper we introduce a 3-dimensional (3D) SIFT descriptor for video or 3D imagery such as MRI data. We also show how this new descriptor is able to better represent the 3D nature of video data in the application of action recognition. This paper will show how 3D SIFT is able to outperform previously used description methods in an elegant and efficient manner. We use a bag of words approach to represent videos, and present a method to discover relationships between spatio-temporal words in order to better describe the video data.

In this work1, we present a method to represent a video with a sequence of words, and learn the temporal sequencing of such words as the key information for predicting and recognizing human actions. We leverage core concepts from the Natural Language Processing (NLP) literature used in sentence classification to solve the problems of action prediction and action recognition. Each frame is converted into a word that is represented as a vector using the Bag of Visual Words (BoW) encoding method. The words are then combined into a sentence to represent the video, as a sentence. The sequence of words in different actions are learned with a simple but effective Temporal Convolutional Neural Network (T-CNN) that captures the temporal sequencing of information in a video sentence. We demonstrate that a key characteristic of the proposed method is its low-latency, i.e. its ability to predict an action accurately with a partial sequence (sentence). Experiments on two datasets, UCF101 and HMDB51 show that the method on average reaches 95% of its accuracy within half the video frames. Results, also demonstrate that our method achieves compatible state-of-the-art performance in action recognition (i.e. at the completion of the sentence) in addition to action prediction.

Due to its low price, webcam has become one of the most promising sensors with the rapid development of computer vision. However, the accuracies of eye tracking and eye movement analysis are largely limited by the quality of the webcam videos. To solve this issue, a novel eye movement analysis model is proposed based on five eye feature points rather than a single point (such as the iris center). First, a single convolutional neural network (CNN) is trained for eye feature point detection, and five eye feature points are detected for obtaining more useful eye movement information. Subsequently, six types of original time-varying eye movement signals can be constructed by feature points of each frame, which can reduce the dependency of the iris center in low quality videos. Finally, behaviors-CNN can be trained by the time-varying eye movement signals for recognizing different eye movement patterns, which is capable of avoiding the influence of errors from the basic eye movement type detection and artificial eye movement feature construction. To validate the performance, a webcam-based visual activity data set was constructed, which contained almost 0.5 million frames collected from 38 subjects. The experimental results on this database have demonstrated that the proposed model can obtain promising results for natural and convenient eye movement-based applications.

ÐA new view-based approach to the representation and recognition of human movement is presented. The basis of the representation is a temporal templateÐa static vector-image where the vector value at each point is a function of the motion properties at the corresponding spatial location in an image sequence. Using aerobics exercises as a test domain, we explore the representational power of a simple, two component version of the templates: The first value is a binary value indicating the presence of motion and the second value is a function of the recency of motion in a sequence. We then develop a recognition method matching temporal templates against stored instances of views of known actions. The method automatically performs temporal segmentation, is invariant to linear changes in speed, and runs in real-time on standard platforms. Index TermsÐMotion recognition, computer vision.


Question: Aggregated motion features for first-person video analysis.

This paper explores the challenge of identifying interaction-level human activities from a first-person perspective, aiming to equip observers, such as robots or wearable cameras, with the capability to discern 'what activities others are engaging in with them' from unbroken video streams. The activities encompass not only benign interactions like 'a person embracing the observer' but also aggressive encounters such as 'striking the observer' or 'projecting objects at the observer', which result in substantial camera ego-motion due to physical contact. The study delves into the use of multi-channel kernels to amalgamate global and local motion data, introducing an innovative activity learning and recognition framework that specifically accounts for the temporal dynamics manifested in first-person activity videos. Through our empirical evaluations, we not only exhibit classification outcomes with segmented videos but also substantiate that our method can accurately pinpoint activities within uninterrupted video recordings.

Due to its low price, webcam has become one of the most promising sensors with the rapid development of computer vision. However, the accuracies of eye tracking and eye movement analysis are largely limited by the quality of the webcam videos. To solve this issue, a novel eye movement analysis model is proposed based on five eye feature points rather than a single point (such as the iris center). First, a single convolutional neural network (CNN) is trained for eye feature point detection, and five eye feature points are detected for obtaining more useful eye movement information. Subsequently, six types of original time-varying eye movement signals can be constructed by feature points of each frame, which can reduce the dependency of the iris center in low quality videos. Finally, behaviors-CNN can be trained by the time-varying eye movement signals for recognizing different eye movement patterns, which is capable of avoiding the influence of errors from the basic eye movement type detection and artificial eye movement feature construction. To validate the performance, a webcam-based visual activity data set was constructed, which contained almost 0.5 million frames collected from 38 subjects. The experimental results on this database have demonstrated that the proposed model can obtain promising results for natural and convenient eye movement-based applications.

We study the question of feature sets for robust visual object recognition; adopting linear SVM based human detection as a test case. After reviewing existing edge and gradient based descriptors, we show experimentally that grids of histograms of oriented gradient (HOG) descriptors significantly outperform existing feature sets for human detection. We study the influence of each stage of the computation on performance, concluding that fine-scale gradients, fine orientation binning, relatively coarse spatial binning, and high-quality local contrast normalization in overlapping descriptor blocks are all important for good results. The new approach gives near-perfect separation on the original MIT pedestrian database, so we introduce a more challenging dataset containing over 1800 annotated human images with a large range of pose variations and backgrounds.

The possibility of automatic understanding of customers’ shopping behavior and acting according to their needs is relevant in the marketing domain, attracting a lot of attention lately. In this work, we focus on the task of automatic assessment of customers’ shopping behavior, by proposing a multilevel framework. The framework is supported at low-level by different types of cameras, which are synchronized, facilitating efficient processing of information. A fish-eye camera is used for tracking, while a high-definition one serves for the action recognition task. The experiments are performed on both laboratory and real-life recordings in a supermarket. From the video recordings, we extract features related to the spatio-temporal behavior of trajectories, the dynamics and the time spent in each region of interest (ROI) in the shop and regarding the customer-products interaction patterns. Next ∗Corresponding Author. Phone: +31(0)15 2781537. Fax: +31(0)15 2787141. Email address: m.c.popa@tudelft.nl (M.C. Popa) Preprint submitted to Pattern Recognition Letters December 24, 2011 we analyze the shopping sequences using a Hidden Markov Model (HMM). We conclude that it is possible to accurately classify trajectories (93%), discriminate between different shopping related actions (91.6%), and recognize shopping behavioral types by means of the proposed reasoning model in 95% of the cases.

We propose an algorithm which combines the discriminative information from depth images as well as from 3D joint positions to achieve high action recognition accuracy. To avoid the suppression of subtle discriminative information and also to handle local occlusions, we compute a vector of many independent local features. Each feature encodes spatiotemporal variations of depth and depth gradients at a specific space-time location in the action volume. Moreover, we encode the dominant skeleton movements by computing a local 3D joint position difference histogram. For each joint, we compute a 3D space-time motion volume which we use as an importance indicator and incorporate in the feature vector for improved action discrimination. To retain only the discriminant features, we train a random decision forest (RDF). The proposed algorithm is evaluated on three standard datasets and compared with nine state-of-the-art algorithms. Experimental results show that, on the average, the proposed algorithm outperform all other algorithms in accuracy and have a processing speed of over 112 frames/second.

Recent advances on human motion analysis have made the extraction of human skeleton structure feasible, even from single depth images. This structure has been proven quite informative for discriminating actions in a recognition scenario. In this context, we propose a local skeleton descriptor that encodes the relative position of joint quadruples. Such a coding implies a similarity normalisation transform that leads to a compact (6D) view-invariant skeletal feature, referred to as skeletal quad. Further, the use of a Fisher kernel representation is suggested to describe the skeletal quads contained in a (sub)action. A Gaussian mixture model is learnt from training data, so that the generation of any set of quads is encoded by its Fisher vector. Finally, a multi-level representation of Fisher vectors leads to an action description that roughly carries the order of sub-action within each action sequence. Efficient classification is here achieved by linear SVMs. The proposed action representation is tested on widely used datasets, MSRAction3D and HDM05. The experimental evaluation shows that the proposed method outperforms state-of-the-art algorithms that rely only on joints, while it competes with methods that combine joints with extra cues.

Vast quantities of videos are now being captured at astonishing rates, but the majority of these are not labelled. To cope with such data, we consider the task of content-based activity recognition in videos without any manually labelled examples, also known as zero-shot video recognition. To achieve this, videos are represented in terms of detected visual concepts, which are then scored as relevant or irrelevant according to their similarity with a given textual query. In this paper, we propose a more robust approach for scoring concepts in order to alleviate many of the brittleness and low precision problems of previous work. Not only do we jointly consider semantic relatedness, visual reliability, and discriminative power. To handle noise and non-linearities in the ranking scores of the selected concepts, we propose a novel pairwise order matrix approach for score aggregation. Extensive experiments on the large-scale TRECVID Multimedia Event Detection data show the superiority of our approach.

Despite the steady progress in video analysis led by the adoption of convolutional neural networks (CNNs), the relative improvement has been less drastic as that in 2D static image classification. Three main challenges exist including spatial (image) feature representation, temporal information representation, and model/computation complexity. It was recently shown by Carreira and Zisserman that 3D CNNs, inflated from 2D networks and pretrained on ImageNet, could be a promising way for spatial and temporal representation learning. However, as for model/computation complexity, 3D CNNs are much more expensive than 2D CNNs and prone to overfit. We seek a balance between speed and accuracy by building an effective and efficient video classification system through systematic exploration of critical network design choices. In particular, we show that it is possible to replace many of the 3D convolutions by low-cost 2D convolutions. Rather surprisingly, best result (in both speed and accuracy) is achieved when replacing the 3D convolutions at the bottom of the network, suggesting that temporal representation learning on high-level “semantic” features is more useful. Our conclusion generalizes to datasets with very different properties. When combined with several other cost-effective designs including separable spatial/temporal convolution and feature gating, our system results in an effective video classification system that that produces very competitive results on several action classification benchmarks (Kinetics, Something-something, UCF101 and HMDB), as well as two action detection (localization) benchmarks (JHMDB and UCF101-24).

Our objective is to index people in a TV-Content. In this context, because of multi-face shots and non-speaking face shots, it is difficult to determine which face is speaking. There is no guaranteed synchronization between sequences of a person’s appearance and sequences of his or her speech. In this work, we want to separate talking and non-talking faces by detecting lip motion. We propose a method to detect the lip motion by measuring the degree of disorder of pixel directions around the lip. Results of experiments on a TV-Show database show that a high correct classification rate can be achieved by the proposed method.

Intelligent agents, such as robots, have to serve a multitude of autonomous functions. Examples are, e.g., collision avoidance, navigation and route planning, active sensing of its environment, or the interaction and non-verbal communication with people in the extended reach space. Here, we focus on the recognition of the action of a human agent based on a biologically inspired visual architecture of analyzing articulated movements. The proposed processing architecture builds upon coarsely segregated streams of sensory processing along different pathways which separately process form and motion information (Layher et al., 2014). Action recognition is performed in an event-based scheme by identifying representations of characteristic pose configurations (key poses) in an image sequence. In line with perceptual studies, key poses are selected unsupervised utilizing a feature-driven criterion which combines extrema in the motion energy with the horizontal and the vertical extendedness of a body shape. Per class representations of key pose frames are learned using a deep convolutional neural network consisting of 15 convolutional layers. The network is trained using the energy-efficient deep neuromorphic networks (Eedn) framework (Esser et al., 2016), which realizes the mapping of the trained synaptic weights onto the IBM Neurosynaptic System platform (Merolla et al., 2014). After the mapping, the trained network achieves real-time capabilities for processing input streams and classify input images at about 1,000 frames per second while the computational stages only consume about 70 mW of energy (without spike transduction). Particularly regarding mobile robotic systems, a low energy profile might be crucial in a variety of application scenarios. Cross-validation results are reported for two different datasets and compared to state-of-the-art action recognition approaches. The results demonstrate, that (I) the presented approach is on par with other key pose based methods described in the literature, which select key pose frames by optimizing classification accuracy, (II) compared to the training on the full set of frames, representations trained on key pose frames result in a higher confidence in class assignments, and (III) key pose representations show promising generalization capabilities in a cross-dataset evaluation.

In this paper, we study the problem of video-based person re-identification. This is more challenging, and of greater practical interest, than conventional image-based person re-identification. To address this problem, we propose the use of convolutional Long Short Term Memory (LSTM) based networks to learn a video-based representation for person re-identification. To this end, we propose to jointly leverage deep Convolutional Neural Networks (CNNs) and LSTM networks. Given sequential video frames of a person, the spatial information encoded in the frames is first extracted by a set of CNNs. An encoderdecoder framework derived from LSTMs is employed to encode the resulting temporal of CNN outputs. This approach leads to a refined feature representation that is able to explicitly model the video as an ordered sequence, while preserving the spatial information. Comparative experiments demonstrate that our approach achieves the state-of-the-art performance for video-based person re-identification on iLIDS-VID and PRID 2011, the two primary public datasets for this purpose.