Question: Hop-by-hop message authentication and source privacy preservation in wireless sensor networks.

Innovative Strategies in Cryptography for Enhanced Security

Contemporary advancements in cryptography are scrutinized, focusing on two primary areas: the expansion of telecommunication applications and the requirement for cryptographic systems that minimize the reliance on secure key distribution channels while providing the functionality of written signatures. Currently, for encrypted communications to maintain privacy, the communicating entities must share a secret key, typically exchanged through secure channels like private couriers or registered mail. This traditional method, however, is not always feasible, particularly in scenarios where immediate, secure communication between parties with no prior contact is necessary, such as in spontaneous business interactions. Delaying the establishment of business relationships to accommodate key distribution via physical means is impractical and costly. The key distribution challenge, therefore, presents a significant hurdle to the seamless integration of secure communications in business environments.

This paper explores novel approaches to address these unresolved issues in cryptography. It highlights how emerging theories in communication and computation are furnishing tools to tackle long-standing cryptographic dilemmas, including the development of cryptographic systems that do not necessitate the pre-established sharing of keys through secure channels. This evolution in cryptographic techniques is pivotal in overcoming the logistical and financial barriers associated with key distribution, facilitating more secure and efficient communication in various sectors, including business.

The focus shifts towards innovative cryptographic mechanisms that enable secure, immediate communications without the prerequisite of a shared key, thereby streamlining the process of initiating secure interactions between parties, particularly in the business domain. This shift is critical in enhancing the practicality and efficiency of cryptographic solutions, making them more adaptable to the dynamic and immediate nature of modern communication needs.

Advances in processor, memory and radio technology will enable small and cheap nodes capable of sensing, communication and computation. Networks of such nodes can coordinate to perform distributed sensing of environmental phenomena. In this paper, we explore the directed diffusion paradigm for such coordination. Directed diffusion is datacentric in that all communication is for named data. All nodes in a directed diffusion-based network are application-aware. This enables diffusion to achieve energy savings by selecting empirically good paths and by caching and processing data in-network. We explore and evaluate the use of directed diffusion for a simple remote-surveillance sensor network.

The IEEE wireless standard 802.15.4 gets widespread attention because of its adoption in sensor networks, home automation, and other networked systems. The goal of the project is to implement an enand decoding block for the IEEE 802.15.4 protocol in GNU Radio, an open source solution for software defined radios. This report will give an insight into the working of GNU Radio and some of its hardware components. Additionally, it gives details about the implementation of the enand decoding blocks. At the end, we will verify the implementation by sending and receiving messages to and from an actual IEEE 802.15.4 radio chip, the CC2420 from ChipCon, and give a small bandwidth comparison of the two solutions.

Devices with integrated Wi-Fi chips broadcast beacons for network connection management purposes. Such information can be captured with inexpensive monitors and used to extract user behavior. To understand the behavior of visitors, we deployed our passive monitoring system---CrowdProbe, in a multi-floor museum for six months. We used a Hidden Markov Models (HMM) based trajectory inference algorithm to infer crowd movement using more than 1.7 million opportunistically obtained probe request frames.
 However, as more devices adopt schemes to randomize their MAC addresses in the passive probe session to protect user privacy, it becomes more difficult to track crowd and understand their behavior. In this paper, we try to make use of historical transition probability to reason about the movement of those randomized devices with spatial and temporal constraints. With CrowdProbe, we are able to achieve sufficient accuracy to understand the movement of visitors carrying devices with randomized MAC addresses.

Connected health care has several applications including telecare medicine information system, personally controlled health records system, and patient monitoring. In such applications, user authentication can ensure the legality of patients. In user authentication for such applications, only the legal user/patient himself/herself is allowed to access the remote server, and no one can trace him/her according to transmitted data. Chang et al. proposed a uniqueness-and-anonymity-preserving remote user authentication scheme for connected health care (Chang et al., J Med Syst 37:9902, 2013). Their scheme uses the user’s personal biometrics along with his/her password with the help of the smart card. The user’s biometrics is verified using BioHashing. Their scheme is efficient due to usage of one-way hash function and exclusive-or (XOR) operations. In this paper, we show that though their scheme is very efficient, their scheme has several security weaknesses such as (1) it has design flaws in login and authentication phases, (2) it has design flaws in password change phase, (3) it fails to protect privileged insider attack, (4) it fails to protect the man-in-the middle attack, and (5) it fails to provide proper authentication. In order to remedy these security weaknesses in Chang et al.’s scheme, we propose an improvement of their scheme while retaining the original merit of their scheme. We show that our scheme is efficient as compared to Chang et al.’s scheme. Through the security analysis, we show that our scheme is secure against possible attacks. Further, we simulate our scheme for the formal security verification using the widely-accepted AVISPA (Automated Validation of Internet Security Protocols and Applications) tool to ensure that our scheme is secure against passive and active attacks. In addition, after successful authentication between the user and the server, they establish a secret session key shared between them for future secure communication.

Modern mobile phones and PDAs are equipped with positioning capabilities (e.g., GPS). Users can access public location-based services (e.g., Google Maps) and ask spatial queries. Although communication is encrypted, privacy and confidentiality remain major concerns, since the queries may disclose the location and identity of the user. Commonly, spatial K-anonymity is employed to hide the query initiator among a group of K users. However, existing work either fails to guarantee privacy, or exhibits unacceptably long response time. In this paper we propose MobiHide, a Peer-to-Peer system for anonymous location-based queries, which addresses these problems. MobiHide employs the Hilbert space-filling curve to map the 2-D locations of mobile users to 1-D space. The transformed locations are indexed by a Chord-based distributed hash table, which is formed by the mobile devices. The resulting Peer-to-Peer system is used to anonymize a query by mapping it to a random group of K users that are consecutive in the 1-D space. Compared to existing state-of-the-art, MobiHide does not provide theoretical anonymity guarantees for skewed query distributions. Nevertheless, it achieves strong anonymity in practice, and it eliminates system hotspots. Our experimental evaluation shows that MobiHide has good load balancing and fault tolerance properties, and is applicable to real-life scenarios with numerous mobile users.

In the elemental technologies, it is necessary to realize the Internet service of things (IoT), sensors and devices, network, platform (hardware platforms, open software platform, such as specific OS platforms). Web services, data analysis and prediction, big data processing, such as security and privacy protection technology, there are a variety of techniques. These elements technology provide a specific function. The element technology is integrated with each other. However, by several techniques are integrated, it can be problems with integration of security technologies that existed for each element technology. Even if individual technologies basic security features are constituting Internet Services (CIA: Confidentiality, integrity, authentication or authorization). It also offers security technology not connected to each other. Therefore, I will look at the security technology and proposes a lightweight routing protocol indispensable for realizing a secure Internet services things.

Intended for network-wide dissemination of commands, configurations and code binaries, flooding has been investigated extensively in wireless networks. However, little work has yet been done on low-duty-cycle wireless sensor networks in which nodes stay asleep most of time and wake up asynchronously. In this type of network, a broadcasting packet is rarely received by multiple nodes simultaneously, a unique constraining feature that makes existing solutions unsuitable. Combined with unreliable links, flooding in low-duty-cycle networks is a new challenging issue.
 In this paper, we introduce Opportunistic Flooding, a novel design tailored for low-duty-cycle networks with unreliable wireless links and predetermined working schedules. The key idea is to make probabilistic forwarding decisions at a sender based on the delay distribution of next-hop nodes. Only opportunistically early packets are forwarded using links outside the energy optimal tree to reduce the flooding delay and redundancy in transmission. To improve performance further, we propose a forwarder selection method to alleviate the hidden terminal problem and a link-quality-based backoff method to resolve simultaneous forwarding operations. We evaluate Opportunistic Flooding with extensive simulation and a test-bed implementation consisting of 30 MicaZ nodes. Evaluation shows our design is close to the optimal performance achievable by oracle flooding designs. Compared with improved traditional flooding, our design achieves significantly shorter flooding delay while consuming only 20% ~ 60% of the transmission energy in various low-duty-cycle network settings.

In the past few years, wireless sensor networks (WSNs) have been gaining increasing attention because of their potential of enabling of novel and attractive solutions in areas such as industrial automation, environmental monitoring, transportation business, health-care etc. If we add this collection of sensor derived data to various Web-based social networks or virtual communities, blogs etc., we can have a remarkable transformation in our ability to "see" ourselves and our planet. Our primary goal is to facilitate connecting sensors, people and software objects to build community-centric sensing applications. However, the computational tools needed to launch this exploration may be more appropriately built from the data center "Cloud" computing model than the traditional HPC approaches. In this paper, we propose a framework to enable this exploration by integrating sensor networks to the emerging data center "cloud" model of computing. But there are many challenges to enable this framework. We propose a pub-sub based model which simplifies the integration of sensor networks with cloud based community-centric applications. Also there is a need for internetworking cloud providers in case of violation of service level agreement with users. We discussed these issues and proposed reasonable solutions.

This report aims at giving a short overview on rank function analysis of authentication protocols concentrating on the results of Schneider in [4] and Heather in [2]. Therefore it shows a standard form of protocol implementation in CSP, shows how authentication properties can be captured as trace specifications, gives basic definitions necessary to reason about the unbounded set of messages, reformulates the central rank function theorem and restates it proof. Additionally it gives hints on applying this approach to verify secrecy protocols as well, a short overview on tool support available and some hints on interesting properties of rank functions. This report will use notations introduced in earlier reports [11-16] throughout and only reintroduce notations where they are considered vital.

Jimmy Lin, Adam Roegiest, Luchen Tan, Richard McCreadie, Ellen Voorhees, and Fernando Diaz 1 David R. Cheriton School of Computer Science, University of Waterloo, Ontario, Canada 2 School of Computing Science, University of Glasgow, Scotland, the United Kingdom 3 National Institute for Standards and Technology, Maryland, USA 4 Microsoft Research, New York, USA {jimmylin, aroegies, luchen.tan}@uwaterloo.ca richard.mccreadie@glasgow.ac.uk, ellen.voorhees@nist.gov, fdiaz@microsoft.com


Question: Hop-by-hop message authentication and source privacy preservation in wireless sensor networks.

This paper presents an enhanced method for calculating logarithms over the Galois Field GF(p) and discusses its implications for cryptography. The proposed algorithm improves upon existing techniques for logarithm computation in finite fields, which is a critical operation in various cryptographic protocols. By optimizing the computational process, the algorithm not only accelerates the calculation but also enhances the security of cryptographic systems that rely on the difficulty of the discrete logarithm problem. The cryptographic significance of this improvement lies in its potential to strengthen encryption schemes and protect against potential attacks. Specifically, the enhanced algorithm can contribute to more robust key exchange mechanisms and secure communication channels in cryptographic applications. This advancement in logarithm computation over GF(p) represents a valuable contribution to the field of cryptography, offering a more efficient and secure solution for cryptographic operations.

In recent years, there has been a growing interest in wireless sensor networks. One of the major issues in wireless sensor network is developing an energy-efficient clustering protocol. Hierarchical clustering algorithms are very important in increasing the network’s life time. Each clustering algorithm is composed of two phases, the setup phase and steady state phase. The hot point in these algorithms is the cluster head selection. In this paper, we study the impact of heterogeneity of nodes in terms of their energy in wireless sensor networks that are hierarchically clustered. We assume that a percentage of the population of sensor nodes is equipped with the additional energy resources. We also assume that the sensor nodes are randomly distributed and are not mobile, the coordinates of the sink and the dimensions of the sensor field are known. Homogeneous clustering protocols assume that all the sensor nodes are equipped with the same amount of energy and as a result, they cannot take the advantage of the presence of node heterogeneity. Adapting this approach, we introduce an energy efficient heterogeneous clustered scheme for wireless sensor networks based on weighted election probabilities of each node to become a cluster head according to the residual energy in each node. Finally, the simulation results demonstrate that our proposed heterogeneous clustering approach is more effective in prolonging the network lifetime compared with LEACH. 2008 Elsevier B.V. All rights reserved.

We give cryptographic schemes that help trace the source of leaks when sensitive or proprietary data is made available to a large set of parties. A very relevant application is in the context of pay television, where only paying customers should be able to view certain programs. In this application, the programs are normally encrypted, and then the sensitive data is the decryption keys that are given to paying customers. If a pirate decoder is found, it is desirable to reveal the source of its decryption keys. We describe fully resilient schemes which can be used against any decoder which decrypts with nonnegligible probability. Since there is typically little demand for decoders which decrypt only a small fraction of the transmissions (even if it is nonnegligible), we further introduce threshold tracing schemes which can only be used against decoders which succeed in decryption with probability greater than some threshold. Threshold schemes are considerably more efficient than fully resilient schemes.

We consider the problem of password-authenticated group Diffie-Hellman key exchange among N parties, N−1 clients and a singleserver, using different passwords. Most password-authenticated key exchange schemes in the literature have focused on an authenticated key exchange using a shared password between a client and a server. With a rapid change in modern communication environment such as ad-hoc networks and ubiquitous computing, it is necessary to construct a secure end-to-end channel between clients, which is a quite different paradigm from the existing ones. To achieve this end-to-end security, only a few schemes of three-party setting have been presented where two clients exchange a key using their own passwords with the help of a server. However, up until now, no formally treated and round efficient protocols which enable group members to generate a common session key with clients’ distinct passwords have been suggested. In this paper we securely and efficiently extend three-party case to Nparty case with a formal proof of security. Two provably secure N-party EKE protocols are suggested; N-party EKE-U in the unicast network and N-party EKE-M in the multicast network. The proposed N-party EKE-M is provable secure and provides forward secrecy. Especially, the scheme is of constant-round, hence scalable and practical.

Reliable flooding in wireless sensor networks (WSNs) is desirable for a broad range of applications and network operations. However, it is a challenging problem to ensure 100% flooding coverage efficiently considering the combined effects of low-duty-cycle operation and unreliable wireless transmission. In this work, we propose a novel dynamic switching-based reliable flooding (DSRF) framework, which is designed as an enhancement layer to provide efficient and reliable flooding over a variety of existing flooding tree structures in low-duty-cycle WSNs. Through comprehensive simulations, we demonstrate that DSRF can effectively improve both flooding energy efficiency and latency.

RPL is a routing protocol for low-power and lossy constrained node networks. A malicious node can manipulate header options used by RPL to track DODAG inconsistencies, thereby causing denial of service attacks, increased control message overhead, and black-holes at the targeted node. RPL counteracts DODAG inconsistencies by using a fixed threshold, upon reaching which all subsequent packets with erroneous header options are ignored. However, the fixed threshold is arbitrary and does not resolve the black-hole issue either. To address this we present a mitigation strategy that allows nodes to dynamically adapt against a DODAG inconsistency attack. We also present the forced black-hole attack problem and a solution that can be used to mitigate it. Results from our experiments show that our proposed approach mitigates these attacks without any significant overhead.

Many location-based services require a mobile user to continuously prove his location. In absence of a secure mechanism, malicious users may lie about their locations to get these services. Mobility trace, a sequence of past mobility points, provides evidence for the user's locations. In this paper, we propose a Continuous Location Integrity and Provenance (CLIP) Scheme to provide authentication for mobility trace, and protect users' privacy. CLIP uses low-power inertial accelerometer sensor with a light-weight entropy-based commitment mechanism and is able to authenticate the user's mobility trace without any cost of trusted hardware. CLIP maintains the user's privacy, allowing the user to submit a portion of his mobility trace with which the commitment can be also verified. Wireless Access Points (APs) or colocated mobile devices are used to generate the location proofs. We also propose a light-weight spatial-temporal trust model to detect fake location proofs from collusion attacks. The prototype implementation on Android demonstrates that CLIP requires low computational and storage resources. Our extensive simulations show that the spatial-temporal trust model can achieve high (> 0.9) detection accuracy against collusion attacks.

Self-configuring wireless sensor networks can be invaluable in many civil and military applications for collecting, processing, and disseminating wide ranges of complex environmental data. Because of this, they have attracted considerable research attention in the last few years. The WINS [1] and SmartDust [2] projects, for instance, aim to integrate sensing, computing, and wireless communication capabilities into a small form factor to enable low-cost production of these tiny nodes in large numbers. Several other groups are investigating efficient hardware/software system architectures, signal processing algorithms, and network protocols for wireless sensor networks [3]-[5]. Sensor nodes are battery driven and hence operate on an extremely frugal energy budget. Further, they must have a lifetime on the order of months to years, since battery replacement is not an option for networks with thousands of physically embedded nodes. In some cases, these networks may be required to operate solely on energy scavenged from the environment through seismic, photovoltaic, or thermal conversion. This transforms energy consumption into the most important factor that determines sensor node lifetime. Conventional low-power design techniques [6] and hardware architectures only provide point solutions which are insufficient for these highly energy-constrained systems. Energy optimization, in the case of sensor networks, is much more complex, since it involves not only reducing the energy consumption of a single sensor node but also maximizing the lifetime of an entire network. The network lifetime can be maximized

The Internet of Things (IoT) being a promising technology of the future is expected to connect billions of devices. The increased number of communication is expected to generate mountains of data and the security of data can be a threat. The devices in the architecture are essentially smaller in size and low powered. Conventional encryption algorithms are generally computationally expensive due to their complexity and requires many rounds to encrypt, essentially wasting the constrained energy of the gadgets. Less complex algorithm, however, may compromise the desired integrity. In this paper we propose a lightweight encryption algorithm named as Secure IoT (SIT). It is a 64-bit block cipher and requires 64-bit key to encrypt the data. The architecture of the algorithm is a mixture of feistel and a uniform substitution-permutation network. Simulations result shows the algorithm provides substantial security in just five encryption rounds. The hardware implementation of the algorithm is done on a low cost 8-bit micro-controller and the results of code size, memory utilization and encryption/decryption execution cycles are compared with benchmark encryption algorithms. The MATLAB code for relevant simulations is available online at https://goo.gl/Uw7E0W. Keywords—IoT; Security; Encryption; Wireless Sensor Network WSN; Khazad

Wireless Sensor Networks (WSNs) are composed self-organized wireless ad hoc networks which comprise of a large number of resource constrained sensor nodes. The major areas of research in WSN is going on hardware, and operating system of WSN, deployment, architecture, localization, synchronization, programming models, data aggregation and dissemination, database querying, architecture, middleware, quality of service and security. This paper study highlights ongoing research activities and issues that affect the design and performance of Wireless Sensor Network.

Cost related to the wiring of sensors and actuators in the manufacturing industry is rapidly increasing with the number of field devices. Complex engineering as well as cable wear and tear are important factors. For over a decade, fieldbuses have been the dominating approach to simplify installation, reduce engineering efforts, and increase reliability at the field level. The next step in this process is to introduce wireless communication in certain problematic parts of the field network. In order to do this, a reliable real-time communication system, designed to cope with the harsh environments where factory automation systems are employed, is required. The wireless interface for sensors and actuators (WISA), with its two subconcepts WISA-power and WISA-com, is such a system. This paper discusses how the WISA concept can be efficiently integrated into wired field networks for factory automation-both state-of-the-art [process fieldbus decentral peripherals (PROFIBUS DP)] and new (PROFINET IO) networks. We propose amendments to WISA, which will improve the 802.11b/g coexistence and harmonize the integration of WISA for sensor/actuator communication in, e.g., PROFINET IO systems. The suggested improvements relate to both the radio protocol in order to improve coexistence with other wireless technologies as well as concepts for the integration of WISA into the field network of PROFINET/PROFIBUS.


Question: A multi-layer encoder for text summarization tasks.

This paper delves into the realm of abstractive text summarization, employing sequence-to-sequence recurrent neural networks (RNNs) and exploring innovative advancements beyond this foundational approach. Authored by Ramesh Nallapati, Bowen Zhou, and Cicero dos Santos from IBM Watson, the research aims to enhance the efficiency and effectiveness of text summarization techniques. By leveraging sequence-to-sequence RNNs, a powerful method for generating concise summaries from lengthy texts, the authors build upon existing models and introduce novel strategies to further refine the summarization process. This includes the investigation of additional neural network architectures and algorithms that complement and improve upon the capabilities of RNNs, ultimately aiming to produce more accurate and contextually relevant summaries. The work by Nallapati, Zhou, and dos Santos represents a significant step forward in the field of abstractive text summarization, showcasing the potential for advanced neural network techniques to revolutionize the way we synthesize and understand large volumes of textual information.

The main objective of a text summarization system is to identify the most important information from the given text and present it to the end users. In this paper, Wikipedia articles are given as input to system and extractive text summarization is presented by identifying text features and scoring the sentences accordingly. The text is first pre-processed to tokenize the sentences and perform stemming operations. We then score the sentences using the different text features. Two novel approaches implemented are using the citations present in the text and identifying synonyms. These features along with the traditional methods are used to score the sentences. The scores are used to classify the sentence to be in the summary text or not with the help of a neural network. The user can provide what percentage of the original text should be in the summary. It is found that scoring the sentences based on citations gives the best results.

In this paper we describe a work in progress on designing the continuous vector space word representations able to map unseen data adequately. We propose a LSTM-based feature extraction layer that reads in a sequence of characters corresponding to a word and outputs a single fixed-length real-valued vector. We then test our model on a POS tagging task on four typologically different languages. The results of the experiments suggest that the model can offer a solution to the out-of-vocabulary words problem, as in a comparable setting its OOV accuracy improves over that of a state of the art tagger.

In this work, we propose a novel method for training neural networks to perform singledocument extractive summarization without heuristically-generated extractive labels. We call our approach BANDITSUM as it treats extractive summarization as a contextual bandit (CB) problem, where the model receives a document to summarize (the context), and chooses a sequence of sentences to include in the summary (the action). A policy gradient reinforcement learning algorithm is used to train the model to select sequences of sentences that maximize ROUGE score. We perform a series of experiments demonstrating that BANDITSUM is able to achieve ROUGE scores that are better than or comparable to the state-of-the-art for extractive summarization, and converges using significantly fewer update steps than competing approaches. In addition, we show empirically that BANDITSUM performs significantly better than competing approaches when good summary sentences appear late in the source document.

Deep learning yields great results across many fields, from speech recognition, image classification, to translation. But for each problem, getting a deep model to work well involves research into the architecture and a long period of tuning. We present a single model that yields good results on a number of problems spanning multiple domains. In particular, this single model is trained concurrently on ImageNet, multiple translation tasks, image captioning (COCO dataset), a speech recognition corpus, and an English parsing task. Our model architecture incorporates building blocks from multiple domains. It contains convolutional layers, an attention mechanism, and sparsely-gated layers. Each of these computational blocks is crucial for a subset of the tasks we train on. Interestingly, even if a block is not crucial for a task, we observe that adding it never hurts performance and in most cases improves it on all tasks. We also show that tasks with less data benefit largely from joint training with other tasks, while performance on large tasks degrades only slightly if at all.

Multi-task learning in text classification leverages implicit correlations among related tasks to extract common features and yield performance gains. However, most previous works treat labels of each task as independent and meaningless onehot vectors, which cause a loss of potential information and makes it difficult for these models to jointly learn three or more tasks. In this paper, we propose Multi-Task Label Embedding to convert labels in text classification into semantic vectors, thereby turning the original tasks into vector matching tasks. We implement unsupervised, supervised and semisupervised models of Multi-Task Label Embedding, all utilizing semantic correlations among tasks and making it particularly convenient to scale and transfer as more tasks are involved. Extensive experiments on five benchmark datasets for text classification show that our models can effectively improve performances of related tasks with semantic representations of labels and additional information from each other.

Extractive text summarization aims to create a condensed version of one or more source documents by selecting the most informative sentences. Research in text summarization has therefore often focused on measures of the usefulness of sentences for a summary. We present an approach to sentence extraction that maps sentences to nodes of a hierarchical ontology. By considering ontology attributes we are able to improve the semantic representation of a sentence's information content. The classifier that maps sentences to the taxonomy is trained using search engines and is therefore very flexible and not bound to a specific domain. In our experiments, we train an SVM classifier to identify summary sentences using ontology-based sentence features. Our experimental results show that the ontology-based extraction of sentences outperforms baseline classifiers, leading to higher Rouge scores of summary extracts.

Quickly moving to a new area of research is painful for researchers due to the vast amount of scientific literature in each field of study. One possible way to overcome this problem is to summarize a scientific topic. In this paper, we propose a model of summarizing a single article, which can be further used to summarize an entire topic. Our model is based on analyzing others’ viewpoint of the target article’s contributions and the study of its citation summary network using a clustering approach.

This paper targets at the first step towards generating citation summaries - to identify the reference scope (i.e., cited text spans) for citances. We present a novel classification-based method that converts the task into binary classification which distinguishes cited and non-cited pairs of citances and reference sentences. The method models pairs of citances and reference sentences as feature vectors where citation-dependent and citation-independent features based on the semantic similarity between texts and the significance of texts are explored. Such vector representations are utilized to train a binary classifier. For a citance, once the set of reference sentences classified as the cited sentences are collected, a heuristic-based filtering strategy is applied to refine the output. The method is evaluated using the CL-SciSumm 2016 datasets and found to perform well with competitive results.

We describe SOCKEYE,1 an open-source sequence-to-sequence toolkit for Neural Machine Translation (NMT). SOCKEYE is a production-ready framework for training and applying models as well as an experimental platform for researchers. Written in Python and built on MXNET, the toolkit offers scalable training and inference for the three most prominent encoder-decoder architectures: attentional recurrent neural networks, self-attentional transformers, and fully convolutional networks. SOCKEYE also supports a wide range of optimizers, normalization and regularization techniques, and inference improvements from current NMT literature. Users can easily run standard training recipes, explore different model settings, and incorporate new ideas. In this paper, we highlight SOCKEYE’s features and benchmark it against other NMT toolkits on two language arcs from the 2017 Conference on Machine Translation (WMT): English–German and Latvian–English. We report competitive BLEU scores across all three architectures, including an overall best score for SOCKEYE’s transformer implementation. To facilitate further comparison, we release all system outputs and training scripts used in our experiments. The SOCKEYE toolkit is free software released under the Apache 2.0 license.

Most multi-document summarization systems follow the extractive framework based on various features. While more and more sophisticated features are designed, the reasonable combination of features becomes a challenge. Usually the features are combined by a linear function whose weights are tuned manually. In this task, Support Vector Regression (SVR) model is used for automatically combining the features and scoring the sentences. Two important problems are inevitably involved. The first one is how to acquire the training data. Several automatic generation methods are introduced based on the standard reference summaries generated by human. Another indispensable problem in SVR application is feature selection, where various features will be picked out and combined into different feature sets to be tested. With the aid of DUC 2005 and 2006 data sets, comprehensive experiments are conducted with consideration of various SVR kernels and feature sets. Then the trained SVR model is used in the main task of DUC 2007 to get the extractive summaries.


Question: A multi-layer encoder for summarizing textual content.

This paper investigates the use of deep generative models for text representation, where the latent variable of a document is sampled from a discrete language model. We present a variational autoencoder framework for this model and apply it to sentence compression tasks. In this scenario, the generative model initially selects a latent summary sentence from a background language model, followed by generating the actual sentence conditioned on this latent summary. Through extensive experiments, we demonstrate that both abstractive and extractive generative models for compression achieve state-of-the-art performance when trained on substantial supervised datasets. Moreover, we delve into semi-supervised compression settings, illustrating that our model can attain results on par with fully supervised models while utilizing significantly less supervised data, thereby highlighting its efficiency and adaptability in resource-constrained environments.

Deep learning methods have recently achieved great empirical success on machine translation, dialogue response generation, summarization, and other text generation tasks. At a high level, the technique has been to train end-to-end neural network models consisting of an encoder model to produce a hidden representation of the source text, followed by a decoder model to generate the target. While such models have significantly fewer pieces than earlier systems, significant tuning is still required to achieve good performance. For text generation models in particular, the decoder can behave in undesired ways, such as by generating truncated or repetitive outputs, outputting bland and generic responses, or in some cases producing ungrammatical gibberish. This paper is intended as a practical guide for resolving such undesired behavior in text generation models, with the aim of helping enable real-world applications.

We report on research in multi-document summarization and on evaluation of summarization in the framework of cross-lingual information retrieval. This work was carried out during a summer workshop on Language Engineering held at Johns Hopkins University by a team of nine researchers from seven universities. The goals of the research were as follows: (1) to develop a toolkit for evaluation of single-document and multi-document summarizers, (2) to develop a modular multi-document summarizer, called MEAD, that works in both English and Chinese, and (3) to perform a meta-evaluation of four automatic summarizers, including MEAD, using several types of evaluation measures: some currently used by summarization researchers and a couple of novel techniques. Central to the experiments in this workshop was the cross-lingual experimental setup based on a large-scale Chinese and English parallel corpus. An extensive set of human judgments were specifically prepared by the Linguistic Data Consortium for our research. These human judgments include a) which documents are relevant to a certain query and b) which sentences in the relevant documents are most relevant to the query and which therefore constitute a good summary of the cluster. These judgments were used to construct variable-length multiand single document summaries as model summaries. Since one of the novel evaluation metrics that we used, Relevance Correlation, is based on the premise that good summaries preserve query relevance both within a language and across languages, we made use of a cross-lingual Information Retrieval (IR) engine. We evaluated the quality of the automatic summaries using co-selection and content-based evaluation, two established techniques. A relatively new metric, relative utility, was also extensively tested. Part of the new scientific contribution is the measurement of relevance correlation, which we introduced and systematically examined in this workshop. Relevance correlation measures the quality of summaries in comparison to the entire documents as a function of how much document relevance drops if summaries are indexed instead of documents. Our results show that this measure is sensible, in that it correlates with more established evaluation measures. Another contribution is the cross-lingual setup which allows us to automatically translate English queries into Chinese, perform Chinese IR with or without summarization. This allows us to calculate relevance correlation for English and for Chinese in parallel (i.e., for the same queries) and to take direct cross-lingual comparisons of evaluations. Additionally, an alternative way of constructing Chinese model summaries from English ones was implemented which relies on the sentence alignment of English and Chinese documents. The results of our large-scale meta-evaluation are numerous, but some of the highlights are the following: (1) All evaluation measures rank human summaries first, which is an appropriate and expected property of such measures, (2) Both relevance correlation and the content-based measures place leading sentence extracts ahead of the more sophisticated summarizers, (3) Relative utility ranks our system, MEAD, as the best summarizer for shorter summaries, although for longer summaries, lead-based summaries outperform MEAD, (4) Co-selection measurements show overall low agreement amongst humans (above chance), whereas relative utility reports higher numbers on the same data (but does not normalize for chance). The deliverable resources and software include: (1) a turn-key extractive multi-document summarizer, MEAD, which allows users to add their own features based on single sentences or pairs of sentences, (2) a large corpus of summaries produced by several automatic methods, including baseline and random summaries, (3) a collection of manual summaries produced by the Linguistic Data Consortium (LDC), (4) a battery of evaluation routines, (5) a collection of IR queries in English and Chinese and the corresponding relevance judgments from the Hong Kong news collection, (6) SMART relevance outputs for both full documents and summaries, (7) XML tools for processing of documents and summaries. JHU 2001 Summer workshop final report Evaluation of Text Summarization

Existing neural machine translation (NMT) models generally translate sentences in isolation, missing the opportunity to take advantage of document-level information. In this work, we propose to augment NMT models with a very light-weight cache-like memory network, which stores recent hidden representations as translation history. The probability distribution over generated words is updated online depending on the translation history retrieved from the memory, endowing NMT models with the capability to dynamically adapt over time. Experiments on multiple domains with different topics and styles show the effectiveness of the proposed approach with negligible impact on the computational cost.

We propose a fully automatic technique for evaluating text summaries without the need to prepare the gold standard summaries manually. A standard and popular summary evaluation techniques or tools are not fully automatic; they all need some manual process or manual reference summary. Using recognizing textual entailment (TE), automatically generated summaries can be evaluated completely automatically without any manual preparation process. We use a TE system based on a combination of lexical entailment module, lexical distance module, Chunk module, Named Entity module and syntactic text entailment (TE) module. The documents are used as text (T) and summary of these documents are taken as hypothesis (H). Therefore, the more information of the document is entailed by its summary the better the summary. Comparing with the ROUGE 1.5.5 evaluation scores over TAC 2008 (formerly DUC, conducted by NIST) dataset, the proposed evaluation technique predicts the ROUGE scores with a accuracy of 98.25% with respect to ROUGE-2 and 95.65% with respect to ROUGE-SU4.

Teaching machines to read natural language documents remains an elusive challenge. Machine reading systems can be tested on their ability to answer questions posed on the contents of documents that they have seen, but until now large scale training and test datasets have been missing for this type of evaluation. In this work we define a new methodology that resolves this bottleneck and provides large scale supervised reading comprehension data. This allows us to develop a class of attention based deep neural networks that learn to read real documents and answer complex questions with minimal prior knowledge of language structure.

Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. For instance, while the current state-of-the-art BLEU-1 score (the higher the better) on the Pascal dataset is 25, our approach yields 59, to be compared to human performance around 69. We also show BLEU-1 score improvements on Flickr30k, from 56 to 66, and on SBU, from 19 to 28. Lastly, on the newly released COCO dataset, we achieve a BLEU-4 of 27.7, which is the current state-of-the-art.

We consider matching with pre-trained contextualized word vectors for multi-turn response selection in retrieval-based chatbots. When directly applied to the task, state-ofthe-art models, such as CoVe and ELMo, do not work as well as they do on other tasks, due to the hierarchical nature, casual language, and domain-specific word use of conversations. To tackle the challenges, we propose pre-training a sentence-level and a sessionlevel contextualized word vectors by learning a dialogue generation model from large-scale human-human conversations with a hierarchical encoder-decoder architecture. The two levels of vectors are then integrated into the input layer and the output layer of a matching model respectively. Experimental results on two benchmark datasets indicate that the proposed contextualized word vectors can significantly and consistently improve the performance of existing matching models for response selection.

We aim to automatically generate natural language narratives about an input structured knowledge base (KB). We build our generation framework based on a pointer network which can copy facts from the input KB, and add two attention mechanisms: (i) slot-aware attention to capture the association between a slot type and its corresponding slot value; and (ii) a new table position self-attention to capture the inter-dependencies among related slots. For evaluation, besides standard metrics including BLEU, METEOR, and ROUGE, we also propose a KB reconstruction based metric by extracting a KB from the generation output and comparing it with the input KB. We also create a new data set which includes 106,216 pairs of structured KBs and their corresponding natural language descriptions for two distinct entity types. Experiments show that our approach significantly outperforms state-of-the-art methods. The reconstructed KB achieves 68.8% 72.6% F-score.1

We demonstrate that replacing an LSTM encoder with a self-attentive architecture can lead to improvements to a state-ofthe-art discriminative constituency parser. The use of attention makes explicit the manner in which information is propagated between different locations in the sentence, which we use to both analyze our model and propose potential improvements. For example, we find that separating positional and content information in the encoder can lead to improved parsing accuracy. Additionally, we evaluate different approaches for lexical representation. Our parser achieves new state-ofthe-art results for single models trained on the Penn Treebank: 93.55 F1 without the use of any external data, and 95.13 F1 when using pre-trained word representations. Our parser also outperforms the previous best-published accuracy figures on 8 of the 9 languages in the SPMRL dataset.

How to model a pair of sentences is a critical issue in many NLP tasks such as answer selection (AS), paraphrase identification (PI) and textual entailment (TE). Most prior work (i) deals with one individual task by fine-tuning a specific system; (ii) models each sentence’s representation separately, rarely considering the impact of the other sentence; or (iii) relies fully on manually designed, task-specific linguistic features. This work presents a general Attention Based Convolutional Neural Network (ABCNN) for modeling a pair of sentences. We make three contributions. (i) The ABCNN can be applied to a wide variety of tasks that require modeling of sentence pairs. (ii) We propose three attention schemes that integrate mutual influence between sentences into CNNs; thus, the representation of each sentence takes into consideration its counterpart. These interdependent sentence pair representations are more powerful than isolated sentence representations. (iii) ABCNNs achieve state-of-the-art performance on AS, PI and TE tasks. We release code at: https://github.com/yinwenpeng/Answer_Selection.


Question: A multi-layer encoder for summarizing text content.

This paper presents an innovative approach to Abstractive Sentence Summarization, which aims to distill a concise version of a sentence without sacrificing its core meaning. We propose the utilization of a conditional recurrent neural network (RNN) for generating summaries of input sentences. A distinguishing feature of our model is the incorporation of a convolutional attention-based encoder, which serves as a conditioning mechanism. This encoder ensures that the RNN decoder is directed to focus on the most pertinent words in the input sentence at each stage of the summary generation process. Our model is designed to operate solely on learned features, offering the advantage of straightforward end-to-end training on extensive datasets. Experimental results demonstrate that our model outperforms recent state-of-the-art methods on the Gigaword corpus, while also showing strong performance on the DUC-2004 shared task. This showcases the model's effectiveness in abstractive summarization tasks.

Summarization and Keyword Selection are two important tasks in NLP community. Although both aim to summarize the source articles, they are usually treated separately by using sentences or words. In this paper, we propose a two-level graph based ranking algorithm to generate summarization and extract keywords at the same time. Previous works have reached a consensus that important sentence is composed by important keywords. In this paper, we further study the mutual impact between them through context analysis. We use Wikipedia to build a two-level concept-based graph, instead of traditional term-based graph, to express their homogenous relationship and heterogeneous relationship. We run PageRank and HITS rank on the graph to adjust both homogenous and heterogeneous relationships. A more reasonable relatedness value will be got for key sentence selection and keyword selection. We evaluate our algorithm on TAC 2011 data set. Traditional term-based approach achieves a score of 0.255 in ROUGE-1 and a score of 0.037 and ROUGE-2 and our approach can improve them to 0.323 and 0.048 separately.

Extracting summaries via integer linear programming and submodularity are popular and successful techniques in extractive multi-document summarization. However, many interesting optimization objectives are neither submodular nor factorizable into an integer linear program. We address this issue and present a general optimization framework where any function of input documents and a system summary can be plugged in. Our framework includes two kinds of summarizers – one based on genetic algorithms, the other using a swarm intelligence approach. In our experimental evaluation, we investigate the optimization of two information-theoretic summary evaluation metrics and find that our framework yields competitive results compared to several strong summarization baselines. Our comparative analysis of the genetic and swarm summarizers reveals interesting complementary properties.

Attention mechanisms in neural networks have proved useful for problems in which the input and output do not have fixed dimension. Often there exist features that are locally translation invariant and would be valuable for directing the model’s attention, but previous attentional architectures are not constructed to learn such features specifically. We introduce an attentional neural network that employs convolution on the input tokens to detect local time-invariant and long-range topical attention features in a context-dependent way. We apply this architecture to the problem of extreme summarization of source code snippets into short, descriptive function name-like summaries. Using those features, the model sequentially generates a summary by marginalizing over two attention mechanisms: one that predicts the next summary token based on the attention weights of the input tokens and another that is able to copy a code token as-is directly into the summary. We demonstrate our convolutional attention neural network’s performance on 10 popular Java projects showing that it achieves better performance compared to previous attentional mechanisms.

Inspired by recent advances in multimodal learning and machine translation, we introduce an encoder-decoder pipeline that learns (a): a multimodal joint embedding space with images and text and (b): a novel language model for decoding distributed representations from our space. Our pipeline effectively unifies joint image-text embedding models with multimodal neural language models. We introduce the structure-content neural language model that disentangles the structure of a sentence to its content, conditioned on representations produced by the encoder. The encoder allows one to rank images and sentences while the decoder can generate novel descriptions from scratch. Using LSTM to encode sentences, we match the state-of-the-art performance on Flickr8K and Flickr30K without using object detections. We also set new best results when using the 19-layer Oxford convolutional network. Furthermore we show that with linear encoders, the learned embedding space captures multimodal regularities in terms of vector space arithmetic e.g. *image of a blue car* "blue" + "red" is near images of red cars. Sample captions generated for 800 images are made available for comparison.

With the huge amount of information available electronically, there is an increasing demand for automatic text summarization systems. The use of machine learning techniques for this task allows one to adapt summaries to the user needs and to the corpus characteristics. These desirable properties have motivated an increasing amount of work in this field over the last few years. Most approaches attempt to generate summaries by extracting sentence segments and adopt the supervised learning paradigm which requires to label documents at the text span level. This is a costly process, which puts strong limitations on the applicability of these methods. We investigate here the use of semi-supervised algorithms for summarization. These techniques make use of few labeled data together with a larger amount of unlabeled data. We propose new semi-supervised algorithms for training classification models for text summarization. We analyze their performances on two data sets - the Reuters news-wire corpus and the Computation and Language (cmp_lg) collection of TIPSTER SUMMAC. We perform comparisons with a baseline - non learning - system, and a reference trainable summarizer system.

Teaching machines to read natural language documents remains an elusive challenge. Machine reading systems can be tested on their ability to answer questions posed on the contents of documents that they have seen, but until now large scale training and test datasets have been missing for this type of evaluation. In this work we define a new methodology that resolves this bottleneck and provides large scale supervised reading comprehension data. This allows us to develop a class of attention based deep neural networks that learn to read real documents and answer complex questions with minimal prior knowledge of language structure.

Conversational machine comprehension requires a deep understanding of the conversation history. To enable traditional, single-turn models to encode the history comprehensively, we introduce FLOW, a mechanism that can incorporate intermediate representations generated during the process of answering previous questions, through an alternating parallel processing structure. Compared to shallow approaches that concatenate previous questions/answers as input, FLOW integrates the latent semantics of the conversation history more deeply. Our model, FLOWQA, shows superior performance on two recently proposed conversational challenges (+7.2% F1 on CoQA and +4.0% on QuAC). The effectiveness of FLOW also shows in other tasks. By reducing sequential instruction understanding to conversational machine comprehension, FLOWQA outperforms the best models on all three domains in SCONE, with +1.8% to +4.4% improvement in accuracy.

As part of evaluating a summary automatically, it is usual to determine how much of the contents of one or more human-produced ‘ideal’ summaries it contains. Past automated methods such as ROUGE compare using fixed word ngrams, which are not ideal for a variety of reasons. In this paper we describe a framework in which summary evaluation measures can be instantiated and compared, and we implement a specific evaluation method using very small units of content, called Basic Elements, that address some of the shortcomings of ngrams. This method is tested on DUC 2003, 2004, and 2005 systems and produces very good correlations with human judgments.

This paper describes a technique to compare large text sources using word vector representations (word2vec) and dimensionality reduction (tSNE) and how it can be implemented using Python. The technique provides a bird’s-eye view of text sources, e.g. text summaries and their source material, and enables users to explore text sources like a geographical map. Word vector representations capture many linguistic properties such as gender, tense, plurality and even semantic concepts like "capital city of". Using dimensionality reduction, a 2D map can be computed where semantically similar words are close to each other. The technique uses the word2vec model from the gensim Python library and t-SNE from scikit-learn.

We present an effective end-to-end memory network model that jointly (i) predicts whether a given document can be considered as relevant evidence for a given claim, and (ii) extracts snippets of evidence that can be used to reason about the factuality of the target claim. Our model combines the advantages of convolutional and recurrent neural networks as part of a memory network. We further introduce a similarity matrix at the inference level of the memory network in order to extract snippets of evidence for input claims more accurately. Our experiments on a public benchmark dataset, FakeNewsChallenge, demonstrate the effectiveness of our approach.


Question: A multi-layer encoder for text abstraction and summarization.

This paper introduces a novel selective encoding model designed to enhance the sequence-to-sequence framework for abstractive sentence summarization tasks. Our proposed model integrates a sentence encoder, a selective gate network, and an attention-based decoder. The sentence encoder and decoder are constructed using recurrent neural networks (RNNs). The selective gate network plays a crucial role by regulating the flow of information from the encoder to the decoder, thereby constructing a refined second-level sentence representation specifically optimized for the summarization task. This tailored representation significantly boosts the model's performance. We validate the effectiveness of our selective encoding model on several benchmark datasets: the English Gigaword, DUC 2004, and MSR abstractive sentence summarization datasets. The empirical results demonstrate that our model surpasses existing state-of-the-art baseline models, showcasing its superior capability in abstractive sentence summarization.

We propose Neural Responding Machine (NRM), a neural networ k-based response generator for Short-Text Conversation. NRM takes the general encoder-de coder framework: it formalizes the generation of response as a decoding process based on the lat ent representation of the input text, while both encoding and decoding are realized with recurren t n ural networks (RNN). The NRM is trained with a large amount of one-round conversation dat a collected from a microblogging service. Empirical study shows that NRM can generate gramma tically correct and content-wise appropriate responses to over 75% of the input text, outperf orming state-of-the-arts in the same setting, including retrieval-based and SMT-based models.

Neural network architectures such as long short-term memory (LSTM) have been proven to be powerful models for processing sequences including text, audio and video. On the basis of vanilla LSTM, multi-modal attention mechanisms are proposed in this paper to synthesize the time and semantic information of input sequences. First, we reconstruct the forget and input gates of the LSTM unit from the perspective of attention model in the temporal dimension. Then the memory content of the LSTM unit is recalculated using a cluster-based attention mechanism in semantic space. Experiments on acoustic scene classification tasks show performance improvements of the proposed methods when compared with vanilla LSTM. The classification errors on LITIS ROUEN dataset and DCASE2016 dataset are reduced by 16.5% and 7.7% relatively. We get a second place in the Kaggle’s YouTube-8M video understanding challenge, and multi-modal attention based LSTM model is one of our bestperforming single systems.

We show how to learn low-dimensional representations (embeddings) of a wide range of concepts in medicine, including diseases (e.g., ICD9 codes), medications, procedures, and laboratory tests. We expect that these embeddings will be useful across medical informatics for tasks such as cohort selection and patient summarization. These embeddings are learned using a technique called neural language modeling from the natural language processing community. However, rather than learning the embeddings solely from text, we show how to learn the embeddings from claims data, which is widely available both to providers and to payers. We also show that with a simple algorithmic adjustment, it is possible to learn medical concept embeddings in a privacy preserving manner from co-occurrence counts derived from clinical narratives. Finally, we establish a methodological framework, arising from standard medical ontologies such as UMLS, NDF-RT, and CCS, to further investigate the embeddings and precisely characterize their quantitative properties.

This paper targets at the first step towards generating citation summaries - to identify the reference scope (i.e., cited text spans) for citances. We present a novel classification-based method that converts the task into binary classification which distinguishes cited and non-cited pairs of citances and reference sentences. The method models pairs of citances and reference sentences as feature vectors where citation-dependent and citation-independent features based on the semantic similarity between texts and the significance of texts are explored. Such vector representations are utilized to train a binary classifier. For a citance, once the set of reference sentences classified as the cited sentences are collected, a heuristic-based filtering strategy is applied to refine the output. The method is evaluated using the CL-SciSumm 2016 datasets and found to perform well with competitive results.

How to model a pair of sentences is a critical issue in many NLP tasks such as answer selection (AS), paraphrase identification (PI) and textual entailment (TE). Most prior work (i) deals with one individual task by fine-tuning a specific system; (ii) models each sentence’s representation separately, rarely considering the impact of the other sentence; or (iii) relies fully on manually designed, task-specific linguistic features. This work presents a general Attention Based Convolutional Neural Network (ABCNN) for modeling a pair of sentences. We make three contributions. (i) The ABCNN can be applied to a wide variety of tasks that require modeling of sentence pairs. (ii) We propose three attention schemes that integrate mutual influence between sentences into CNNs; thus, the representation of each sentence takes into consideration its counterpart. These interdependent sentence pair representations are more powerful than isolated sentence representations. (iii) ABCNNs achieve state-of-the-art performance on AS, PI and TE tasks. We release code at: https://github.com/yinwenpeng/Answer_Selection.

We discuss technology to help a person monitor changes in news coverage over time. We define temporal summaries of news stories as extracting a single sentence from each event within a news topic, where the stories are presented one at a time and sentences from a story must be ranked before the next story can be considered. We explain a method for evaluation, and describe an evaluation corpus that we have built. We also propose several methods for constructing temporal summaries and evaluate their effectiveness in comparison to degenerate cases. We show that simple approaches are effective, but that the problem is far from solved.

Automatic text summarization is a major area of research in the domain of information systems. Most of the methods requires domain knowledge in order to produce a coherent and meaningful summary. In Extractive text summarization, sentences are scored on some features. A large number of feature based scoring methods have been proposed for extractive automatic text summarization by researchers. This paper reviews features for sentence scoring. The results on combinations of various features for scoring are discussed. ROUGE-N is used to evaluate generated summary with abstractive summary of DUC 2002 dataset.

We propose a hierarchical attention network for document classification. Our model has two distinctive characteristics: (i) it has a hierarchical structure that mirrors the hierarchical structure of documents; (ii) it has two levels of attention mechanisms applied at the wordand sentence-level, enabling it to attend differentially to more and less important content when constructing the document representation. Experiments conducted on six large scale text classification tasks demonstrate that the proposed architecture outperform previous methods by a substantial margin. Visualization of the attention layers illustrates that the model selects qualitatively informative words and sentences.

Deep learning methods have recently achieved great empirical success on machine translation, dialogue response generation, summarization, and other text generation tasks. At a high level, the technique has been to train end-to-end neural network models consisting of an encoder model to produce a hidden representation of the source text, followed by a decoder model to generate the target. While such models have significantly fewer pieces than earlier systems, significant tuning is still required to achieve good performance. For text generation models in particular, the decoder can behave in undesired ways, such as by generating truncated or repetitive outputs, outputting bland and generic responses, or in some cases producing ungrammatical gibberish. This paper is intended as a practical guide for resolving such undesired behavior in text generation models, with the aim of helping enable real-world applications.


Question: A multi-layer encoder architecture for text summarization tasks.

This paper introduces a novel regularization strategy specifically tailored for Recurrent Neural Networks (RNNs) featuring Long Short-Term Memory (LSTM) units. Despite its proven efficacy in regularizing neural networks, the dropout technique has shown limited success when applied to RNNs and LSTMs. Herein, we elucidate the proper application of dropout to LSTM architectures and demonstrate its significant impact in mitigating overfitting across a spectrum of computational tasks. These tasks encompass language modeling, speech recognition, and machine translation, where our regularization method significantly enhances model performance by reducing overfitting.

**Question**: An enhanced particle swarm optimization for economic dispatch problem with valve-point effect

**Documents**: A Novel Particle Swarm Optimization Algorithm for Multi-Objective Combinatorial Optimization Problem
This paper presents a novel particle swarm optimization algorithm for solving multi-objective combinatorial optimization problems. The new algorithm is called multi-objective particle swarm optimization with Pareto-adaptive ε-dominance (MOPSO-PAED). In MOPSO-PAED, each particle is assigned a Pareto-adaptive ε-dominance value, which is used to guide the search process. The algorithm is tested on a set of benchmark problems, including the multi-objective knapsack problem, multi-objective traveling salesman problem, and multi-objective set covering problem. The experimental results show that MOPSO-PAED is able to find high-quality Pareto solutions for these problems.

**Similar Documents**: This paper unveils a new particle swarm optimization (PSO) algorithm designed to address multi-objective combinatorial optimization challenges, termed the Multi-Objective Particle Swarm Optimization with Pareto-adaptive ε-dominance (MOPSO-PAED). In MOPSO-PAED, particles are endowed with a Pareto-adaptive ε-dominance value to steer the optimization process. The efficacy of the algorithm is assessed through a series of benchmark problems, encompassing the multi-objective knapsack, traveling salesman, and set covering problems. Experimental outcomes highlight MOPSO-PAED's capability to efficiently generate high-quality Pareto solutions for these complex optimization tasks.

The following report introduces ideas augmenting standard Long Short Term Memory (LSTM) architecture with multiple memory cells per hidden unit in order to improve its generalization capabilities. It considers both deterministic and stochastic variants of memory operation. It is shown that the nondeterministic Array-LSTM approach improves stateof-the-art performance on character level text prediction achieving 1.402 BPC† on enwik8 dataset. Furthermore, this report estabilishes baseline neural-based results of 1.12 BPC and 1.19 BPC for enwik9 and enwik10 datasets respectively.

This paper presents a novel multi-document summarization approach based on personalized pagerank (PPRSum). In this algorithm, we uniformly integrate various kinds of information in the corpus. At first, we train a salience model of sentence global features based on Naive Bayes Model. Secondly, we generate a relevance model for each corpus utilizing the query of it. Then, we compute the personalized prior probability for each sentence in the corpus utilizing the salience model and the relevance model both. With the help of personalized prior probability, a Personalized PageRank ranking process is performed depending on the relationships among all sentences in the corpus. Additionally, the redundancy penalty is imposed on each sentence. The summary is produced by choosing the sentences with both high query-focused information richness and high information novelty. Experiments on DUC2007 are performed and the ROUGE evaluation results show that PPRSum ranks between the 1st and the 2nd systems on DUC2007 main task.

Previous work combines word-level and character-level representations using concatenation or scalar weighting, which is suboptimal for high-level tasks like reading comprehension. We present a fine-grained gating mechanism to dynamically combine word-level and character-level representations based on properties of the words. We also extend the idea of fine-grained gating to modeling the interaction between questions and paragraphs for reading comprehension. Experiments show that our approach can improve the performance on reading comprehension tasks, achieving new state-of-the-art results on the Children’s Book Test and Who Did What datasets. To demonstrate the generality of our gating mechanism, we also show improved results on a social media tag prediction task.1

Recently a variety of LSTM-based conditional language models (LM) have been applied across a range of language generation tasks. In this work we study various model architectures and different ways to represent and aggregate the source information in an endto-end neural dialogue system framework. A method called snapshot learning is also proposed to facilitate learning from supervised sequential signals by applying a companion cross-entropy objective function to the conditioning vector. The experimental and analytical results demonstrate firstly that competition occurs between the conditioning vector and the LM, and the differing architectures provide different trade-offs between the two. Secondly, the discriminative power and transparency of the conditioning vector is key to providing both model interpretability and better performance. Thirdly, snapshot learning leads to consistent performance improvements independent of which architecture is used.

We present two approaches that use unlabeled data to improve sequ nce learning with recurrent networks. The first approach is to predict wha t comes next in a sequence, which is a conventional language model in natural language processing. The second approach is to use a sequence autoencoder, which r eads the input sequence into a vector and predicts the input sequence again. T hese two algorithms can be used as a “pretraining” step for a later supervised seq uence learning algorithm. In other words, the parameters obtained from the unsu pervised step can be used as a starting point for other supervised training model s. In our experiments, we find that long short term memory recurrent networks after b eing pretrained with the two approaches are more stable and generalize bette r. With pretraining, we are able to train long short term memory recurrent network s up to a few hundred timesteps, thereby achieving strong performance in ma ny text classification tasks, such as IMDB, DBpedia and 20 Newsgroups.

Summarization is the way towards lessening the content of a text file to make it brief that holds all the critical purposes in the content of original text file. In the process of extractive summarization, one extracts only those sentences which are the most relevant sentences in the text document and that conveys the moral of the content. The extractive summarization techniques usually revolve around the idea of discovering most relevant and frequent keywords and then extract the sentences based on those keywords. Manual extraction or explanation of relevant keywords are a dreary procedure overflowing with errors including loads of manual exertion and time. In this paper, we proposed a hybrid approach to extract keyword automatically for multi-document text summarization in enewspaper articles. The performance of the proposed approach is compared with three additional keyword extraction techniques namely, term frequency-inverse document frequency (TF-IDF), term frequency-adaptive inverse document frequency (TF-AIDF), and a number of false alarm (NFA) for automatic keyword extraction and summarization in e-newspapers articles for better analysis. Finally, we showed that our proposed techniques had been outperformed over other techniques for automatic keyword extraction and summarization.

ACL 2018 • Baseline: 6 layer Transformer, 512 hidden units • Data sets: IWSLT'16, WMT'17 • Metrics: BLEU (and METEOR in the paper) • 3 runs, reporting mean and standard deviation Encoder/Decoder Combinations • Architecture Definition Language • Allows easy experimentation of Neural Machine Translation (NMT) architecture variations • Detailed analysis of NMT architecture combinations • Multiple source attention layers and residual feedforward layer are key • Self-attention more important on the source side Introduction

ROUGE is a widely adopted, automatic evaluation measure for text summarization. While it has been shown to correlate well with human judgements, it is biased towards surface lexical similarities. This makes it unsuitable for the evaluation of abstractive summarization, or summaries with substantial paraphrasing. We study the effectiveness of word embeddings to overcome this disadvantage of ROUGE. Specifically, instead of measuring lexical overlaps, word embeddings are used to compute the semantic similarity of the words used in summaries instead. Our experimental results show that our proposal is able to achieve better correlations with human judgements when measured with the Spearman and Kendall rank co-

How should one apply deep learning to tasks such as morphological reinflection, which stochastically edit one string to get another? A recent approach to such sequence-to-sequence tasks is to compress the input string into a vector that is then used to generate the output string, using recurrent neural networks. In contrast, we propose to keep the traditional architecture, which uses a finite-state transducer to score all possible output strings, but to augment the scoring function with the help of recurrent networks. A stack of bidirectional LSTMs reads the input string from leftto-right and right-to-left, in order to summarize the input context in which a transducer arc is applied. We combine these learned features with the transducer to define a probability distribution over aligned output strings, in the form of a weighted finite-state automaton. This reduces hand-engineering of features, allows learned features to examine unbounded context in the input string, and still permits exact inference through dynamic programming. We illustrate our method on the tasks of morphological reinflection and lemmatization.

Common-sense or background knowledge is required to understand natural language, but in most neural natural language understanding (NLU) systems, the requisite background knowledge is indirectly acquired from static corpora. We develop a new reading architecture for the dynamic integration of explicit background knowledge in NLU models. A new task-agnostic reading module provides refined word representations to a task-specific NLU architecture by processing background knowledge in the form of free-text statements, together with the taskspecific inputs. Strong performance on the tasks of document question answering (DQA) and recognizing textual entailment (RTE) demonstrate the effectiveness and flexibility of our approach. Analysis shows that our models learn to exploit knowledge selectively and in a semantically appropriate way.


Question: Information retrieval - a text mining methodology

This paper introduces a novel method for Information Extraction (IE) using a relational learning approach to automatically generate pattern-match rules. IE is a critical text mining technique that identifies and extracts specific items of interest from natural language documents, making it an essential tool for data analysis and information management. However, traditional IE systems are often domain-specific and require extensive manual rule creation, which is both time-consuming and challenging. To address these limitations, we propose a machine learning-based solution that leverages the inherent patterns in text data. Our system, RAPIER, is designed to learn from document-template pairs, where the templates represent the desired output format. The learning algorithm is a hybrid that combines techniques from various inductive logic programming systems, enabling it to learn complex, unbounded patterns. These patterns include constraints on the surrounding words and part-of-speech tags, crucial for accurately identifying the desired information. We validate the effectiveness of our approach through a case study on extracting information from computer job postings found on the newsgroup misc.jobs.offered. The results are promising, demonstrating the potential of RAPIER to automate the information extraction process in a variety of domains.

Ontology can be seen as a formal representation of knowledge. They have been investigated in many arti ficial intelligence studies including semantic web, softwa re engineering, and information retrieval. The aim of ontology is t o develop knowledge representations that can be shared and re used. This research project is concerned with the use of assoc iati n rules to extract the Quran ontology. The manual acquisition of ontologies from Quran verses can be very costly; therefore, we ne d an intelligent system for Quran ontology construction using patternbased schemes and associations rules to discover Qu ran concepts and semantics relations from Quran verses. Our system i s based on the combination of statistics and linguistics methods t o extract concepts and conceptual relations from Quran. In particular, linguistic pattern-based approach is exploited to extract spec ific concepts from the Quran, while the conceptual relations are found based on association rules technique. The Quran ontology wil l offer a new and powerful representation of Quran knowledge, and the association rules will help to represent the relations between all classes of connected concepts in the Quran ontology. Keywords— Ontology; Concepts Extraction ; Quran; Association Rules; Text Mining.

In this paper, we address the problem of ranking clinical documents using centrality based approach. We model the documents to be ranked as nodes in a graph and place edges between documents based on their similarity. Given a query, we compute similarity of the query with respect to every document in the graph. Based on these similarity values, documents are ranked for a given query. Initially, Lucene is used to retrieve top fifty documents that are relevant to the query and then our proposed approach is applied on these retrieved documents to rerank them. Experimental results show that our approach did not perform well as the documents retrieved by Lucene are not among the top 50 documents in the Gold Standard.

Two well-known indexing methods are inverted files and signature files. We have undertaken a detailed comparison of these two approaches in the context of text indexing, paying particular attention to query evaluation speed and space requirements. We have examined their relative performance using both experimentation and a refined approach to modeling of signature files, and demonstrate that inverted files are distinctly superior to signature files. Not only can inverted files be used to evaluate typical queries in less time than can signature files, but inverted files require less space and provide greater functionality. Our results also show that a synthetic text database can provide a realistic indication of the behavior of an actual text database. The tools used to generate the synthetic database have been made publicly available

The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. I dedicate this book to my grandchildren. Preface The intent of this book is to describe some recent data mining tools that have proven effective in dealing with data sets which often involve uncertain description or other complexities that cause difficulty for the conventional approaches of logistic regression, neural network models, and decision trees. Among these traditional algorithms, neural network models often have a relative advantage when data is complex. We will discuss methods with simple examples, review applications, and evaluate relative advantages of several contemporary methods. Our intent is to cover the fundamental concepts of data mining, to demonstrate the potential of gathering large sets of data, and analyzing these data sets to gain useful business understanding. We have organized the material into three parts. Part I introduces concepts. Part II contains chapters on a number of different techniques often used in data mining. Part III focuses on business applications of data mining. Not all of these chapters need to be covered, and their sequence could be varied at instructor design. The book will include short vignettes of how specific concepts have been applied in real practice. A series of representative data sets will be generated to demonstrate specific methods and concepts. References to data mining software and sites such as www.kdnuggets.com will be provided. Chapter 1 gives an overview of data mining, and provides a description of the data mining process. An overview of useful business applications is provided. Chapter 2 presents the data mining process in more detail. It demonstrates this process with a typical set of data. Visualization of data through data mining software is addressed. Chapter 3 presents memory-based reasoning methods of data mining. Major real applications are described. Algorithms are demonstrated with prototypical data based on real applications. Chapter 4 discusses association rule methods. Application in the form of market basket analysis is discussed. A real data set is described, and a simplified version used to demonstrate association rule methods. Chapter 5 presents fuzzy data mining approaches. Fuzzy decision tree approaches are described, as well as fuzzy association rule applications. Real data mining applications are described and demonstrated Chapter 6 presents Rough …

Today, a huge amount of text is being generated for social purposes on social networking services on the Web. Unlike traditional documents, such text is usually extremely short and tends to be informal. Analysis of such text benefit many applications such as advertising, search, and content filtering. In this work, we study one traditional text mining task on such new form of text, that is extraction of meaningful keywords. We propose several intuitive yet useful features and experiment with various classification models. Evaluation is conducted on Facebook data. Performances of various features and models are reported and compared.

Paper presents a survey of methods and approaches for keyword extraction task. In addition to the systematization of methods, the paper gathers a comprehensive review of existing research. Related work on keyword extraction is elaborated for supervised and unsupervised methods, with special emphasis on graphbased methods as well as Croatian keyword extraction. Selectivity-based keyword extraction method is proposed as a new unsupervised graph-based keyword extraction method which extracts nodes from a complex network as keyword candidates. The paper provides guidelines for future research and development of new graph-based approaches for keyword extraction.

Text document clustering plays an important role in providing intuitive navigation and browsing mechanisms by organizing large sets of documents into a small number of meaningful clusters. The bag of words representation used for these clustering methods is often unsatisfactory as it ignores relationships between important terms that do not cooccur literally. In order to deal with the problem, we integrate core ontologies as background knowledge into the process of clustering text documents. Our experimental evaluations compare clustering techniques based on precategorizations of texts from Reuters newsfeeds and on a smaller domain of an eLearning course about Java. In the experiments, improvements of results by background knowledge compared to a baseline without background knowledge can be shown in many interesting combinations.

The worldwide adoption of Twitter turned it into one of the most popular platforms for content analysis as it serves as a gauge of the public’s feeling and opinion on a variety of topics. This is particularly true of political discussions and lawmakers’ actions and initiatives. Yet, one common but unrealistic assumption is that the data of interest for analysis is readily available in a comprehensive and accurate form. Data need to be retrieved, but due to the brevity and noisy nature of Twitter content, it is difficult to formulate user queries that match relevant posts that use different terminology without introducing a considerable volume of unwanted content. This problem is aggravated when the analysis must contemplate multiple and related topics of interest, for which comments are being concurrently posted. This article presents Active Tweet Retrieval Visualization (ATR-Vis), a user-driven visual approach for the retrieval of Twitter content applicable to this scenario. The method proposes a set of active retrieval strategies to involve an analyst in such a way that a major improvement in retrieval coverage and precision is attained with minimal user effort. ATR-Vis enables non-technical users to benefit from the aforementioned active learning strategies by providing visual aids to facilitate the requested supervision. This supports the exploration of the space of potentially relevant tweets, and affords a better understanding of the retrieval results. We evaluate our approach in scenarios in which the task is to retrieve tweets related to multiple parliamentary debates within a specific time span. We collected two Twitter datasets, one associated with debates in the Canadian House of Commons during a particular week in May 2014, and another associated with debates in the Brazilian Federal Senate during a selected week in May 2015. The two use cases illustrate the effectiveness of ATR-Vis for the retrieval of relevant tweets, while quantitative results show that our approach achieves high retrieval quality with a modest amount of supervision. Finally, we evaluated our tool with three external users who perform searching in social media as part of their professional work.

This paper presents a novel framework for predicting stock trends and making financial trading decisions based on a combination of data and text mining techniques. The prediction models of the proposed system are based on the textual content of time-stamped Web documents in addition to traditional numerical time series data, which is also available from the Web. The financial trading system based on the model predictions (ADMIRAL) is using three different trading strategies. In this paper, the ADMIRAL system is simulated and evaluated on real-world series of news stories and stocks data using the C4.5 decision tree induction algorithm. The main performance measures are the predictive accuracy of the induced models and, more importantly, the profitability of each trading strategy using these predictions

Text Summarization plays an important role in the area of text mining and natural language processing. As the information resources are increasing tremendously, readers are overloaded with loads of information. Finding out the relevant data and manually summarizing it in short time is much more difficult, challenging and tedious task for a human being. Text Summarization aims to compress the source text into a shorter and concise form with preserving its information content and overall meaning. Summarization can be classified into two main categories i.e. extractive summarization and abstractive summarization. This paper presents a novel approach to generate abstractive summary from extractive summary using WordNet ontology. An experimental result shows the generated summary in well-compressed, grammatically correct and human readable format.