Question: Client-oriented network-level QoE equity for encrypted 'DASH-S' content delivery.

This paper introduces Mininet, a groundbreaking system designed to facilitate rapid prototyping of large-scale networks on the limited resources of a single laptop. By leveraging OS-level virtualization features, such as processes and network namespaces, Mininet achieves impressive scalability, capable of simulating hundreds of nodes. Initial user experiences highlight the transformative impact of Mininet's real-time run, poke, and debug capabilities on workflow efficiency. Drawing from a wide range of case studies involving over 100 users across 18 institutions engaged in Software-Defined Networks (SDN) development, the paper underscores Mininet's potential to revolutionize collaborative network research. The system's self-contained SDN prototypes, downloadable and executable on any PC, offer an unprecedented platform for evaluation, exploration, modification, and further development, thus significantly enhancing the accessibility and interoperability of network research.

Data center networks continually seek higher network performance to meet the ever increasing application demand. Recently, researchers are exploring the method to enhance the data center network performance by intelligent caching and increasing the access points for hot data chunks. Motivated by this, we come up with a simple yet useful caching mechanism for generic data centers, i.e., a server caches a data chunk after an application on it reads the chunk from the file system, and then uses the cached chunk to serve subsequent chunk requests from nearby servers. To turn the basic idea above into a practical system and address the challenges behind it, we design content-centric data center networks CCDNs, which exploits an innovative combination of content-based forwarding and location [Internet Protocol IP]-based forwarding in switches, to correctly locate the target server for a data chunk on a fully distributed basis. Furthermore, CCDN enhances traditional content-based forwarding to determine the nearest target server, and enhances traditional location IP-based forwarding to make high utilization of the precious memory space in switches. Extensive simulations based on real-world workloads and experiments on a test bed built with NetFPGA prototypes show that, even with a small portion of the server’s storage as cache e.g., 3% and with a modest content forwarding information base size e.g., 1000 entries in switches, CCDN can improve the average throughput to get data chunks by 43% compared with a pure Hadoop File System HDFS system in a real data center.

http://dx.doi.org/10.1016/j.infsof.2014.09.012 0950-5849/ 2014 Elsevier B.V. All rights reserved. ⇑ Corresponding author at: Center of Informatics, Federal University of Pernambuco (CIn/UFPE), Av. Jornalista Anibal Fernandes, S/N, Cidade Universitária, 50. Recife, PE, Brazil. Tel.: +55 81 8250 0470. E-mail addresses: fss4@cin.ufpe.br (F. Selleri Silva), fsfs@cin.ufpe.br (F.S.F. Soares), alp5@cin.ufpe.br (A.L. Peres), ima2@cin.ufpe.br (I.M.d. Azevedo), aplfv@ci (A.P.L.F. Vasconcelos), fkk@cin.ufpe.br (F.K. Kamei), srlm@cin.ufpe.br (S.R.d.L. Meira). Fernando Selleri Silva a,b,⇑, Felipe Santana Furtado Soares , Angela Lima Peres , Ivanildo Monteiro de Azevedo , Ana Paula L.F. Vasconcelos , Fernando Kenji Kamei , Silvio Romero de Lemos Meira a,c

Efficient and scalable live-streaming overlay construction has become a hot topic recently. In order to improve the performance metrics, such as startup delay, source-to-end delay, and playback continuity, most previous studies focused on intra-overlay optimization. Such approaches have drawbacks including low resource utilization, high startup and source-to-end delay, and unreasonable resource assignment in global P2P networks. Anysee is a peer-to-peer live streaming system and adopts an inter-overlay optimization scheme, in which resources can join multiple overlays, so as to (1) improve global resource utilization and distribute traffic to all physical links evenly; (2) assign resources based on their locality and delay; (3) guarantee streaming service quality by using the nearest peers, even when such peers might belong to different overlays; and (4) balance the load among the group members. We compare the performance of our design with existing approaches based on comprehensive trace driven simulations. Results show that AnySee outperforms previous schemes in resource utilization and the QoS of streaming services. AnySee has been implemented as an Internet based live streaming system, and was successfully released in the summer of 2004 in CERNET of China. Over 60,000 users enjoy massive entertainment programs, including TV programs, movies, and academic conferences. Statistics prove that this design is scalable and robust, and we believe that the wide deployment of AnySee will soon benefit many more Internet users.

Data collection is required to be safe and efficient considering both data privacy and system performance. In this paper, we study a new problem: distributed data sharing with privacy-preserving requirements. Given a data demander requesting data from multiple distributed data providers, the objective is to enable the data demander to access the distributed data without knowing the privacy of any individual provider. The problem is challenged by two questions: how to transmit the data safely and accurately; and how to efficiently handle data streams? As the first study, we propose a practical method, Shadow Coding, to preserve the privacy in data transmission and ensure the recovery in data collection, which achieves privacy preserving computation in a data-recoverable, efficient, and scalable way. We also provide practical techniques to make Shadow Coding efficient and safe in data streams. Extensive experimental study on a large-scale real-life dataset offers insight into the performance of our schema. The proposed schema is also implemented as a pilot system in a city to collect distributed mobile phone data.

This paper describes a new approach to end-to-end congestion control on a multi-user network. Rather than manually formulate each endpoint's reaction to congestion signals, as in traditional protocols, we developed a program called Remy that generates congestion-control algorithms to run at the endpoints.
 In this approach, the protocol designer specifies their prior knowledge or assumptions about the network and an objective that the algorithm will try to achieve, e.g., high throughput and low queueing delay. Remy then produces a distributed algorithm---the control rules for the independent endpoints---that tries to achieve this objective.
 In simulations with ns-2, Remy-generated algorithms outperformed human-designed end-to-end techniques, including TCP Cubic, Compound, and Vegas. In many cases, Remy's algorithms also outperformed methods that require intrusive in-network changes, including XCP and Cubic-over-sfqCoDel (stochastic fair queueing with CoDel for active queue management).
 Remy can generate algorithms both for networks where some parameters are known tightly a priori, e.g. datacenters, and for networks where prior knowledge is less precise, such as cellular networks. We characterize the sensitivity of the resulting performance to the specificity of the prior knowledge, and the consequences when real-world conditions contradict the assumptions supplied at design-time.

In order to fully utilize the stable edge transmission capability of CDN and the scalable last-mile transmission capability of P2P, while at the same time avoiding ISP-unfriendly policies and unlimited usage of P2P delivery, some researches have begun focusing on CDN-P2P-hybrid architecture and ISP-friendly P2P content delivery technology in recent years. In this paper, we first survey CDN-P2P-hybrid architecture technology, including current industry efforts and academic efforts in this field. Second, we make comparisons between CDN and P2P. And then we explore and analyze main issues, including overlay route hybrid issues, and playing buffer hybrid issues. After that we focus on CDN-P2P-hybrid model analysis and design, we compare the tightlycoupled hybrid model with the loosely-coupled hybrid model, and we propose that there are some main common models which need further study. At last, we analyze the prospective research direction and propose our future work. KeywordsCDN, P2P, P2P Streaming, CDN-P2P-hybrid Architecture, Live Streaming, VoD Streaming Note: This work is supported by 2009 National Science Foundation of China (60903164): Research on Model and Algorithm of New-Generation Controllable, Trustworthy, Network-friendly CDN-P2P hybrid Content Delivery.

Multipath TCP, as proposed by the IETF working group mptcp, allows a single data stream to be split across multiple paths. This has obvious benefits for reliability, and it can also lead to more efficient use of networked resources. We describe the design of a multipath congestion control algorithm, we implement it in Linux, and we evaluate it for multihomed servers, data centers and mobile clients. We show that some ‘obvious’ solutions for multipath congestion control can be harmful, but that our algorithm improves throughput and fairness compared to single-path TCP. Our algorithm is a drop-in replacement for TCP, and we believe it is safe to deploy.

The blockchain-based digital content distribution system was developed. Decentralized and pear-to-pear authentication mechanism can be considered as the ideal rights management mechanism. The blockchain has the potential to realize this ideal content distribution system. This is the successful model of the Superdistribution concept which was announced almost 30 years ago. The proposed system was demonstrated and got a lot of feedback for the future practical system.

With the growing realization that current Internet protocols are reaching the limits of their senescence, several ongoing research efforts aim to design potential next-generation Internet architectures. Although they vary in maturity and scope, in order to avoid past pitfalls, these efforts seek to treat security and privacy as fundamental requirements. Resilience to Denialof-Service (DoS) attacks that plague today’s Internet is a major issue for any new architecture and deserves full attention. In this paper, we focus on DoS in Named Data Networking (NDN) – a specific candidate for next-generation Internet architecture designs. By naming data instead of its locations, NDN transforms data into a first-class entity and makes itself an attractive and viable approach to meet the needs for many current and emerging applications. It also incorporates some basic security features that mitigate classes of attacks that are commonly seen today. However, NDN’s resilience to DoS attacks has not been analyzed to-date. This paper represents a first step towards assessment and possible mitigation of DoS in NDN. After identifying and analyzing several new types of attacks, it investigates their variations, effects and counter-measures. This paper also sheds some light on the debate about relative virtues of self-certifying, as opposed to human-readable, names in the context of content-centric networking.

Functional encryption (FE) is an extremely powerful cryptographic mechanism that lets an authorized entity compute on encrypted data, and learn the results in the clear. However, all current cryptographic instantiations for general FE are too impractical to be implemented. We construct IRON, a provably secure, and practical FE system using Intel's recent Software Guard Extensions (SGX). We show that IRON can be applied to complex functionalities, and even for simple functions, outperforms the best known cryptographic schemes. We argue security by modeling FE in the context of hardware elements, and prove that IRON satisfies the security model.


Question: Client-oriented network-level QoE equity for encrypted 'DASH-S' streams.

This paper delves into the network characteristics of video streaming, a dominant component of Internet traffic. Despite its prevalence, detailed knowledge regarding the specific network traits of this traffic remains surprisingly scarce. Our research focuses on the two leading video streaming platforms, Netflix and YouTube, to analyze their network behavior. We uncover that the streaming tactics employed vary based on the application type (Web browser versus native mobile application) and the container technology (Silverlight, Flash, or HTML5) used for streaming. Specifically, we pinpoint three distinct streaming methodologies that generate traffic patterns ranging from non-acknowledgment (non-ack) clocked ON-OFF cycles to extensive Transmission Control Protocol (TCP) data transfers. To further our understanding, we develop an analytical framework to assess the potential implications of these streaming strategies on the overall network traffic. Based on our findings, we propose tailored recommendations to optimize network performance in response to these varying strategies.

**Question**: A Direct Search Method to solve Economic Dispatch Problem with Valve-Point Effect

**Documents**: An efficient metaheuristic algorithm for economic dispatch problem
The economic dispatch (ED) problem is one of the most important optimization problems in power systems. It aims to schedule the committed generating units in order to meet the required load demand at minimum cost while satisfying system and unit constraints. This paper presents a new metaheuristic algorithm called the Teaching-Learning-Based Optimization (TLBO) algorithm for solving the ED problem. The proposed algorithm is based on the effect of the influence of a teacher on the output of learners in a class which is expressed in terms of results improvement by a teacher. The performance of the proposed algorithm is evaluated on standard test systems with valve-point effect and compared with other algorithms available in the literature.

**Similar Documents**: An Innovative Metaheuristic Approach for Solving the Economic Dispatch Problem
The economic dispatch (ED) problem is a critical optimization task within power systems, focusing on the scheduling of operational generating units to fulfill load demand at the lowest possible cost, while adhering to operational constraints. This paper introduces an innovative metaheuristic technique, the Teaching-Learning-Based Optimization (TLBO) algorithm, to tackle the ED problem efficiently. Inspired by the educational process, where a teacher's influence significantly impacts learners' outcomes, the TLBO algorithm employs this concept to enhance the optimization process. The algorithm's effectiveness is demonstrated through its application to benchmark test systems, which include the valve-point effect—a common challenge in practical ED problems. Comparative analysis with existing algorithms from the literature underscores the superior performance and robustness of the TLBO

Rate Control for Communication Networks: Shadow Prices, Proportional Fairness and Stability Author(s): F. P. Kelly, A. K. Maulloo and D. K. H. Tan Source:  The Journal of the Operational Research Society, Vol. 49, No. 3 (Mar., 1998), pp. 237-252 Published by: on behalf of the Palgrave Macmillan Journals Operational Research Society Stable URL: http://www.jstor.org/stable/3010473 Accessed: 07-10-2015 21:10 UTC

Twitch is a live video streaming platform used for broadcasting video gameplay, ranging from amateur players to eSports tournaments. This platform has gathered a substantial world wide community, reaching more than 1.7 million broadcasters and 100 million visitors every month. Twitch is fundamentally different from “static” content distribution platforms such as YouTube and Netflix, as streams are generated and consumed in real time. In this paper, we explore the Twitch infrastructure to understand how it manages live streaming delivery to an Internet-wide audience. We found Twitch manages a geo-distributed infrastructure, with presence in four continents. Our findings show that Twitch dynamically allocates servers to channels depending on their popularity. Additionally, we explore the redirection strategy of clients to servers depending on their region and the specific channel.

Content Delivery Networks (CDNs) are increasingly deployed for their efficient content delivery and are often integrated with Software Defined Networks (SDNs) to achieve centrality and programmability of the network. However, these networks are also an attractive target for network attackers whose main goal is to exhaust network resources. One attack approach is to over-flood the OpenFlow switch tables containing routing information. Due to the increasing number of different flooding attacks such as DDoS, it becomes difficult to distinguish these attacks from normal traffic when evaluated with traditional attack detection methods. This paper proposes an architectural method that classifies and defends all possible forms of DDoS attack and legitimate Flash Crowd traffic using a segregated dimension functioning cognitive process based in a controller module. Our results illustrate that the proposed model yields significantly enhanced performance with minimal false positives and false negatives when classified with optimal Support Vector Machine and Logistic Regression algorithms. The traffic classifications initiate deployment of security rules to the OpenFlow switches, preventing new forms of flooding attacks. To the best of our knowledge, this is the first work conducted on SDN-driven CDNi used to detect and defend against all possible DDoS attacks through traffic segregated dimension functioning coupled with cognitive classification.

Detecting encrypted traffic is increasingly important for deep packet inspection (DPI) to improve the performance of intrusion detection systems. We propose a machine learning approach with several randomness tests to achieve high accuracy detection of encrypted traffic while requiring low overhead incurred by the detection procedure. To demonstrate how effective the proposed approach is, the performance of four classification methods (Näıve Bayesian, Support Vector Machine, CART and AdaBoost) are explored. Our recommendation is to use CART which is not only capable of achieving an accuracy of 99.9% but also up to about 2.9 times more efficient than the second best candidate (Näıve Bayesian).

The use of wireless technology based on Vehicular Ad hoc Networks (VANETs) for information exchange can influence the drivers' behavior towards improving driving performance and reducing road accidents. This information can even be more relevant if it is presented as a video stream. In this paper we propose a system that relies on VANET and video-streaming technology: the See-Through System (STS). The system enhances driver's visibility and supports the driver's overtaking decision in challenging situations, such as overtaking a vision-obstructing vehicle. The use of the See-Through System provides the driver with an additional tool for determining if traffic conditions permit starting an overtaking maneuver thus reducing the risk of overtaking. We tested the See-Through System on an experimental vehicle on the road as well as in the context of a driving simulator for real world environment. Results are promising, since the use of the 802.11p standard wireless communication protocol allows a vehicle-to-vehicle connection without significant delay and the totality of the participants regarded the information provided by the STS as useful.

A novel framework for remote service discovery and access of IP cameras with Network address Translation (NAT) traversal is presented in this paper. The proposed protocol, termed STDP (Service Trader Discovery Protocol), is a hybrid combination of Zeroconf and SIP (Session Initial Protocol). The Zeroconf is adopted for the discovery and/or publication of local services; whereas, the SIP is used for the delivery of local services to the remote nodes. In addition, both the SIP-ALG (Application Layer Gateway) and UPnP (Universal Plug and Play)-IGD (Internet Gateway Device) protocols are used for NAT traversal. The proposed framework is well-suited for high mobility applications where the fast deployment and low administration efforts of IP cameras are desired.

The IoT Internet of Things being a promising technology of the future. It is expected to connect billions of devices. The increased communication number is expected to generate data mountain and the data security can be a threat. The devices in the architecture are fundamentally smaller in size and low powered. In general, classical encryption algorithms are computationally expensive and this due to their complexity and needs numerous rounds for encrypting, basically wasting the constrained energy of the gadgets. Less complex algorithm, though, may compromise the desired integrity. In this paper we apply a lightweight encryption algorithm named as Secure IoT (SIT) to a quantized speech image for Secure IoT. It is a 64-bit block cipher and requires 64-bit key to encrypt the data. This quantized speech image is constructed by first quantizing a speech signal and then splitting the quantized signal into frames. Then each of these frames is transposed for obtaining the different columns of this quantized speech image. Simulations result shows the algorithm provides substantial security in just five encryption rounds. Keywords—IoT, Security, Encryption, quantized speech image, SNR, PESQ, Histogram, Entropy, Correlation.

In this paper we study how to develop an efficient real-time auditing and proof of violation (POV) scheme for cloud storage systems. A POV scheme enables a user or a service provider to produce cryptographic proofs which can be used to prove either the occurrence of the violation of properties or the innocence of the service provider. POV schemes are solutions for obtaining mutual nonrepudiation between users and the service provider in the cloud. After each file operation, a real-time auditing should be performed so that the violation of the service provider can be found instantly. Existing solutions need to cache the hash values of files in client devices and thus the overhead for storing and synchronizing hash values in client devices which share files is huge. We propose a novel scheme in which client devices do not need to cache any hash values of files. A small portion called slice of a binary hash tree is transferred to the client device for real-time auditing and it can support POV whenever the audit does not pass. Experimental results are presented that demonstrate the feasibility of the proposed scheme and show that our scheme outperforms previous work by one to two order of magnitude. Service providers of cloud storage can use the proposed scheme to provide a mutual nonrepudiation guarantee in their service-level agreements.

In this paper, we develop, analyze and implement a congestion control scheme in a noncooperative game framework, where each user's cost function is composed of a pricing function proportional to the queueing delay experienced by the user, and a fairly general utility function which captures the user demand for bandwidth. Using a network model based on fluid approximations and through a realistic modeling of queues, we establish the existence of a unique equilibrium as well as its global asymptotic stability for a general network topology, where boundary effects are also taken into account. We also provide sufficient conditions for system stability when there is a bottleneck link shared by multiple users experiencing nonnegligible communication delays. In addition, we study an adaptive pricing scheme using hybrid systems concepts. Based on these theoretical foundations, we implement a window-based, end-to-end congestion control scheme, and simulate it in ns-2 network simulator on various network topologies with sizable propagation delays.

This paper demonstrates complete AES key recovery from known-plaintext timings of a network server on another computer. This attack should be blamed on the AES design, not on the particular AES library used by the server; it is extremely difficult to write constant-time high-speed AES software for common general-purpose computers. This paper discusses several of the obstacles in detail.


Question: NEURAL NETWORK-BASED SUBWORD MODELING FOR LANGUAGE APPLICATIONS

This paper presents an efficient Hierarchical Distributed Language Model (HDLN) that significantly accelerates the training and testing processes of Neural Probabilistic Language Models (NPLMs). Despite their high accuracy in comparison to traditional n-gram models, NPLMs have been limited by their time-consuming computational requirements. Morin and Bengio previously developed a hierarchical model structured around a binary word tree, which, although faster than its non-hierarchical counterpart, suffered from reduced performance despite utilizing expert-designed word trees. In this work, we propose a novel approach to create a fast hierarchical language model alongside an automated algorithm for constructing word trees based on features extracted from the data. Our method for generating word trees does not rely on expert knowledge but rather on the intrinsic patterns within the data itself. We demonstrate that the resulting hierarchical models not only surpass the performance of non-hierarchical neural models but also outperform the most advanced n-gram models. This showcases the potential of hierarchical structures in enhancing the efficiency and effectiveness of language models.

In spite of the great success of the deep neural network (DNN) in speech processing, it is still unclear what kind of underlying mechanisms are involved in this achievement. This preliminary study attempts to find an answer by investigating the functions of DNN's hidden layers in representing speech articulations. Two sets of experiments are performed on the hidden layers in speech recognition. The layer removing experiment is conducted on the English TIMIT database, and the layer replacing experiment is to substitute a layer in an English DNN by the corresponding layer in a Japanese DNN. It is found that the different layers seem to be responsible for different phoneme groups according to the place of articulation. The lower layers are responsible for the back vowels, and the higher layers are responsible for the front vowels. The second layer (i.e. the first hidden layer) of the seven-layer network has major responsibility for more than half of the consonants with the constriction located in the front of the vocal tract, while the other consonants rely on the middle and higher layers. The layer replacing experiment demonstrated that the above relation was language independent. It is necessary to design elaborate studies to discover more details in the future.

We present two methods for unsupervised segmentation of words into morphemelike units. The model utilized is especially suited for languages with a rich morphology, such as Finnish. The first method is based on the Minimum Description Length (MDL) principle and works online. In the second method, Maximum Likelihood (ML) optimization is used. The quality of the segmentations is measured using an evaluation method that compares the segmentations produced to an existing morphological analysis. Experiments on both Finnish and English corpora show that the presented methods perform well compared to a current stateof-the-art system.

This paper describes the neural dependency parser submitted by Stanford to the CoNLL 2017 Shared Task on parsing Universal Dependencies. Our system uses relatively simple LSTM networks to produce part of speech tags and labeled dependency parses from segmented and tokenized sequences of words. In order to address the rare word problem that abounds in languages with complex morphology, we include a character-based word representation that uses an LSTM to produce embeddings from sequences of characters. Our system was ranked first according to all five relevant metrics for the system: UPOS tagging (93.09%), XPOS tagging (82.27%), unlabeled attachment score (81.30%), labeled attachment score (76.30%), and content word labeled attachment score (72.57%).

In this paper, we explore a novel bilingual word alignment approach based on DNN (Deep Neural Network), which has been proven to be very effective in various machine learning tasks (Collobert et al., 2011). We describe in detail how we adapt and extend the CD-DNNHMM (Dahl et al., 2012) method introduced in speech recognition to the HMMbased word alignment model, in which bilingual word embedding is discriminatively learnt to capture lexical translation information, and surrounding words are leveraged to model context information in bilingual sentences. While being capable to model the rich bilingual correspondence, our method generates a very compact model with much fewer parameters. Experiments on a large scale EnglishChinese word alignment task show that the proposed method outperforms the HMM and IBM model 4 baselines by 2 points in F-score.

In this paper we present a cross-lingual extension of a neural tensor network model for knowledge base completion. We exploit multilingual synsets from BabelNet to translate English triples to other languages and then augment the reference knowledge base with cross-lingual triples. We project monolingual embeddings of different languages to a shared multilingual space and use them for network initialization (i.e., as initial concept embeddings). We then train the network with triples from the cross-lingually augmented knowledge base. Results on WordNet link prediction show that leveraging cross-lingual information yields significant gains over exploiting only monolingual triples.

Perplexity is a widely used measure to evaluate word prediction power of a word-based language model. It can be computed independently and has shown good correlation with word error rate (WER) in speech recognition. However, for character based languages, character error rate (CER) is commonly used instead of WER as the measure for speech recognition, although language model is still word based. Due to the fact that different word segmentation strategies may result in different word vocabulary for the same text corpus, in many cases, wordbased perplexity is incompetent to evaluate the combined effect of word segmentation and language model training to predict final CER. In this paper, a new word-based language model evaluation measure is proposed to account for the effect of word segmentation and the goal of predicting CER. Experiments were conducted on Chinese speech recognition. Compared to the traditional word-based perplexity, the new measure is more robust to word segmentation and shows much more consistent correlation with CER in a large vocabulary continuous Chinese speech recognition task.

Neural Machine translation has shown promising results in recent years. In order to control the computational complexity, NMT has to employ a small vocabulary, and massive rare words outside the vocabulary are all replaced with a single unk symbol. Besides the inability to translate rare words, this kind of simple approach leads to much increased ambiguity of the sentences since meaningless unks break the structure of sentences, and thus hurts the translation and reordering of the in-vocabulary words. To tackle this problem, we propose a novel substitution-translation-restoration method. In substitution step, the rare words in a testing sentence are replaced with similar in-vocabulary words based on a similarity model learnt from monolingual data. In translation and restoration steps, the sentence will be translated with a model trained on new bilingual data with rare words replaced, and finally the translations of the replaced words will be substituted by that of original ones. Experiments on Chinese-to-English translation demonstrate that our proposed method can achieve more than 4 BLEU points over the attention-based NMT. When compared to the recently proposed method handling rare words in NMT, our method can also obtain an improvement by nearly 3 BLEU points.

This study proposes a new neural machine translation (NMT) model based on the encoderdecoder model that incorporates named entity (NE) tags of source-language sentences. Conventional NMT models have two problems enumerated as follows: (i) they tend to have difficulty in translating words with multiple meanings because of the high ambiguity, and (ii) these models’ ability to translate compound words seems challenging because the encoder receives a word, a part of the compound word, at each time step. To alleviate these problems, the encoder of the proposed model encodes the input word on the basis of its NE tag at each time step, which could reduce the ambiguity of the input word. Furthermore, the encoder introduces a chunk-level LSTM layer over a word-level LSTM layer and hierarchically encodes a source-language sentence to capture a compound NE as a chunk on the basis of the NE tags. We evaluate the proposed model on an English-to-Japanese translation task with the ASPEC, and English-to-Bulgarian and English-to-Romanian translation tasks with the Europarl corpus. The evaluation results show that the proposed model achieves up to 3.11 point improvement in BLEU.

End-to-end neural network models for named entity recognition (NER) have shown to achieve effective performances on general domain datasets (e.g. newswire), without requiring additional hand-crafted features. However, in biomedical domain, recent studies have shown that handengineered features (e.g. orthographic features) should be used to attain effective performance, due to the complexity of biomedical terminology (e.g. the use of acronyms and complex gene names). In this work, we propose a novel approach that allows a neural network model based on a long short-term memory (LSTM) to automatically learn orthographic features and incorporate them into a model for biomedical NER. Importantly, our bi-directional LSTM model learns and leverages orthographic features on an end-to-end basis. We evaluate our approach by comparing against existing neural network models for NER using three well-established biomedical datasets. Our experimental results show that the proposed approach consistently outperforms these strong baselines across all of the three datasets.

We present a state-of-the-art speech recognition system developed using end-toend deep learning. Our architecture is significantly simpler than traditional speech systems, which rely on laboriously engineered processing pipelines; these traditional systems also tend to perform poorly when used in noisy environments. In contrast, our system does not need hand-designed components to model background noise, reverberation, or speaker variation, but instead directly learns a function that is robust to such effects. We do not need a phoneme dictionary, nor even the concept of a “phoneme.” Key to our approach is a well-optimized RNN training system that uses multiple GPUs, as well as a set of novel data synthesis techniques that allow us to efficiently obtain a large amount of varied data for training. Our system, called Deep Speech, outperforms previously published results on the widely studied Switchboard Hub5’00, achieving 16.0% error on the full test set. Deep Speech also handles challenging noisy environments better than widely used, state-of-the-art commercial speech systems.


Question: NEURAL NETWORK-BASED SUBWORD MODELING FOR LANGUAGE APPLICATIONS

This paper introduces various enhancements to the conventional recurrent neural network language model (RNN LM), which has previously demonstrated superior accuracy in language modeling tasks over numerous competing methods. Despite its efficacy, the RNN LM has faced criticism for its high computational demand, a challenge this work addresses by proposing modifications that achieve a more than 15-fold increase in speed for both training and testing stages. The significance of employing a backpropagation through time algorithm is also highlighted, alongside a comparative study with feedforward networks. Moreover, the paper explores strategies for decreasing the model's parameter count, resulting in a streamlined RNN LM that is not only more compact and faster during both training and testing, but also maintains or improves upon the accuracy of the original model.

One of the key problems in spoken language understanding (SLU) is the task of slot filling. In light of the recent success of applying deep neural network technologies in domain detection and intent identification, we carried out an in-depth investigation on the use of recurrent neural networks for the more difficult task of slot filling involving sequence discrimination. In this work, we implemented and compared several important recurrent-neural-network architectures, including the Elman-type and Jordan-type recurrent networks and their variants. To make the results easy to reproduce and compare, we implemented these networks on the common Theano neural network toolkit, and evaluated them on the ATIS benchmark. We also compared our results to a conditional random fields (CRF) baseline. Our results show that on this task, both types of recurrent networks outperform the CRF baseline substantially, and a bi-directional Jordantype network that takes into account both past and future dependencies among slots works best, outperforming a CRFbased baseline by 14% in relative error reduction.

In this work, we propose a novel decoding approach for neural machine translation (NMT) based on continuous optimisation. The resulting optimisation problem can then be tackled using a whole range of continuous optimisation algorithms which have been developed and used in the literature mainly for training. Our approach is general and can be applied to other sequence-to-sequence neural models as well. We make use of this powerful decoding approach to intersect an underlying NMT with a language model, to intersect left-to-right and right-to-left NMT models, and to decode with soft constraints involving coverage and fertility of the source sentence words. The experimental results show the promise of the proposed framework.

Languages such as Chinese and Japanese have a significantly large number (several thousands) of alphabets as compared to other languages, and each of their sentences consists of several concatenated words with wide varieties of inflected forms; thus appropriate word segmentation is quite difficult. Therefore, recently proposed sophisticated language-processing methods designed for languages such as English cannot be applied. In this paper, we address those issues and propose a new and efficient document classification technique for such languages. The proposed method is characterized into a new “image-based character embedding” method and character-level convolutional neural networks method with “wildcard training.” The first method encodes each character based on its pictorial structures and preserves them. Further, the second method treats some of the input characters as wildcards in the classification stage and functions as efficient data augmentation. We confirmed that our proposed method showed superior performance when compared conventional methods for Japanese document classification problems.

Biomedical named entity recognition (bio-NER) is a crucial and basic step in many biomedical information extraction tasks. However, traditional NER systems are mainly based on complex hand-designed features which are derived from various linguistic analyses and maybe only adapted to specified area. In this paper, we construct Recurrent Neural Network to identify entity names with word embeddings input rather than hand-designed features. Our contributions mainly include three aspects: 1) we adapt a deep learning architecture Recurrent Neural Network (RNN) to entity names recognition; 2) based on the original RNNs such as Elman-type and Jordan-type model, an improved RNN model is proposed; 3) considering that both past and future dependencies are important information, we combine bidirectional recurrent neural networks based on information entropy at the top layer. The experiments conducted on the BioCreative II GM data set demonstrate RNN models outperform CRF and deep neural networks (DNN), furthermore, the improved RNN model performs better than two original RNN models and the combined method is effective.

Deep learning is becoming a mainstream technology for speech recognition at industrial scale. In this paper, we provide an overview of the work by Microsoft speech researchers since 2009 in this area, focusing on more recent advances which shed light to the basic capabilities and limitations of the current deep learning technology. We organize this overview along the feature-domain and model-domain dimensions according to the conventional approach to analyzing speech systems. Selected experimental results, including speech recognition and related applications such as spoken dialogue and language modeling, are presented to demonstrate and analyze the strengths and weaknesses of the techniques described in the paper. Potential improvement of these techniques and future research directions are discussed.

We present two methods for unsupervised segmentation of words into morphemelike units. The model utilized is especially suited for languages with a rich morphology, such as Finnish. The first method is based on the Minimum Description Length (MDL) principle and works online. In the second method, Maximum Likelihood (ML) optimization is used. The quality of the segmentations is measured using an evaluation method that compares the segmentations produced to an existing morphological analysis. Experiments on both Finnish and English corpora show that the presented methods perform well compared to a current stateof-the-art system.

In a conversation or a dialogue process, attention and intention play intrinsic roles. This paper proposes a neural network based approach that models the attention and intention processes. It essentially consists of three recurrent networks. The encoder network is a word-level model representing source side sentences. The intention network is a recurrent network that models the dynamics of the intention process. The decoder network is a recurrent network that produces responses to the input from the source side. It is a language model that is dependent on the intention and has an attention mechanism to attend to particular source side words, when predicting a symbol in the response. The model is trained end-toend without labeling data. Experiments show that this model generates natural responses to user inputs.

This paper describes the DCU-UVT team’s participation in the Language Identification in Code-Switched Data shared task in the Workshop on Computational Approaches to Code Switching. Wordlevel classification experiments were carried out using a simple dictionary-based method, linear kernel support vector machines (SVMs) with and without contextual clues, and a k-nearest neighbour approach. Based on these experiments, we select our SVM-based system with contextual clues as our final system and present results for the Nepali-English and Spanish-English datasets.

Attention mechanism has been a key component in Recurrent Neural Networks (RNNs) based sequence to sequence learning framework, which has been adopted in many text understanding tasks, such as neural machine translation and abstractive summarization. In these tasks, the attention mechanism models how important each part of the source sentence is to generate a target side word. To compute such importance scores, the attention mechanism summarizes the source side information in the encoder RNN hidden states (i.e., ht), and then builds a context vector for a target side word upon a subsequence representation of the source sentence, since ht actually summarizes the information of the subsequence containing the first t-th words in the source sentence. We in this paper, show that an additional attention mechanism called word attention, that builds itself upon word level representations, significantly enhances the performance of sequence to sequence learning. Our word attention can enrich the source side contextual representation by directly promoting the clean word level information in each step. Furthermore, we propose to use contextual gates to dynamically combine the subsequence level and word level contextual information. Experimental results on abstractive summarization and neural machine translation show that word attention significantly improve over strong baselines. In particular, we achieve the state-of-the-art result on WMT’14 English-French translation task with 12M training data.

In this paper we describe a work in progress on designing the continuous vector space word representations able to map unseen data adequately. We propose a LSTM-based feature extraction layer that reads in a sequence of characters corresponding to a word and outputs a single fixed-length real-valued vector. We then test our model on a POS tagging task on four typologically different languages. The results of the experiments suggest that the model can offer a solution to the out-of-vocabulary words problem, as in a comparable setting its OOV accuracy improves over that of a state of the art tagger.


Question: NEURAL NETWORK-BASED SUBWORD MODELING FOR LANGUAGE APPLICATIONS

This paper introduces an innovative learning technique, back-propagation, tailored for artificial neural networks consisting of units emulating neurones. The procedure involves iteratively modifying the connection weights within the network to minimize the discrepancy between the network's actual output and the desired output. Through this process, internal 'hidden' units, not directly involved in input or output, evolve to encapsulate significant characteristics of the task domain. The interactions among these hidden units effectively capture the underlying patterns in the task, setting back-propagation apart from previous, more rudimentary methods such as the perceptron-convergence algorithm.

In the context of subword language modeling with neural networks, back-propagation plays a pivotal role in training models to understand and generate human language at a subword level. By iteratively adjusting the weights of connections in the neural network, back-propagation enables the model to learn representations that are sensitive to the context and semantics of language, even at the subword level. This capability is crucial for handling the complexity and variability of natural language, particularly in scenarios where words are broken down into smaller, meaningful units. The technique's adaptability and precision in refining the network's weights make it indispensable for advancing the field of subword language modeling.

This paper describes a new method for synthesizing speech by concatenating sub-word units from a database of labelled speech. A large unit inventory is created by automatically clustering units of the same phone class based on their phonetic and prosodic context. The appropriate cluster is then selected for a target unit offering a small set of candidate units. An optimal path is found through the candidate units based on their distance from the cluster center and an acoustically based join cost. Details of the method and justification are presented. The results of experiments using two different databases are given, optimising various parameters within the system. Also a comparison with other existing selection based synthesis techniques is given showing the advantages this method has over existing ones. The method is implemented within a full text-to-speech system offering efficient natural sounding speech synthesis.

Neural language models learn word representations that capture rich linguistic and conceptual information. Here we investigate the embeddings learned by neural machine translation models. We show that translation-based embeddings outperform those learned by cutting-edge monolingual models at single-language tasks requiring knowledge of conceptual similarity and/or syntactic role. The findings suggest that, while monolingual models learn information about how concepts are related, neural-translation models better capture their true ontological status. It is well known that word representations can be learned from the distributional patterns in corpora. Originally, such representations were constructed by counting word co-occurrences, so that the features in one word’s representation corresponded to other words [11, 17]. Neural language models, an alternative means to learn word representations, use language data to optimise (latent) features with respect to a language modelling objective. The objective can be to predict either the next word given the initial words of a sentence [4, 14, 8], or simply a nearby word given a single cue word [13, 15]. The representations learned by neural models (sometimes called embeddings) generally outperform those acquired by co-occurrence counting models when applied to NLP tasks [3]. Despite these clear results, it is not well understood how the architecture of neural models affects the information encoded in their embeddings. Here, we explore this question by considering the embeddings learned by architectures with a very different objective function to monolingual language models: neural machine translation models. We show that translation-based embeddings outperform monolingual embeddings on two types of task: those that require knowledge of conceptual similarity (rather than simply association or relatedness), and those that require knowledge of syntactic role. We discuss what the findings indicate about the information content of different embeddings, and suggest how this content might emerge as a consequence of the translation objective. 1 Learning embeddings from language data Both neural language models and translation models learn real-valued embeddings (of specified dimension) for words in some pre-specified vocabulary, V , covering many or all words in their training corpus. At each training step, a ‘score’ for the current training example (or batch) is computed based on the embeddings in their current state. This score is compared to the model’s objective function, and the error is backpropagated to update both the model weights (affecting how the score is computed from the embeddings) and the embedding features. At the end of this process, the embeddings should encode information that enables the model to optimally satisfy its objective. 1.1 Monolingual models In the original neural language model [4] and subsequent variants [8], each training example consists of n subsequent words, of which the model is trained to predict the n-th word given the first n − 1 ar X iv :1 41 0. 07 18 v2 [ cs .C L ] 1 3 N ov 2 01 4 1 words. The model first represents the input as an ordered sequence of embeddings, which it transforms into a single fixed length ‘hidden’ representation by, e.g., concatenation and non-linear projection. Based on this representation, a probability distribution is computed over the vocabulary, from which the model can sample a guess at the next word. The model weights and embeddings are updated to maximise the probability of correct guesses for all sentences in the training corpus. More recent work has shown that high quality word embeddings can be learned via models with no nonlinear hidden layer [13, 15]. Given a single word in the corpus, these models simply predict which other words will occur nearby. For each word w in V , a list of training cases (w, c) : c ∈ V is extracted from the training corpus. For instance, in the skipgram approach [13], for each ‘cue word’ w the ‘context words’ c are sampled from windows either side of tokens of w in the corpus (with c more likely to be sampled if it occurs closer to w).1 For each w in V , the model initialises both a cue-embedding, representing the w when it occurs as a cue-word, and a context-embedding, used when w occurs as a context-word. For a cue word w, the model can use the corresponding cueembedding and all context-embeddings to compute a probability distribution over V that reflects the probability of a word occurring in the context of w. When a training example (w, c) is observed, the model updates both the cue-word embedding of w and the context-word embeddings in order to increase the conditional probability of c. 1.2 Translation-based embeddings Neural translation models generate an appropriate sentence in their target language St given a sentence Ss in their source language [see, e.g., 16, 6]. In doing so, they learn distinct sets of embeddings for the vocabularies Vs and Vt in the source and target languages respectively. Observing a training case (Ss, St), such a model represents Ss as an ordered sequence of embeddings of words from Vs. The sequence for Ss is then encoded into a single representation RS .2 Finally, by referencing the embeddings in Vt, RS and a representation of what has been generated thus far, the model decodes a sentence in the target language word by word. If at any stage the decoded word does not match the corresponding word in the training target St, the error is recorded. The weights and embeddings in the model, which together parameterise the encoding and decoding process, are updated based on the accumulated error once the sentence decoding is complete. Although neural translation models can differ in low-level architecture [7, 2], the translation objective exerts similar pressure on the embeddings in all cases. The source language embeddings must be such that the model can combine them to form single representations for ordered sequences of multiple words (which in turn must enable the decoding process). The target language embeddings must facilitate the process of decoding these representations into correct target-language sentences. 2 Comparing Mono-lingual and Translation-based Embeddings To learn translation-based embeddings, we trained both the RNN encoder-decoder [RNNenc, 7] and the RNN Search architectures [2] on a 300m word corpus of English-French sentence pairs. We conducted all experiments with the resulting (English) source embeddings from these models. For comparison, we trained a monolingual skipgram model [13] and its Glove variant [15] for the same number of epochs on the English half of the bilingual corpus. We also extracted embeddings from a full-sentence language model [CW, 8] trained for several months on a larger 1bn word corpus. As in previous studies [1, 5, 3], we evaluate embeddings by calculating pairwise (cosine) distances and correlating these distances with (gold-standard) human judgements. Table 1 shows the correlations of different model embeddings with three such gold-standard resources, WordSim-353 [1], MEN [5] and SimLex-999 [10]. Interestingly, translation embeddings perform best on SimLex-999, while the two sets of monolingual embeddings perform better on modelling the MEN and WordSim353. To interpret these results, it should be noted that SimLex-999 evaluation quantifies conceptual similarity (dog wolf ), whereas MEN and WordSim-353 (despite its name) quantify more general relatedness (dog collar) [10]. The results seem to indicate that translation-based embeddings better capture similarity, while monolingual embeddings better capture relatedness. 1 Subsequent variants use different algorithms for selecting the (w, c) from the training corpus [9, 12] Alternatively, subsequences (phrases) of Ss may be encoded at this stage in place of the whole sentence [2].

This paper presents a time-delayed neural network (TDNN) model that has the capability of learning and predicting the dynamic behavior of nonlinear elements that compose a wireless communication system. This model could help speeding up system deployment by reducing modeling time. This paper presents results of effective application of the TDNN model to an amplifier, part of a wireless transmitter.

This report documents the program and the outcomes of Dagstuhl Seminar 17042 “From Characters to Understanding Natural Language (C2NLU): Robust End-to-End Deep Learning for NLP”. The seminar brought together researchers from different fields, including natural language processing, computational linguistics, deep learning and general machine learning. 31 participants from 22 academic and industrial institutions discussed advantages and challenges of using characters, i.e., “raw text”, as input for deep learning models instead of language-specific tokens. Eight talks provided overviews of different topics, approaches and challenges in current natural language processing research. In five working groups, the participants discussed current natural language processing/understanding topics in the context of character-based modeling, namely, morphology, machine translation, representation learning, end-to-end systems and dialogue. In most of the discussions, the need for a more detailed model analysis was pointed out. Especially for character-based input, it is important to analyze what a deep learning model is able to learn about language – about tokens, morphology or syntax in general. For an efficient and effective understanding of language, it might furthermore be beneficial to share representations learned from multiple objectives to enable the models to focus on their specific understanding task instead of needing to learn syntactic regularities of language first. Therefore, benefits and challenges of transfer learning were an important topic of the working groups as well as of the panel discussion and the final plenary discussion. Seminar January 22–27, 2017 – http://www.dagstuhl.de/17042 1998 ACM Subject Classification I.2 Artificial Intelligence, I.2.7 Natural Language Processing

This paper describes the DCU-UVT team’s participation in the Language Identification in Code-Switched Data shared task in the Workshop on Computational Approaches to Code Switching. Wordlevel classification experiments were carried out using a simple dictionary-based method, linear kernel support vector machines (SVMs) with and without contextual clues, and a k-nearest neighbour approach. Based on these experiments, we select our SVM-based system with contextual clues as our final system and present results for the Nepali-English and Spanish-English datasets.

We introduce an encoder-decoder recurrent neural network model called Recurrent Neural Aligner (RNA) that can be used for sequence to sequence mapping tasks. Like connectionist temporal classification (CTC) models, RNA defines a probability distribution over target label sequences including blank labels corresponding to each time step in input. The probability of a label sequence is calculated by marginalizing over all possible blank label positions. Unlike CTC, RNA does not make a conditional independence assumption for label predictions; it uses the predicted label at time t−1 as an additional input to the recurrent model when predicting the label at time t. We apply this model to end-to-end speech recognition. RNA is capable of streaming recognition since the decoder does not employ attention mechanism. The model is trained on transcribed acoustic data to predict graphemes and no external language and pronunciation models are used for decoding. We employ an approximate dynamic programming method to optimize negative log likelihood, and a sampling-based sequence discriminative training technique to fine-tune the model to minimize expected word error rate. We show that the model achieves competitive accuracy without using an external language model nor doing beam search decoding.

State-of-the-art sequence labeling systems traditionally require large amounts of taskspecific knowledge in the form of handcrafted features and data pre-processing. In this paper, we introduce a novel neutral network architecture that benefits from both wordand character-level representations automatically, by using combination of bidirectional LSTM, CNN and CRF. Our system is truly end-to-end, requiring no feature engineering or data preprocessing, thus making it applicable to a wide range of sequence labeling tasks. We evaluate our system on two data sets for two sequence labeling tasks — Penn Treebank WSJ corpus for part-of-speech (POS) tagging and CoNLL 2003 corpus for named entity recognition (NER). We obtain state-of-the-art performance on both datasets — 97.55% accuracy for POS tagging and 91.21% F1 for NER.

We recast syntactic parsing as a language modeling problem and use recent advances in neural network language modeling to achieve a new state of the art for constituency Penn Treebank parsing — 93.8 F1 on section 23, using 2-21 as training, 24 as development, plus tri-training. When trees are converted to Stanford dependencies, UAS and LAS are 95.9% and 94.1%.

This paper describes digital circuit architectures for automatic speech recognition (ASR) and voice activity detection (VAD) with improved accuracy, programmability, and scalability. Our ASR architecture is designed to minimize off-chip memory bandwidth, which is the main driver of system power consumption. A SIMD processor with 32 parallel execution units efficiently evaluates feed-forward deep neural networks (NNs) for ASR, limiting memory usage with a sparse quantized weight matrix format. We argue that VADs should prioritize accuracy over area and power, and introduce a VAD circuit that uses an NN to classify modulation frequency features with 22.3-<inline-formula> <tex-math notation="LaTeX">$\mu \text{W}$ </tex-math></inline-formula> power consumption. The 65-nm test chip is shown to perform a variety of ASR tasks in real time, with vocabularies ranging from 11 words to 145 000 words and full-chip power consumption ranging from 172 <inline-formula> <tex-math notation="LaTeX">$\mu \text{W}$ </tex-math></inline-formula> to 7.78 mW.

We present two approaches that use unlabeled data to improve sequ nce learning with recurrent networks. The first approach is to predict wha t comes next in a sequence, which is a conventional language model in natural language processing. The second approach is to use a sequence autoencoder, which r eads the input sequence into a vector and predicts the input sequence again. T hese two algorithms can be used as a “pretraining” step for a later supervised seq uence learning algorithm. In other words, the parameters obtained from the unsu pervised step can be used as a starting point for other supervised training model s. In our experiments, we find that long short term memory recurrent networks after b eing pretrained with the two approaches are more stable and generalize bette r. With pretraining, we are able to train long short term memory recurrent network s up to a few hundred timesteps, thereby achieving strong performance in ma ny text classification tasks, such as IMDB, DBpedia and 20 Newsgroups.


Question: Neural network-based subword language modeling approach.

This paper explores the challenges associated with long-term memory storage in recurrent neural networks, particularly focusing on the issue of decaying error backflow, which significantly hinders the learning process. To address this, we introduce a new method called Long Short-Term Memory (LSTM), a gradient-based approach designed to efficiently manage information over extended time intervals. LSTM mitigates the problem of insufficient error backflow by implementing a mechanism that allows for constant error flow through specific units, known as constant error carousels. These units are equipped with multiplicative gate units that learn to control access to the constant error flow, enabling the network to remember information for over 1000 discrete-time steps.

LSTM is characterized by its local space and time complexity, with a computational complexity per time step and weight of O.1. Our experimental evaluations, conducted with artificial data featuring local, distributed, real-valued, and noisy pattern representations, highlight the superior performance of LSTM. Compared to other recurrent network algorithms such as real-time recurrent learning, backpropagation through time, recurrent cascade correlation, Elman nets, and neural sequence chunking, LSTM demonstrates a higher success rate and faster learning speed. Moreover, LSTM is capable of solving intricate, artificial long-time-lag tasks that have eluded previous recurrent network algorithms, showcasing its effectiveness in overcoming the limitations of long-term memory storage in neural networks.

We introduce a new neural architecture to learn the conditional probability of an output sequence with elements that are discrete tokens corresponding to positions in an input sequence. Such problems cannot be trivially addressed by existent approaches such as sequence-to-sequence [1] and Neural Turing Machines [2], because the number of target classes in each step of the output depends on the length of the input, which is variable. Problems such as sorting variable sized sequences, and various combinatorial optimization problems belong to this class. Our model solves the problem of variable size output dictionaries using a recently proposed mechanism of neural attention. It differs from the previous attention attempts in that, instead of using attention to blend hidden units of an encoder to a context vector at each decoder step, it uses attention as a pointer to select a member of the input sequence as the output. We call this architecture a Pointer Net (Ptr-Net). We show Ptr-Nets can be used to learn approximate solutions to three challenging geometric problems – finding planar convex hulls, computing Delaunay triangulations, and the planar Travelling Salesman Problem – using training examples alone. Ptr-Nets not only improve over sequence-to-sequence with input attention, but also allow us to generalize to variable size output dictionaries. We show that the learnt models generalize beyond the maximum lengths they were trained on. We hope our results on these tasks will encourage a broader exploration of neural learning for discrete problems.

Neural net advances improve computers' language ability in many fields.

As a fundamental task in the natural language processing field, language modeling aims to estimate the distribution of the word sequences. However, the most existing algorithms have focused on the main texts while often ignoring the vastly-accessible semantic attributes of the documents, e.g., titles, authors, sentiments and tags. To address this issue, we propose Semantic Attribute Modulated (SAM) language modeling, a novel language modeling framework that incorporates the various semantic attributes. Attributes are selected automatically with an attribute attention mechanism. We build three text datasets with a diversity of semantic attributes. On the three text datasets, we empirically examine the language model perplexities of several typical corpora, and then demonstrate the superiority of our model with the different combinations of the attributes. Extensive qualitative results, including word semantic analysis, attention values and an interesting lyric generation, further demonstrate the effectiveness of our SAMmethod.

This paper makes three significant extensions to a noisy channel speller designed for standard written text to target the challenging domain of search queries. First, the noisy channel model is subsumed by a more general ranker, which allows a variety of features to be easily incorporated. Second, a distributed infrastructure is proposed for training and applying Web scale n-gram language models. Third, a new phrase-based error model is presented. This model places a probability distribution over transformations between multi-word phrases, and is estimated using large amounts of query-correction pairs derived from search logs. Experiments show that each of these extensions leads to significant improvements over the stateof-the-art baseline methods.

Neural machine translation (NMT) conducts end-to-end translation with a source language encoder and a target language decoder, making promising translation performance. However, as a newly emerged approach, the method has some limitations. An NMT system usually has to apply a vocabulary of certain size to avoid the time-consuming training and decoding, thus it causes a serious out-of-vocabulary problem. Furthermore, the decoder lacks a mechanism to guarantee all the source words to be translated and usually favors short translations, resulting in fluent but inadequate translations. In order to solve the above problems, we incorporate statistical machine translation (SMT) features, such as a translation model and an n-gram language model, with the NMT model under the log-linear framework. Our experiments show that the proposed method significantly improves the translation quality of the state-of-the-art NMT system on Chinese-toEnglish translation tasks. Our method produces a gain of up to 2.33 BLEU score on NIST open test sets.

We present KenLM, a library that implements two data structures for efficient language model queries, reducing both time and memory costs. The PROBING data structure uses linear probing hash tables and is designed for speed. Compared with the widelyused SRILM, our PROBING model is 2.4 times as fast while using 57% of the memory. The TRIE data structure is a trie with bit-level packing, sorted records, interpolation search, and optional quantization aimed at lower memory consumption. TRIE simultaneously uses less memory than the smallest lossless baseline and less CPU than the fastest baseline. Our code is open-source1, thread-safe, and integrated into the Moses, cdec, and Joshua translation systems. This paper describes the several performance techniques used and presents benchmarks against alternative implementations.

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 Englishto-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.2 after training for 4.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

We introduce an autoregressive attention mechanism for parallelizable characterlevel sequence modeling. We use this method to augment a neural model consisting of blocks of causal convolutional layers connected by highway network skip connections. We denote the models with and without the proposed attention mechanism respectively as Highway Causal Convolution (Causal Conv) and Autoregressive-attention Causal Convolution (ARA-Conv). The autoregressive attention mechanism crucially maintains causality in the decoder, allowing for parallel implementation. We demonstrate that these models, compared to their recurrent counterparts, enable fast and accurate learning in character-level NLP tasks. In particular, these models outperform recurrent neural network models in natural language correction and language modeling tasks, and run in a fraction of the time.

Because of their superior ability to preserve sequence information over time, Long Short-Term Memory (LSTM) networks, a type of recurrent neural network with a more complex computational unit, have obtained strong results on a variety of sequence modeling tasks. The only underlying LSTM structure that has been explored so far is a linear chain. However, natural language exhibits syntactic properties that would naturally combine words to phrases. We introduce the Tree-LSTM, a generalization of LSTMs to tree-structured network topologies. TreeLSTMs outperform all existing systems and strong LSTM baselines on two tasks: predicting the semantic relatedness of two sentences (SemEval 2014, Task 1) and sentiment classification (Stanford Sentiment Treebank).

Recently, distributed word embeddings trained by neural language models are commonly used for text classification with Convolutional Neural Networks (CNNs). In this paper, we propose a novel neural language model, Topic-based Skip-gram, to learn topic-based word embeddings for biomedical literature indexing with CNNs. Topic-based Skip-gram leverages textual content with topic models, e.g., Latent Dirichlet Allocation (LDA), to capture precise topic-based word relationship and then integrate it into distributed word embedding learning. We then describe two multimodal CNN architectures, which are able to employ different kinds of word embeddings at the same time for text classification. Through extensive experiments conducted on several real-world datasets, we demonstrate that combination of our Topic-based Skip-gram and multimodal CNN architectures outperforms state-of-the-art methods in biomedical literature indexing, clinical note annotation and general textual benchmark dataset classification.


Question: NEURAL NETWORK-BASED SUBWORD MODELING FOR LANGUAGE APPLICATIONS

This paper explores and critically assesses a technique for the automatic grouping of English words based on their distribution within specific syntactic environments. The method employs deterministic annealing to identify the sets of clusters that result in the least distortion. As the annealing parameter rises, previously stable clusters become less so, leading to their division and the creation of a hierarchical, "soft" classification of the dataset. These clusters serve as the foundation for class models of word co-occurrence. The efficacy of these models is evaluated by testing them against data that has been withheld from the training process. This approach not only enhances the understanding of linguistic patterns but also provides a more nuanced model for predicting word co-occurrence, which is crucial for tasks like language modeling and natural language processing.

To improve speech recognition performance, a combination between TANDEM and bottleneck Deep Neural Networks (DNN) is investigated. In particular, exploiting a feature combination performed by means of a multi-stream hierarchical processing, we show a performance improvement by combining the same input features processed by different neural networks. The experiments are based on the spontaneous telephone recordings of the Cantonese IARPA Babel corpus using both standard MFCCs and Gabor as input features.

This paper addresses the problems of word spotting and word recognition on images. In word spotting, the goal is to find all instances of a query word in a dataset of images. In recognition, the goal is to recognize the content of the word image, usually aided by a dictionary or lexicon. We describe an approach in which both word images and text strings are embedded in a common vectorial subspace. This is achieved by a combination of label embedding and attributes learning, and a common subspace regression. In this subspace, images and strings that represent the same word are close together, allowing one to cast recognition and retrieval tasks as a nearest neighbor problem. Contrary to most other existing methods, our representation has a fixed length, is low dimensional, and is very fast to compute and, especially, to compare. We test our approach on four public datasets of both handwritten documents and natural images showing results comparable or better than the state-of-the-art on spotting and recognition tasks.

Accurate, real-time Automatic Speech Recognition (ASR) requires huge memory storage and computational power. The main bottleneck in state-of-the-art ASR systems is the Viterbi search on a Weighted Finite State Transducer (WFST). The WFST is a graph-based model created by composing an Acoustic Model (AM) and a Language Model (LM) offline. Offline composition simplifies the implementation of a speech recognizer as only one WFST has to be searched. However, the size of the composed WFST is huge, typically larger than a Gigabyte, resulting in a large memory footprint and memory bandwidth requirements.
 In this paper, we take a completely different approach and propose a hardware accelerator for speech recognition that composes the AM and LM graphs on-the-fly. In our ASR system, the fully-composed WFST is never generated in main memory. On the contrary, only the subset required for decoding each input speech fragment is dynamically generated from the AM and LM models. In addition to the direct benefits of this on-the-fly composition, the resulting approach is more amenable to further reduction in storage requirements through compression techniques.
 The resulting accelerator, called UNFOLD, performs the decoding in real-time using the compressed AM and LM models, and reduces the size of the datasets from more than one Gigabyte to less than 40 Megabytes, which can be very important in small form factor mobile and wearable devices.
 Besides, UNFOLD improves energy-efficiency by orders of magnitude with respect to CPUs and GPUs. Compared to a state-of-the-art Viterbi search accelerators, the proposed ASR system outperforms by providing 31x reduction in memory footprint and 28% energy savings on average.

In this paper, we apply a bidirectional Long Short-Term Memory with a Conditional Random Field to the task of disfluency detection. Long-range dependencies is one of the core problems for disfluency detection. Our model handles long-range dependencies by both using the Long Short-Term Memory and hand-crafted discrete features. Experiments show that utilizing the hand-crafted discrete features significantly improves the model’s performance by achieving the state-of-the-art score of 87.1% on the Switchboard corpus.

We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including: part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements.

How should one apply deep learning to tasks such as morphological reinflection, which stochastically edit one string to get another? A recent approach to such sequence-to-sequence tasks is to compress the input string into a vector that is then used to generate the output string, using recurrent neural networks. In contrast, we propose to keep the traditional architecture, which uses a finite-state transducer to score all possible output strings, but to augment the scoring function with the help of recurrent networks. A stack of bidirectional LSTMs reads the input string from leftto-right and right-to-left, in order to summarize the input context in which a transducer arc is applied. We combine these learned features with the transducer to define a probability distribution over aligned output strings, in the form of a weighted finite-state automaton. This reduces hand-engineering of features, allows learned features to examine unbounded context in the input string, and still permits exact inference through dynamic programming. We illustrate our method on the tasks of morphological reinflection and lemmatization.

This paper introduces NEMWEL, a system that performs Never-Ending MultiWord Expressions Learning. Instead of using a static corpus and classifier, NEMWEL applies supervised learning on automatically crawled news texts. Moreover, it uses its own results to periodically retrain the classifier, bootstrapping on its own results. In addition to a detailed description of the system’s architecture and its modules, we report the results of a manual evaluation. It shows that NEMWEL is capable of learning new expressions over time with improved precision.

Large Vocabulary Continuous Speech Recognition (LVCSR), which is characterized by a high variability of the speech, is the most challenging task in automatic speech recognition (ASR). Believing that the evaluation of ASR systems on relevant and common speech corpora is one of the key factors that help accelerating research, we present, in this paper, a benchmark comparison of the performances of the current state-of-the-art LVCSR systems over different speech recognition tasks. Furthermore, we put objectively into evidence the best performing technologies and the best accuracy achieved so far in each task. The benchmarks have shown that the Deep Neural Networks and Convolutional Neural Networks have proven their efficiency on several LVCSR tasks by outperforming the traditional Hidden Markov Models and Guaussian Mixture Models. They have also shown that despite the satisfying performances in some LVCSR tasks, the problem of large-vocabulary speech recognition is far from being solved in some others, where more research efforts are still needed.

Semantic matching is of central importance to many natural language tasks [2, 28]. A successful matching algorithm needs to adequately model the internal structures of language objects and the interaction between them. As a step toward this goal, we propose convolutional neural network models for matching two sentences, by adapting the convolutional strategy in vision and speech. The proposed models not only nicely represent the hierarchical structures of sentences with their layerby-layer composition and pooling, but also capture the rich matching patterns at different levels. Our models are rather generic, requiring no prior knowledge on language, and can hence be applied to matching tasks of different nature and in different languages. The empirical study on a variety of matching tasks demonstrates the efficacy of the proposed model on a variety of matching tasks and its superiority to competitor models.

Deep Bidirectional LSTM (DBLSTM) recurrent neural networks have recently been shown to give state-of-the-art performance on the TIMIT speech database. However, the results in that work relied on recurrent-neural-network-specific objective functions, which are difficult to integrate with existing large vocabulary speech recognition systems. This paper investigates the use of DBLSTM as an acoustic model in a standard neural network-HMM hybrid system. We find that a DBLSTM-HMM hybrid gives equally good results on TIMIT as the previous work. It also outperforms both GMM and deep network benchmarks on a subset of the Wall Street Journal corpus. However the improvement in word error rate over the deep network is modest, despite a great increase in framelevel accuracy. We conclude that the hybrid approach with DBLSTM appears to be well suited for tasks where acoustic modelling predominates. Further investigation needs to be conducted to understand how to better leverage the improvements in frame-level accuracy towards better word error rates.


Question: An overview of resource management strategies in Internet of Things operating systems.

This paper delves into the critical role of intelligent device-to-device (D2D) communication within the Internet of Things (IoT) framework, paralleling the human interaction with the Internet. In the IoT landscape, devices are envisioned as the primary actors, engaging in autonomous communication devoid of centralized governance. This peer-to-peer interaction facilitates the aggregation, dissemination, and forwarding of data across multiple hops, essential for the IoT's operational efficacy. Central to this dynamic is the real-time acquisition of pertinent information, which, when processed, transforms into actionable intelligence, fostering a smarter environment. The sophistication of devices directly influences the quality of information gathered, setting the stage for a networked ecosystem where devices adhere to diverse standards, encounter sporadic connectivity, and often grapple with resource limitations. These distinct traits pose novel networking hurdles that conventional routing protocols are ill-equipped to address. As a result, there is a pressing need for devices to adopt intelligent routing protocols to ensure effective D2D communication. We provide a comprehensive review of strategies to achieve intelligent D2D communication within the IoT context, with a specific emphasis on contemporary routing algorithms' potential to optimize D2D interactions.

ed as devices to provide services at low-levels as network discovery services, metadata exchange services, and asynchronous publish and subscribe event in (Kranenburg et al. 2011; Sundmaeker et al. 2010); In (Peris-Lopez et al. 2006), a representational state transfer (REST) is defined to increase interoperability between Page 14 of 32 http://mc.manuscriptcentral.com/intr Internet Research 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 or Peer Rview loosely coupled services and distributed applications. In (Hernandez-Castro et al. 2013), the services layer introduced a service provisioning process (SPP) that can provide the interaction between applications and services. It is important to design an effective security strategy to protect services against attacks in the service layer. The security requirements in the service layer include: • Authorization, service authentication, group authentication, privacy protection, integrity, integrity, security of keys, non-repudiation, anti-replay, availability,

In the emerging Internet of Things rich data on real-world objects and events will be generated in vast amounts and stored in widely distributed databases. In truly global and dynamic application scenarios, intermediate brokers are needed to find these data, even if the exact location and form of storage are initially unknown to the requester. Discovery Services are aimed to fill this gap: they respond to requests for data on specific objects with a list of corresponding data providers. In this paper, we frame functional requirements for Discovery Services, and perform an overview and analysis of five established approaches for implementing Discovery Services that are taken from literature and industrial practice. In order to compare their characteristics, we develop a quality framework based on literature review and an ISO standard for software quality.

As cities continue to grow, numerous initiatives for Smart Cities are being conducted. The concept of Smart City encompasses several concepts being governance, economy, management, infrastructure, technology and people. This means that a Smart City can have different communication needs. Wireless technologies such as WiFi, ZigBee, Bluetooth, WiMax, 4G or LTE (Long Term Evolution) have presented themselves as solutions to the communication needs of Smart City initiatives. However, as most of them employ unlicensed bands, interference and coexistence problems are increasing. In this paper, the wireless technologies available nowadays for IoT (Internet of Things) in Smart Cities are presented. Our contribution is a review of wireless technologies, their comparison and the problems that difficult coexistence among them. In order to do so, the characteristics and adequacy of wireless technologies to each domain are considered. The problems derived of over-crowded unlicensed spectrum and coexistence difficulties among each technology are discussed as well. Finally, power consumption concerns are addressed.

The functional scope of today's software-defined data centers (SDDC) has expanded to such an extent that servers face a growing amount of critical background operational tasks like load monitoring, logging, migration, and duplication, etc. These ancillary operations, which we refer to as management operations, often nibble the stringent data center power envelope and exert a tremendous amount of pressure on front-end user tasks. However, existing power capping, peak shaving, and time shifting mechanisms mainly focus on managing data center power demand at the "macro level" -- they do not distinguish ancillary background services from user tasks, and therefore often incur significant performance degradation and energy overhead.
 In this study we explore "micro-level" power management in SDDC: tuning a specific set of critical loads for the sake of overall system efficiency and performance. Specifically, we look at management operations that can often lead to resource contention and energy overhead in an IaaS SDDC. We assess the feasibility of this new power management paradigm by characterizing the resource and power impact of various management operations. We propose HOPE, a new system optimization framework for eliminating the potential efficiency bottleneck caused by the management operations in the SDDC. HOPE is implemented on a customized OpenStack cloud environment with heavily instrumented power infrastructure. We thoroughly validate HOPE models and optimization efficacy under various user workload scenarios. Our deployment experiences show that the proposed technique allows SDDC to reduce energy consumption by 19%, reduce management operation execution time by 25.4%, and in the meantime improve workload performance by 30%.

The Internet of Things has increased the number of connected devices by an order of magnitude. This increase comes with a corresponding increase in complexity, which has led to centralized architectures that do not respect the privacy of the users and incur high development and deployment costs. We propose the Open Thing Platform as a new distributed architecture which is built around ThingTalk, a new high level declarative programming language that can express the interaction between multiple physical devices and online services. The architecture abstracts away the details of the communication without sacrificing privacy. We experiment with ThingTalk in multiple domains and we show that several applications can be written in a few lines of code and with no need for any server infrastructure.

Wireless sensor networks (WSNs) are becoming increasingly popular with the advent of the Internet of things (IoT). Various real-world applications of WSNs such as in smart grids, smart farming and smart health would require a potential deployment of thousands or maybe hundreds of thousands of sensor nodes/actuators. To ensure proper working order and network efficiency of such a network of sensor nodes, an effective WSN management system has to be integrated. However, the inherent challenges of WSNs such as sensor/actuator heterogeneity, application dependency and resource constraints have led to challenges in implementing effective traditional WSN management. This difficulty in management increases as the WSN becomes larger. Software Defined Networking (SDN) provides a promising solution in flexible management WSNs by allowing the separation of the control logic from the sensor nodes/actuators. The advantage with this SDN-based management in WSNs is that it enables centralized control of the entire WSN making it simpler to deploy network-wide management protocols and applications on demand. This paper highlights some of the recent work on traditional WSN management in brief and reviews SDN-based management techniques for WSNs in greater detail while drawing attention to the advantages that SDN brings to traditional WSN management. This paper also investigates open research challenges in coming up with mechanisms for flexible and easier SDN-based WSN configuration and management.

Internet of Things is an up-and-coming information and technology industry, and in such an environment of the concept and the content and the extension of Internet of Things are not very distinctive, the project of Internet of Things which is a piece of small area with small-scale and self-system obtain gratifying achievement and bright future, it can promote the development of Internet of Things in some extent. But there are some serious hidden danger and potential crisis problems. The paper focuses on the application of Internet of Things in the nation and even in the global in the future, analysing the existed security risks of the Internet of Things 's network points, transmission, finally we propose some suggestive solutions due to these problems.

With the increase of energy consumption associated with IT infrastructures, energy management is becoming a priority in the design and operation of complex service-based systems. At the same time, service providers need to comply with Service Level Agreement (SLA) contracts which determine the revenues and penalties on the basis of the achieved performance level. This paper focuses on the resource allocation problem in multitier virtualized systems with the goal of maximizing the SLAs revenue while minimizing energy costs. The main novelty of our approach is to address-in a unifying framework-service centers resource management by exploiting as actuation mechanisms allocation of virtual machines (VMs) to servers, load balancing, capacity allocation, server power state tuning, and dynamic voltage/frequency scaling. Resource management is modeled as an NP-hard mixed integer nonlinear programming problem, and solved by a local search procedure. To validate its effectiveness, the proposed model is compared to top-performing state-of-the-art techniques. The evaluation is based on simulation and on real experiments performed in a prototype environment. Synthetic as well as realistic workloads and a number of different scenarios of interest are considered. Results show that we are able to yield significant revenue gains for the provider when compared to alternative methods (up to 45 percent). Moreover, solutions are robust to service time and workload variations.

The rapid development of Internet of things (IoT) technology makes it possible for connecting various smart objects together through the Internet and providing more data interoperability methods for application purpose. Recent research shows more potential applications of IoT in information intensive industrial sectors such as healthcare services. However, the diversity of the objects in IoT causes the heterogeneity problem of the data format in IoT platform. Meanwhile, the use of IoT technology in applications has spurred the increase of real-time data, which makes the information storage and accessing more difficult and challenging. In this research, first a semantic data model is proposed to store and interpret IoT data. Then a resource-based data accessing method (UDA-IoT) is designed to acquire and process IoT data ubiquitously to improve the accessibility to IoT data resources. Finally, we present an IoT-based system for emergency medical services to demonstrate how to collect, integrate, and interoperate IoT data flexibly in order to provide support to emergency medical services. The result shows that the resource-based IoT data accessing method is effective in a distributed heterogeneous data environment for supporting data accessing timely and ubiquitously in a cloud and mobile computing platform.

The emergence of IoT (Internet of Things) has brought various studies on low-power techniques back to embedded systems. In general, minimizing power consumed by program executions is the main consideration of system design. While running programs, executing data-dependent program code results in a large number of memory accesses which consume huge amount of power. Further, most embedded systems consist of multiple types of memory devices, i.e., heterogeneous memory system, to benefit the different characteristics of memory devices. In this paper, we conduct a research on low-power techniques to reduce the power consumption of heterogeneous memory to achieve ultra-low-power in the system level. This study proposes a segment-aware energy-efficient management to improve the power efficiency considering the characteristics and structures of the memory devices. In the proposed approach, the technique migrates program code from allocated memory device to another if the consumed power is considered to be less. We also analyze and evaluate the comprehensive effects on energy efficiency by applying the technique as well. Compared to the unmodified program code, our model reduces power consumption up to 12.98% by migrating functions.