Question: Advancing 3D face recognition in real-world scenarios: A registration-free method employing fine-grained matching of 3D keypoint descriptors.

In this innovative work, we have engineered a real-time computational framework capable of detecting, tracking, and identifying individuals through facial recognition, achieving a level of accuracy and efficiency suitable for practical applications. Inspired by physiological principles and guided by the demands of real-time performance, our method eschews the complexity of three-dimensional geometry reconstruction. Instead, it leverages the inherent upright orientation of faces, thereby focusing on a limited set of characteristic two-dimensional (2-D) views. The system's core functionality hinges on the projection of facial images into a feature space that encapsulates the salient variations among known faces. This space is defined by "eigenfaces," which are the eigenvectors—or principal components—of the face image set. These eigenfaces do not necessarily align with traditional facial features like eyes or noses. By representing a face as a weighted sum of eigenface components, our system can swiftly and accurately identify individuals by comparing these weights to those of known faces. Notably, our approach enables the unsupervised learning and recognition of new faces, and its implementation is streamlined through a neural network architecture, highlighting its adaptability and ease of use.

We have developed a near-real-time computer system that can locate and track a subject's head, and then recognize the person by comparing characteristics of the face to those of known individuals. The computational approach taken in this system is motivated by both physiology and information theory, as well as by the practical requirements of near-real-time performance and accuracy. Our approach treats the face recognition problem as an intrinsically two-dimensional (2-D) recognition problem rather than requiring recovery of three-dimensional geometry, taking advantage of the fact that faces are normally upright and thus may be described by a small set of 2-D characteristic views. The system functions by projecting face images onto a feature space that spans the significant variations among known face images. The significant features are known as "eigenfaces," because they are the eigenvectors (principal components) of the set of faces; they do not necessarily correspond to features such as eyes, ears, and noses. The projection operation characterizes an individual face by a weighted sum of the eigenface features, and so to recognize a particular face it is necessary only to compare these weights to those of known individuals. Some particular advantages of our approach are that it provides for the ability to learn and later recognize new faces in an unsupervised manner, and that it is easy to implement using a neural network architecture.

The state-of-the-art of face recognition has been significantly advanced by the emergence of deep learning. Very deep neural networks recently achieved great success on general object recognition because of their superb learning capacity. This motivates us to investigate their effectiveness on face recognition. This paper proposes two very deep neural network architectures, referred to as DeepID3, for face recognition. These two architectures are rebuilt from stacked convolution and inception layers proposed in VGG net [10] and GoogLeNet [16] to make them suitable to face recognition. Joint face identification-verification supervisory signals are added to both intermediate and final feature extraction layers during training. An ensemble of the proposed two architectures achieves 99.53% LFW face verification accuracy and 96.0% LFW rank-1 face identification accuracy, respectively. A further discussion of LFW face verification result is given in the end.

Traditionally, the indoor scene classification problem has been approached from a 2D image recognition point of view. In most visual scene classification systems, a descriptor for the input image is generated to obtain a suitable representation that includes features related to color, shape or spatial information. Techniques based on the use of a spatial pyramid have proven to be adequate to perform this step. In the past years, on the other hand, 3D sensors have become widely available, which allows to include new information sources to the framework previously described. In this work we rely on RGB-D data to extend the spatial pyramid approach, aimed at building descriptors that can lead to a more robust representation against changing lighting conditions. The proposed descriptors are evaluated on the RobotVision@ImageCLEF-2013 benchmark dataset, remarkably outperforming state-of-the-art 3D local and global descriptors.

In this paper, we describe the acquisition and contents of a large-scale Chinese face database: the CAS-PEAL face database. The goals of creating the CAS-PEAL face database include the following: 1) providing the worldwide researchers of face recognition with different sources of variations, particularly pose, expression, accessories, and lighting (PEAL), and exhaustive ground-truth information in one uniform database; 2) advancing the state-of-the-art face recognition technologies aiming at practical applications by using off-the-shelf imaging equipment and by designing normal face variations in the database; and 3) providing a large-scale face database of Mongolian. Currently, the CAS-PEAL face database contains 99 594 images of 1040 individuals (595 males and 445 females). A total of nine cameras are mounted horizontally on an arc arm to simultaneously capture images across different poses. Each subject is asked to look straight ahead, up, and down to obtain 27 images in three shots. Five facial expressions, six accessories, and 15 lighting changes are also included in the database. A selected subset of the database (CAS-PEAL-R1, containing 30 863 images of the 1040 subjects) is available to other researchers now. We discuss the evaluation protocol based on the CAS-PEAL-R1 database and present the performance of four algorithms as a baseline to do the following: 1) elementarily assess the difficulty of the database for face recognition algorithms; 2) preference evaluation results for researchers using the database; and 3) identify the strengths and weaknesses of the commonly used algorithms.

Efficient keypoint-based object detection methods are used in many real-time computer vision applications. These approaches often model an object as a collection of keypoints and associated descriptors, and detection then involves first constructing a set of correspondences between object and image keypoints via descriptor matching, and subsequently using these correspondences as input to a robust geometric estimation algorithm such as RANSAC to find the transformation of the object in the image. In such approaches, the object model is generally constructed offline, and does not adapt to a given environment at runtime. Furthermore, the feature matching and transformation estimation stages are treated entirely separately. In this paper, we introduce a new approach to address these problems by combining the overall pipeline of correspondence generation and transformation estimation into a single structured output learning framework. Following the recent trend of using efficient binary descriptors for feature matching, we also introduce an approach to approximate the learned object model as a collection of binary basis functions which can be evaluated very efficiently at runtime. Experiments on challenging video sequences show that our algorithm significantly improves over state-of-the-art descriptor matching techniques using a range of descriptors, as well as recent online learning based approaches.

Establishing correspondences between 3D geometries is essential to a large variety of graphics and vision applications, including 3D reconstruction, localization, and shape matching. Despite significant progress, geometric matching on real-world 3D data is still a challenging task due to the noisy, low-resolution, and incomplete nature of scanning data. These difficulties limit the performance of current state-of-art methods which are typically based on histograms over geometric properties. In this paper, we introduce 3DMatch1, a data-driven local feature learner that jointly learns a geometric feature representation and an associated metric function from a large collection of realworld scanning data. We represent 3D geometry using accumulated distance fields around key-point locations. This representation is suited to handle noisy and partial scanning data, and concurrently supports deep learning with convolutional neural networks directly in 3D. To train the networks, we propose a way to automatically generate correspondence labels for deep learning by leveraging existing RGB-D reconstruction algorithms. In our results, we demonstrate that we are able to outperform state-of-theart approaches by a significant margin. In addition, we show the robustness of our descriptor in a purely geometric sparse bundle adjustment pipeline for 3D reconstruction. 1All of our code, datasets, and trained models are publicly available: http://3dmatch.cs.princeton.edu/

As the training of Adaboost is complicated or does not work well in the multi-view face detection with large plane-rotated angle, this paper proposes a rotation invariant multi-view color face detection method combining skin color segmentation and multi-view Adaboost algorithm. First the possible face region is fast detected by skin color table, and the skin-color background adhesion of face is separated by color clustering. After region merging, the candidate face region is received. Then the face direction in plane is calculated by K-L transform, and finally the candidate face region corrected by rotating is scanned by multi-view Adaboost classifier to locate the face accurately. The experiments show that the method can effectively detect the plane large-angle multi-view face image which the conventional Adaboost can not do. It can be effectively applied to the cases of multi-view and multi-face image with complex background.

Automatic face recognition technologies have seen significant improvements in performance due to a combination of advances in deep learning and availability of larger datasets for training deep networks. Since recognizing faces is a task that humans are believed to be very good at, it is only natural to compare the relative performance of automated face recognition and humans when processing fully unconstrained facial imagery. In this work, we expand on previous studies of the recognition accuracy of humans and automated systems by performing several novel analyses utilizing unconstrained face imagery. We examine the impact on performance when human recognizers are presented with varying amounts of imagery per subject, immutable attributes such as gender, and circumstantial attributes such as occlusion, illumination, and pose. Results indicate that humans greatly outperform state of the art automated face recognition algorithms on the challenging IJB-A dataset.

This paper presents a literature survey of automatic 3D surface registration techniques emphasizing the mathematical and algorithmic underpinnings of the subject. The relevance of surface registration to medical imaging is that there is much useful anatomical information in the form of collected surface points which originate from complimentary modalities and which must be reconciled. Surface registration can be roughly partitioned into three issues: choice of transformation, elaboration of surface representation and similarity criterion, and matching and global optimization. The first issue concerns the assumptions made about the nature of relationships between the two modalities, e.g. whether a rigid-body assumption applies, and if not, what type and how general a relation optimally maps one modality onto the other. The second issue determines what type of information we extract from the 3D surfaces, which typically characterizes their local or global shape, and how we organize this information into a representation of the surface which will lead to improved efficiency and robustness in the last stage. The last issue pertains to how we exploit this information to estimate the transformation which best aligns local primitives in a globally consistent manner or which maximizes a measure of the similarity in global shape of two surfaces. Within this framework, this paper discusses in detail each surface registration issue and reviews the state-of-the-art among existing techniques.

In this paper, we address the problem of extreme head pose estimation from intensity images, in a monocular setup. We introduce a novel fusion pipeline to integrate into a dedicated Kalman Filter the pose estimated from a tracking scheme in the prediction stage and the pose estimated from a detection scheme in the correction stage. To that end, the measurement covariance of the Kalman Filter is updated in every frame. The tracking scheme is performed using a set of keypoints extracted in the area of the head along with a simple 3D geometric model. The detection scheme, on the other hand, relies on the alignment of facial landmarks in each frame combined with 3D features extracted on a head mesh. The head pose in each scheme is estimated by minimizing the reprojection error from the 3D-2D correspondences. By combining both frameworks, we extend the applicability of head pose estimation from facial landmarks to cases where these features are no longer visible. We compared the proposed method to other related approaches, showing that it can achieve state-of-the-art performance. We also demonstrate that our approach is suitable for cases with extreme head rotations and (self-) occlusions, besides being suitable for real time applications.


Question: Deep recurrent models incorporating fast-forward connections for neural machine translation tasks.

This contribution presents extensive 5-gram counts and language models derived from the Common Crawl corpus, a vast repository comprising over 9 billion web pages. Our dataset significantly advances upon the Google n-gram counts by addressing two critical limitations: the incorporation of low-frequency entries and the elimination of duplicates to minimize boilerplate content. By retaining singleton occurrences, we successfully applied Kneser-Ney smoothing to construct robust, high-capacity language models. This paper elaborates on the processing techniques employed on this massive dataset, highlighting the challenges encountered when dealing with such scale. The resulting unpruned Kneser-Ney English 5-gram language model, constructed from 975 billion deduplicated tokens, encompasses more than 500 billion distinct n-grams. We demonstrate improvements of 0.5 to 1.4 BLEU points by leveraging these extensive language models in translating across various languages, showcasing their efficacy in enhancing translation quality.

Deep learning yields great results across many fields, from speech recognition, image classification, to translation. But for each problem, getting a deep model to work well involves research into the architecture and a long period of tuning. We present a single model that yields good results on a number of problems spanning multiple domains. In particular, this single model is trained concurrently on ImageNet, multiple translation tasks, image captioning (COCO dataset), a speech recognition corpus, and an English parsing task. Our model architecture incorporates building blocks from multiple domains. It contains convolutional layers, an attention mechanism, and sparsely-gated layers. Each of these computational blocks is crucial for a subset of the tasks we train on. Interestingly, even if a block is not crucial for a task, we observe that adding it never hurts performance and in most cases improves it on all tasks. We also show that tasks with less data benefit largely from joint training with other tasks, while performance on large tasks degrades only slightly if at all.

Semantic slot filling is one of the most challenging problems in spoken language understanding (SLU). In this paper, we propose to use recurrent neural networks (RNNs) for this task, and present several novel architectures designed to efficiently model past and future temporal dependencies. Specifically, we implemented and compared several important RNN architectures, including Elman, Jordan, and hybrid variants. To facilitate reproducibility, we implemented these networks with the publicly available Theano neural network toolkit and completed experiments on the well-known airline travel information system (ATIS) benchmark. In addition, we compared the approaches on two custom SLU data sets from the entertainment and movies domains. Our results show that the RNN-based models outperform the conditional random field (CRF) baseline by 2% in absolute error reduction on the ATIS benchmark. We improve the state-of-the-art by 0.5% in the Entertainment domain, and 6.7% for the movies domain.

In this paper a doubly attentive transformer machine translation model (DATNMT) is presented in which a doubly-attentive transformer decoder normally joins spatial visual features obtained via pretrained convolutional neural networks, conquering any gap between image captioning and translation. In this framework, the transformer decoder figures out how to take care of source-language words and parts of an image freely by methods for two separate attention components in an Enhanced Multi-Head Attention Layer of doubly attentive transformer, as it generates words in the target language. We find that the proposed model can effectively exploit not just the scarce multimodal machine translation data, but also large general-domain textonly machine translation corpora, or imagetext image captioning corpora. The experimental results show that the proposed doublyattentive transformer-decoder performs better than a single-decoder transformer model, and gives the state-of-the-art results in the EnglishGerman multimodal machine translation task.

Multivariate time series forecasting is extensively studied throughout the years with ubiquitous applications in areas such as finance, traffic, environment, etc. Still, concerns have been raised on traditional methods for incapable of modeling complex patterns or dependencies lying in real word data. To address such concerns, various deep learning models, mainly Recurrent Neural Network (RNN) based methods, are proposed. Nevertheless, capturing extremely long-term patterns while effectively incorporating information from other variables remains a challenge for time-series forecasting. Furthermore, lack-of-explainability remains one serious drawback for deep neural network models. Inspired by Memory Network proposed for solving the question-answering task, we propose a deep learning based model named Memory Timeseries network (MTNet) for time series forecasting. MTNet consists of a large memory component, three separate encoders, and an autoregressive component to train jointly. Addtionally, the attention mechanism designed enable MTNet to be highly interpretable. We can easily tell which part of the historic data is referenced the most.

We propose to exploit the potential of multiple word clusterings in class-based recurrent neural network (RNN) language models for ensemble RNN language modeling. By varying the clustering criteria and the space of word embedding, different word clusterings are obtained to define different word/class factorizations. For each such word/class factorization, several base RNNLMs are learned, and the word prediction probabilities of the base RNNLMs are then combined to form an ensemble prediction. We use a greedy backward model selection procedure to select a subset of models and combine these models for word prediction. The proposed ensemble language modeling method has been evaluated on Penn Treebank test set as well as Wall Street Journal (WSJ) Eval 92 and 93 test sets, where it improved test set perplexity and word error rate over the state-of-the-art single RNNLMs as well as multiple RNNLMs produced by varying RNN learning conditions.

Given that in practice training data is scarce for all but a small set of problems, a core question is how to incorporate prior knowledge into a model. In this paper, we consider the case of prior procedural knowledge for neural networks, such as knowing how a program should traverse a sequence, but not what local actions should be performed at each step. To this end, we present an end-to-end differentiable interpreter for the programming language Forth which enables programmers to write program sketches with slots that can be filled with behaviour trained from program input-output data. We can optimise this behaviour directly through gradient descent techniques on user-specified objectives, and also integrate the program into any larger neural computation graph. We show empirically that our interpreter is able to effectively leverage different levels of prior program structure and learn complex behaviours such as sequence sorting and addition. When connected to outputs of an LSTM and trained jointly, our interpreter achieves state-of-the-art accuracy for end-to-end reasoning about quantities expressed in natural language stories.

Researchers have recently started investigating deep neural networks for dialogue applications. In particular, generative sequence-to-sequence (Seq2Seq) models have shown promising results for unstructured tasks, such as word-level dialogue response generation. The hope is that such models will be able to leverage massive amounts of data to learn meaningful natural language representations and response generation strategies, while requiring a minimum amount of domain knowledge and hand-crafting. An important challenge is to develop models that can effectively incorporate dialogue context and generate meaningful and diverse responses. In support of this goal, we review recently proposed models based on generative encoder-decoder neural network architectures, and show that these models have better ability to incorporate long-term dialogue history, to model uncertainty and ambiguity in dialogue, and to generate responses with high-level compositional structure.

We present SummaRuNNer, a Recurrent Neural Network (RNN) based sequence model for extractive summarization of documents and show that it achieves performance better than or comparable to state-of-the-art. Our model has the additional advantage of being very interpretable, since it allows visualization of its predictions broken up by abstract features such as information content, salience and novelty. Another novel contribution of our work is abstractive training of our extractive model that can train on human generated reference summaries alone, eliminating the need for sentence-level extractive labels.

We describe a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense (grammatically and semantically) using a language model. The entire network is trained jointly on all these tasks using weight-sharing, an instance of multitask learning. All the tasks use labeled data except the language model which is learnt from unlabeled text and represents a novel form of semi-supervised learning for the shared tasks. We show how both multitask learning and semi-supervised learning improve the generalization of the shared tasks, resulting in state-of-the-art-performance.

In this paper, we explore a novel bilingual word alignment approach based on DNN (Deep Neural Network), which has been proven to be very effective in various machine learning tasks (Collobert et al., 2011). We describe in detail how we adapt and extend the CD-DNNHMM (Dahl et al., 2012) method introduced in speech recognition to the HMMbased word alignment model, in which bilingual word embedding is discriminatively learnt to capture lexical translation information, and surrounding words are leveraged to model context information in bilingual sentences. While being capable to model the rich bilingual correspondence, our method generates a very compact model with much fewer parameters. Experiments on a large scale EnglishChinese word alignment task show that the proposed method outperforms the HMM and IBM model 4 baselines by 2 points in F-score.


Question: Deep recurrent models featuring fast-forward connections for neural machine translation tasks.

This paper presents an innovative method to enhance the performance of neural networks by mitigating the issue of co-adaptation among feature detectors. In scenarios where a large feedforward neural network is trained using a small dataset, the network often exhibits poor performance on unseen test data due to overfitting. This challenge is significantly alleviated by randomly disabling half of the feature detectors for each training instance. This technique, known as "dropout," prevents intricate co-adaptations, where a feature detector is only effective in the presence of other specific detectors. Consequently, each neuron is compelled to learn a feature that is universally beneficial for producing the correct output, regardless of the various internal contexts in which it must operate. The random dropout strategy has been shown to significantly enhance performance on numerous benchmark tasks, setting new records in speech and object recognition. In a feedforward artificial neural network, the network's ability to predict the correct output given an input vector is achieved by adjusting the weights on the connections of the non-linear hidden units. When the relationship between input and output is complex, and the network possesses sufficient hidden units to model it accurately, there may be numerous weight configurations that can almost perfectly model the training set. However, these configurations often perform poorly on test data due to feature detectors being finely tuned to work together on the training data rather than on unseen data. Overfitting is effectively reduced by employing dropout to prevent complex co-adaptations on the training data. During each presentation of a training instance, each hidden unit is randomly excluded from the network with a 0.5 probability, ensuring that hidden units do not depend on the presence of others. The dropout procedure can also be viewed as an efficient model averaging technique for neural networks. A reliable way to minimize test set error is to average the predictions made by a large number of different networks. Traditionally, this is achieved by training multiple networks separately and applying each to the test data, a process that is computationally intensive. Random dropout allows for the training of a vast number of networks in a reasonable time frame, with each training instance potentially corresponding to a different network, all sharing weights for the present hidden units. The standard stochastic gradient descent method is employed for training dropout neural networks on mini-batches of training cases, with a modification to the penalty term typically used to prevent excessive weight growth. Instead of penalizing the squared length (L2 norm) of the entire weight vector, an upper bound is set on the L2 norm of the incoming weight vector for each individual hidden unit. If a weight update would

Self-attention is a method of encoding sequences of vectors by relating these vectors to each-other based on pairwise similarities. These models have recently shown promising results for modeling discrete sequences, but they are non-trivial to apply to acoustic modeling due to computational and modeling issues. In this paper, we apply self-attention to acoustic modeling, proposing several improvements to mitigate these issues: First, self-attention memory grows quadratically in the sequence length, which we address through a downsampling technique. Second, we find that previous approaches to incorporate position information into the model are unsuitable and explore other representations and hybrid models to this end. Third, to stress the importance of local context in the acoustic signal, we propose a Gaussian biasing approach that allows explicit control over the context range. Experiments find that our model approaches a strong baseline based on LSTMs with networkin-network connections while being much faster to compute. Besides speed, we find that interpretability is a strength of selfattentional acoustic models, and demonstrate that self-attention heads learn a linguistically plausible division of labor.

Recurrent neural networks that can be trained end-to-end on sequence learning tasks provide promising benefits over traditional recognition systems. In this paper, we demonstrate the application of an attention-based long short-term memory decoder network for offline handwriting recognition and analyze the segmentation, classification and decoding errors produced by the model. We further extend the decoding network by a bidirectional topology together with an integrated length estimation procedure and show that it is superior to unidirectional decoder networks. Results are presented for the word and text line recognition tasks of the RIMES handwriting recognition database. The software used in the experiments is freely available for academic research purposes.

Andor, D., Alberti, C., Weiss, D., Severyn, A., Presta, A., Ganchev, K., Petrov, S., and Collins, M. (2016). Globally normalized transition-based neural networks. In Association for Computational Linguistics. Ballesteros, M., Goldberg, Y., Dyer, C., and Smith, N. A. (2016). Training with exploration improves a greedy stack-LSTM parser. Proceedings of the conference on empirical methods in natural language processing. Chen, D. and Manning, C. D. (2014). A fast and accurate dependency parser using neural networks. In Proceedings of the conference on empirical methods in natural language processing, pages 740–750. Cheng, H., Fang, H., He, X., Gao, J., and Deng, L. (2016). Bi-directional attention with agreement for dependency parsing. arXiv preprint arXiv:1608.02076. Hashimoto, K., Xiong, C., Tsuruoka, Y., and Socher, R. (2016). A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01587. Kiperwasser, E. and Goldberg, Y. (2016). Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4:313–327. Kuncoro, A., Ballesteros, M., Kong, L., Dyer, C., Neubig, G., and Smith, N. A. (2016). What do recurrent neural network grammars learn about syntax? CoRR, abs/1611.05774. McDonald, R. T. and Pereira, F. C. (2006). Online learning of approximate dependency parsing algorithms. In EACL. Nivre, J., Hall, J., and Nilsson, J. (2006). Maltparser: A data-driven parser-generator for dependency parsing. In Proceedings of LREC, volume 6, pages 2216–2219.

ACL 2018 • Baseline: 6 layer Transformer, 512 hidden units • Data sets: IWSLT'16, WMT'17 • Metrics: BLEU (and METEOR in the paper) • 3 runs, reporting mean and standard deviation Encoder/Decoder Combinations • Architecture Definition Language • Allows easy experimentation of Neural Machine Translation (NMT) architecture variations • Detailed analysis of NMT architecture combinations • Multiple source attention layers and residual feedforward layer are key • Self-attention more important on the source side Introduction

We investigate the task of building open domain, conversational dialogue systems based on large dialogue corpora using generative models. Generative models produce system responses that are autonomously generated word-by-word, opening up the possibility for realistic, flexible interactions. In support of this goal, we extend the recently proposed hierarchical recurrent encoder-decoder neural network to the dialogue domain, and demonstrate that this model is competitive with state-of-the-art neural language models and backoff n-gram models. We investigate the limitations of this and similar approaches, and show how its performance can be improved by bootstrapping the learning from a larger questionanswer pair corpus and from pretrained word embeddings.

We present a memory augmented neural network for natural language understanding: Neural Semantic Encoders. NSE is equipped with a novel memory update rule and has a variable sized encoding memory that evolves over time and maintains the understanding of input sequences through read, compose and write operations. NSE can also access multiple and shared memories. In this paper, we demonstrated the effectiveness and the flexibility of NSE on five different natural language tasks: natural language inference, question answering, sentence classification, document sentiment analysis and machine translation where NSE achieved state-of-the-art performance when evaluated on publically available benchmarks. For example, our shared-memory model showed an encouraging result on neural machine translation, improving an attention-based baseline by approximately 1.0 BLEU.

This paper develops a model that addresses sentence embedding, a hot topic in current natural language processing research, using recurrent neural networks (RNN) with Long Short-Term Memory (LSTM) cells. The proposed LSTM-RNN model sequentially takes each word in a sentence, extracts its information, and embeds it into a semantic vector. Due to its ability to capture long term memory, the LSTM-RNN accumulates increasingly richer information as it goes through the sentence, and when it reaches the last word, the hidden layer of the network provides a semantic representation of the whole sentence. In this paper, the LSTM-RNN is trained in a weakly supervised manner on user click-through data logged by a commercial web search engine. Visualization and analysis are performed to understand how the embedding process works. The model is found to automatically attenuate the unimportant words and detect the salient keywords in the sentence. Furthermore, these detected keywords are found to automatically activate different cells of the LSTM-RNN, where words belonging to a similar topic activate the same cell. As a semantic representation of the sentence, the embedding vector can be used in many different applications. These automatic keyword detection and topic allocation abilities enabled by the LSTM-RNN allow the network to perform document retrieval, a difficult language processing task, where the similarity between the query and documents can be measured by the distance between their corresponding sentence embedding vectors computed by the LSTM-RNN. On a web search task, the LSTM-RNN embedding is shown to significantly outperform several existing state of the art methods. We emphasize that the proposed model generates sentence embedding vectors that are specially useful for web document retrieval tasks. A comparison with a well known general sentence embedding method, the Paragraph Vector, is performed. The results show that the proposed method in this paper significantly outperforms Paragraph Vector method for web document retrieval task.

The baseline BiLSTM model consists of two LSTM components, which process the input in the forward left-to-right and the backward rightto-left directions, respectively. In each direction, the reading of input words is modelled as a recurrent process with a single hidden state. Given an initial value, the state changes its value recurrently, each time consuming an incoming word. Take the forward LSTM component for example. Denoting the initial state as h 0, which is a model parameter, the recurrent state transition step for calculating h 1, . . . , h n+1 is defined as follows (Graves and Schmidhuber, 2005):

Neural network architectures such as long short-term memory (LSTM) have been proven to be powerful models for processing sequences including text, audio and video. On the basis of vanilla LSTM, multi-modal attention mechanisms are proposed in this paper to synthesize the time and semantic information of input sequences. First, we reconstruct the forget and input gates of the LSTM unit from the perspective of attention model in the temporal dimension. Then the memory content of the LSTM unit is recalculated using a cluster-based attention mechanism in semantic space. Experiments on acoustic scene classification tasks show performance improvements of the proposed methods when compared with vanilla LSTM. The classification errors on LITIS ROUEN dataset and DCASE2016 dataset are reduced by 16.5% and 7.7% relatively. We get a second place in the Kaggle’s YouTube-8M video understanding challenge, and multi-modal attention based LSTM model is one of our bestperforming single systems.

We present a novel deep Recurrent Neural Network (RNN) model for acoustic modelling in Automatic Speech Recognition (ASR). We term our contribution as a TC-DNN-BLSTM-DNN model, the model combines a Deep Neural Network (DNN) with Time Convolution (TC), followed by a Bidirectional LongShort Term Memory (BLSTM), and a final DNN. The first DNN acts as a feature processor to our model, the BLSTM then generates a context from the sequence acoustic signal, and the final DNN takes the context and models the posterior probabilities of the acoustic states. We achieve a 3.47 WER on the Wall Street Journal (WSJ) eval92 task or more than 8% relative improvement over the baseline DNN models


Question: Deep recurrent models enhanced with fast-forward connections for neural machine translation tasks.

This paper introduces an innovative phrase-based translation model and decoding methodology, designed to facilitate a comprehensive evaluation and comparison of various existing phrase-based translation models. Within this framework, we conduct extensive experiments to gain deeper insights into the reasons behind the superior performance of phrase-based models over their word-based counterparts. Our empirical findings, which are consistent across multiple language pairs, reveal that achieving peak performance can be accomplished through straightforward techniques: employing heuristics to learn phrase translations from word-level alignments, and incorporating lexical weighting into phrase translations. Interestingly, our results indicate that expanding the learning scope to include phrases longer than three words, or deriving phrases from high-accuracy word-level alignment models, does not significantly enhance performance. Conversely, restricting the learning process to syntactically motivated phrases leads to a decline in system performance. These discoveries contribute to a more nuanced understanding of the factors that influence the effectiveness of phrase-based translation models.

We present a perceptron-style discriminative approach to machine translation in which large feature sets can be exploited. Unlike discriminative reranking approaches, our system can take advantage of learned features in all stages of decoding. We first discuss several challenges to error-driven discriminative approaches. In particular, we explore different ways of updating parameters given a training example. We find that making frequent but smaller updates is preferable to making fewer but larger updates. Then, we discuss an array of features and show both how they quantitatively increase BLEU score and how they qualitatively interact on specific examples. One particular feature we investigate is a novel way to introduce learning into the initial phrase extraction process, which has previously been entirely heuristic.

We propose Neural Responding Machine (NRM), a neural networ k-based response generator for Short-Text Conversation. NRM takes the general encoder-de coder framework: it formalizes the generation of response as a decoding process based on the lat ent representation of the input text, while both encoding and decoding are realized with recurren t n ural networks (RNN). The NRM is trained with a large amount of one-round conversation dat a collected from a microblogging service. Empirical study shows that NRM can generate gramma tically correct and content-wise appropriate responses to over 75% of the input text, outperf orming state-of-the-arts in the same setting, including retrieval-based and SMT-based models.

We propose SEARNN, a novel training algorithm for recurrent neural networks (RNNs) inspired by the “learning to search” (L2S) approach to structured prediction. RNNs have been widely successful in structured prediction applications such as machine translation or parsing, and are commonly trained using maximum likelihood estimation (MLE). Unfortunately, this training loss is not always an appropriate surrogate for the test error: by only maximizing the ground truth probability, it fails to exploit the wealth of information offered by structured losses. Further, it introduces discrepancies between training and predicting (such as exposure bias) that may hurt test performance. Instead, SEARNN leverages test-alike search space exploration to introduce global-local losses that are closer to the test error. We demonstrate improved performance over MLE on three different tasks: OCR, spelling correction and text chunking. Finally, we propose a subsampling strategy to enable SEARNN to scale to large vocabulary sizes.

We present a memory augmented neural network for natural language understanding: Neural Semantic Encoders. NSE is equipped with a novel memory update rule and has a variable sized encoding memory that evolves over time and maintains the understanding of input sequences through read, compose and write operations. NSE can also access multiple and shared memories. In this paper, we demonstrated the effectiveness and the flexibility of NSE on five different natural language tasks: natural language inference, question answering, sentence classification, document sentiment analysis and machine translation where NSE achieved state-of-the-art performance when evaluated on publically available benchmarks. For example, our shared-memory model showed an encouraging result on neural machine translation, improving an attention-based baseline by approximately 1.0 BLEU.

Neural machine translation NMT has been prominent in many machine translation tasks. However, in some domain-specific tasks, only the corpora from similar domains can improve translation performance. If out-of-domain corpora are directly added into the in-domain corpus, the translation performance may even degrade. Therefore, domain adaptation techniques are essential to solve the NMT domain problem. Most existing methods for domain adaptation are designed for the conventional phrase-based machine translation. For NMT domain adaptation, there have been only a few studies on topics such as fine tuning, domain tags, and domain features. In this paper, we have four goals for sentence level NMT domain adaptation. First, the NMT's internal sentence embedding is exploited and the sentence embedding similarity is used to select out-of-domain sentences that are close to the in-domain corpus. Second, we propose three sentence weighting methods, i.e., sentence weighting, domain weighting, and batch weighting, to balance the data distribution during NMT training. Third, in addition, we propose dynamic training methods to adjust the sentence selection and weighting during NMT training. Fourth, to solve the multidomain problem in a real-world NMT scenario where the domain distributions of training and testing data often mismatch, we proposed a multidomain sentence weighting method to balance the domain distributions of training data and match the domain distributions of training and testing data. The proposed methods are evaluated in international workshop on spoken language translation IWSLT English-to-French/German tasks and a multidomain English-to-French task. Empirical results show that the sentence selection and weighting methods can significantly improve the NMT performance, outperforming the existing baselines.

End-to-end (E2E) models, which directly predict output character sequences given input speech, are good candidates for on-device speech recognition. E2E models, however, present numerous challenges: In order to be truly useful, such models must decode speech utterances in a streaming fashion, in real time; they must be robust to the long tail of use cases; they must be able to leverage user-specific context (e.g., contact lists); and above all, they must be extremely accurate. In this work, we describe our efforts at building an E2E speech recognizer using a recurrent neural network transducer. In experimental evaluations, we find that the proposed approach can outperform a conventional CTC-based model in terms of both latency and accuracy in a number of evaluation categories.

We study the topmost weight matrix of neural network language models. We show that this matrix constitutes a valid word embedding. When training language models, we recommend tying the input embedding and this output embedding. We analyze the resulting update rules and show that the tied embedding evolves in a more similar way to the output embedding than to the input embedding in the untied model. We also offer a new method of regularizing the output embedding. Our methods lead to a significant reduction in perplexity, as we are able to show on a variety of neural network language models. Finally, we show that weight tying can reduce the size of neural translation models to less than half of their original size without harming their performance.

We present SummaRuNNer, a Recurrent Neural Network (RNN) based sequence model for extractive summarization of documents and show that it achieves performance better than or comparable to state-of-the-art. Our model has the additional advantage of being very interpretable, since it allows visualization of its predictions broken up by abstract features such as information content, salience and novelty. Another novel contribution of our work is abstractive training of our extractive model that can train on human generated reference summaries alone, eliminating the need for sentence-level extractive labels.

Attention is an increasingly popular mechanism used in a wide range of neural architectures. Because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for attention architectures for natural language processing, with a focus on architectures designed to work with vector representation of the textual data. We discuss the dimensions along which proposals differ, the possible uses of attention, and chart the major research activities and open challenges in the area.

Graph-structured data is important in modeling relationships between multiple entities, and can be used to represent states of the world as well as many data structures. Li et al. (2016) describe a model known as a Gated Graph Sequence Neural Network (GGS-NN) that produces sequences from graph-structured input. In this work I introduce the Gated Graph Transformer Neural Network (GGTNN), an extension of GGS-NNs that uses graph-structured data as an intermediate representation. The model can learn to construct and modify graphs in sophisticated ways based on textual input, and also to use the graphs to produce a variety of outputs. For example, the model successfully learns to solve almost all of the bAbI tasks (Weston et al., 2016), and also discovers the rules governing graphical formulations of a simple cellular automaton and a family of Turing machines.


Question: Deep recurrent models incorporating fast-forward connections for neural machine translation tasks.

This paper investigates the application of deep learning models augmented with memory components or attention mechanisms to the question answering (QA) domain. We evaluate and contrast three models: Neural Machine Translation (NMT), Neural Turing Machines (NTM), and Memory Networks (MemNets), using a simulated QA dataset. Notably, this study is pioneering in its application of NMT and NTM for QA tasks. Our findings indicate that integrating attention mechanisms with memory components holds promise for addressing specific QA challenges, demonstrating the potential of these combined approaches.

**Question**: Evaluating the Performance of Convolutional Neural Networks in Image Recognition

**Documents**: A Comparative Study of Deep Learning Techniques for Image Classification
This paper presents a comparative study of various deep learning techniques for image classification. We compare Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Deep Belief Networks (DBN) on a benchmark dataset of natural images. Our results show that CNN outperforms RNN and DBN in terms of classification accuracy.

**Similar Documents**: This study conducts a thorough comparison of different deep learning methodologies for image classification tasks. We assess the performance of Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Deep Belief Networks (DBN) using a standard dataset of natural images. Our findings reveal that CNN surpasses RNN and DBN in classification accuracy, highlighting its effectiveness in image recognition applications.

Recent progress in using recurrent neural networks (RNNs) for image description has motivated the exploration of their application for video description. However, while images are static, working with videos requires modeling their dynamic temporal structure and then properly integrating that information into a natural language description model. In this context, we propose an approach that successfully takes into account both the local and global temporal structure of videos to produce descriptions. First, our approach incorporates a spatial temporal 3-D convolutional neural network (3-D CNN) representation of the short temporal dynamics. The 3-D CNN representation is trained on video action recognition tasks, so as to produce a representation that is tuned to human motion and behavior. Second we propose a temporal attention mechanism that allows to go beyond local temporal modeling and learns to automatically select the most relevant temporal segments given the text-generating RNN. Our approach exceeds the current state-of-art for both BLEU and METEOR metrics on the Youtube2Text dataset. We also present results on a new, larger and more challenging dataset of paired video and natural language descriptions.

Recently a variety of LSTM-based conditional language models (LM) have been applied across a range of language generation tasks. In this work we study various model architectures and different ways to represent and aggregate the source information in an endto-end neural dialogue system framework. A method called snapshot learning is also proposed to facilitate learning from supervised sequential signals by applying a companion cross-entropy objective function to the conditioning vector. The experimental and analytical results demonstrate firstly that competition occurs between the conditioning vector and the LM, and the differing architectures provide different trade-offs between the two. Secondly, the discriminative power and transparency of the conditioning vector is key to providing both model interpretability and better performance. Thirdly, snapshot learning leads to consistent performance improvements independent of which architecture is used.

We propose a selective encoding model to extend the sequence-to-sequence framework for abstractive sentence summarization. It consists of a sentence encoder, a selective gate network, and an attention equipped decoder. The sentence encoder and decoder are built with recurrent neural networks. The selective gate network constructs a second level sentence representation by controlling the information flow from encoder to decoder. The second level representation is tailored for sentence summarization task, which leads to better performance. We evaluate our model on the English Gigaword, DUC 2004 and MSR abstractive sentence summarization datasets. The experimental results show that the proposed selective encoding model outperforms the state-ofthe-art baseline models.

Long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997) can solve numerous tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset. Without resets, the state may grow indefinitely and eventually cause the network to break down. Our remedy is a novel, adaptive forget gate that enables an LSTM cell to learn to reset itself at appropriate times, thus releasing internal resources. We review illustrative benchmark problems on which standard LSTM outperforms other RNN algorithms. All algorithms (including LSTM) fail to solve continual versions of these problems. LSTM with forget gates, however, easily solves them, and in an elegant way.

The success of deep learning often derives from well-chosen operational building blocks. In this work, we revise the temporal convolution operation in CNNs to better adapt it to text processing. Instead of concatenating word representations, we appeal to tensor algebra and use low-rank n-gram tensors to directly exploit interactions between words already at the convolution stage. Moreover, we extend the n-gram convolution to non-consecutive words to recognize patterns with intervening words. Through a combination of lowrank tensors, and pattern weighting, we can efficiently evaluate the resulting convolution operation via dynamic programming. We test the resulting architecture on standard sentiment classification and news categorization tasks. Our model achieves state-of-the-art performance both in terms of accuracy and training speed. For instance, we obtain 51.2% accuracy on the fine-grained sentiment classification task.1

ACL 2018 • Baseline: 6 layer Transformer, 512 hidden units • Data sets: IWSLT'16, WMT'17 • Metrics: BLEU (and METEOR in the paper) • 3 runs, reporting mean and standard deviation Encoder/Decoder Combinations • Architecture Definition Language • Allows easy experimentation of Neural Machine Translation (NMT) architecture variations • Detailed analysis of NMT architecture combinations • Multiple source attention layers and residual feedforward layer are key • Self-attention more important on the source side Introduction

We propose Neural Responding Machine (NRM), a neural networ k-based response generator for Short-Text Conversation. NRM takes the general encoder-de coder framework: it formalizes the generation of response as a decoding process based on the lat ent representation of the input text, while both encoding and decoding are realized with recurren t n ural networks (RNN). The NRM is trained with a large amount of one-round conversation dat a collected from a microblogging service. Empirical study shows that NRM can generate gramma tically correct and content-wise appropriate responses to over 75% of the input text, outperf orming state-of-the-arts in the same setting, including retrieval-based and SMT-based models.

Deep Bidirectional LSTM (DBLSTM) recurrent neural networks have recently been shown to give state-of-the-art performance on the TIMIT speech database. However, the results in that work relied on recurrent-neural-network-specific objective functions, which are difficult to integrate with existing large vocabulary speech recognition systems. This paper investigates the use of DBLSTM as an acoustic model in a standard neural network-HMM hybrid system. We find that a DBLSTM-HMM hybrid gives equally good results on TIMIT as the previous work. It also outperforms both GMM and deep network benchmarks on a subset of the Wall Street Journal corpus. However the improvement in word error rate over the deep network is modest, despite a great increase in framelevel accuracy. We conclude that the hybrid approach with DBLSTM appears to be well suited for tasks where acoustic modelling predominates. Further investigation needs to be conducted to understand how to better leverage the improvements in frame-level accuracy towards better word error rates.

This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time. The approach is demonstrated for text (where the data are discrete) and online handwriting (where the data are real-valued). It is then extended to handwriting synthesis by allowing the network to condition its predictions on a text sequence. The resulting system is able to generate highly realistic cursive handwriting in a wide variety of styles.

While natural languages are compositional, how state-of-the-art neural models achieve compositionality is still unclear. We propose a deep network, which not only achieves competitive accuracy for text classification, but also exhibits compositional behavior. That is, while creating hierarchical representations of a piece of text, such as a sentence, the lower layers of the network distribute their layer-specific attention weights to individual words. In contrast, the higher layers compose meaningful phrases and clauses, whose lengths increase as the networks get deeper until fully composing the sentence.


Question: Deep recurrent models incorporating fast-forward connections for neural machine translation tasks.

This brief report introduces an enhancement to the conventional long short-term memory (LSTM) networks by incorporating a depth gate to link memory cells across consecutive layers. This approach establishes a linear relationship between the recurrent units of lower and upper layers. Notably, this linear connection is regulated by a gating mechanism, which we refer to as the depth gate. The depth gate's functionality is determined by the lower layer's memory cell, the current input, and the previous memory cell of the same layer. Our empirical studies have substantiated that this novel LSTM architecture significantly enhances performance in the domains of machine translation and language modeling.

Introduction Statistical language models estimate the probability for a given sequence of words. Given a sentence s with n words such as s = (w1,w2 . . .wn), the language model assigns P(s). Statistical language models assess good word sequence estimations based on the sequence probability estimation. Building robust, fast and accurate language models is one of the main factors in the success of building systems such as machine translation systems and automatic speech recognition systems. Statistical language models can be estimated based on various approaches. Classical language models are estimated based on n-gram word sequences such as P(s) = P(w1,w2 . . .wn) = ∏n i=1 P(wi|wi−1) and can be approximated based on the Markov concept for shorter contexts (for example, bigram if n = 2 or trigram if n = 3 and so on). Recent researchers have applied neural networks different architectures to build and estimate language models. The classical feed-forward neural network-based language models have been continuously reporting good results among the traditional n-gram language modeling techniques [1]. It Abstract

Deep learning methods have recently achieved great empirical success on machine translation, dialogue response generation, summarization, and other text generation tasks. At a high level, the technique has been to train end-to-end neural network models consisting of an encoder model to produce a hidden representation of the source text, followed by a decoder model to generate the target. While such models have significantly fewer pieces than earlier systems, significant tuning is still required to achieve good performance. For text generation models in particular, the decoder can behave in undesired ways, such as by generating truncated or repetitive outputs, outputting bland and generic responses, or in some cases producing ungrammatical gibberish. This paper is intended as a practical guide for resolving such undesired behavior in text generation models, with the aim of helping enable real-world applications.

Recurrent neural network language models RNNLMs are becoming increasingly popular for a range of applications including automatic speech recognition. An important issue that limits their possible application areas is the computational cost incurred in training and evaluation. This paper describes a series of new efficiency improving approaches that allows RNNLMs to be more efficiently trained on graphics processing units GPUs and evaluated on CPUs. First, a modified RNNLM architecture with a nonclass-based, full output layer structure F-RNNLM is proposed. This modified architecture facilitates a novel spliced sentence bunch mode parallelization of F-RNNLM training using large quantities of data on a GPU. Second, two efficient RNNLM training criteria based on variance regularization and noise contrastive estimation are explored to specifically reduce the computation associated with the RNNLM output layer softmax normalisation term. Finally, a pipelined training algorithm utilizing multiple GPUs is also used to further improve the training speed. Initially, RNNLMs were trained on a moderate dataset with 20M words from a large vocabulary conversational telephone speech recognition task. The training time of RNNLM is reduced by up to a factor of 53 on a single GPU over the standard CPU-based RNNLM toolkit. A 56 times speed up in test time evaluation on a CPU was obtained over the baseline F-RNNLMs. Consistent improvements in both recognition accuracy and perplexity were also obtained over C-RNNLMs. Experiments on Google's one billion corpus also reveals that the training of RNNLM scales well.

In this paper we show that a simple beam approximation of the joint distribution between attention and output is an easy, accurate, and efficient attention mechanism for sequence to sequence learning. The method combines the advantage of sharp focus in hard attention and the implementation ease of soft attention. On five translation and two morphological inflection tasks we show effortless and consistent gains in BLEU compared to existing attention mechanisms.

In a conversation or a dialogue process, attention and intention play intrinsic roles. This paper proposes a neural network based approach that models the attention and intention processes. It essentially consists of three recurrent networks. The encoder network is a word-level model representing source side sentences. The intention network is a recurrent network that models the dynamics of the intention process. The decoder network is a recurrent network that produces responses to the input from the source side. It is a language model that is dependent on the intention and has an attention mechanism to attend to particular source side words, when predicting a symbol in the response. The model is trained end-toend without labeling data. Experiments show that this model generates natural responses to user inputs.

The success of deep neural networks has inspired many to wonder whether other learners could benefit from deep, layered architectures. We present a general framework called forward thinking for deep learning that generalizes the architectural flexibility and sophistication of deep neural networks while also allowing for (i) different types of learning functions in the network, other than neurons, and (ii) the ability to adaptively deepen the network as needed to improve results. This is done by training one layer at a time, and once a layer is trained, the input data are mapped forward through the layer to create a new learning problem. The process is then repeated, transforming the data through multiple layers, one at a time, rendering a new dataset, which is expected to be better behaved, and on which a final output layer can achieve good performance. In the case where the neurons of deep neural nets are replaced with decision trees, we call the result a Forward Thinking Deep Random Forest (FTDRF). We demonstrate a proof of concept by applying FTDRF on the MNIST dataset. We also provide a general mathematical formulation, called Forward Thinking that allows for other types of deep learning problems to be considered.

Very deep CNNs with small 3 × 3 kernels have recently been shown to achieve very strong performance as acoustic models in hybrid NN-HMM speech recognition systems. In this paper we investigate how to efficiently scale these models to larger datasets. Specifically, we address the design choice of pooling and padding along the time dimension which renders convolutional evaluation of sequences highly inefficient. We propose a new CNN design without timepadding and without timepooling, which is slightly suboptimal for accuracy, but has two significant advantages: it enables sequence training and deployment by allowing efficient convolutional evaluation of full utterances, and, it allows for batch normalization to be straightforwardly adopted to CNNs on sequence data. Through batch normalization, we recover the lost peformance from removing the time-pooling, while keeping the benefit of efficient convolutional evaluation. We demonstrate the performance of our models both on larger scale data than before, and after sequence training. Our very deep CNN model sequence trained on the 2000h switchboard dataset obtains 9.4 word error rate on the Hub5 test-set, matching with a single model the performance of the 2015 IBM system combination, which was the previous best published result.

Recurrent neural network language models (RNNLMs) have recently demonstrated state-of-the-art performance across a variety of tasks. In this paper, we improve their performance by providing a contextual real-valued input vector in association with each word. This vector is used to convey contextual information about the sentence being modeled. By performing Latent Dirichlet Allocation using a block of preceding text, we achieve a topic-conditioned RNNLM. This approach has the key advantage of avoiding the data fragmentation associated with building multiple topic models on different data subsets. We report perplexity results on the Penn Treebank data, where we achieve a new state-of-the-art. We further apply the model to the Wall Street Journal speech recognition task, where we observe improvements in word-error-rate.

Recurrent Neural Networks (RNN) are a type of statistical model designed to handle sequential data. The model reads a sequence one symbol at a time. Each symbol is processed based on information collected from the previous symbols. With existing RNN architectures, each symbol is processed using only information from the previous processing step. To overcome this limitation, we propose a new kind of RNN model that computes a recurrent weighted average (RWA) over every past processing step. Because the RWA can be computed as a running average, the computational overhead scales like that of any other RNN architecture. The approach essentially reformulates the attention mechanism into a stand-alone model. The performance of the RWA model is assessed on the variable copy problem, the adding problem, classification of artificial grammar, classification of sequences by length, and classification of the MNIST images (where the pixels are read sequentially one at a time). On almost every task, the RWA model is found to fit the data significantly faster than a standard LSTM model.

We describe a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense (grammatically and semantically) using a language model. The entire network is trained jointly on all these tasks using weight-sharing, an instance of multitask learning. All the tasks use labeled data except the language model which is learnt from unlabeled text and represents a novel form of semi-supervised learning for the shared tasks. We show how both multitask learning and semi-supervised learning improve the generalization of the shared tasks, resulting in state-of-the-art-performance.


Question: Reflections on Vehicular Ad Hoc Networks (VANETs) in real-world traffic situations.

VANETs: An In-depth Examination of Challenges and Future Directions
Vehicular Ad Hoc Networks (VANETs), as a specialized subset of Mobile Ad Hoc Networks (MANETs), hold significant promise for enhancing future intelligent transportation systems (ITS). These networks, devoid of fixed infrastructure, leverage vehicles as nodes to create a dynamic communication framework. Yet, VANETs face unique challenges, primarily stemming from vehicular mobility, driver actions, and the inherently unstable network topology, that starkly contrast with traditional MANETs. This paper delves into these challenges, highlighting key issues and exploring proposed solutions. It also provides an overview of the current research landscape and outlines potential future developments. By reading this article, one gains a deeper insight into the intricacies of vehicular ad hoc networking and the evolving research trends in this domain.

This contribution provides a review of fundamental goals, development and future perspectives of driver assistance systems. Mobility is a fundamental desire of mankind. Virtually any society strives for safe and efficient mobility at low ecological and economic costs. Nevertheless, its technical implementation significantly differs among societies, depending on their culture and their degree of industrialization. A potential evolutionary roadmap for driver assistance systems is discussed. Emerging from systems based on proprioceptive sensors, such as ABS or ESC, we review the progress incented by the use of exteroceptive sensors such as radar, video, or lidar. While the ultimate goal of automated and cooperative traffic still remains a vision of the future, intermediate steps towards that aim can be realized through systems that mitigate or avoid collisions in selected driving situations. Research extends the state-of-the-art in automated driving in urban traffic and in cooperative driving, the latter addressing communication and collaboration between different vehicles, as well as cooperative vehicle operation by its driver and its machine intelligence. These steps are considered important for the interim period, until reliable unsupervised automated driving for all conceivable traffic situations becomes available. The prospective evolution of driver assistance systems will be stimulated by several technological, societal and market trends. The paper closes with a view on current research fields.

On-road vehicle detection and rear-end crash prevention are demanding subjects in both academia and automotive industry. The paper focuses on monocular vision-based vehicle detection under challenging lighting conditions, being still an open topic in the area of driver assistance systems. The paper proposes an effective vehicle detection method based on multiple features analysis and Dempster-Shafer-based fusion theory. We also utilize a new idea of Adaptive Global Haar-like (AGHaar) features as a promising method for feature classification and vehicle detection in both daylight and night conditions. Validation tests and experimental results show superior detection results for day, night, rainy, and challenging conditions compared to state-of-the-art solutions.

Autonomous vehicles promise numerous improvements to vehicular traffic: an increase in both highway capacity and traffic flow because of faster response times, less fuel consumption and pollution thanks to more foresighted driving, and hopefully fewer accidents thanks to collision avoidance systems. In addition, drivers can save time for more useful activities. In order for these vehicles to safely operate in everyday traffic or in harsh off-road environments, a multitude of problems in perception, navigation, and control have to be solved. This paper gives an overview of the most current trends in autonomous vehicles, highlighting the concepts common to most successful systems as well as their differences. It concludes with an outlook into the promising future of autonomous vehicles.

Currently autonomous or self-driving vehicles are at the heart of academia and industry research becauseof itsmulti-faceted advantages that includes improved safety, reduced congestion, lower emissions and greater mobility. Software is the key driving factor underpinning autonomy within which planning algorithms that are responsible for mission-critical decision making hold a significant position. While transporting passengers or goods from a givenorigin to agivendestination,motionplanningmethods incorporate searching for apath to follow, avoiding obstacles and generating the best trajectory that ensures safety, comfort and efficiency. A range of different planning approaches have beenproposed in the literature. Thepurpose of this paper is to reviewexisting approaches and then compare and contrast different methods employed for the motion planning of autonomous on-road driving that consists of (1)findingapath, (2) searching for the safestmanoeuvre and (3)determining themost feasible trajectory. Methods developed by researchers in each of these three levels exhibit varying levels of complexity and performance accuracy. This paper presents a critical evaluation of each of these methods, in terms of their advantages/disadvantages, inherent limitations, feasibility, optimality, handling of obstacles and testing operational environments. Based on a critical review of existingmethods, research challenges to address current limitations are identified and future research directions are suggested so as to enhance the performanceofplanningalgorithmsat all three levels. Somepromising areasof future focushave been identified as the use of vehicular communications (V2V and V2I) and the incorporation of transport engineering aspects in order to improve the look-ahead horizon of current sensing technologies that are essential for planning with the aim of reducing the total cost of driverless vehicles. This critical reviewon planning techniques presented in this paper, along with theassociateddiscussions on their constraints and limitations, seek to assist researchers in accelerating development in the emerging field of autonomous vehicle research. Crown Copyright 2015 Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

Road traffic accidents (RTAs) are a major public health concern, resulting in an estimated 1.2 million deaths and 50 million injuries worldwide each year. In the developing world, RTAs are among the leading cause of death and injury; Ethiopia in particular experiences the highest rate of such accidents. Thus, methods to reduce accident severity are of great interest to traffic agencies and the public at large. In this work, we applied data mining technologies to link recorded road characteristics to accident severity in Ethiopia, and developed a set of rules that could be used by the Ethiopian Traffic Agency to improve safety. Problem Statement The costs of fatalities and injuries due to road traffic accidents (RTAs) have a tremendous impact on societal well-being and socioeconomic development. RTAs are among the leading causes of death and injury worldwide, causing an estimated 1.2 million deaths and 50 million injuries each year (World Health Organization, 2004). Ethiopia has the highest rate of RTAs, owing to the fact that road transport is the major transportation system in the country. The Ethiopian traffic control system archives data on various aspects of the traffic system, such as traffic volume, concentration, and vehicle accidents. With more vehicles and traffic, the capital city of Addis Ababa takes the lion’s share of the risk, with an average of 20 accidents being recorded every day and even more going unreported. The basic hypothesis of this research is that accidents are not randomly scattered along the road network, and that drivers are not involved in accidents at random. There are complex circumstantial relationships between several characteristics (driver, road, car, etc.) and the accident occurrence. As such, one cannot improve safety without successfully relating accident frequency and severity to the causative variables (Kononov and Janson 2002). We will attempt to extend the authors’ previous work in this area by generating additional attributes and focusing on the contribution of road-related factors to accident severity in Ethiopia. This will help to identify the parts of a road that are risky, thus supporting traffic accident data analysis in decision-making processes. The general objective of the research is to investigate the role of road-related factors in accident severity, using RTA data from Ethiopia and predictive models. Our three specific objectives include: 1) exploring the underlying variables (especially road-related ones) that impact car accident severity, 2) predicting accident severity using different data mining techniques, and 3) comparing standard classification models for this task. Literature Review Various studies have addressed the different aspects of RTAs, with most focusing on predicting or establishing the critical factors influencing injury severity (Chong, A. et al. 2005). Numerous data mining-related studies have been undertaken to analyze RTA data locally and globally, with results frequently varying depending on the socio-economic conditions and infrastructure of a given location. Ossenbruggen, Pendharkar et al. (2001) used a logistic regression model to identify the prediction factors of crashes and crash-related injuries, using models to perform a risk assessment of a given region. These models included attributes describing a site by its land use activity, roadside design, use of traffic control devices, and traffic exposure. Their study illustrated that village sites were less hazardous than residential or shopping sites. Abdalla et al. (1997) also studied the relationship between casualty frequency and the distance of an accident from residential zones. Not surprisingly, casualty frequencies were higher in accidents that occurred nearer to residential zones, possibly due to higher exposure. The

For autonomous parking applications to become possible, knowledge about the parking environment is required. Therefore, a real-time algorithm for detecting parked vehicles from radar data is presented. These data are first accumulated in an occupancy grid from which objects are detected by applying techniques borrowed from the computer vision field. Two random forest classifiers are trained to recognize two categories of objects: parallel-parked vehicles and cross-parked vehicles. Performances of the classifiers are evaluated as well as the capacity of the complete system to detect parked vehicles in real world scenarios.

Full autonomy of ground vehicles is a major goal of the ITS (Intelligent Transportation Systems) community. However, reaching such highest autonomy level in all situations (weather, traffic, ...) may seem difficult in practice, despite recent results regarding driverless cars (e.g., Google Cars). In addition, an automated vehicle should also self-assess its own perception abilities, and not only perceive its environment. In this paper, we propose an intermediate approach towards full automation, by defining a spectrum of automation layers, from fully manual (the car is driven by a driver) to fully automated (the car is driven by a computer), based on an ontological model for representing knowledge. We also propose a second ontology for situation assessment (what does the automated car perceive?), including the sensors/actuators state, environmental conditions and driver's state. Finally, we also define inference rules to link the situation assessment ontology to the automation level one. Both ontological models have been built and first results are presented.

Moving vehicles are detected and tracked automatically in monocular image sequences from road traffic scenes recorded by a stationary camera. In order to exploit the a priori knowledge about shape and motion of vehicles in traffic scenes, a parameterized vehicle model is used for an intraframe matching process and a recursive estimator based on a motion model is used for motion estimation. An interpretation cycle supports the intraframe matching process with a state MAP-update step. Initial model hypotheses are generated using an image segmentation component which clusters coherently moving image features into candidate representations of images of a moving vehicle. The inclusion of an illumination model allows taking shadow edges of the vehicle into account during the matching process. Only such an elaborate combination of various techniques has enabled us to track vehicles under complex illumination conditions and over long (over 400 frames) monocular image sequences. Results on various real-world road traffic scenes are presented and open problems as well as future work are outlined.

Many Advanced Driver Assistance Systems (ADAS) have been developed to improve car safety. However, it is still a challenging problem to make autonomous vehicles to drive safely on urban streets such as uncontrolled intersections (without traffic lights) and narrow roads. In this paper, we introduce a decision making system that can assist autonomous vehicles at uncontrolled intersections and narrow roads. We constructed a machine understandable ontology-based Knowledge Base, which contains maps and traffic regulations. The system makes decisions in comply with traffic regulations such as Right-Of-Way rules when it receives a collision warning signal. The decisions are sent to a path planning system to change the route or stop to avoid collisions.

Commercial automotive radars used today are based on frequency modulated continuous wave signals due to the simple and robust detection method and good accuracy. However, the increase in both the number of radars deployed per vehicle and the number of such vehicles leads to mutual interference, cutting short future plans for autonomous driving and active safety functionality. We propose and analyze a radar communication (RadCom) approach to reduce this mutual interference while simultaneously offering communication functionality. We achieve this by frequency division multiplexing radar and communication, where communication is built on a decentralized carrier sense multiple access protocol and is used to adjust the timing of radar transmissions. Our simulation results indicate that radar interference can be significantly reduced, at no cost in radar accuracy.


Question: Reflections on the application of Vehicular Ad Hoc Networks (VANETs) in real-world traffic situations.

This paper delves into the intricacies of vehicular safety communication systems, focusing on the Dedicated Short-Range Communication (DSRC) technology operating at 5.9 GHz. As the automotive sector advances towards enhanced active safety features, DSRC emerges as a critical technology for enabling next-generation safety applications based on communication. A pivotal aspect of vehicular safety communication involves the continuous exchange of messages between all equipped vehicles, necessitating effective channel congestion control and optimization of broadcast performance. These concerns are integral to the comprehensive design of communication protocols. Additionally, the inherent multichannel characteristic of DSRC demands a coordinated operational framework to accommodate both safety and non-safety applications simultaneously. This article offers a detailed perspective on DSRC-based vehicular safety communications and introduces a suite of protocols designed to tackle the aforementioned challenges, thereby ensuring seamless and safe vehicular communication in real-world traffic scenarios.

Avoiding high computational costs and calibration issues involved in stereo-vision-based algorithms, this paper proposes real-time monocular-vision-based techniques for simultaneous vehicle detection and inter-vehicle distance estimation, in which the performance and robustness of the system remain competitive, even for highly challenging benchmark datasets. This paper develops a collision warning system by detecting vehicles ahead and, by identifying safety distances to assist a distracted driver, prior to occurrence of an imminent crash. We introduce adaptive global Haar-like features for vehicle detection, tail-light segmentation, virtual symmetry detection, intervehicle distance estimation, as well as an efficient single-sensor multifeature fusion technique to enhance the accuracy and robustness of our algorithm. The proposed algorithm is able to detect vehicles ahead at both day or night and also for short- and long-range distances. Experimental results under various weather and lighting conditions (including sunny, rainy, foggy, or snowy) show that the proposed algorithm outperforms state-of-the-art algorithms.

The development of radar signal processing algorithms for target tracking and higher-level automotive applications is mainly done based on real radar data. A data basis has to be acquired during cost-expensive and time-consuming test runs. For a comparably simple application like the adaptive cruise control (ACC), the variety of significant traffic situations can sufficiently be covered by test runs. But for more advanced applications like intersection assistance, the effort for the acquisition of a representative set of radar data will be unbearable. In this paper, we propose a way of simulating radar target lists in a realistic but computationally undemanding way, which will allow to significantly reduce the amount of real radar data needed

Electronic toll collection transponders, e.g., E-ZPass, are a widely-used wireless technology. About 70% to 89% of the cars in US have these devices, and some states plan to make them mandatory. As wireless devices however, they lack a basic function: a MAC protocol that prevents collisions. Hence, today, they can be queried only with directional antennas in isolated spots. However, if one could interact with e-toll transponders anywhere in the city despite collisions, it would enable many smart applications. For example, the city can query the transponders to estimate the vehicle flow at every intersection. It can also localize the cars using their wireless signals, and detect those that run a red-light. The same infrastructure can also deliver smart street-parking, where a user parks anywhere on the street, the city localizes his car, and automatically charges his account. This paper presents Caraoke, a networked system for delivering smart services using e-toll transponders. Our design operates with existing unmodified transponders, allowing for applications that communicate with, localize, and count transponders, despite wireless collisions. To do so, Caraoke exploits the structure of the transponders' signal and its properties in the frequency domain. We built Caraoke reader into a small PCB that harvests solar energy and can be easily deployed on street lamps. We also evaluated Caraoke on four streets on our campus and demonstrated its capabilities.

Vehicles of higher automation levels require the creation of situation awareness. One important aspect of this situation awareness is an understanding of the current risk of a driving situation. In this work, we present a novel approach for the dynamic risk assessment of driving situations based on images of a front stereo camera using deep learning. To this end, we trained a deep neural network with recorded monocular images, disparity maps and a risk metric for diverse traffic scenes. Our approach can be used to create the aforementioned situation awareness of vehicles of higher automation levels and can serve as a heterogeneous channel to systems based on radar or lidar sensors that are used traditionally for the calculation of risk metrics.

Automated driving requires decision making in dynamic and uncertain environments. The uncertainty from the prediction originates from the noisy sensor data and from the fact that the intention of human drivers cannot be directly measured. This problem is formulated as a partially observable Markov decision process (POMDP) with the intended route of the other vehicles as hidden variables. The solution of the POMDP is a policy determining the optimal acceleration of the ego vehicle along a preplanned path. Therefore, the policy is optimized for the most likely future scenarios resulting from an interactive, probabilistic motion model for the other vehicles. Considering possible future measurements of the surrounding cars allows the autonomous car to incorporate the estimated change in future prediction accuracy in the optimal policy. A compact representation results in a low-dimensional state-space. Thus, the problem can be solved online for varying road layouts and number of vehicles. This is done with a point-based solver in an anytime fashion on a continuous state-space. Our evaluation is threefold: At first, the convergence of the algorithm is evaluated and it is shown how the convergence can be improved with an additional search heuristic. Second, we show various planning scenarios to demonstrate how the introduction of different considered uncertainties results in more conservative planning. At the end, we show online simulations for the crossing of complex (unsignalized) intersections. We can demonstrate that our approach performs nearly as good as with full prior information about the intentions of the other vehicles and clearly outperforms reactive approaches.

The automotive industry is moving aggressively in the direction of advanced active safety. Dedicated short-range communication (DSRC) is a key enabling technology for the next generation of communication-based safety applications. One aspect of vehicular safety communication is the routine broadcast of messages among all equipped vehicles. Therefore, channel congestion control and broadcast performance improvement are of particular concern and need to be addressed in the overall protocol design. Furthermore, the explicit multichannel nature of DSRC necessitates a concurrent multichannel operational scheme for safety and non-safety applications. This article provides an overview of DSRC based vehicular safety communications and proposes a coherent set of protocols to address these requirements

Due to the widespread popularity in both academia and industry, vehicular ad hoc networks (VANETs) have been used in a wide range of applications starting from intelligent transportation to e-health and itinerary planning. This paper proposes a new decentralized lightweight authentication and key agreement scheme for VANETs. In the proposed scheme, there are three types of mutual authentications: 1) between vehicles; 2) between vehicles and their respective cluster heads; and 3) between cluster heads and their respective roadside units. Apart from these authentications, the proposed scheme also maintains secret keys between roadside units for their secure communications. The rigorous formal and informal security analysis shows that the proposed scheme is capable to defend various malicious attacks. Moreover, the ns-2 simulation demonstrates the practicability of the proposed scheme in VANET environment.

Cooperative driving behavior is essential for driving in traffic, especially for ramp merging, lane changing or navigating intersections. Autonomous vehicles should also manage these situations by behaving cooperatively and naturally. The challenge of cooperative driving is estimating other vehicles' intentions. In this paper, we present a novel method to estimate other human-driven vehicles' intentions with the aim of achieving a natural and amenable cooperative driving behavior, without using wireless communication. The new approach allows the autonomous vehicle to cooperate with multiple observable merging vehicles on the ramp with a leading vehicle ahead of the autonomous vehicle in the same lane. To avoid calculating trajectories, simplify computation, and take advantage of mature Level-3 components, the new method reacts to merging cars by determining a following target for an off-the-shelf distance keeping module (ACC) which governs speed control of the autonomous vehicle. We train and evaluate the proposed model using real traffic data. Results show that the new approach has a lower collision rate than previous methods and generates more human driver-like behaviors in terms of trajectory similarity and time-to-collision to leading vehicles.

Introduction of automated driving increases complexity of automotive systems. As a result, architecture design becomes a major concern for ensuring non-functional requirements such as safety, and modifiability. In the ISO 26262 standard, architecture patterns are recommended for system development. However, the existing architecture patterns may not be able to answer requirements of automated driving completely. When applying these patterns in the automated driving context, modification and analysis of these patterns are needed. In this paper, we present a novel architecture pattern for safety critical automated driving functions. In addition, we propose a generic approach to compare our pattern with a number of existing ones. The comparison results can be used as a basis for project specific architectural decisions. Our Safety Channel pattern is validated by its implementation for a real-life truck platooning application.

Internet of vehicles (IoV) is a branch of the internet of things (IoT) which is used for communication among vehicles. As vehicular nodes are considered always in motion, hence it causes the frequent changes in the topology. These changes cause major issues in IoV like scalability, dynamic topology changes, and shortest path for routing. Clustering is among one of the solutions for such type of issues. In this paper, the stability of IoV topology in a dynamic environment is focused. The proposed metaheuristic dragonfly-based clustering algorithm CAVDO is used for cluster-based packet route optimization to make stable topology, and mobility aware dynamic transmission range algorithm (MA-DTR) is used with CAVDO for transmission range adaptation on the basis of traffic density. The proposed CAVDO with MA-DTR is compared with the progressive baseline techniques ant colony optimization (ACO) and comprehensive learning particle swarm optimization (CLPSO). Numerous experiments were performed keeping in view the transmission dynamics for stable topology. CAVDO performed better in many cases providing minimum number of clusters according to current channel condition. Considerable important parameters involved in clustering process are: number of un-clustered nodes as a re-clustering criterion, clustering time, re-clustering delay, dynamic transmission range, direction, and speed. According to these parameters, results indicate that CAVDO outperformed ACO-based clustering and CLPSO in various network settings. Additionally, to improve the network availability and to incorporate the functionalities of next-generation network infrastructure, 5G-enabled architecture is also utilized.