Question: Traffic light identification in complex environments using integrated detection techniques.

This paper delves into the development of a sophisticated 3D object proposal method designed to enhance the precision of object class detection in complex environments. Initially, we outline the intricate architecture of our object detection network, which is crucial for understanding its operational capabilities. Subsequently, we detail our innovative road estimation technique, which plays a pivotal role in the detection process. Additionally, we introduce a class-agnostic version of our approach, expanding its utility across various object types. Our investigation includes an exhaustive evaluation of object detection and orientation estimation, complemented by recall statistics for both 2D and 3D bounding box proposals. To provide a holistic perspective, we present a series of successful detections alongside instances where the system encountered challenges, accompanied by a meticulous error analysis to identify and address the limitations of our object detection algorithm. This comprehensive analysis not only showcases the strengths of our approach but also highlights areas for future improvement.

Human sensing enables intelligent vehicles to provide driver-adaptive support by classifying perceived workload into multiple levels. Objective of this study is to classify driver workload associated with traffic complexity into five levels. We conducted driving experiments in systematically varied traffic complexity levels in a simulator. We recorded driver physiological signals including electrocardiography, electrodermal activity, and electroencephalography. In addition, we integrated driver performance and subjective workload measures. Deep learning based models outperform statistical machine learning methods when dealing with dynamic time-series data with variable sequence lengths. We show that our long short-term memory based recurrent neural network model can classify driver perceived-workload into five classes with an accuracy of 74.5%. Since perceived workload differ between individual drivers for the same traffic situation, our results further highlight the significance of including driver characteristics such as driving style and workload sensitivity to achieve higher classification accuracy.

Intersection management is one of the most challenging problems within the transport system. Traffic light-based methods have been efficient but are not able to deal with the growing mobility and social challenges. On the other hand, the advancements of automation and communications have enabled cooperative intersection management, where road users, infrastructure, and traffic control centers are able to communicate and coordinate the traffic safely and efficiently. Major techniques and solutions for cooperative intersections are surveyed in this paper for both signalized and nonsignalized intersections, whereas focuses are put on the latter. Cooperative methods, including time slots and space reservation, trajectory planning, and virtual traffic lights, are discussed in detail. Vehicle collision warning and avoidance methods are discussed to deal with uncertainties. Concerning vulnerable road users, pedestrian collision avoidance methods are discussed. In addition, an introduction to major projects related to cooperative intersection management is presented. A further discussion of the presented works is given with highlights of future research topics. This paper serves as a comprehensive survey of the field, aiming at stimulating new methods and accelerating the advancement of automated and cooperative intersections.

This paper presents an algorithm implemented in real time in a laboratory robot, designed to trace and detect the color of an object in an indoor environment using the Kalman filter. The image capture is done by a kinect camera installed in the robot. The algorithm is programmed into a single board computer, BeagleBone Black that uses the ROS — OpenCV system in image processing.

Many driver assistant and safety systems depend on an accurate environmental model containing the positions of stationary objects, the states of dynamic objects and information about valid driving corridors. Therefore, a robust differentiation between moving and stationary objects is required. This is challenging for laser scanners, because these sensors are not able to measure the velocity of objects directly. Therefore, an advanced occupancy grid approach, the online map, is introduced, which enables the robust separation of moving and stationary objects. The online map is used for the robust detection of the road boundaries for the determination of driving corridors in urban and highway scenarios. An algorithm for the detection of arbitrary moving objects using the online map is proposed.

This paper describes a machine learning approach for visual object detection which is capable of processing images extremely rapidly and achieving high detection rates. This work is distinguished by three key contributions. The first is the introduction of a new image representation called the “Integral Image” which allows the features used by our detector to be computed very quickly. The second is a learning algorithm, based on AdaBoost, which selects a small number of critical visual features from a larger set and yields extremely efficient classifiers[6]. The third contribution is a method for combining increasingly more complex classifiers in a “cascade” which allows background regions of the image to be quickly discarded while spending more computation on promising object-like regions. The cascade can be viewed as an object specific focus-of-attention mechanism which unlike previous approaches provides statistical guarantees that discarded regions are unlikely to contain the object of interest. In the domain of face detection the system yields detection rates comparable to the best previous systems. Used in real-time applications, the detector runs at 15 frames per second without resorting to image differencing or skin color detection.

This work investigated two different machine learning techniques: Cascade Learning and Deep Learning, to find out which algorithm performs better to detect the number plate of vehicles registered in Bangladesh. To do this, we created a dataset of about 1000 images collected from a security camera of Independent University, Bangladesh. Each image in the dataset were then labelled manually by selecting the Region of Interest (ROI). In the Cascade Learning approach, a sliding window technique was used to detect objects. Then a cascade classifier was employed to determine if the window contained object of interest or not. In the Deep Learning approach, CIFAR-10 dataset was used to pre-train a 15-layer Convolutional Neural Network (CNN). Using this pretrained CNN, a Regions with CNN (R-CNN) was then trained using our dataset. We found that the Deep Learning approach (maximum accuracy 99.60% using 566 training images) outperforms the detector constructed using Cascade classifiers (maximum accuracy 59.52% using 566 positive and 1022 negative training images) for 252 test images.

Enormous incorporation of information technologies into all aspects of recent life caused demand for processing vehicles as conceptual resources in information systems. This can be achieved by a human agent, or by special intelligent equipment which is be able to recognize vehicles by their number plates in a real environment and reflect it into conceptual resources. Because of this, various recognition techniques have been developed and number plate recognition systems are today used in various traffic and security applications, such as parking, access and border control, or tracking of stolen cars. This work deals with problematic from field of artificial intelligence, machine vision and neural networks in construction of an automatic car number plate Recognition (ACNPR). This techniques deals with enhanced shadow removal algorithm, along with horizontal and vertical projection method for number plate localization & segmentation..Work comparatively deals with methods achieving invariance of systems towards image illumination, translations and various light conditions during the capture.

Pedestrians are important participants in urban traffic environments, and thus act as an interesting category of objects for autonomous cars. Automatic pedestrian detection is an essential task for protecting pedestrians from collision. In this thesis, we investigate and develop novel approaches by interpreting spatial and temporal characteristics of pedestrians, in three different aspects: shape, cognition and motion. The special up-right human body shape, especially the geometry of the head and shoulder area, is the most discriminative characteristic for pedestrians from other object categories. Inspired by the success of Haar-like features for detecting human faces, which also exhibit a uniform shape structure, we propose to design particular Haar-like features for pedestrians. Tailored to a pre-defined statistical pedestrian shape model, Haar-like templates with multiple modalities are designed to describe local difference of the shape structure. Cognition theories aim to explain how human visual systems process input visual signals in an accurate and fast way. By emulating the center-surround mechanism in human visual systems, we design multi-channel, multi-direction and multi-scale contrast features, and boost them to respond to the appearance of pedestrians. In this way, our detector is considered as a top-down saliency system. In the last part of this thesis, we exploit the temporal characteristics for moving pedestrians and then employ motion information for feature design, as well as for regions of interest (ROIs) selection. Motion segmentation on optical flow fields enables us to select those blobs most probably containing moving pedestrians; a combination of Histogram of Oriented Gradients (HOG) and motion self difference features further enables robust detection. We test our three approaches on image and video data captured in urban traffic scenes, which are rather challenging due to dynamic and complex backgrounds. The achieved results demonstrate that our approaches reach and surpass state-of-the-art performance, and can also be employed for other applications, such as indoor robotics or public surveillance.

The knowledge about lanes and the exact position on the road is fundamental for many advanced driver assistance systems. In this paper, a novel iterative histogram based approach with occupancy grids for the detection of multiple lanes is proposed. In highway scenarios, our approach is highly suitable to determine the correct number of all existing lanes on the road. Additionally, the output of the laserscannner based lane detection is fused with a production-available vision based system. It is shown that both sensor systems perfectly complement each other to increase the robustness of a lane tracking system. The achieved accuracy of the fusion system, the laserscannner and video based system is evaluated with a highly accurate DGPS to investigate the performance with respect to lateral vehicle control applications.

A driver assistance system (DAS) should support the driver by monitoring road and vehicle events and presenting relevant and timely information to the driver. It is impossible to know what a driver is thinking, but we can monitor the driver’s gaze direction and compare it with the position of information in the driver’s viewfield to make inferences. In this way, not only do we monitor the driver’s actions, we monitor the driver’s observations as well. In this paper we present the automated detection and recognition of road signs, combined with the monitoring of the driver’s response. We present a complete system that reads speed signs in real-time, compares the driver’s gaze, and provides immediate feedback if it appears the sign has been missed by t ©


Question: Traffic light identification in intricate settings utilizing combined detection techniques.

This paper presents an innovative method for object detection that significantly reduces computational time without compromising accuracy. By leveraging the principle that multi-resolution image features can be efficiently approximated through extrapolation from nearby scales, rather than direct computation, we have developed a detection algorithm that outperforms state-of-the-art techniques in speed while maintaining high precision. The traditional bottleneck in object detection systems is the time-consuming process of computing features at every scale of a densely sampled image pyramid. Our novel approach circumvents this issue by demonstrating that computing features at octave-spaced scales is sufficient to accurately approximate those on a finely-sampled pyramid for a wide range of feature types. The extrapolation process is computationally less intensive compared to direct feature computation, resulting in substantial speedups with minimal impact on detection accuracy. We have successfully integrated our fast feature pyramid approximation into three distinct visual recognition systems and evaluated their performance on pedestrian detection tasks using the Caltech, INRIA, TUD-Brussels, and ETH data sets, as well as on general object detection using the PASCAL VOC data set. The results confirm that our method offers substantial speed improvements without compromising detection accuracy. This approach is versatile and can be widely applied to vision algorithms requiring fine-grained multi-scale analysis. It is important to note that the effectiveness of our approximation is contingent upon the broad spectral characteristics of the images, performing optimally on most natural images but potentially less accurately on images with narrow band-pass spectra, such as periodic textures.

Humans detect and identify faces in a scene with little or no effort . However, building an automated system that accomplishes this task is very difficult . There are several related Subproblems : detection of a pattern as a face . identification of the face, analysis of facial expressions, and classification based on physical features of the face . A system that performs these operations will find many applications, e .g . criminal identification, authentication in secure systems, etc . Most of the work to date has been in identification . This paper surveys the past work in solving these problems . The capability of the human visual system with respect to these problems is also discussed . It is meant to serve as a guide for an automated system . Some new approaches to these problems are also briefly discussed . Face detection Face identification Facial expressions Classification Facial features

With the increasing need for road lane detection used in lane departure warning systems and autonomous vehicles, many studies have been conducted to turn road lane detection into a virtual assistant to improve driving safety and reduce car accidents. Most of the previous research approaches detect the central line of a road lane and not the accurate left and right boundaries of the lane. In addition, they do not discriminate between dashed and solid lanes when detecting the road lanes. However, this discrimination is necessary for the safety of autonomous vehicles and the safety of vehicles driven by human drivers. To overcome these problems, we propose a method for road lane detection that distinguishes between dashed and solid lanes. Experimental results with the Caltech open database showed that our method outperforms conventional methods.

Coded structured light is considered one of the most reliable techniques for recovering the surface of objects. This technique is based on projecting a light pattern and viewing the illuminated scene from one or more points of view. Since the pattern is coded, correspondences between image points and points of the projected pattern can be easily found. The decoded points can be triangulated and 3D information is obtained. We present an overview of the existing techniques, as well as a new and de nitive classi cation of patterns for structured light sensors. We have implemented a set of representative techniques in this eld and present some comparative results. The advantages and constraints of the di5erent patterns are also discussed. ? 2003 Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved.

Person re-identification is about recognizing people who have passed by a sensor earlier. Previous work is mainly based on RGB data, but in this work we for the first time present a system where we combine RGB, depth, and thermal data for re-identification purposes. First, from each of the three modalities, we obtain some particular features: from RGB data, we model color information from different regions of the body, from depth data, we compute different soft body biometrics, and from thermal data, we extract local structural information. Then, the three information types are combined in a joined classifier. The tri-modal system is evaluated on a new RGB-D-T dataset, showing successful results in re-identification scenarios.

While the RGB, HSV and YUV (YCbCr) are standard models used in various colour imaging applications, not all of their information are necessary to classify skin colour. This paper presents a novel skin colour model, RGB-H-CbCr for the detection of human faces. Skin regions are extracted using a set of bounding rules based on the skin colour distribution obtained from a training set. The segmented face regions are further classified using a parallel combination of simple morphological operations. Experimental results on a large photo data set have demonstrated that the proposed model is able to achieve good detection success rates for near-frontal faces of varying orientations, skin colour and background environment. The results are also comparable to that of the AdaBoost face classifier.

We present a novel lane detection and tracking system using a fusion of Maximally Stable Extremal Regions (MSER) and Progressive Probabilistic Hough Transform (PPHT). First, MSER is applied to obtain a set of blobs including noisy pixels (e.g., trees, cars and traffic signs) and the candidate lane markings. A scanning refinement algorithm is then introduced to enhance the results of MSER and filter out noisy data. After that, to achieve the requirements of real-time systems, the PPHT is applied. Compared to Hough transform which returns the parameters ρ and Θ, PPHT returns two end-points of the detected line markings. To track lane markings, two kalman trackers are used to track both end-points. Several experiments are conducted in Ottawa roads to test the performance of our framework. The detection rate of the proposed system averages 92.7% and exceeds 84.9% in poor conditions.

Intelligent visual surveillance for road vehicles is the key to developing autonomous intelligent traffic systems. Recently, traffic incident detection employing computer vision and image processing has attracted much attention. In this paper, a probabilistic model for predicting traffic accidents using three-dimensional (3-D) model-based vehicle tracking is proposed. Sample data including motion trajectories are first obtained by 3-D model-based vehicle tracking. A fuzzy self-organizing neural network algorithm is then applied to learn activity patterns from the sample trajectories. Finally, vehicle activity is predicted by locating and matching each partial trajectory with the learned activity patterns, and the occurrence probability of a traffic accident is determined. Experiments show the effectiveness of the proposed algorithms.

In this article, the three most used object detection approaches, Linear Binary Pattern cascade, Haarlike cascade, and Histogram of Oriented Gradients with Support Vector Machine are applied to automatic runway detection in high resolution satellite imagery and their results are compared. They have been employed predominantly for human feature recognition and this paper tests theirs applicability to runways. The results show that they can be indeed employed for this purpose with LBP and Haar approaches performing better than HOG+SVM.

Enormous incorporation of information technologies into all aspects of recent life caused demand for processing vehicles as conceptual resources in information systems. This can be achieved by a human agent, or by special intelligent equipment which is be able to recognize vehicles by their number plates in a real environment and reflect it into conceptual resources. Because of this, various recognition techniques have been developed and number plate recognition systems are today used in various traffic and security applications, such as parking, access and border control, or tracking of stolen cars. This work deals with problematic from field of artificial intelligence, machine vision and neural networks in construction of an automatic car number plate Recognition (ACNPR). This techniques deals with enhanced shadow removal algorithm, along with horizontal and vertical projection method for number plate localization & segmentation..Work comparatively deals with methods achieving invariance of systems towards image illumination, translations and various light conditions during the capture.

• Lighting. When an object is in bright light, it looks brighter than when it’s in shadow, so a program can’t just look at image intensity values. • Within-class variation. Different instances of the same kind of object can look quite different to one another. For example, a green station wagon and a red convertible are both cars, so a program can’t simply compare a picture to one example. • Aspect. The same object can look very different when viewed at from different directions—pick up a book and compare its cover and its spine to see this effect. Again, this means that a program might need to have many examples of each type of object. • Deformation. Many objects can change their appearance significantly without their identity changing. For example, you can move your limbs around, change clothes, paint your face, or have your hair cut. You will look very different indeed, but you will still be a person.


Question: Traffic light identification in complex scenarios using integrated detection techniques.

This paper introduces a refined geodesic weighted Bayesian model designed for the precise detection of salient objects in complex scenes, addressing a common limitation in existing approaches. Typically, Bayesian-based salient object detection methods, while effective, overlook the importance of spatial relationships, leading to the inadvertent highlighting of background regions that bear a resemblance to the object of interest. To tackle this issue, we have developed a novel model that emphasizes regions more likely to be integral to the salient object, thereby effectively diminishing the prominence of irrelevant background areas. Our methodology involves several key steps: initially, we employ a combination of features to learn a comprehensive similarity measure that accurately assesses the likeness between adjacent regions. Subsequently, this combined similarity measure is utilized as the edge weight in an undirected weighted graph, facilitating the computation of geodesic distances. Finally, these geodesic distances are leveraged to weight the observation likelihood, enabling the inference of a more accurate saliency map. The efficacy of our proposed model is substantiated through comprehensive experiments conducted on multiple benchmark datasets, showcasing its superiority in navigating the complexities of salient object detection in intricate scenes.

License plate recognition(LPR)system contributed to facilitate the human work and make it easier. It is an image processing technology, which uses number plate to identify the vehicles. This paper aims to investigate in the existing LPR research by categorizing existing methods by analyzing the pros/cons of these features, and comparing them in terms of the algorithms that involve and recognition performance. We focus on the recent studies and their limitations, since each country has a different environment and format of licence plate.

Human Vision system is the most complex and accurate system. In order to extract better features about Synthetic Aperture Radar (SAR) targets, a SAR automatic target recognition (ATR) algorithm based on human visual cortical system is proposed. This algorithm contains three stages: (1) Image preprocessing (we use a Kuan filter to do the enhancement and an adaptive Intersecting Cortical Model (ICM) to do the segmentation) (2) Feature extraction using a sparse autoencoder. (3) Classification using a softmax regression classifier. Experiment result of MSTAR public data shows a better performance of recognition.

Future driver assistance systems are likely to use a multisensor approach with heterogeneous sensors for tracking dynamic objects around the vehicle. The quality and type of data available for a data fusion algorithm depends heavily on the sensors detecting an object. This article presents a general framework which allows the use sensor specific advantages while abstracting the specific details of a sensor. Different tracking models are used depending on the current set of sensors detecting the object. A sensor independent algorithm for classifying objects regarding their current and past movement state is presented. The described architecture and algorithms have been successfully implemented in Tartan racingpsilas autonomous vehicle for the urban grand challenge. Results are presented and discussed.

In this paper, we investigate the robustness of traffic sign recognition algorithms under challenging conditions. Existing datasets are limited in terms of their size and challenging condition coverage, which motivated us to generate the Challenging Unreal and Real Environments for Traffic Sign Recognition (CURE-TSR) dataset. It includes more than two million traffic sign images that are based on real-world and simulator data. We benchmark the performance of existing solutions in realworld scenarios and analyze the performance variation with respect to challenging conditions. We show that challenging conditions can decrease the performance of baseline methods significantly, especially if these challenging conditions result in loss or misplacement of spatial information. We also investigate the effect of data augmentation and show that utilization of simulator data along with real-world data enhance the average recognition performance in real-world scenarios. The dataset is publicly available at https://ghassanalregib.com/cure-tsr/.

Many Advanced Driver Assistance Systems (ADAS) have been developed to improve car safety. However, it is still a challenging problem to make autonomous vehicles to drive safely on urban streets such as uncontrolled intersections (without traffic lights) and narrow roads. In this paper, we introduce a decision making system that can assist autonomous vehicles at uncontrolled intersections and narrow roads. We constructed a machine understandable ontology-based Knowledge Base, which contains maps and traffic regulations. The system makes decisions in comply with traffic regulations such as Right-Of-Way rules when it receives a collision warning signal. The decisions are sent to a path planning system to change the route or stop to avoid collisions.

Vision-based object detection is one of the fundamental functions in numerous traffic scene applications such as self-driving vehicle systems and advance driver assistance systems (ADAS). However, it is also a challenging task due to the diversity of traffic scene and the storage, power and computing source limitations of the platforms for traffic scene applications. This paper presents a generalized Haar filter based deep network which is suitable for the object detection tasks in traffic scene. In this approach, we first decompose a object detection task into several easier local regression tasks. Then, we handle the local regression tasks by using several tiny deep networks which simultaneously output the bounding boxes, categories and confidence scores of detected objects. To reduce the consumption of storage and computing resources, the weights of the deep networks are constrained to the form of generalized Haar filter in training phase. Additionally, we introduce the strategy of sparse windows generation to improve the efficiency of the algorithm. Finally, we perform several experiments to validate the performance of our proposed approach. Experimental results demonstrate that the proposed approach is both efficient and effective in traffic scene compared with the state-of-the-art.

Support Vector Machine (SVM) classifier with Histogram of Oriented Gradients (HOG) feature become one of the most popular techniques used for vehicle detection in recent years. And the computing time of SVM is a main obstacle to get real time implementation which is important for Advanced Driver Assistance Systems (ADAS) applications. One of the effective ways to reduce the computing complexity of SVM is to reduce the dimension of HOG feature. In this paper, we examine the effect of the number of HOG bins on the vehicle detection and the symmetric characteristics of HOG feature of vehicle. And we successfully demonstrate the speed-up of SVM classifier for vehicle detection by about three times while maintaining the detection performance.

We present a novel lane detection and tracking system using a fusion of Maximally Stable Extremal Regions (MSER) and Progressive Probabilistic Hough Transform (PPHT). First, MSER is applied to obtain a set of blobs including noisy pixels (e.g., trees, cars and traffic signs) and the candidate lane markings. A scanning refinement algorithm is then introduced to enhance the results of MSER and filter out noisy data. After that, to achieve the requirements of real-time systems, the PPHT is applied. Compared to Hough transform which returns the parameters ρ and Θ, PPHT returns two end-points of the detected line markings. To track lane markings, two kalman trackers are used to track both end-points. Several experiments are conducted in Ottawa roads to test the performance of our framework. The detection rate of the proposed system averages 92.7% and exceeds 84.9% in poor conditions.

Color is an efficient feature for object detection as it has the advantage of being invariant to changes in scaling, rotation, and partial occlusion. Skin color detection is an essential required step in various applications related to computer vision. The rapidly-growing research in human skin detection is based on the premise that information about individuals, intent, mode, and image contents can be extracted from colored images, and computers can then respond in an appropriate manner. Detecting human skin in complex images has proven to be a challenging problem because skin color can vary dramatically in its appearance due to many factors such as illumination, race, aging, imaging conditions, and complex background. However, many methods have been developed to deal with skin detection problem in color images. The purpose of this study is to provide an up-to-date survey on skin color modeling and detection methods. We also discuss relevant issues such as color spaces, cost and risks, databases, testing, and benchmarking. After investigating these methods and identifying their strengths and limitations, we conclude with several implications for future direction.

This paper describes the obstacle detection and tracking algorithms developed for Boss, which is Carnegie Mellon University 's winning entry in the 2007 DARPA Urban Challenge. We describe the tracking subsystem and show how it functions in the context of the larger perception system. The tracking subsystem gives the robot the ability to understand complex scenarios of urban driving to safely operate in the proximity of other vehicles. The tracking system fuses sensor data from more than a dozen sensors with additional information about the environment to generate a coherent situational model. A novel multiple-model approach is used to track the objects based on the quality of the sensor data. Finally, the architecture of the tracking subsystem explicitly abstracts each of the levels of processing. The subsystem can easily be extended by adding new sensors and validation algorithms.


Question: Traffic Light Identification in Complex Scenarios using Fusion Detection Techniques

This paper presents an innovative incremental framework designed for video-based traffic sign detection, tracking, and recognition, a crucial component for advanced intelligent transport systems. While state-of-the-art methods, particularly those employing deep learning, have achieved commendable results on public data sets, they often demand substantial computational resources and primarily focus on single image tasks, limiting their practicality in real-world scenarios. To address these limitations, we propose a unified incremental computational framework, suitable for traffic sign detection, tracking, and recognition tasks using a monocular camera mounted on a moving vehicle in dynamic environments. The key contributions of this work are threefold: Firstly, to enhance detection accuracy, we introduce a novel approach that leverages the spatial distribution prior of traffic signs to incorporate contextual information. Secondly, to improve tracking performance and localization precision in non-stationary conditions, an efficient incremental framework is devised, integrating an off-line detector, an online detector, and a motion model predictor for concurrent detection and tracking of traffic signs. Lastly, to achieve more stable classification outcomes, we propose a scale-based intra-frame fusion technique. Our method is validated on two public data sets, demonstrating comparable results to deep learning methods but with reduced computational requirements and near-real-time processing capabilities.

The increasing availability of video data, through existing traffic cameras or dedicated field data collection, paves the way for the collection of massive datasets about the microscopic behaviour of road users using computer vision techniques. Analysis of such datasets helps to understand the normal road user behaviour, and it can be used for realistic prediction of future motion and computing surrogate safety indicators. A multi-level motion pattern learning framework is developed that enables automated scene interpretation, anomalous behaviour detection and surrogate safety analysis. Firstly, Points of interest (POI) are learnt based on Gaussian Mixture Model and the Expectation Maximization algorithm and then used to form activity paths (APs). Secondly, motion patterns, represented by trajectory prototypes, are learnt from road users’ trajectories in each AP using a two-stage trajectory clustering method based on spatial then temporal (speed) information. Finally, motion prediction relies on matching at each instant partial trajectories to the learnt prototypes to evaluate the potential for collision by computing indicators. An intersection case study demonstrates the framework ability in many ways: it helps to reduce the computation cost up to 90 %, clean trajectories dataset from tracking outliers, use actual trajectories as prototypes without any preand post-processing and predict future motion realistically to compute surrogate safety indicators.

This paper addresses automatic interpretation of images of outdoor scenes. The method allows instances of objects from a number of generic classes to be identi ed: vegetation, buildings, vehicles, roads, etc., thereby enabling image databases to be queried on scene content. The feature set is based, in part, on psychophysical principles and includes measures of colour, texture and shape. Using a large database of ground-truth labelled images, a neural network is trained as a pattern classi er. The method is demonstrated on a large test set to provide highly accurate image interpretations, with over 90% of the image area labelled correctly. Scene Understanding Object Recognition Classi cation Image Segmentation Neural Networks

Vehicles of higher automation levels require the creation of situation awareness. One important aspect of this situation awareness is an understanding of the current risk of a driving situation. In this work, we present a novel approach for the dynamic risk assessment of driving situations based on images of a front stereo camera using deep learning. To this end, we trained a deep neural network with recorded monocular images, disparity maps and a risk metric for diverse traffic scenes. Our approach can be used to create the aforementioned situation awareness of vehicles of higher automation levels and can serve as a heterogeneous channel to systems based on radar or lidar sensors that are used traditionally for the calculation of risk metrics.

This paper presents an e cient shape-based object detection method based on Distance Transforms and describes its use for real-time vision on-board vehicles. The method uses a template hierarchy to capture the variety of object shapes; e cient hierarchies can be generated o ine for given shape distributions using stochastic optimization techniques (i.e. simulated annealing). Online, matching involves a simultaneous coarse-tone approach over the shape hierarchy and over the transformation parameters. Very large speedup factors are typically obtained when comparing this approach with the equivalent brute-force formulation; we have measured gains of several orders of magnitudes. We present experimental results on the real-time detection of tra c signs and pedestrians from a moving vehicle. Because of the highly time sensitive nature of these vision tasks, we also discuss some hardwarespeci c implementations of the proposed method as far as SIMD parallelism is concerned.

The autonomous vehicle, as an emerging and rapidly growing field, has received extensive attention for its futuristic driving experiences. Although the fast developing depth sensors and machine learning methods have given a huge boost to self-driving research, existing autonomous driving vehicles do meet with several avoidable accidents during their road testings. The major cause is the misunderstanding between self-driving systems and human drivers. To solve this problem, we propose a humanlike driving system in this paper to give autonomous vehicles the ability to make decisions like a human. In our method, a convolutional neural network model is used to detect, recognize, and abstract the information in the input road scene, which is captured by the on-board sensors. And then a decision-making system calculates the specific commands to control the vehicles based on the abstractions. The biggest advantage of our work is that we implement a decision-making system which can well adapt to real-life road conditions, in which a massive number of human drivers exist. In addition, we build our perception system with only the depth information, rather than the unstable RGB data. The experimental results give a good demonstration of the efficiency and robustness of the proposed method.

Simultaneous target tracking and identification through feature association, attribute matching, or blob analysis is dependent on spatio-temporal measurements. Improved track maintenance should be achievable by maintaining coarse sensor resolutions on maneuvering targets and utilizing finer sensor resolutions to resolve closely-spaced targets. There are inherent optimal resolutions for sensors and restricted altitudes that constrain operational performance that a sensor manager must optimize for both wide-area surveillance and precision tracking. The advent of better optics, coordinated sensor management, and fusion strategies provide an opportunity to enhance simultaneous tracking and identification algorithms. We investigate utilizing electro-optical (EO) and infrared (IR) sensors operating at various resolutions to optimize target tracking and identification. We use a target-dense maneuvering scenario to highlight the performance gains with the multiresolution EO/IR data association (MEIDA) algorithm in tracking crossing targets.

Vision-based vehicle detection is an important issue for advanced driver assistance systems. In this paper, we presented an improved multi-vehicle detection and tracking method using cascade Adaboost and Adaptive Kalman filter(AKF) with target identity awareness. A cascade Adaboost classifier using Haar-like features was built for vehicle detection, followed by a more comprehensive verification process which could refine the vehicle hypothesis in terms of both location and dimension. In vehicle tracking, each vehicle was tracked with independent identity by an Adaptive Kalman filter in collaboration with a data association approach. The AKF adaptively adjusted the measurement and process noise covariance through on-line stochastic modelling to compensate the dynamics changes. The data association correctly assigned different detections with tracks using global nearest neighbour(GNN) algorithm while considering the local validation. During tracking, a temporal context based track management was proposed to decide whether to initiate, maintain or terminate the tracks of different objects, thus suppressing the sparse false alarms and compensating the temporary detection failures. Finally, the proposed method was tested on various challenging real roads, and the experimental results showed that the vehicle detection performance was greatly improved with higher accuracy and robustness.

The proliferation of the video surveillance cameras, which accounts for the vast majority of cameras worldwide, has resulted in the need to nd methods and algorithms for dealing with the huge amount of information that is gathered every second. This encompasses processing tasks, such as raising an alarm or detouring moving objects, as well as some semantic tasks like event monitoring, trajectory or ow analysis, counting people, etc.

At this moment Automous cars are probably the biggest and most talked about technology in the Robotics Reserach Community. In spite of great technolgical advances over past few years a full fledged autonomous car is still far from reality. This article talks about the existing system and discusses the possibility of a Computer Vision enabled driving being superior than the LiDar based system. A detailed overview of privacy violations that might arise from autonomous driving has been discussed in detail both from a technical as well as legal perspective. It has been proved through evidence and arguments that efficient and accurate estimation and efficient solution of the contraint satisfaction problem adressed in the case of autonomous cars are negatively correlated with the preserving the privacy of the user. It is a very difficult trade-off since both are very important aspects and has to be taken into account. The fact that one cannot compromise with the safety issues of the car makes it inevidable to run into serious privacy concerns that might have adverse social and political effects.

Bicycles that share the road with intelligent vehicles present particular challenges for automated perception systems. Bicycle detection is important because bicycles share the road with vehicles and can move at comparable speeds in urban environments. From a computer vision standpoint, bicycle detection is challenging as bicycle's appearance can change dramatically between viewpoints and a person riding on the bicycle is a non-rigid object. In this paper, we present a vision-based framework to detect and track bicycles that takes into account these issues. A mixture model of multiple viewpoints is defined and trained via a Support Vector Machine (SVM) to detect bicycles under a variety of circumstances. Each component of the model uses a part-based representation and known geometric context is used to improve overall detection efficiency. An extended Kalman filter (EKF) is used to estimate the position and velocity of the bicycle in vehicle coordinates. We demonstrate the effectiveness of this approach through a series of experiments run on video data of moving bicycles captured from a vehicle-mounted camera


Question: Traffic Light Identification in Complex Environments Using Fusion Detection Techniques

This document outlines a cutting-edge real-time traffic light recognition system specifically designed for integration with on-vehicle camera systems. The proposed method has been rigorously tested and has proven effective in urban environments, demonstrating its robustness and reliability. A key feature of this system is the utilization of "Adaptive Templates," a versatile tool that enables the recognition of diverse traffic light configurations found in different countries. This adaptability ensures that the system can be universally applied, enhancing its practicality and utility in a wide range of scenarios.

Classical object tracking approaches use a Kalman-filter with a single dynamic model which is therefore optimised to a single driving maneuver. In contrast the interacting multiple model (IMM) filter allows for several parallel models which are combined to a weighted estimate. Choosing models for different driving modes, such as constant speed, acceleration and strong acceleration changes, the object state estimation can be optimised for highly dynamic driving maneuvers. The paper describes the analysis of Stop&Go situations and the systematic parametrisation of the IMM method based on these statistics. The evaluation of the IMM approach is presented based on real sensor measurements of laser scanners, a radar and a video image processing unit.

An increasing safety and reducing road accidents, thereby saving lives are one of great interest in the context of Advanced Driver Assistance Systems. Apparently, among the complex and challenging tasks of future road vehicles is road lane detection or road boundaries detection. It is based on lane detection (which includes the localization of the road, the determination of the relative position between vehicle and road, and the analysis of the vehiclepsilas heading direction). One of the principal approaches to detect road boundaries and lanes using vision system on the vehicle. However, lane detection is a difficult problem because of the varying road conditions that one can encounter while driving. In this paper, a vision-based lane detection approach capable of reaching real time operation with robustness to lighting change and shadows is presented. The system acquires the front view using a camera mounted on the vehicle then applying few processes in order to detect the lanes. Using a pair of hyperbolas which are fitting to the edges of the lane, those lanes are extracted using Hough transform. The proposed lane detection system can be applied on both painted and unpainted road as well as curved and straight road in different weather conditions. This approach was tested and the experimental results show that the proposed scheme was robust and fast enough for real time requirements. Eventually, a critical overview of the methods were discussed, their potential for future deployment were assist.

Surveillance systems depend greatly on the robustness and availability of the video streams. The cameras must deliver reliable streams from an angle corresponding to the correct viewpoint. In other words, the field of view and video quality must remain unchanged after the initial installation of a surveillance camera. The paper proposes an approach to detect changes (such as displacement, out of focus, obstruction) automatically in a difficult environment with illumination variations and dynamic background and foreground objects.

Many Advanced Driver Assistance Systems (ADAS) have been developed to improve car safety. However, it is still a challenging problem to make autonomous vehicles to drive safely on urban streets such as uncontrolled intersections (without traffic lights) and narrow roads. In this paper, we introduce a decision making system that can assist autonomous vehicles at uncontrolled intersections and narrow roads. We constructed a machine understandable ontology-based Knowledge Base, which contains maps and traffic regulations. The system makes decisions in comply with traffic regulations such as Right-Of-Way rules when it receives a collision warning signal. The decisions are sent to a path planning system to change the route or stop to avoid collisions.

• Lighting. When an object is in bright light, it looks brighter than when it’s in shadow, so a program can’t just look at image intensity values. • Within-class variation. Different instances of the same kind of object can look quite different to one another. For example, a green station wagon and a red convertible are both cars, so a program can’t simply compare a picture to one example. • Aspect. The same object can look very different when viewed at from different directions—pick up a book and compare its cover and its spine to see this effect. Again, this means that a program might need to have many examples of each type of object. • Deformation. Many objects can change their appearance significantly without their identity changing. For example, you can move your limbs around, change clothes, paint your face, or have your hair cut. You will look very different indeed, but you will still be a person.

This paper describes an approach to overcome a situation of monitoring and managing a parking area using a vision based automated parking system. With the rapid increase of cars the need to find available parking space in the most efficient manner, to avoid traffic congestion in a parking area, is becoming a necessity in car park management. Current car park management is dependent on either human personnel keeping track of the available car park spaces or a sensor based system that monitors the availability of each car park space or the overall number of available car park spaces. In both situations, the information available was only the total number of car park spaces available and not the actual location available. In addition, the installation and maintenance cost of a sensor based system is dependent on the number of sensors used in a car park. This paper shows a vision based system that is able to detect and indicate the available parking spaces in a car park. The methods utilized to detect available car park spaces were based on coordinates to indicate the regions of interest and a car classifier. This paper shows that the initial work done here has an accuracy that ranges from 90% to 100% for a 4 space car park. The work done indicated that the application of a vision based car park management system would be able to detect and indicate the available car park spaces

MOVING OBJECT DETECTION, TRACKING AND CLASSIFICATION FOR SMART VIDEO SURVEILLANCE Yiğithan Dedeoğlu M.S. in Computer Engineering Supervisor: Assist. Prof. Dr. Uğur Güdükbay August, 2004 Video surveillance has long been in use to monitor security sensitive areas such as banks, department stores, highways, crowded public places and borders. The advance in computing power, availability of large-capacity storage devices and high speed network infrastructure paved the way for cheaper, multi sensor video surveillance systems. Traditionally, the video outputs are processed online by human operators and are usually saved to tapes for later use only after a forensic event. The increase in the number of cameras in ordinary surveillance systems overloaded both the human operators and the storage devices with high volumes of data and made it infeasible to ensure proper monitoring of sensitive areas for long times. In order to filter out redundant information generated by an array of cameras, and increase the response time to forensic events, assisting the human operators with identification of important events in video by the use of “smart” video surveillance systems has become a critical requirement. The making of video surveillance systems “smart” requires fast, reliable and robust algorithms for moving object detection, classification, tracking and activity analysis. In this thesis, a smart visual surveillance system with real-time moving object detection, classification and tracking capabilities is presented. The system operates on both color and gray scale video imagery from a stationary camera. It can handle object detection in indoor and outdoor environments and under changing illumination conditions. The classification algorithm makes use of the shape of the detected objects and temporal tracking results to successfully categorize objects into pre-defined classes like human, human group and vehicle. The system is also able to detect the natural phenomenon fire in various scenes reliably. The proposed tracking algorithm successfully tracks video objects even in full occlusion cases. In addition to these, some important needs of a robust iii

This paper demonstrates a generalised multi-scale feature learning approach to multi-class segmentation, applied to the estimation of fruit yield on treecrops. The learning approach makes the algorithm flexible and adaptable to different classification problems, and hence applicable to a wide variety of tree-crop applications. Extensive experiments were performed on a dataset consisting of 8000 colour images collected in an apple orchard. This paper shows that the algorithm was able to segment apples with different sizes and colours in an outdoor environment with natural lighting conditions, with a single model obtained from images captured using a monocular colour camera. The segmentation results are applied to the problem of fruit counting and the results are compared against manual counting. The results show a squared correlation coefficient of R2 = 0.81.

Vision-based vehicle detection is an important issue for advanced driver assistance systems. In this paper, we presented an improved multi-vehicle detection and tracking method using cascade Adaboost and Adaptive Kalman filter(AKF) with target identity awareness. A cascade Adaboost classifier using Haar-like features was built for vehicle detection, followed by a more comprehensive verification process which could refine the vehicle hypothesis in terms of both location and dimension. In vehicle tracking, each vehicle was tracked with independent identity by an Adaptive Kalman filter in collaboration with a data association approach. The AKF adaptively adjusted the measurement and process noise covariance through on-line stochastic modelling to compensate the dynamics changes. The data association correctly assigned different detections with tracks using global nearest neighbour(GNN) algorithm while considering the local validation. During tracking, a temporal context based track management was proposed to decide whether to initiate, maintain or terminate the tracks of different objects, thus suppressing the sparse false alarms and compensating the temporary detection failures. Finally, the proposed method was tested on various challenging real roads, and the experimental results showed that the vehicle detection performance was greatly improved with higher accuracy and robustness.

Sudden pedestrian crossing (SPC) is the major reason for pedestrian-vehicle crashes. In this paper, we focus on detecting SPCs at night for supporting an advanced driver assistance system using a far-infrared (FIR) camera mounted on the front roof of a vehicle. Although the thermal temperature of the road is similar to or higher than that of the pedestrians during summer nights, many previous researches have focused on pedestrian detection during the winter, spring, or autumn seasons. However, our research concentrates on SPC during the hot summer season because the number of collisions between pedestrians and vehicles in Korea is higher at that time than during the other seasons. For real-time processing, we first decide the optimal levels of image scaling and search area. We then use our proposed method for detecting virtual reference lines that are associated with road segmentation without using color information and change these lines according to the turning direction of the vehicle. Pedestrian detection is conducted using a cascade random forest with low-dimensional Haar-like features and oriented center-symmetric local binary patterns. The SPC is predicted based on the likelihood and the spatiotemporal features of the pedestrians, such as their overlapping ratio with virtual reference lines, as well as the direction and magnitude of each pedestrian’s movement. The proposed algorithm was successfully applied to various pedestrian data sets captured by an FIR camera, and the results show that its SPC detection performance is better than those of other methods.


Question: Categorizing sentences from various viewpoints using a Category Expert Attention Network.

This paper outlines a convolutional neural network architecture, specifically the Dynamic Convolutional Neural Network (DCNN), tailored for the semantic analysis of sentences. Central to this network is the Dynamic k-Max Pooling mechanism, which enables global pooling over linear sequences, effectively processing sentences of variable lengths. This architecture facilitates the construction of a feature graph that adeptly captures both short and long-range dependencies within sentences, without the need for a parse tree, thus making it universally applicable across languages. The DCNN's efficacy is demonstrated through a series of experiments: binary and multi-class sentiment prediction on a small scale, six-way question classification, and sentiment prediction on Twitter data using distant supervision. In these tasks, the DCNN exhibits outstanding performance, notably achieving a more than 25% reduction in error rate compared to the strongest baseline in the Twitter sentiment prediction experiment.

Andor, D., Alberti, C., Weiss, D., Severyn, A., Presta, A., Ganchev, K., Petrov, S., and Collins, M. (2016). Globally normalized transition-based neural networks. In Association for Computational Linguistics. Ballesteros, M., Goldberg, Y., Dyer, C., and Smith, N. A. (2016). Training with exploration improves a greedy stack-LSTM parser. Proceedings of the conference on empirical methods in natural language processing. Chen, D. and Manning, C. D. (2014). A fast and accurate dependency parser using neural networks. In Proceedings of the conference on empirical methods in natural language processing, pages 740–750. Cheng, H., Fang, H., He, X., Gao, J., and Deng, L. (2016). Bi-directional attention with agreement for dependency parsing. arXiv preprint arXiv:1608.02076. Hashimoto, K., Xiong, C., Tsuruoka, Y., and Socher, R. (2016). A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01587. Kiperwasser, E. and Goldberg, Y. (2016). Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4:313–327. Kuncoro, A., Ballesteros, M., Kong, L., Dyer, C., Neubig, G., and Smith, N. A. (2016). What do recurrent neural network grammars learn about syntax? CoRR, abs/1611.05774. McDonald, R. T. and Pereira, F. C. (2006). Online learning of approximate dependency parsing algorithms. In EACL. Nivre, J., Hall, J., and Nilsson, J. (2006). Maltparser: A data-driven parser-generator for dependency parsing. In Proceedings of LREC, volume 6, pages 2216–2219.

Traditional machine learning techniques, including support vector machine (SVM), random walk, and so on, have been applied in various tasks of text sentiment analysis, which makes poor generalization ability in terms of complex classification problem. In recent years, deep learning has made a breakthrough in the research of Natural Language Processing. Convolutional neural network (CNN) and recurrent neural networks (RNNs) are two mainstream methods of deep learning in document and sentence modeling. In this paper, a model of capturing deep sentiment representation based on CNN and long short-term memory recurrent neural network (LSTM) is proposed. The model uses the pre-trained word vectors as input and employs CNN to gain significant local features of the text, then features are fed to two-layer LSTMs, which can extract context-dependent features and generate sentence representation for sentiment classification. We evaluate the proposed model by conducting a series of experiments on dataset. The experimental results show that the model we designed outperforms existing CNN, LSTM, CNN-LSTM (our implement of one-layer LSTM directly stacked on one-layer CNN) and SVM (support vector machine).

This paper describes the PASCAL Network of Excellence first Recognising Textual Entailment (RTE-1) Challenge benchmark. The RTE task is defined as recognizing, given two text fragments, whether the meaning of one text can be inferred (entailed) from the other. This application-independent task is suggested as capturing major inferences about the variability of semantic expression which are commonly needed across multiple applications. The Challenge has raised noticeable attention in the research community, attracting 17 submissions from diverse groups, suggesting the generic relevance of the task.

This article introduces a sentiment analysis approach that adopts the way humans read, interpret, and extract sentiment from text. Our motivation builds on the assumption that human interpretation should lead to the most accurate assessment of sentiment in text. We call this automated process Human Reading for Sentiment (HRS). Previous research in sentiment analysis has produced many frameworks that can fit one or more of the HRS aspects; however, none of these methods has addressed them all in one approach. HRS provides a meta-framework for developing new sentiment analysis methods or improving existing ones. The proposed framework provides a theoretical lens for zooming in and evaluating aspects of any sentiment analysis method to identify gaps for improvements towards matching the human reading process. Key steps in HRS include the automation of humans low-level and high-level cognitive text processing. This methodology paves the way towards the integration of psychology with computational linguistics and machine learning to employ models of pragmatics and discourse analysis for sentiment analysis. HRS is tested with two state-of-the-art methods; one is based on feature engineering, and the other is based on deep learning. HRS highlighted the gaps in both methods and showed improvements for both.

Aspect-level sentiment classification aims at identifying the sentiment polarity of specific target in its context. Previous approaches have realized the importance of targets in sentiment classification and developed various methods with the goal of precisely modeling their contexts via generating target-specific representations. However, these studies always ignore the separate modeling of targets. In this paper, we argue that both targets and contexts deserve special treatment and need to be learned their own representations via interactive learning. Then, we propose the interactive attention networks (IAN) to interactively learn attentions in the contexts and targets, and generate the representations for targets and contexts separately. With this design, the IAN model can well represent a target and its collocative context, which is helpful to sentiment classification. Experimental results on SemEval 2014 Datasets demonstrate the effectiveness of our model.

Natural language processing systems have the ability to analyse not only the sentiment of human language, but also the stance of the speaker. Representing this information visually from unevenly distributed and potentially sparse datasets is challenging, in particular when trying to facilitate exploration and knowledge discovery. We present work on a novel visualisation approach for scalable visualisation of sentiment and stance and provide a language resource of e-government public engagement of 9,278 user comments with stance explicitly declared by the author.

Grammatical error correction (GEC) is the task of automatically correcting grammatical errors in written text. Previous research has mainly focussed on individual error types and current commercial proofreading tools only target limited error types. As sentences produced by learners may contain multiple errors of different types, a practical error correction system should be able to detect and correct all errors. In this thesis, we investigate GEC for learners of English as a Second Language (ESL). Specifically, we treat GEC as a translation task from incorrect into correct English, explore new models for developing end-to-end GEC systems for all error types, study system performance for each error type, and examine model generali-sation to different corpora. First, we apply Statistical Machine Translation (SMT) to GEC and prove that it can form the basis of a competitive all-errors GEC system. We implement an SMT-based GEC system which contributes to our winning system submitted to a shared task in 2014. Next, we propose a ranking model to re-rank correction candidates generated by an SMT-based GEC system. This model introduces new linguistic information and we show that it improves correction quality. Finally, we present the first study using Neural Machine Translation (NMT) for GEC. We demonstrate that NMT can be successfully applied to GEC and help capture new errors missed by an SMT-based GEC system. While we focus on GEC for English, our methods presented in this thesis can be easily applied to any language. Acknowledgements First and foremost, I owe a huge debt of gratitude to my supervisor, Ted Briscoe, who has patiently guided me through my PhD and always been very helpful, understanding and supportive. I cannot thank him enough for providing me with opportunities that helped me grow as a researcher and a critical thinker. I am immensely grateful to my examiners, Paula Buttery and Stephen Pulman, for their thorough reading of my thesis, their valuable comments and an enjoyable viva. My appreciation extends to my fellow members of the Natural Language and Information Processing research group, with whom I have always enjoyed discussing our work and other random things. My gratitude goes to Stephen Clark and Ann Copestake for giving me early feedback on my work as well as Christopher Bryant for generously reading my thesis draft. I would especially like to thank Mariano Felice for being not just a great colleague but also a dear friend. A special mention has …

We apply Support Vector Machines (SVMs) to identify English base phrases (chunks). SVMs are known to achieve high generalization performance even with input data of high dimensional feature spaces. Furthermore, by the Kernel principle, SVMs can carry out training with smaller computational overhead independent of their dimensionality. We apply weighted voting of 8 SVMsbased systems trained with distinct chunk representations. Experimental results show that our approach achieves higher accuracy than previous approaches.

Languages such as Chinese and Japanese have a significantly large number (several thousands) of alphabets as compared to other languages, and each of their sentences consists of several concatenated words with wide varieties of inflected forms; thus appropriate word segmentation is quite difficult. Therefore, recently proposed sophisticated language-processing methods designed for languages such as English cannot be applied. In this paper, we address those issues and propose a new and efficient document classification technique for such languages. The proposed method is characterized into a new “image-based character embedding” method and character-level convolutional neural networks method with “wildcard training.” The first method encodes each character based on its pictorial structures and preserves them. Further, the second method treats some of the input characters as wildcards in the classification stage and functions as efficient data augmentation. We confirmed that our proposed method showed superior performance when compared conventional methods for Japanese document classification problems.

A common feature of many online review sites is the use of an overall rating that summarizes the opinions expressed in a review. Unfortunately, these document-level ratings do not provide any information about the opinions contained in the review that concern a specific aspect (e.g., cleanliness) of the product being reviewed (e.g., a hotel). In this paper we study the finer-grained problem of aspect-oriented opinion mining at the sentence level, which consists of predicting, for all sentences in the review, whether the sentence expresses a positive, neutral, or negative opinion (or no opinion at all) about a specific aspect of the product. For this task we propose a set of increasingly powerful models based on conditional random fields (CRFs), including a hierarchical multi-label CRFs scheme that jointly models the overall opinion expressed in the review and the set of aspect-specific opinions expressed in each of its sentences. We evaluate the proposed models against a dataset of hotel reviews (which we here make publicly available) in which the set of aspects and the opinions expressed concerning them are manually annotated at the sentence level. We find that both hierarchical and multi-label factors lead to improved predictions of aspect-oriented opinions.


Question: Categorize sentences from various viewpoints using a Category Expert Attention Network.

This paper presents a novel global logbilinear regression model for generating word vectors that captures both semantic and syntactic regularities, addressing the obscurity surrounding the emergence of these patterns in previous models. Our analysis highlights the essential model features necessary for the manifestation of such regularities within word vector spaces. Inspired by this insight, we introduce a new model that synthesizes the strengths of two predominant approaches in the literature: global matrix factorization and local context window techniques. Unlike traditional models that either operate on the entire sparse matrix or individual context windows within a large corpus, our approach optimizes computational efficiency by focusing solely on the non-zero elements of a word-word co-occurrence matrix. This innovative strategy allows the model to effectively harness statistical information, resulting in a vector space that exhibits meaningful substructure. Our model's efficacy is substantiated by its impressive performance on word analogy tasks, achieving a 75% accuracy rate, and outperforming competing models in similarity tasks and named entity recognition. This breakthrough offers a deeper understanding of how regularities arise in word vectors and provides a more efficient and accurate method for their representation.

This article introduces a sentiment analysis approach that adopts the way humans read, interpret, and extract sentiment from text. Our motivation builds on the assumption that human interpretation should lead to the most accurate assessment of sentiment in text. We call this automated process Human Reading for Sentiment (HRS). Previous research in sentiment analysis has produced many frameworks that can fit one or more of the HRS aspects; however, none of these methods has addressed them all in one approach. HRS provides a meta-framework for developing new sentiment analysis methods or improving existing ones. The proposed framework provides a theoretical lens for zooming in and evaluating aspects of any sentiment analysis method to identify gaps for improvements towards matching the human reading process. Key steps in HRS include the automation of humans low-level and high-level cognitive text processing. This methodology paves the way towards the integration of psychology with computational linguistics and machine learning to employ models of pragmatics and discourse analysis for sentiment analysis. HRS is tested with two state-of-the-art methods; one is based on feature engineering, and the other is based on deep learning. HRS highlighted the gaps in both methods and showed improvements for both.

We propose a sentence generation strategy that describes images by predicting the most likely nouns, verbs, scenes and prepositions that make up the core sentence structure. The input are initial noisy estimates of the objects and scenes detected in the image using state of the art trained detectors. As predicting actions from still images directly is unreliable, we use a language model trained from the English Gigaword corpus to obtain their estimates; together with probabilities of co-located nouns, scenes and prepositions. We use these estimates as parameters on a HMM that models the sentence generation process, with hidden nodes as sentence components and image detections as the emissions. Experimental results show that our strategy of combining vision and language produces readable and descriptive sentences compared to naive strategies that use vision alone.

Answer selection in community question answering (cQA) is a challenging task in natural language processing. The difficulty lies in that it not only needs the consideration of semantic matching between question answer pairs but also requires a serious modeling of contextual factors. In this letter, we propose an attentive deep neural network architecture so as to learn the deterministic information for answer selection. The architecture can support various input formats through the organization of convolutional neural networks, attention-based long short-term memory, and conditional random fields. Experiments are carried out on the SemEval-2015 cQA dataset. We attain 58.35% on macroaveraged $\mathrm{F_1}$, which outperforms the Top-1 system in the shared task by 1.16% and improves the state-of-the-art deep-neural-network-based method by 2.21%.

Deep models are the defacto standard in visual decision problems due to their impressive performance on a wide array of visual tasks. However, they are frequently seen as opaque and are unable to explain their decisions. In contrast, humans can justify their decisions with natural language and point to the evidence in the visual world which supports their decisions. We propose a method which incorporates a novel explanation attention mechanism; our model is trained using textual rationals, and infers latent attention to visually ground explanations. We collect two novel datasets in domains where it is interesting and challenging to explain decisions. First, we extend the visual question answering task to not only provide an answer but also visual and natural language explanations for the answer. Second, we focus on explaining human activities in a contemporary activity recognition dataset. We extensively evaluate our model, both on the justification and pointing tasks, by comparing it to prior models and ablations using both automatic and human evaluations.

In this paper we propose to solve the problem of Visual Question Answering by using a novel segmentation guided attention based network which we call SegAttendNet. We use image segmentation maps, generated by a Fully Convolutional Deep Neural Network to refine our attention maps and use these refined attention maps to make the model focus on the relevant parts of the image to answer a question. The refined attention maps are used by the LSTM network to learn to produce the answer. We presently train our model on the visual7W dataset and do a category wise evaluation of the 7 question categories. We achieve state of the art results on this dataset and beat the previous benchmark on this dataset by a 1.5% margin improving the question answering accuracy from 54.1% to 55.6% and demonstrate improvements in each of the question categories. We also visualize our generated attention maps and note their improvement over the attention maps generated by the previous best approach.

This paper introduces a new general-purpose sentiment lexicon called WKWSCI Sentiment Lexicon, and compares it with five existing lexicons: Hu & Liu Opinion Lexicon, MPQA Subjectivity Lexicon, General Inquirer, NRC Word-Sentiment Association Lexicon and SO-CAL lexicon. The effectiveness of the sentiment lexicons for sentiment categorization at the document-level and sentence-level was evaluated using an Amazon product review dataset and a news headlines dataset. WKWSCI, MPQA, Hu & Liu and SO-CAL lexicons are equally good for product review sentiment categorization, obtaining accuracy rates of 75% to 77% when appropriate weights are used for different categories of sentiment words. However, when a training corpus is not available, Hu & Liu obtained the best accuracy with a simple-minded approach of counting positive and negative words, for both document-level and sentencelevel sentiment categorization. The WKWSCI lexicon obtained the best accuracy of 69% on the news headlines sentiment categorization task, and the sentiment strength values obtained a Pearson correlation of 0.57 with human assigned sentiment values. It is recommended that the Hu & Liu lexicon be used for product review texts, and the WKWSCI lexicon for non-review texts.

Automatically generating a natural language description of an image is a task close to the heart of image understanding. In this paper, we present a multi-model neural network method closely related to the human visual system that automatically learns to describe the content of images. Our model consists of two sub-models: an object detection and localization model, which extract the information of objects and their spatial relationship in images respectively; Besides, a deep recurrent neural network (RNN) based on long short-term memory (LSTM) units with attention mechanism for sentences generation. Each word of the description will be automatically aligned to different objects of the input image when it is generated. This is similar to the attention mechanism of the human visual system. Experimental results on the COCO dataset showcase the merit of the proposed method, which outperform previous benchmark models.

This paper proposes method for aspect identification and sentiment analysis. This paper deals with the problem of sentence level. We use semantic syntax tree to extract NP nodes, a method based on slack to identify aspect and mini-support function to narrow aspect; to analyze the sentiment for the aspect we use dictionary-based method. Experimental results show that the proposed approach achieves the state-of-the-art.

This paper describes our system submission to the CogALex-2016 Shared Task on Corpus-Based Identification of Semantic Relations. Our system won first place for Task-1 and second place for Task-2. The evaluation results of our system on the test set is 88.1% (79.0% for TRUE only) f-measure for Task-1 on detecting semantic similarity, and 76.0% (42.3% when excluding RANDOM) for Task-2 on identifying fine-grained semantic relations. In our experiments, we try word analogy, linear regression, and multi-task Convolutional Neural Networks (CNNs) with word embeddings from publicly available word vectors. We found that linear regression performs better in the binary classification (Task-1), while CNNs have better performance in the multi-class semantic classification (Task-2). We assume that word analogy is more suited for deterministic answers rather than handling the ambiguity of one-to-many and many-to-many relationships. We also show that classifier performance could benefit from balancing the distribution of labels in the training data.

Accurate dependency recovery has recently been reported for a number of wide-coverage statistical parsers using Combinatory Categorial Grammar (CCG). However, overall figures give no indication of a parser’s performance on specific constructions, nor how suitable a parser is for specific applications. In this paper we give a detailed evaluation of a CCG parser on object extraction dependencies found in WSJ text. We also show how the parser can be used to parse questions for Question Answering. The accuracy of the original parser on questions is very poor, and we propose a novel technique for porting the parser to a new domain, by creating new labelled data at the lexical category level only. Using a supertagger to assign categories to words, trained on the new data, leads to a dramatic increase in question parsing accuracy.


Question: Categorize sentences from various angles using a category expert attention network.

This document explores the nuanced task of rating inference, a step beyond binary sentiment analysis, where the objective is to accurately gauge an author's sentiment across a multi-point scale, akin to a one to five "star" rating system. Unlike traditional multi-class text categorization, this task is characterized by varying degrees of similarity between class labels. For instance, "three stars" is conceptually closer to "four stars" than to "one star." To assess the complexity of the task, we first examine human performance. Subsequently, we employ a meta-algorithm, grounded in a metric labeling formulation, designed to adjust an n-ary classifier's output. This adjustment is strategically aimed at ensuring that reviews with similar content receive labels that reflect their proximity on the rating scale. Utilizing a tailored similarity measure, we demonstrate that our meta-algorithm outperforms both multi-class and regression versions of Support Vector Machines (SVMs), yielding substantial improvements. This research was published in the Proceedings of the ACL in 2005.

Natural language inference aims to predict whether a premise sentence can infer another hypothesis sentence. Recent progress on this task only relies on a shallow interaction between sentence pairs, which is insufficient for modeling complex relations. In this paper, we present an attention-fused deep matching network (AF-DMN) for natural language inference. Unlike existing models, AF-DMN takes two sentences as input and iteratively learns the attention-aware representations for each side by multi-level interactions. Moreover, we add a selfattention mechanism to fully exploit local context information within each sentence. Experiment results show that AF-DMN achieves state-of-the-art performance and outperforms strong baselines on Stanford natural language inference (SNLI), multigenre natural language inference (MultiNLI), and Quora duplicate questions datasets.

Community question answering aims at choosing the most appropriate answer for a given question, which is important in many NLP applications. Previous neural network-based methods consider several different aspects of information through calculating attentions. These different kinds of attentions are always simply summed up and can be seen as a “single view”, causing severe information loss. To overcome this problem, we propose a Multi-View Fusion Neural Network, where each attention component generates a “view” of the QA pair and a fusion RNN integrates the generated views to form a more holistic representation. In this fusion RNN method, a filter gate collects important information of input and directly adds it to the output, which borrows the idea of residual networks. Experimental results on the WikiQA and SemEval-2016 CQA datasets demonstrate that our proposed model outperforms the state-of-the-art methods.

The last decade has witnessed the success of traditional feature-based methods for exploiting the discrete structures such as words or lexical patterns to extract relations from text. Recently, convolutional and recurrent neural networks have been shown to capture effective hidden structures within sentences via continuous representations, thereby significantly advancing the performance of relation extraction. The advantage of convolutional neural networks is their capacity to generalize the consecutive k-grams in the sentences while recurrent neural networks are effective to encode long range sentence context. This paper proposes to combine the traditional feature-based method, the convolutional and recurrent neural networks to simultaneously benefit from their advantages. Our systematic evaluation of different network architectures and combination methods demonstrates the effectiveness of this approach and results in the state-ofthe-art performance on the ACE 2005 and SemEval datasets.

Classifying short texts to one category or clustering semantically related texts is challenging, and the importance of both is growing due to the rise of microblogging platforms, digital news feeds, and the like. We can accomplish this classifying and clustering with the help of a deep neural network which produces compact binary representations of a short text, and can assign the same category to texts that have similar binary representations. But problems arise when there is little contextual information on the short texts, which makes it difficult for the deep neural network to produce similar binary codes for semantically related texts. We propose to address this issue using semantic enrichment. This is accomplished by taking the nouns, and verbs used in the short texts and generating the concepts and co-occurring words with the help of those terms. The nouns are used to generate concepts within the given short text, whereas the verbs are used to prune the ambiguous context (if any) present in the text. The enriched text then goes through a deep neural network to produce a prediction label for that short text representing it’s category.

As the amount of unstructured text data that humanity produces overall and on the Internet grows, so does the need to intelligently to process it and extract different types of knowledge from it. Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have been applied to natural language processing systems with comparative, remarkable results. The CNN is a noble approach to extract higher level features that are invariant to local translation. However, it requires stacking multiple convolutional layers in order to capture long-term dependencies, due to the locality of the convolutional and pooling layers. In this paper, we describe a joint CNN and RNN framework to overcome this problem. Briefly, we use an unsupervised neural language model to train initial word embeddings that are further tuned by our deep learning network, then, the pre-trained parameters of the network are used to initialize the model. At a final stage, the proposed framework combines former information with a set of feature maps learned by a convolutional layer with long-term dependencies learned via long-short-term memory. Empirically, we show that our approach, with slight hyperparameter tuning and static vectors, achieves outstanding results on multiple sentiment analysis benchmarks. Our approach outperforms several existing approaches in term of accuracy; our results are also competitive with the state-of-the-art results on the Stanford Large Movie Review data set with 93.3% accuracy, and the Stanford Sentiment Treebank data set with 48.8% fine-grained and 89.2% binary accuracy, respectively. Our approach has a significant role in reducing the number of parameters and constructing the convolutional layer followed by the recurrent layer as a substitute for the pooling layer. Our results show that we were able to reduce the loss of detailed, local information and capture long-term dependencies with an efficient framework that has fewer parameters and a high level of performance.

Attention-based recurrent neural network models for joint intent detection and slot filling have achieved the state-of-the-art performance, while they have independent attention weights. Considering that slot and intent have the strong relationship, this paper proposes a slot gate that focuses on learning the relationship between intent and slot attention vectors in order to obtain better semantic frame results by the global optimization. The experiments show that our proposed model significantly improves sentence-level semantic frame accuracy with 4.2% and 1.9% relative improvement compared to the attentional model on benchmark ATIS and Snips datasets respectively1.

This study proposes a new neural machine translation (NMT) model based on the encoderdecoder model that incorporates named entity (NE) tags of source-language sentences. Conventional NMT models have two problems enumerated as follows: (i) they tend to have difficulty in translating words with multiple meanings because of the high ambiguity, and (ii) these models’ ability to translate compound words seems challenging because the encoder receives a word, a part of the compound word, at each time step. To alleviate these problems, the encoder of the proposed model encodes the input word on the basis of its NE tag at each time step, which could reduce the ambiguity of the input word. Furthermore, the encoder introduces a chunk-level LSTM layer over a word-level LSTM layer and hierarchically encodes a source-language sentence to capture a compound NE as a chunk on the basis of the NE tags. We evaluate the proposed model on an English-to-Japanese translation task with the ASPEC, and English-to-Bulgarian and English-to-Romanian translation tasks with the Europarl corpus. The evaluation results show that the proposed model achieves up to 3.11 point improvement in BLEU.

An obstacle to the development of many natural language processing products is the vast amount of training examples necessary to get satisfactory results. The generation of these examples is often a tedious and timeconsuming task. This paper this paper proposes a method to transform the sentiment of sentences in order to limit the work necessary to generate more training data. This means that one sentence can be transformed to an opposite sentiment sentence and should reduce by half the work required in the generation of text. The proposed pipeline consists of a sentiment classifier with an attention mechanism to highlight the short phrases that determine the sentiment of a sentence. Then, these phrases are changed to phrases of the opposite sentiment using a baseline model and an autoencoder approach. Experiments are run on both the separate parts of the pipeline as well as on the end-to-end model. The sentiment classifier is tested on its accuracy and is found to perform adequately. The autoencoder is tested on how well it is able to change the sentiment of an encoded phrase and it was found that such a task is possible. We use human evaluation to judge the performance of the full (end-to-end) pipeline and that reveals that a model using word vectors outperforms the encoder model. Numerical evaluation shows that a success rate of 54.7% is achieved on the sentiment change.

This paper studies the automated categorization and extraction of scientific concepts from titles of scientific articles, in order to gain a deeper understanding of their key contributions and facilitate the construction of a generic academic knowledgebase. Towards this goal, we propose an unsupervised, domain-independent, and scalable two-phase algorithm to type and extract key concept mentions into aspects of interest (e.g., Techniques, Applications, etc.). In the first phase of our algorithm we proposePhraseType, a probabilistic generative model which exploits textual features and limited POS tags to broadly segment text snippets into aspect-typed phrases. We extend this model to simultaneously learn aspect-specific features and identify academic domains in multi-domain corpora, since the two tasks mutually enhance each other. In the second phase, we propose an approach based on adaptor grammars to extract fine grained concept mentions from the aspect-typed phrases without the need for any external resources or human effort, in a purely data-driven manner. We apply our technique to study literature from diverse scientific domains and show significant gains over state-of-the-art concept extraction techniques. We also present a qualitative analysis of the results obtained.

Sentiment analysis using deep learning models is a leading subject of interest in Natural Language Processing that is as powerful as it is opaque. Current state-ofthe-art models can produce accurate predictions, but they provide little insight as to why the model predicted this sentiment. Businesses relying on these models might be less likely to act on insight given the lack of evidence for predictions. These people would be more likely to trust such predictions if a brief explanation of the outcome is provided. Recent work by Lei et al [4]. has set forth a framework for a multi-aspect sentiment analysis concurrently providing text rationalization with each prediction. This framework sets forth a two-part approach, which summarizes a review and predicts a sentiment. In this paper, we explore the performance of this framework, seeking to recreate and improve upon it in TensorFlow.