Question: Modeling and performance evaluation of millimeter wave cellular networks.

The IEEE 802.11ad: Paving the Way for Directional 60 GHz Communication and Multi-Gigabit-per-Second Wi-Fi [Invited Paper]

Following the December 2012 ratification of the IEEE 802.11ad amendment to the 802.11 standard, a significant advancement has been made in bringing consumer wireless communication into the millimeter wave (mm-Wave) frequency range. The promise of multi-gigabit-per-second data rates and minimized interference is achieved with a trade-off: signal propagation characteristics become more challenging, necessitating a fundamental reevaluation of traditional Wi-Fi communication paradigms. This article delves into the design considerations that informed the development of the IEEE 802.11ad standard and the innovative techniques introduced to tackle the complexities of mm-Wave communication. Specifically, we explore the shift from omnidirectional to highly directional communication and its implications for the IEEE 802.11ad standard's design.

In the realm of mm-Wave communication, the IEEE 802.11ad standard marks a pivotal moment, introducing directional 60 GHz communication to achieve multi-gigabit-per-second Wi-Fi speeds. This advancement, however, comes with the challenge of managing the unique signal propagation issues associated with the higher frequency band. The standard's design had to account for these adverse conditions while still delivering on the promise of enhanced throughput and reduced interference. By adopting highly directional communication, the IEEE 802.11ad standard overcomes many of the obstacles inherent to mm-Wave communication, demonstrating a significant leap forward in wireless technology.

Millimeter-wave communication achieves multi-Gbps data rates via highly directional beamforming to overcome pathloss and provide the desired SNR. Unfortunately, establishing communication with sufficiently narrow beamwidth to obtain the necessary link budget is a high overhead procedure in which the search space scales with device mobility and the product of the sender-receiver beam resolution. In this paper, we design, implement, and experimentally evaluate Blind Beam Steering (BBS) a novel architecture and algorithm that removes in-band overhead for directional mm-Wave link establishment. Our system architecture couples mm-Wave and legacy 2.4/5 GHz bands using out-of-band direction inference to establish (overhead-free) multi-Gbps mm-Wave communication. Further, BBS evaluates direction estimates retrieved from passively overheard 2.4/5 GHz frames to assure highest mm-Wave link quality on unobstructed direct paths. By removing in-band overhead, we leverage mm-Wave's very high throughput capabilities, beam-width scalability and provide robustness to mobility. We demonstrate that BBS achieves 97.8% accuracy estimating direction between pairing nodes using at least 5 detection band antennas. Further, BBS successfully detects unobstructed direct path conditions with an accuracy of 96.5% and reduces the IEEE 802.11ad beamforming training overhead by 81%.

For the successful design of low-cost and high-performance radar systems accurate and efficient system simulation is a key requirement. In this paper we present a new versatile simulation environment for frequency-modulated continuous-wave radar systems. Besides common hardware simulation it covers integrated system simulation and concept analysis from signal synthesis to baseband. It includes a flexible scenario generator, accurate noise modeling, and efficiently delivers simulation data for development and testing of signal processing algorithms. A comparison of simulations and measurement results for an integrated 77-GHz radar prototype shows the capabilities of the simulator on two different scenarios.

For small cell technology to significantly increase the capacity of tower-based cellular networks, mobile users will need to be actively pushed onto the more lightly loaded tiers (corresponding to, e.g., pico and femtocells), even if they offer a lower instantaneous SINR than the macrocell base station (BS). Optimizing a function of the long-term rate for each user requires (in general) a massive utility maximization problem over all the SINRs and BS loads. On the other hand, an actual implementation will likely resort to a simple biasing approach where a BS in tier j is treated as having its SINR multiplied by a factor Aj ≥ 1, which makes it appear more attractive than the heavily-loaded macrocell. This paper bridges the gap between these approaches through several physical relaxations of the network-wide association problem, whose solution is NP hard. We provide a low-complexity distributed algorithm that converges to a near-optimal solution with a theoretical performance guarantee, and we observe that simple per-tier biasing loses surprisingly little, if the bias values Aj are chosen carefully. Numerical results show a large (3.5x) throughput gain for cell-edge users and a 2x rate gain for median users relative to a maximizing received power association.

This paper presents a new design concept of a beam steerable high gain phased array antenna based on WR28 waveguide at 29 GHz frequency for fifth generation (5G) full dimension multiple input multiple output (FD-MIMO) system. The 8×8 planar phased array is fed by a three dimensional beamformer to obtain volumetric beam scanning ranging from −60 to +60 degrees both in azimuth and elevation direction. Beamforming network (BFN) is designed using 16 set of 8×8 Butler matrix beamformer to get 64 beam states, which control the horizontal and vertical angle. This is a new concept to design waveguide based high power three-dimensional beamformer for volumetric multibeam in Ka band for 5G application. The maximum gain of phased array is 28.5 dBi that covers 28.9 GHz to 29.4 GHz frequency band.

In this paper, a novel approach for the design of a compact broadband planar microstrip patch antenna is presented for millimetre wave wireless applications. This antenna structure was realized on a Quartz crystal substrate of dielectric constant 3.8 with the thickness of 0.8mm with ground plane. In order to examine, the performances of this antenna it was designed at the frequency 30 GHz and simulated by IE3D software package of Zeland. This type of antenna is composed of two slots of different dimensions and fed by single coaxial feed. Appropriate design of the antenna structure resulted in three discontinuous resonant bands. The proposed antenna has a −10dB return loss impedance bandwidth of 10.3GHz. The performance of this antenna was verified and characterised in terms of the antenna return loss, radiation pattern, surface current distribution and power gain. Also, the numerical results show that the proposed antenna has good impedance bandwidth and radiation characteristics over the entire operating bands. Details of the proposed antenna configurations and design procedures are given.

This article discusses the benefits and challenges associated with the design of multi-function scalable phased arrays at millimeter wave frequencies. First, applications for phased arrays with tens to hundreds of elements are discussed. Existing solutions for scaling silicon-based phased arrays from microwave to terahertz frequencies are reviewed. The challenges and tradeoffs associated with multiple integration options for W-band phased arrays are analyzed, with special consideration given to packaging and antenna performance. Finally, a solution based on SiGe ICs and organic packages for a 64-element dual-polarized 94 GHz phased array is described, along with associated measurement results.

Significant advances in the development of millimeter-wave and terahertz (30-10 000 GHz) technologies have been made to cope with the increasing interest in this still not fully explored electromagnetic spectrum. The nature of electromagnetic waves over this frequency range is well suited for the development of high-resolution imaging applications, molecular-sensitive spectroscopic devices, and ultrabroadband wireless communications. In this paper, millimeter-wave and terahertz antenna technologies are overviewed including the conventional and nonconventional planar/nonplanar antenna structures based on different platforms. As a promising technological platform, substrate-integrated circuits (SICs) attract more and more attention. Various substrate-integrated waveguide (SIW) schemes and other synthesized guide techniques have been widely employed in the design of antennas and arrays. Different types of substrate-integrated antennas and beamforming networks are discussed with respect to theoretical and experimental results in connection with electrical and mechanical performances.

This paper presents phase compensated techniques for 60 GHz communication. A phase compensation method for antenna embedded in a mobile device and a planar lens for base station antenna are introduced. For the Line-of Sight (LoS) scenario, end-fire antenna arrays are adapted for mobile device. 8×1 end-fire patch antenna arrays are designed in Low temperature co-fired ceramic (LTCC) substrate. Simulated max gain of the antenna arrays is 13.03 dBi. When the antenna module is mounted, scattering problem is occurred and its max gain is severely degraded to 6.4 dBi. By using phase compensated technique, max gain of the antenna arrays are enhanced to 12.5 dBi. The beam steered gain of the arrays is also simulated and good coverage can be achieved. A planar lens antenna is also studied for high gain of the base station antenna.

In this paper, we discuss the application of machine learning techniques for performance prediction problems in wireless networks. These problems often involve using existing measurement data to predict network performance where direct measurements are not available. We explore the performance of existing machine learning algorithms for these problems and propose a simple taxonomy of main problem categories. As an example, we use an extensive real-world drive test data set to show that classical machine learning methods such as Gaussian process regression, exponential smoothing of time series, and random forests can yield excellent prediction results. Applying these methods to the management of wireless mobile networks has the potential to significantly reduce operational costs while simultaneously improving user experience. We also discuss key challenges for future work, especially with the focus on practical deployment of machine learning techniques for performance prediction in mobile wireless networks.

We develop channel estimation agorithms for millimeter wave (mmWave) multiple input multiple output (MIMO) systems with one-bit analog-to-digital converters (ADCs). Since the mmWave MIMO channel is sparse due to the propagation characteristics, the estimation problem is formulated as a one-bit compressed sensing problem. We propose a modified EM algorithm that exploits sparsity and has better performance than the conventional EM algorithm. We also present a second solution using the generalized approximate message passing (GAMP) algorithm to solve this optimization problem. The simulation results show that GAMP can reduce mean squared error in the important low and medium SNR regions.


Question: A decision-making structure for autonomous driving in highway scenarios.

This paper outlines the development and implementation of an advanced decision-making framework designed for automated driving systems in highway environments. Drawing inspiration from Carnegie Mellon University's pioneering work on autonomous driving, particularly highlighted in their Urban Challenge entry with the vehicle 'Boss', the framework emphasizes robust navigation and decision-making capabilities in complex urban settings. The team, led by Chris Urmson and comprising experts such as Joshua Anhalt, Drew Bagnell, Christopher Baker, and others, has been instrumental in pushing the boundaries of autonomous vehicle technology. Their research and development efforts have centered around enhancing the vehicle's perception, planning, and control systems, enabling it to navigate through the intricate and often unpredictable landscape of urban environments. Key to their success has been the integration of sophisticated algorithms and machine learning techniques, which allow 'Boss' to interpret its surroundings, make informed decisions, and execute maneuvers with precision. This comprehensive approach has not only proven effective in navigating the challenges of city driving but also lays the groundwork for the future of automated driving on highways, where the decision-making framework's capabilities will be crucial for ensuring safety, efficiency, and compliance with traffic regulations.

Modern cars are incorporating an increasing number of driver assist features, among which automatic lane keeping. The latter allows the car to properly position itself within the road lanes, which is also crucial for any subsequent lane departure or trajectory planning decision in fully autonomous cars. Traditional lane detection methods rely on a combination of highly-specialized, hand-crafted features and heuristics, usually followed by post-processing techniques, that are computationally expensive and prone to scalability due to road scene variations. More recent approaches leverage deep learning models, trained for pixel-wise lane segmentation, even when no markings are present in the image due to their big receptive field. Despite their advantages, these methods are limited to detecting a pre-defined, fixed number of lanes, e.g. ego-lanes, and can not cope with lane changes. In this paper, we go beyond the aforementioned limitations and propose to cast the lane detection problem as an instance segmentation problem - in which each lane forms its own instance - that can be trained end-to-end. To parametrize the segmented lane instances before fitting the lane, we further propose to apply a learned perspective transformation, conditioned on the image, in contrast to a fixed ”bird’s-eye view” transformation. By doing so, we ensure a lane fitting which is robust against road plane changes, unlike existing approaches that rely on a fixed, predefined transformation. In summary, we propose a fast lane detection algorithm, running at 50 fps, which can handle a variable number of lanes and cope with lane changes. We verify our method on the tuSimple dataset and achieve competitive results.

Lane estimation plays a central role for Driver Assistance Systems, therefore many approaches have been proposed to measure its performance. However, no commonly agreed metric exists. In this work, we first present a detailed survey of the current measures. Most of them apply pixel-level benchmarks on camera images and require a time-consuming and fault-prone labeling process. Moreover, these metrics cannot be used to assess other sources such as the detected guardrails, curbs or other vehicles. Therefore, we introduce an efficient and sensor-independent metric, which provides an objective and intuitive self-assessment for the entire road estimation process at multiple levels: individual detectors, lane estimation itself, and the target applications (e.g., lane keeping system). Our metric does not require a high labeling effort and can be used both online and offline. By selecting the evaluated points in specific distances, it can be applied to any road model representation. By comparing in 2D vehicle coordinate system, two possibilities exist to generate the ground-truth: the human-driven path or the expensive alternative with DGPS and detailed maps. This paper applies both methods and reveals that the human-driven path also qualifies for this task and it is applicable to scenarios without GPS signal, e.g., tunnel. Although the lateral offset between reference and detection is widely used in the majority of works, this paper shows that another criterion, the angle deviation, is more appropriate. Finally, we compare our metric with other state-of-the-art metrics using real data recordings from different scenarios.

The use of wireless technology based on Vehicular Ad hoc Networks (VANETs) for information exchange can influence the drivers' behavior towards improving driving performance and reducing road accidents. This information can even be more relevant if it is presented as a video stream. In this paper we propose a system that relies on VANET and video-streaming technology: the See-Through System (STS). The system enhances driver's visibility and supports the driver's overtaking decision in challenging situations, such as overtaking a vision-obstructing vehicle. The use of the See-Through System provides the driver with an additional tool for determining if traffic conditions permit starting an overtaking maneuver thus reducing the risk of overtaking. We tested the See-Through System on an experimental vehicle on the road as well as in the context of a driving simulator for real world environment. Results are promising, since the use of the 802.11p standard wireless communication protocol allows a vehicle-to-vehicle connection without significant delay and the totality of the participants regarded the information provided by the STS as useful.

Intelligent visual surveillance for road vehicles is the key to developing autonomous intelligent traffic systems. Recently, traffic incident detection employing computer vision and image processing has attracted much attention. In this paper, a probabilistic model for predicting traffic accidents using three-dimensional (3-D) model-based vehicle tracking is proposed. Sample data including motion trajectories are first obtained by 3-D model-based vehicle tracking. A fuzzy self-organizing neural network algorithm is then applied to learn activity patterns from the sample trajectories. Finally, vehicle activity is predicted by locating and matching each partial trajectory with the learned activity patterns, and the occurrence probability of a traffic accident is determined. Experiments show the effectiveness of the proposed algorithms.

The detection of a traffic light on the road is important for the safety of persons who occupy a vehicle, in a normal vehicles or an autonomous land vehicle. In normal vehicle, a system that helps a driver to perceive the details of traffic signals, necessary to drive, could be critical in a delicate driving manoeuvre (i.e crossing an intersection of roads). Furthermore, traffic lights detection by an autonomous vehicle is a special case of perception, because it is important for the control that the autonomous vehicle must take. Multiples authors have used image processing as a base for achieving traffic light detection. However, the image processing presents a problem regarding conditions for capturing scenes, and therefore, the traffic light detection is affected. For this reason, this paper proposes a method that links the image processing with an estimation state routine formed by Hidden Markov Models (HMM). This method helps to determine the current state of the traffic light detected, based on the obtained states by image processing, aiming to obtain the best performance in the determination of the traffic light states. With the proposed method in this paper, we obtained 90.55% of accuracy in the detection of the traffic light state, versus a 78.54% obtained using solely image processing. The recognition of traffic lights using image processing still has a large dependence on the capture conditions of each frame from the video camera. In this context, the addition of a pre-processing stage before image processing could contribute to improve this aspect, and could provide a better results in determining the traffic light state.

Conflict-free routing is an important problem to be addressed in AGV Systems, in particular when they are bi-directional. This paper considers the problem of routing AGVs in the presence of contingencies. The importance of this work stems from the realization that path-planning approaches are ill suited to AGVSs subject to contingencies. In this paper, a two stage closed-loop control strategy is proposed. In the first control stage, a pre-planning method proposed by Kim and Tanchoco (1991) is used to establish the fastest conflict-free routes for AGVs. The objective of the second stage is the avoidance of deadlocks in the presence of interruptions while maintaining the planned routes. Copyright © 2005 IFAC

This contribution provides a review of fundamental goals, development and future perspectives of driver assistance systems. Mobility is a fundamental desire of mankind. Virtually any society strives for safe and efficient mobility at low ecological and economic costs. Nevertheless, its technical implementation significantly differs among societies, depending on their culture and their degree of industrialization. A potential evolutionary roadmap for driver assistance systems is discussed. Emerging from systems based on proprioceptive sensors, such as ABS or ESC, we review the progress incented by the use of exteroceptive sensors such as radar, video, or lidar. While the ultimate goal of automated and cooperative traffic still remains a vision of the future, intermediate steps towards that aim can be realized through systems that mitigate or avoid collisions in selected driving situations. Research extends the state-of-the-art in automated driving in urban traffic and in cooperative driving, the latter addressing communication and collaboration between different vehicles, as well as cooperative vehicle operation by its driver and its machine intelligence. These steps are considered important for the interim period, until reliable unsupervised automated driving for all conceivable traffic situations becomes available. The prospective evolution of driver assistance systems will be stimulated by several technological, societal and market trends. The paper closes with a view on current research fields.

Autonomous or semi-autonomous systems are deployed in environments where contact with programmers or technicians is infrequent or undesirable. To operate reliably, such systems should be able to adapt to new circumstances on their own. This paper describes our combined approach for adaptable software architecture and task synthesis from high-level goals, which is based on a three-layer model. In the uppermost layer, reactive plans are generated from goals expressed in a temporal logic. The middle layer is responsible for plan execution and assembling a configuration of domain-specific software components, which reside in the lowest layer. Moreover, the middle layer is responsible for selecting alternative components when the current configuration is no longer viable for the circumstances that have arisen. The implementation demonstrates that the approach enables us to handle non-determinism in the environment and unexpected failures in software components.

Despite the availability of lane departure and lane keeping systems for highway assistance, unmarked and winding rural roads still pose challenges to lane recognition systems. To detect an upcoming curve as soon as possible, the viewing range of image-based lane recognition systems has to be extended. This is done by evaluating 3D information obtained from stereo vision or imaging radar in this paper. Both sensors deliver evidence grids as the basis for road course estimation. Besides known Kalman Filter approaches, Particle Filters have recently gained interest since they offer the possibility to employ cues of a road, which can not be described as measurements needed for a Kalman Filter approach. We propose to combine both principles and their benefits in a Kalman Particle Filter. The comparison between the results gained from this recently published filter scheme and the classical approaches using real world data proves the advantages of the Kalman Particle Filter.

This paper provides an efficient way to design an automatic car braking system using Fuzzy Logic. The system could avoid accidents caused by the delays in driver reaction times at critical situations. The proposed Fuzzy Logic Controller is able to brake a car when the car approaches for an obstacle in the very near range. Collision avoidance is achieved by steering the car if the obstacle is in the tolerable range and hence there is no necessity to apply the brakes. Another FLC (which is cascaded with the first FLC for collision avoidance) implements the Anti-lock Braking capability during heavy braking condition. Thus the system is made intelligent since it could take decisions automatically depending upon the inputs from ultrasonic sensors. A simulative study is done using MATLAB and LabVIEW software. The results obtained by the simulation model are compared with the existing system and the proposed model conveys a satisfactory result which has high consumer acceptance. ATMega controller is used for implementation of the proposed system.


Question: A decision-making structure for autonomous driving in highway settings.

This document details the development and initial simulation testing of a maneuver-based trajectory planning algorithm for highly autonomous vehicles on real roads, specifically focusing on interaction with traffic and other drivers in highway environments. The algorithm is designed to be compatible with a fail-safe embedded system featuring limited computational resources, similar to an engine control unit, making it suitable for integration into future commercial vehicles. The system operates on a platform with a clock speed below 150 MHz, 150 kB of RAM, and 3 MB of program memory. The planning process is divided into two stages. The first stage identifies feasible maneuvers that prioritize collision avoidance, resulting in a set of potential actions in the longitudinal (acceleration or deceleration) and lateral (lane change) directions. The second stage involves a detailed assessment of multiple trajectories within these maneuvers, optimizing them based on criteria such as travel time, adherence to traffic regulations, fuel efficiency, and ride comfort. The final output is a recommended trajectory that guides the vehicle's state (position, orientation, velocity, and acceleration) for the upcoming seconds, facilitating safe and efficient autonomous driving in complex highway scenarios.

We address the problem of real-time navigation in dynamic environments for car-like robots. We present an approach to identify controls that will lead to a collision with a moving obstacle at some point in the future. Our approach generalizes the concept of velocity obstacles, which have been used for navigation among dynamic obstacles, and takes into account the constraints of a car-like robot. We use this formulation to find controls that will allow collision free navigation in dynamic environments. Finally, we demonstrate the performance of our algorithm on a simulated car-like robot among moving obstacles.

Future autonomous automobiles will require extremely reliable radar sensors to guarantee the safety of road users. This requires test environments to verify that automotive radars work properly even in complex traffic situations. Radar target simulators can generate virtual scattering centers in front of the radar sensor and are therefore a promising verification technique. However, currently available systems only offer a very basic approach to manipulating scattering centers. With the workflow presented in this publication it becomes possible to decompose high-level descriptions of traffic scenarios into parameters required for radar target simulation. The workflow can therefore be used to generate realistic traffic scenarios and thus increases the reliability of automotive radar sensor tests.

The safety of the planned paths of autonomous cars with respect to the movement of other traffic participants is considered. Therefore, the stochastic occupancy of the road by other vehicles is predicted. The prediction considers uncertainties originating from the measurements and the possible behaviors of other traffic participants. In addition, the interaction of traffic participants, as well as the limitation of driving maneuvers due to the road geometry, is considered. The result of the presented approach is the probability of a crash for a specific trajectory of the autonomous car. The presented approach is efficient as most of the intensive computations are performed offline, which results in a lean online algorithm for real-time application.

Auto-brake systems have been on the market for a few years, but in order to continue improving their performance, multi-target threat assessment is a key component. The goal is to early be able to say that a collision can no longer be avoided by a steering maneuver, and thus braking can be started earlier. This is done by looking at all objects in the road scene, including other vehicles and barriers, and then looking for escape paths. The paper presents a computationally efficient way of determining whether an escape path can be found in a scenario with generally positioned objects, given the limited dynamics of the vehicle.

To reduce roadway congestion and improve traffic safety, accurate traffic metrics, such as number of vehicles travelling through lane-ways, are required. Unfortunately most existing infrastructure, such as loop-detectors and many video detectors, do not feasibly provide accurate vehicle counts. Consequently, a novel method is proposed which models vehicle motion using hidden Markov models (HMM). The proposed method represents a specified small region of the roadway as 'empty', 'vehicle entering', 'vehicle inside', and 'vehicle exiting', and then applies a modified Viterbi algorithm to the HMM sequential estimation framework to initialize and track vehicles. Vehicle observations are obtained using an Adaboost trained Haar-like feature detector. When tested on 88 hours of video, from three distinct locations, the proposed method proved to be robust to changes in lighting conditions, moving shadows, and camera motion, and consistently out-performed Multiple Target Tracking (MTT) and Virtual Detection Line(VDL) implementations. The median vehicle count error of the proposed method is lower than MTT and VDL by 28%, and 70% respectively. As future work, this algorithm will be implemented to provide the traffic industry with improved automated vehicle counting, with the intent to eventually provide real-time counts.

Development and integration of physiological sensors into automotive applications is gaining importance. Assistance systems which possess knowledge about the driver's cognitive state could increase road safety. In this paper we present a flexible framework that enables the development, evaluation and verification of sensors and algorithms for automotive applications using physiological signals under realistic driving conditions. We have integrated a custom capacitive ECG measurement system into a test car and validated its performance in real world driving tests. During first test runs, the capacitive system achieved a sensitivity of up to 95.5% and a precision rate of up to 92.6%. Our system also records synchronized vehicle dynamics. We discuss the road test measurements which suggest that the driving situation highly impacts the quality of ECG signal. Therefore, information on driving dynamics could be used to improve the precision rate of future capacitive ECG measurement.

125 years after Bertha Benz completed the first overland journey in automotive history, the Mercedes Benz S-Class S 500 INTELLIGENT DRIVE followed the same route from Mannheim to Pforzheim, Germany, in fully autonomous manner. The autonomous vehicle was equipped with close-to-production sensor hardware and relied solely on vision and radar sensors in combination with accurate digital maps to obtain a comprehensive understanding of complex traffic situations. The historic Bertha Benz Memorial Route is particularly challenging for autonomous driving. The course taken by the autonomous vehicle had a length of 103 km and covered rural roads, 23 small villages and major cities (e.g. downtown Mannheim and Heidelberg). The route posed a large variety of difficult traffic scenarios including intersections with and without traffic lights, roundabouts, and narrow passages with oncoming traffic. This paper gives an overview of the autonomous vehicle and presents details on vision and radar-based perception, digital road maps and video-based self-localization, as well as motion planning in complex urban scenarios.

Software testing is all too often simply a bug hunt rather than a wellconsidered exercise in ensuring quality. A more methodical approach than a simple cycle of system-level test-fail-patch-test will be required to deploy safe autonomous vehicles at scale. The ISO 26262 development V process sets up a framework that ties each type of testing to a corresponding design or requirement document, but presents challenges when adapted to deal with the sorts of novel testing problems that face autonomous vehicles. This paper identifies five major challenge areas in testing according to the V model for autonomous vehicles: driver out of the loop, complex requirements, non-deterministic algorithms, inductive learning algorithms, and failoperational systems. General solution approaches that seem promising across these different challenge areas include: phased deployment using successively relaxed operational scenarios, use of a monitor/actuator pair architecture to separate the most complex autonomy functions from simpler safety functions, and fault injection as a way to perform more efficient edge case testing. While significant challenges remain in safety-certifying the type of algorithms that provide high-level autonomy themselves, it seems within reach to instead architect the system and its accompanying design process to be able to employ existing software safety approaches.

In urban environments, moving obstacles detection and free space determination are key issues for driving assistance systems and autonomous vehicles. When using lidar sensors scanning in front of the vehicle, uncertainty arises from ignorance and errors. Ignorance is due to the perception of new areas and errors come from imprecise pose estimation and noisy measurements. Complexity is also increased when the lidar provides multi-echo and multi-layer information. This paper presents an occupancy grid framework that has been designed to manage these different sources of uncertainty. A way to address this problem is to use grids projected onto the road surface in global and local frames. The global one generates the mapping and the local one is used to deal with moving objects. A credibilist approach is used to model the sensor information and to do a global fusion with the world-fixed map. Outdoor experimental results carried out with a precise positioning system show that such a perception strategy increases significantly the performance compared to a standard approach.

A driver assistance system (DAS) should support the driver by monitoring road and vehicle events and presenting relevant and timely information to the driver. It is impossible to know what a driver is thinking, but we can monitor the driver’s gaze direction and compare it with the position of information in the driver’s viewfield to make inferences. In this way, not only do we monitor the driver’s actions, we monitor the driver’s observations as well. In this paper we present the automated detection and recognition of road signs, combined with the monitoring of the driver’s response. We present a complete system that reads speed signs in real-time, compares the driver’s gaze, and provides immediate feedback if it appears the sign has been missed by t ©


Question: A decision-making structure for automated driving systems in highway settings.

A Predictive Model for Assessing Collision Risk in Autonomous Driving
This study focuses on enhancing the safety of autonomous vehicles by accurately forecasting the probability of collisions with other road users. A predictive model is developed to estimate the stochastic occupancy of the road, accounting for uncertainties in measurements and the potential actions of surrounding traffic participants. The model also takes into consideration the interplay between vehicles and the constraints imposed by road geometry on possible driving maneuvers. The outcome is a quantifiable risk assessment for any given path the autonomous vehicle intends to follow. Notably, the approach is optimized for real-time application, with the bulk of computationally intensive tasks carried out offline, ensuring a streamlined online algorithm that can be implemented in real-time scenarios.

What machine learning (ML) technique is used for system intelligence implementation in ADAS (advanced driving assistance system)? This paper tries to answer this question. This paper analyzes ADAS and ML independently and then relate which ML technique is applicable to what ADAS component and why. The paper gives a good grasp of the current state-of-the-art. Sample works in supervised, unsupervised, deep and reinforcement learnings are presented; their strengths and rooms for improvements are also discussed. This forms part of the basics in understanding autonomous vehicle. This work is a contribution to the ongoing research in ML aimed at reducing road traffic accidents and fatalities, and the invocation of safe driving.

In this paper, we present the concept of a collision prevention assistant, a system that we believe can significantly contribute to road safety. We propose a new situation assessment algorithm which is tailored to the action of braking and that further accounts for the nonlinearities that arise when vehicles cut out or come to a standstill. The effect of sensor uncertainty on the performance of the proposed algorithm is modelled using a Markov chain and analyzed by means of a Monte Carlo simulation on a typical traffic situation.

A driver assistance system (DAS) should support the driver by monitoring road and vehicle events and presenting relevant and timely information to the driver. It is impossible to know what a driver is thinking, but we can monitor the driver’s gaze direction and compare it with the position of information in the driver’s viewfield to make inferences. In this way, not only do we monitor the driver’s actions, we monitor the driver’s observations as well. In this paper we present the automated detection and recognition of road signs, combined with the monitoring of the driver’s response. We present a complete system that reads speed signs in real-time, compares the driver’s gaze, and provides immediate feedback if it appears the sign has been missed by t ©

The accurate detection and classification of moving objects is a critical aspect of advanced driver assistance systems. We believe that by including the object classification from multiple sensor detections as a key component of the object's representation and the perception process, we can improve the perceived model of the environment. First, we define a composite object representation to include class information in the core object's description. Second, we propose a complete perception fusion architecture based on the evidential framework to solve the detection and tracking of moving objects problem by integrating the composite representation and uncertainty management. Finally, we integrate our fusion approach in a real-time application inside a vehicle demonstrator from the interactIVe IP European project, which includes three main sensors: radar, lidar, and camera. We test our fusion approach using real data from different driving scenarios and focusing on four objects of interest: pedestrian, bike, car, and truck.

Auto-brake systems have been on the market for a few years, but in order to continue improving their performance, multi-target threat assessment is a key component. The goal is to early be able to say that a collision can no longer be avoided by a steering maneuver, and thus braking can be started earlier. This is done by looking at all objects in the road scene, including other vehicles and barriers, and then looking for escape paths. The paper presents a computationally efficient way of determining whether an escape path can be found in a scenario with generally positioned objects, given the limited dynamics of the vehicle.

Recognizing driver awareness is an important prerequisite for the design of advanced automotive safety systems. Since visual attention is constrained to a driver's field of view, knowing where a driver is looking provides useful cues about his activity and awareness of the environment. This work presents an identity-and lighting-invariant system to estimate a driver's head pose. The system is fully autonomous and operates online in daytime and nighttime driving conditions, using a monocular video camera sensitive to visible and near-infrared light. We investigate the limitations of alternative systems when operated in a moving vehicle and compare our approach, which integrates Localized Gradient Orientation histograms with support vector machines for regression. We estimate the orientation of the driver's head in two degrees-of-freedom and evaluate the accuracy of our method in a vehicular testbed equipped with a cinematic motion capture system.

Advanced Driver Assistance Systems have been shown to greatly improve road safety. However, existing systems are typically reactive with an inability to understand complex traffic scenarios. We present a method to predict driver intention as the vehicle enters an intersection using a Long Short Term Memory (LSTM) based Recurrent Neural Network (RNN). The model is learnt using the position, heading and velocity fused from GPS, IMU and odometry data collected by the ego-vehicle. In this paper we focus on determining the earliest possible moment in which we can classify the driver's intention at an intersection. We consider the outcome of this work an essential component for all levels of road vehicle automation.

A robust methodology for detecting and evaluating driver distraction induced by in-vehicle information system using artificial neural network and fuzzy logic is introduced in this paper. An artificial neural network is used to predict driver's performance on a specific road segment. The predicted performance-based measures are compared to the driving with secondary task accomplishment. Fuzzy logic is applied to fuse the variables into a single output, which constitutes a level of driver distraction in percentage. The technique was tested on a vehicle simulator by ten drivers that exploited in-vehicle information system as a secondary activity. The driver-in-the-loop experiment outcomes are discussed.

The development of radar signal processing algorithms for target tracking and higher-level automotive applications is mainly done based on real radar data. A data basis has to be acquired during cost-expensive and time-consuming test runs. For a comparably simple application like the adaptive cruise control (ACC), the variety of significant traffic situations can sufficiently be covered by test runs. But for more advanced applications like intersection assistance, the effort for the acquisition of a representative set of radar data will be unbearable. In this paper, we propose a way of simulating radar target lists in a realistic but computationally undemanding way, which will allow to significantly reduce the amount of real radar data needed

In this paper, we propose an efficient vehicle trajectory prediction framework based on recurrent neural network. Basically, the characteristic of the vehicle's trajectory is different from that of regular moving objects since it is affected by various latent factors including road structure, traffic rules, and driver's intention. Previous state of the art approaches use sophisticated vehicle behavior model describing these factors and derive the complex trajectory prediction algorithm, which requires a system designer to conduct intensive model optimization for practical use. Our approach is data-driven and simple to use in that it learns complex behavior of the vehicles from the massive amount of trajectory data through deep neural network model. The proposed trajectory prediction method employs the recurrent neural network called long short-term memory (LSTM) to analyze the temporal behavior and predict the future coordinate of the surrounding vehicles. The proposed scheme feeds the sequence of vehicles' coordinates obtained from sensor measurements to the LSTM and produces the probabilistic information on the future location of the vehicles over occupancy grid map. The experiments conducted using the data collected from highway driving show that the proposed method can produce reasonably good estimate of future trajectory.


Question: A decision-making structure for automated driving systems in highway settings.

In this document, we explore the development and performance of "Junior," Stanford University's autonomous vehicle entry in the DARPA Urban Challenge. Junior was designed to navigate complex urban environments, demonstrating advanced decision-making capabilities crucial for automated driving systems. The vehicle's architecture integrates a sophisticated perception system, enabling it to interpret its surroundings in real-time. This includes identifying obstacles, understanding traffic signs, and maintaining situational awareness. Junior's decision-making framework is particularly noteworthy, as it allows the vehicle to make safe and efficient driving decisions in dynamic highway settings. The system employs a combination of machine learning algorithms and rule-based systems to handle various driving scenarios. For instance, it can autonomously change lanes, merge into traffic, and maintain a safe following distance from other vehicles. Junior's performance in the Urban Challenge showcased its ability to operate autonomously in a competitive environment, successfully completing the course and demonstrating the potential of automated driving technologies in highway environments. The insights gained from Junior's development and competition have significantly contributed to the advancement of autonomous vehicle technology, particularly in decision-making frameworks for highway driving scenarios.

Despite the availability of lane departure and lane keeping systems for highway assistance, unmarked and winding rural roads still pose challenges to lane recognition systems. To detect an upcoming curve as soon as possible, the viewing range of image-based lane recognition systems has to be extended. This is done by evaluating 3D information obtained from stereo vision or imaging radar in this paper. Both sensors deliver evidence grids as the basis for road course estimation. Besides known Kalman Filter approaches, Particle Filters have recently gained interest since they offer the possibility to employ cues of a road, which can not be described as measurements needed for a Kalman Filter approach. We propose to combine both principles and their benefits in a Kalman Particle Filter. The comparison between the results gained from this recently published filter scheme and the classical approaches using real world data proves the advantages of the Kalman Particle Filter.

Future driver assistance systems are likely to use a multisensor approach with heterogeneous sensors for tracking dynamic objects around the vehicle. The quality and type of data available for a data fusion algorithm depends heavily on the sensors detecting an object. This article presents a general framework which allows the use sensor specific advantages while abstracting the specific details of a sensor. Different tracking models are used depending on the current set of sensors detecting the object. A sensor independent algorithm for classifying objects regarding their current and past movement state is presented. The described architecture and algorithms have been successfully implemented in Tartan racingpsilas autonomous vehicle for the urban grand challenge. Results are presented and discussed.

This paper describes a real-time multi-vehicle motion planning (MVMP) algorithm for the emergency vehicle clearance task. To address the inherent limitations of human drivers in perception, communication, and cooperation, we require that the emergency vehicle and the surrounding normal vehicles are connected and automated vehicles (CAVs). The concerned MVMP task is to find cooperative trajectories such that the emergency vehicle can efficiently pass through the normal vehicles ahead. We use an optimal-control based formulation to describe the MVMP problem, which is centralized, straightforward, and complete. For the online solutions, the centralized MVMP formulation is converted into a multi-period and multi-stage version. Concretely, each period consists of two stages: the emergency vehicle and several normal CAVs ahead try to form a regularized platoon via acceleration or deceleration (stage 1); when a regularized platoon is formed, these vehicles act cooperatively to make way for the emergency vehicle until the emergency vehicle becomes the leader in this local platoon (stage 2). When one period finishes, the subsequent period begins immediately. This sequential process continues until the emergency vehicle finally passes through all the normal CAVs. The subproblem at stage 1 is extremely easy because nearly all the challenging nonlinearity gathers only in stage 2; typical solutions to the subproblem at stage 2 can be prepared offline, and then implemented online directly. Through this, our proposed MVMP algorithm avoids heavy online computations and thus runs in real time.

125 years after Bertha Benz completed the first overland journey in automotive history, the Mercedes Benz S-Class S 500 INTELLIGENT DRIVE followed the same route from Mannheim to Pforzheim, Germany, in fully autonomous manner. The autonomous vehicle was equipped with close-to-production sensor hardware and relied solely on vision and radar sensors in combination with accurate digital maps to obtain a comprehensive understanding of complex traffic situations. The historic Bertha Benz Memorial Route is particularly challenging for autonomous driving. The course taken by the autonomous vehicle had a length of 103 km and covered rural roads, 23 small villages and major cities (e.g. downtown Mannheim and Heidelberg). The route posed a large variety of difficult traffic scenarios including intersections with and without traffic lights, roundabouts, and narrow passages with oncoming traffic. This paper gives an overview of the autonomous vehicle and presents details on vision and radar-based perception, digital road maps and video-based self-localization, as well as motion planning in complex urban scenarios.

This study investigated user acceptance, concerns, and willingness to buy partially, highly, and fully automated vehicles. By means of a 63-question Internet-based survey, we collected 5000 responses from 109 countries (40 countries with at least 25 respondents). We determined cross-national differences, and assessed correlations with personal variables, such as age, gender, and personality traits as measured with a short version of the Big Five Inventory. Results showed that respondents, on average, found manual driving the most enjoyable mode of driving. Responses were diverse: 22% of the respondents did not want to pay more than $0 for a fully automated driving system, whereas 5% indicated they would be willing to pay more than $30,000, and 33% indicated that fully automated driving would be highly enjoyable. 69% of respondents estimated that fully automated driving will reach a 50% market share between now and 2050. Respondents were found to be most concerned about software hacking/misuse, and were also concerned about legal issues and safety. Respondents scoring higher on neuroticism were slightly less comfortable about data transmitting, whereas respondents scoring higher on agreeableness were slightly more comfortable with this. Respondents from more developed countries (in terms of lower accident statistics, higher education, and higher income) were less comfortable with their vehicle transmitting data, with cross-national correlations between q = 0.80 and q = 0.90. The present results indicate the major areas of promise and concern among the international public, and could be useful for vehicle developers and other stakeholders. 2015 Elsevier Ltd. All rights reserved.

Numerous technologies have been deployed to assist and manage transportation. But recent concerted efforts in academia and industry point to a paradigm shift in intelligent transportation systems. Vehicles will carry computing and communication platforms, and will have enhanced sensing capabilities. They will enable new versatile systems that enhance transportation safety and efficiency and will provide infotainment. This article surveys the state-of-the-art approaches, solutions, and technologies across a broad range of projects for vehicular communication systems.

A method is presented for combining two previously proposed algorithms for path-planning and dynamic steering control into a computationally feasible scheme for real-time feedback control of autonomous vehicles in uncertain environments. In the proposed approach to vehicle guidance and control, Path Relaxation is used to compute critical points along a globally desirable path using a priori information and sensor data'. Generalized potential fields are then used for local feedback control to drive the vehicle along a collision-free path using the critical points as subgoals'. Simulation results are presented to demonstrate the control scheme.

The detection of a traffic light on the road is important for the safety of persons who occupy a vehicle, in a normal vehicles or an autonomous land vehicle. In normal vehicle, a system that helps a driver to perceive the details of traffic signals, necessary to drive, could be critical in a delicate driving manoeuvre (i.e crossing an intersection of roads). Furthermore, traffic lights detection by an autonomous vehicle is a special case of perception, because it is important for the control that the autonomous vehicle must take. Multiples authors have used image processing as a base for achieving traffic light detection. However, the image processing presents a problem regarding conditions for capturing scenes, and therefore, the traffic light detection is affected. For this reason, this paper proposes a method that links the image processing with an estimation state routine formed by Hidden Markov Models (HMM). This method helps to determine the current state of the traffic light detected, based on the obtained states by image processing, aiming to obtain the best performance in the determination of the traffic light states. With the proposed method in this paper, we obtained 90.55% of accuracy in the detection of the traffic light state, versus a 78.54% obtained using solely image processing. The recognition of traffic lights using image processing still has a large dependence on the capture conditions of each frame from the video camera. In this context, the addition of a pre-processing stage before image processing could contribute to improve this aspect, and could provide a better results in determining the traffic light state.

Application of high level fusion approaches demonstrate a sequence of significant advantages in multi sensor data fusion and automotive safety fusion systems are no exception to this. High level fusion can be applied to automotive sensor networks with complementary or/and redundant field of views. The advantage of this approach is that it ensures system modularity and allows benchmarking, as it does not permit feedbacks and loops inside the processing. In this paper two specific high level data fusion approaches are described including a brief architectural and algorithmic presentation. These approaches differ mainly in their data association part: (a) track level fusion approach solves it with the point to point association with emphasis on object continuity and multidimensional assignment, and (b) grid based fusion approach that proposes a generic way to model the environment and to perform sensor data fusion. The test case for these approaches is a multi sensor equipped PReVENT/ProFusion2 truck demonstrator vehicle.

Context: Self-Driving cars are getting more and more attention with public demonstration from all important automotive OEMs but also from companies, which do not have a long history in the automotive industry. Fostered by large international competitions in the last decade, several automotive OEMs have already announced to bring this technology to the market around 2020. Objective: International competitions like the 2007 DARPA Urban Challenge did not focus on efficient usage of resources to realize the self-driving vehicular functionality. Since the automotive industry is very cost-sensitive, realizing reliable and robust selfdriving functionality is challenging when expensive and sophisticated sensors mounted very visibly on the vehicle’s roof for example cannot be used. Therefore, the goal for this study is to investigate how architectural design decisions of recent self-driving vehicular technology consider resource-efficiency. Method: In a multiple case study, the architectural design decisions derived for resourceconstrained self-driving miniature cars for the international competition CaroloCup are compared with architectural designs from recent real-scale self-driving cars. Results: Scaling down available resources for realizing self-driving vehicular technology puts additional constraints on the architectural design; especially reusability of software components in platform-independent algorithmic concepts are prevailing. Conclusion: Software frameworks like the robotic operating system (ROS) enable fast prototypical solutions; however, architectural support for resource-constrained devices is limited. Here, architectural design drivers as realized in AUTOSAR are more suitable.


Question: A decision-making structure for autonomous driving in highway settings.

This paper introduces a comprehensive methodology for the classification and tracking of dynamic objects in urban environments, utilizing a multi-sensor approach for advanced driver assistance systems. Future systems are envisioned to incorporate a variety of sensor types to enhance the detection and tracking of objects surrounding the vehicle. The effectiveness of data fusion algorithms is significantly influenced by the capabilities of the sensors involved. To leverage the unique strengths of each sensor while maintaining a generalized approach, a versatile framework is proposed. This framework enables the selection of appropriate tracking models based on the specific sensors currently detecting an object. Additionally, a sensor-agnostic algorithm is outlined for the classification of objects according to their current and historical movement patterns. The proposed architecture and algorithms have been successfully implemented and tested in Tartan Racing's autonomous vehicle, which participated in the Urban Grand Challenge. The results of this implementation are analyzed and discussed, highlighting the effectiveness of the multi-sensor approach in urban autonomous driving scenarios.

A robust methodology for detecting and evaluating driver distraction induced by in-vehicle information system using artificial neural network and fuzzy logic is introduced in this paper. An artificial neural network is used to predict driver's performance on a specific road segment. The predicted performance-based measures are compared to the driving with secondary task accomplishment. Fuzzy logic is applied to fuse the variables into a single output, which constitutes a level of driver distraction in percentage. The technique was tested on a vehicle simulator by ten drivers that exploited in-vehicle information system as a secondary activity. The driver-in-the-loop experiment outcomes are discussed.

In this paper, we aim to explore path following. We implement a path following component by referring to the existing Pure Pursuit algorithm. Using the simulation and field operational test, we identified the problem in the path following component. The main problems identified were with respect to vehicles meandering off the path, turning a corner, and the instability of steering control. Therefore, we apply some modifications to the Pure Pursuit[1] algorithm. We have also conducted the simulation and field operational tests again to evaluate these modifications.

This paper describes an intelligent autonomous airborne flight capability that is being used as a test bed for future technology development. The unmanned air vehicles (UAVs) fly under autonomous control of both an onboard computer and an autopilot. The onboard computer provides the mission control and runs the autonomous Intelligent Controller (IC) software while the autopilot controls the vehicle navigation and flight control. The autonomous airborne flight system is described in detail. An IC architecture directly applicable to the design of unmanned vehicles is also presented. The UAVs may operate independently or in cooperation with one another to carry out a specified mission. The intelligent UAV flight system is used to evaluate and study autonomous UAV control as well as multi-vehicle collaborative control.

Advanced driver assistance systems (ADAS) are becoming more common in safety and convenience applications. The computer vision based ADAS described in this paper, is an add-on system, suitable to variety of cars. It detects vacant legal parking spots, and safely guides the vehicle into the selected parking. Detection can be performed in both indoor parking lots and along roadsides. The system is composed of three standard computer-connected webcams, which are attached to the vehicle. Upon slowing down, the system starts searching automatically for a right hand-side vacant parking spot, while being aware to parking color signs. Once detected, the parking orientation is determined, and the driver is notified. Once a parking is selected by the driver, the relative position between the vehicle and the parking spot is monitored. Vocal and visual parking guidance instructions are presented to the driver. In addition, if during parking, an object is moving on the road towards the car, a safety alert is given. The system is universal in the sense that, as an add-on system, it can be installed on any private 4-wheeled vehicle, and is suited to urban driving environment.

A vision-based vehicle guidance system for road vehicles can have three main roles: 1) road detection; 2) obstacle detection; and 3) sign recognition. The first two have been studied for many years and with many good results, but traffi c sign recognition is a less-studied field. Traffi c signs provide drivers with very valuable informatio n about the road, in order to make drivin g safer and easier. We think that traffi c signs must play the same role for autonomous vehicles. They are designed to be easily recognized by human driver s mainly because their color and shapes are very different from natural environments. The algorithm described in this paper takes advantage of these features. I t has two main parts. The first one, for the detection, uses color thresholding to segment the image and shape analysis to detect the signs. The second one, for the classification, uses a neural network. Some results from natural scenes are shown. On the other hand, the algorithm is valid to detect other kinds of marks that would tell the mobile robot to perform some task at that place.

As we march down the road of automation in robotics and artificial intelligence, we will need to automate an increasing amount of ethical decision-making in order for our devices to operate independently from us. But automating ethical decision-making raises novel questions for engineers and designers, who will have to make decisions about how to accomplish that task. For example, some ethical decisionmaking involves hard moral cases, which in turn requires user input if we are to respect established norms surrounding autonomy and informed consent. The author considers this and other ethical considerations that accompany the automation of ethical decision-making. He proposes some general ethical requirements that should be taken into account in the design room, and sketches a design tool that can be integrated into the design process to help engineers, designers, ethicists, and policymakers decide how best to automate certain forms of ethical decision-making.

The automotive industry is moving aggressively in the direction of advanced active safety. Dedicated short-range communication (DSRC) is a key enabling technology for the next generation of communication-based safety applications. One aspect of vehicular safety communication is the routine broadcast of messages among all equipped vehicles. Therefore, channel congestion control and broadcast performance improvement are of particular concern and need to be addressed in the overall protocol design. Furthermore, the explicit multichannel nature of DSRC necessitates a concurrent multichannel operational scheme for safety and non-safety applications. This article provides an overview of DSRC based vehicular safety communications and proposes a coherent set of protocols to address these requirements

The project aims to build a monocular vision autonomous car prototype using Raspberry Pi as a processing chip. An HD camera along with an ultrasonic sensor is used to provide necessary data from the real world to the car. The car is capable of reaching the given destination safely and intelligently thus avoiding the risk of human errors. Many existing algorithms like lane detection, obstacle detection are combined together to provide the necessary control to the car.

In urban environments, moving obstacles detection and free space determination are key issues for driving assistance systems and autonomous vehicles. When using lidar sensors scanning in front of the vehicle, uncertainty arises from ignorance and errors. Ignorance is due to the perception of new areas and errors come from imprecise pose estimation and noisy measurements. Complexity is also increased when the lidar provides multi-echo and multi-layer information. This paper presents an occupancy grid framework that has been designed to manage these different sources of uncertainty. A way to address this problem is to use grids projected onto the road surface in global and local frames. The global one generates the mapping and the local one is used to deal with moving objects. A credibilist approach is used to model the sensor information and to do a global fusion with the world-fixed map. Outdoor experimental results carried out with a precise positioning system show that such a perception strategy increases significantly the performance compared to a standard approach.

In order to reduce the great number of parking incidences and other collisions in low speed scenarios, an obstacle avoidance algorithm is proposed. Since collision avoidance can be achieved by sole braking when driving slowly this algorithm's objective is a comfort orientated braking routine. Therefore, an optimization problem is formulated leading to jerk and time optimal trajectories for braking. In addition to that, based on the prediction of the future vehicle motion, this algorithm compensates for a significant actuator time delay. Using an occupancy grid for representing the static vehicle environment and the current driving state, possible collision points are determined not only for the vehicle front or rear, but also for both vehicle sides, where no sensors are located. The algorithm's performance is demonstrated in a real-world scenario.


Question: Detecting shadows using conditional generative adversarial networks.

This paper explores the innovative application of Conditional Generative Adversarial Networks (CGANs) for the intricate task of shadow detection in images. Unlike conventional approaches, CGANs are uniquely positioned to discern and isolate shadows by learning from a diverse set of images. The architecture of CGANs integrates a generator and a discriminator, both of which are trained in a supervised manner using labeled data. The generator is tasked with creating shadow maps that closely resemble real shadows, while the discriminator aims to distinguish between actual shadows and those generated by the generator. This adversarial training process ensures that the generator progressively learns to produce highly realistic shadow maps. To enhance the performance of CGANs in shadow detection, the paper introduces several modifications to the standard CGAN framework. These include the use of conditional inputs, which provide contextual information to the generator, allowing it to generate shadows that are consistent with the scene's lighting and objects. Additionally, the paper discusses the effectiveness of using a U-Net-like architecture for the generator component of the CGAN. U-Net, originally developed for biomedical image segmentation, is a convolutional neural network that excels in producing high-resolution outputs. By adapting U-Net's architecture to the generator in CGANs, the model can effectively capture detailed shadow patterns, leading to improved shadow detection accuracy. Through extensive experiments on various datasets, the paper demonstrates that the proposed CGAN-based approach outperforms traditional methods in terms of both quantitative metrics and qualitative visual results. This showcases the potential of CGANs in tackling complex image processing tasks such as shadow detection.

Hand gesture-to-gesture translation in the wild is a challenging task since hand gestures can have arbitrary poses, sizes, locations and self-occlusions. Therefore, this task requires a high-level understanding of the mapping between the input source gesture and the output target gesture. To tackle this problem, we propose a novel hand Gesture Generative Adversarial Network (GestureGAN). GestureGAN consists of a single generator G and a discriminator D, which takes as input a conditional hand image and a target hand skeleton image. GestureGAN utilizes the hand skeleton information explicitly, and learns the gesture-to-gesture mapping through two novel losses, the color loss and the cycle-consistency loss. The proposed color loss handles the issue of "channel pollution" while back-propagating the gradients. In addition, we present the Frechet ResNet Distance (FRD) to evaluate the quality of generated images. Extensive experiments on two widely used benchmark datasets demonstrate that the proposed GestureGAN achieves state-of-the-art performance on the unconstrained hand gesture-to-gesture translation task. Meanwhile, the generated images are in high-quality and are photo-realistic, allowing them to be used as data augmentation to improve the performance of a hand gesture classifier. Our model and code are available at https://github.com/Ha0Tang/GestureGAN.

While humans easily recognize relations between data from different domains without any supervision, learning to automatically discover them is in general very challenging and needs many ground-truth pairs that illustrate the relations. To avoid costly pairing, we address the task of discovering cross-domain relations given unpaired data. We propose a method based on generative adversarial networks that learns to discover relations between different domains (DiscoGAN). Using the discovered relations, our proposed network successfully transfers style from one domain to another while preserving key attributes such as orientation and face identity.

It is a difficult task to classify images with multiple class labels using only a small number of labeled examples, especially when the label (class) distribution is imbalanced. Emotion classification is such an example of imbalanced label distribution, because some classes of emotions like disgusted are relatively rare comparing to other labels like happy or sad. In this paper, we propose a data augmentation method using generative adversarial networks (GAN). It can complement and complete the data manifold and find better margins between neighboring classes. Specifically, we design a framework with a CNN model as the classifier and a cycle-consistent adversarial networks (CycleGAN) as the generator. In order to avoid gradient vanishing problem, we employ the least-squared loss as adversarial loss. We also propose several evaluation methods on three benchmark datasets to validate GAN’s performance. Empirical results show that we can obtain 5%∼10% increase in the classification accuracy after employing the GAN-based data augmentation techniques.

In recent years, there have been tremendous advancements in the field of machine learning. These advancements have been made through both academic as well as industrial research. Lately, a fair amount of research has been dedicated to the usage of generative models in the field of computer vision and image classification. These generative models have been popularized through a new framework called Generative Adversarial Networks. Moreover, many modified versions of this framework have been proposed in the last two years. We study the original model proposed by Goodfellow et al. (2014) as well as modifications over the original model and provide a comparative analysis of these models.

Although it is typically expected that using a large amount of labeled training data would lead to improve performance in deep learning, it is generally difficult to obtain such DataBase (DB). In competitions such as the Detection and Classification of Acoustic Scenes and Events (DCASE) challenge Task 1, participants are constrained to use a relatively small DB as a rule, which is similar to the aforementioned issue. To improve Acoustic Scene Classification (ASC) performance without employing additional DB, this paper proposes to use Generative Adversarial Networks (GAN) based method for generating additional training DB. Since it is not clear whether every sample generated by GAN would have equal impact in classification performance, this paper proposes to use Support Vector Machine (SVM) hyper plane for each class as reference for selecting samples, which have class discriminative information. Based on the crossvalidated experiments on development DB, the usage of the generated features could improve ASC performance.

Humans have the remarkable ability of imagination, where within the human mind virtual simulations are done of scenarios whether visual, auditory or any other senses. These imaginations are based on the experiences during interaction with the real world, where human senses help the mind understand their surroundings. Such level of imagination has not yet been achieved using current algorithms, but a current trend in deep learning architectures known as Generative Adversarial Networks (GANs) have proven capable of generating new and interesting images or videos based on the training data. In that way, GANs can be used to mimic human imagination, where the resulting generated visuals of GANs are based on the data used during training. In this paper, we use a combination of Long Short-Term Memory (LSTM) Networks and 3D GANs to generate videos. We use a 3D Convolutional GAN to generate new human action videos based on trained data. The generated human action videos are used to generate longer videos consisting of a sequence of short actions combined creating longer and more complex activities. To generate the sequence of actions needed we use an LSTM network to translate a simple input description text into the required sequence of actions. The generated chunks are then concatenated using a motion interpolation scheme to form a single video consisting of many generated actions. Hence a visualization of the input text description is generated as a video of a subject performing the activity described.

The ability to segment unknown objects in depth images has potential to enhance robot skills in grasping and object tracking. Recent computer vision research has demonstrated that Mask R-CNN can be trained to segment specific categories of objects in RGB images when massive handlabeled datasets are available. As generating these datasets is time-consuming, we instead train with synthetic depth images. Many robots now use depth sensors, and recent results suggest training on synthetic depth data can generalize well to the real world. We present a method for automated dataset generation and rapidly generate a training dataset of 50k depth images and 320k object masks synthetically using simulated scenes of 3D CAD models. We train a variant of Mask R-CNN on the generated dataset to perform category-agnostic instance segmentation without hand-labeled data. We evaluate the trained network, which we refer to as Synthetic Depth (SD) Mask R-CNN, on a set of real, high-resolution images of challenging, denselycluttered bins containing objects with highly-varied geometry. SD Mask R-CNN outperforms point cloud clustering baselines by an absolute 15% in Average Precision and 20% in Average Recall, and achieves performance levels similar to a Mask RCNN trained on a massive, hand-labeled RGB dataset and fine-tuned on real images from the experimental setup. The network also generalizes well to a lower-resolution depth sensor. We deploy the model in an instance-specific grasping pipeline to demonstrate its usefulness in a robotics application. Code, the synthetic training dataset, and supplementary material are available at https://bit.ly/2letCuE.

Single image reflection separation is an ill-posed problem since two scenes, a transmitted scene and a reflected scene, need to be inferred from a single observation. To make the problem tractable, in this work we assume that categories of two scenes are known. It allows us to address the problem by generating both scenes that belong to the categories while their contents are constrained to match with the observed image. A novel network architecture is proposed to render realistic images of both scenes based on adversarial learning. The network can be trained in a weakly supervised manner, i.e., it learns to separate an observed image without corresponding ground truth images of transmission and reflection scenes which are difficult to collect in practice. Experimental results on real and synthetic datasets demonstrate that the proposed algorithm performs favorably against existing methods.

Many machine learning image classifiers are vulnerable to adversarial attacks, inputs with perturbations designed to intentionally trigger misclassification. Current adversarial methods directly alter pixel colors and evaluate against pixel norm-balls: pixel perturbations smaller than a specified magnitude, according to a measurement norm. This evaluation, however, has limited practical utility since perturbations in the pixel space do not correspond to underlying real-world phenomena of image formation that lead to them and has no security motivation attached. Pixels in natural images are measurements of light that has interacted with the geometry of a physical scene. As such, we propose a novel evaluation measure, parametric normballs, by directly perturbing physical parameters that underly image formation. One enabling contribution we present is a physically-based differentiable renderer that allows us to propagate pixel gradients to the parametric space of lighting and geometry. Our approach enables physically-based adversarial attacks, and our differentiable renderer leverages models from the interactive rendering literature to balance the performance and accuracy trade-offs necessary for a memory-efficient and scalable adversarial data augmentation workflow.

Generative Adversarial Nets (GANs) and Variational Auto-Encoders (VAEs) provide impressive image generations from Gaussian white noise, but the underlying mathematics are not well understood. We compute deep convolutional network generators by inverting a fixed embedding operator. Therefore, they do not require to be optimized with a discriminator or an encoder. The embedding is Lipschitz continuous to deformations so that generators transform linear interpolations between input white noise vectors into deformations between output images. This embedding is computed with a wavelet Scattering transform. Numerical experiments demonstrate that the resulting Scattering generators have similar properties as GANs or VAEs, without learning a discriminative network or an encoder.


Question: Shadow identification utilizing conditional generative adversarial networks.

This paper presents a novel methodology for learning predictive models of object transformations using time-lapse videos, enabling computers to simulate the human ability to envision future states of objects based on their current appearance. Specifically, we investigate the application of generative models for the creation of visual representations of objects at future points in time. Our models tackle various prediction scenarios, including generating a future state from a single image of an object, predicting a future state from two images taken at different times, and recursively generating a sequence of future states in a recurrent model. We assess the performance of our generated outcomes through both qualitative and quantitative analyses, and further conduct a human evaluation to compare different model variants, demonstrating the effectiveness of our approach in capturing temporal transformations from time-lapse video data.

We introduce a simple semi-supervised learning approach for images based on in-painting using an adversarial loss. Images with random patches removed are presented to a generator whose task is to fill in the hole, based on the surrounding pixels. The in-painted images are then presented to a discriminator network that judges if they are real (unaltered training images) or not. This task acts as a regularizer for standard supervised training of the discriminator. Using our approach we are able to directly train large VGG-style networks in a semi-supervised fashion. We evaluate on STL-10 and PASCAL datasets, where our approach obtains performance comparable or superior to existing methods.

Unsupervised learning for visual perception of 3D geometry is of great interest to autonomous systems. Recent works on unsupervised learning have made considerable progress on geometry perception; however, they perform poorly on dynamic objects and scenarios with dark and noisy environments. In contrast, supervised learning algorithms, which are robust, require large labeled geometric dataset. This paper introduces SIGNet, a novel framework that provides robust geometry perception without requiring geometrically informative labels. Specifically, SIGNet integrates semantic information to make unsupervised robust geometric predictions for dynamic objects in low lighting and noisy environments. SIGNet is shown to improve upon the state-of-the-art unsupervised learning for geometry perception by 30% (in squared relative error for depth prediction). In particular, SIGNet improves the dynamic object class performance by 39% in depth prediction and 29% in flow prediction.

In this paper, we propose a novel approach to video captioning based on adversarial learning and long short-term memory (LSTM). With this solution concept, we aim at compensating for the deficiencies of LSTM-based video captioning methods that generally show potential to effectively handle temporal nature of video data when generating captions but also typically suffer from exponential error accumulation. Specifically, we adopt a standard generative adversarial network (GAN) architecture, characterized by an interplay of two competing processes: a “generator” that generates textual sentences given the visual content of a video and a “discriminator” that controls the accuracy of the generated sentences. The discriminator acts as an “adversary” toward the generator, and with its controlling mechanism, it helps the generator to become more accurate. For the generator module, we take an existing video captioning concept using LSTM network. For the discriminator, we propose a novel realization specifically tuned for the video captioning problem and taking both the sentences and video features as input. This leads to our proposed LSTM–GAN system architecture, for which we show experimentally to significantly outperform the existing methods on standard public datasets.

Domain adaptation refers to the problem of leveraging labeled data in a source domain to learn an accurate model in a target domain where labels are scarce or unavailable. A recent approach for finding a common representation of the two domains is via domain adversarial training (Ganin & Lempitsky, 2015), which attempts to induce a feature extractor that matches the source and target feature distributions in some feature space. However, domain adversarial training faces two critical limitations: 1) if the feature extraction function has high-capacity, then feature distribution matching is a weak constraint, 2) in non-conservative domain adaptation (where no single classifier can perform well in both the source and target domains), training the model to do well on the source domain hurts performance on the target domain. In this paper, we address these issues through the lens of the cluster assumption, i.e., decision boundaries should not cross high-density data regions. We propose two novel and related models: 1) the Virtual Adversarial Domain Adaptation (VADA) model, which combines domain adversarial training with a penalty term that punishes violation of the cluster assumption; 2) the Decision-boundary Iterative Refinement Training with a Teacher (DIRT-T)1 model, which takes the VADA model as initialization and employs natural gradient steps to further minimize the cluster assumption violation. Extensive empirical results demonstrate that the combination of these two models significantly improve the state-of-the-art performance on the digit, traffic sign, and Wi-Fi recognition domain adaptation benchmarks.

We present a novel CNN-based image editing method that allows the user to change the semantic information of an image over a user-specified region. Our method makes this possible by combining the idea of manifold projection with spatial conditional batch normalization (sCBN), a version of conditional batch normalization with userspecifiable spatial weight maps. With sCBN and manifold projection, our method lets the user perform (1) spatial class translation that changes the class of an object over an arbitrary region of user’s choice, and (2) semantic transplantation that transplants semantic information contained in an arbitrary region of the reference image to an arbitrary region in the target image. These two transformations can be used simultaneously, and can realize a complex composite image-editing task like “change the nose of a beagle to that of a bulldog, and open her mouth”. The user can also use our method with intuitive copy-paste-style manipulations. We demonstrate the power of our method on various images. Code will be available at https://github. com/pfnet-research/neural-collage.

In recent years, there have been tremendous advancements in the field of machine learning. These advancements have been made through both academic as well as industrial research. Lately, a fair amount of research has been dedicated to the usage of generative models in the field of computer vision and image classification. These generative models have been popularized through a new framework called Generative Adversarial Networks. Moreover, many modified versions of this framework have been proposed in the last two years. We study the original model proposed by Goodfellow et al. (2014) as well as modifications over the original model and provide a comparative analysis of these models.

A large amount of annotated training images is critical for training accurate and robust deep network models but the collection of a large amount of annotated training images is often time-consuming and costly. Image synthesis alleviates this constraint by generating annotated training images automatically by machines which has attracted increasing interest in the recent deep learning research. We develop an innovative image synthesis technique that composes annotated training images by realistically embedding foreground objects of interest (OOI) into background images. The proposed technique consists of two key components that in principle boost the usefulness of the synthesized images in deep network training. The first is context-aware semantic coherence which ensures that the OOI are placed around semantically coherent regions within the background image. The second is harmonious appearance adaptation which ensures that the embedded OOI are agreeable to the surrounding background from both geometry alignment and appearance realism. The proposed technique has been evaluated over two related but very different computer vision challenges, namely, scene text detection and scene text recognition. Experiments over a number of public datasets demonstrate the effectiveness of our proposed image synthesis technique the use of our synthesized images in deep network training is capable of achieving similar or even better scene text detection and scene text recognition performance as compared with using real images.

Three-dimensional geometric data offer an excellent domain for studying representation learning and generative modeling. In this paper, we look at geometric data represented as point clouds. We introduce a deep autoencoder (AE) network with state-of-the-art reconstruction quality and generalization ability. The learned representations outperform existing methods on 3D recognition tasks and enable basic shape editing via simple algebraic manipulations, such as semantic part editing, shape analogies and shape interpolation. We perform a thorough study of different generative models including: GANs operating on the raw point clouds, significantly improved GANs trained in the fixed latent space of our AEs and, Gaussian mixture models (GMM). For our quantitative evaluation we propose measures of sample fidelity and diversity based on matchings between sets of point clouds. Interestingly, our careful evaluation of generalization, fidelity and diversity reveals that GMMs trained in the latent space of our AEs produce the best results.

We introduce several techniques for sampling and visualizing the latent spaces of generative models. Replacing linear interpolation with spherical linear interpolation prevents diverging from a model’s prior distribution and produces sharper samples. J-Diagrams and MINE grids are introduced as visualizations of manifolds created by analogies and nearest neighbors. We demonstrate two new techniques for deriving attribute vectors: bias-corrected vectors with data replication and synthetic vectors with data augmentation. Most techniques are intended to be independent of model type and examples are shown on both Variational Autoencoders and Generative Adversarial Networks. Keywords—Generative; VAE; GAN; Sampling; Manifold

The recent advances in deep learning have made it possible to generate photorealistic images by using neural networks and even to extrapolate video frames from an input video clip. In this paper, for the sake of both furthering this exploration and our own interest in a realistic application, we study image-to-video translation and particularly focus on the videos of facial expressions. This problem challenges the deep neural networks by another temporal dimension comparing to the imageto-image translation. Moreover, its single input image fails most existing video generation methods that rely on recurrent models. We propose a user-controllable approach so as to generate video clips of various lengths from a single face image. The lengths and types of the expressions are controlled by users. To this end, we design a novel neural network architecture that can incorporate the user input into its skip connections and propose several improvements to the adversarial training method for the neural network. Experiments and user studies verify the effectiveness of our approach. Especially, we would like to highlight that even for the face images in the wild (downloaded from the Web and the authors’ own photos), our model can generate high-quality facial expression videos of which about 50% are labeled as real by Amazon Mechanical Turk workers. Please see our project page1 for more demonstrations.