Question: A reverse Yarbus methodology: forecasting the observer's task from eye movement templates.

This paper presents an innovative method for synthesizing 3D faces, capable of automatic generation from photographic images or direct modeling via a user-friendly interface. The technique addresses two primary challenges in computer-assisted face modeling: automatic registration of new images or 3D models through dense, one-to-one correspondence with an internal model, and the regulation of facial naturalness to prevent the creation of unrealistic appearances. The process begins with an ensemble of 3D face models, which are transformed into a vector space representation to create a morphable face model. This model allows for the creation of new faces or expressions by combining elements from the prototypes. Statistical constraints derived from the example faces guide both manual modeling and automated matching, ensuring the natural appearance of the synthesized faces. The paper showcases the effectiveness of the method through 3D reconstructions from single images and demonstrates its utility in photo-realistic image manipulation, as well as in manipulating faces according to parameters such as gender, facial fullness, and distinctiveness.

**Question**: A direct search method to solve the economic dispatch problem with valve-point effect

**Documents**: An evolutionary recurrent network which automates the design of recurrent neural/fuzzy networks using a new evolutionary learning algorithm is proposed in this paper. This new evolutionary learning algorithm is based on a hybrid of genetic algorithm (GA) and particle swarm optimization (PSO), and is thus called HGAPSO. In HGAPSO, individuals in a new generation are created, not only by crossover and mutation operation as in GA, but also by PSO. The concept of elite strategy is adopted in HGAPSO, where the upper-half of the best-performing individuals in a population are regarded as elites. However, instead of being reproduced directly to the next generation, these elites are first enhanced. The group constituted by the elites is regarded as a swarm, and each elite corresponds to a particle within it. In this regard, the elites are enhanced by PSO, an operation which mimics the maturing phenomenon in nature. These enhanced elites constitute half of the population in the new generation, whereas the other half is generated by performing crossover and mutation operation on these enhanced elites. HGAPSO is applied to recurrent neural/fuzzy network design as follows. For recurrent neural network, a fully connected recurrent neural network is designed and applied to a temporal sequence production problem. For recurrent fuzzy network design, a Takagi-Sugeno-Kang-type recurrent fuzzy network is designed and applied to dynamic plant control. The performance of HGAPSO is compared to both GA and PSO

A computer vision system has been implemented that can recognize threedimensional objects from unknown viewpoints in single gray-scale images. Unlike most other approaches, the recognition is accomplished without any attempt to reconstruct depth information bottom-up from the visual input. Instead, three other mechanisms are used that can bridge the gap between the two-dimensional image and knowledge of three-dimensional objects. First, a process of perceptual organization is used to form groupings and structures in the image that are likely to be invariant over a wide range of viewpoints. Second, a probabilistic ranking method is used to reduce the size of the search space during model based matching. Finally, a process of spatial correspondence brings the projections of three-dimensional models into direct correspondence with the image by solving for unknown viewpoint and model parameters. A high level of robustness in the presence of occlusion and missing data can be achieved through full application of a viewpoint consistency constraint. It is argued that similar mechanisms and constraints form the basis for recognition in human vision. This paper has been published in Artificial Intelligence, 31, 3 (March 1987), pp. 355–395.

In the past five years we have observed the rise of incredibly well performing feed-forward neural networks trained supervisedly for vision related tasks. These models have achieved super-human performance on object recognition, localisation, and detection in still images. However, there is a need to identify the best strategy to employ these networks with temporal visual inputs and obtain a robust and stable representation of video data. Inspired by the human visual system, we propose a deep neural network family, CortexNet, which features not only bottom-up feedforward connections, but also it models the abundant top-down feedback and lateral connections, which are present in our visual cortex. We introduce two training schemes — the unsupervised MatchNet and weakly supervised TempoNet modes — where a network learns how to correctly anticipate a subsequent frame in a video clip or the identity of its predominant subject, by learning egomotion clues and how to automatically track several objects in the current scene. Find the project website at tinyurl.com/CortexNet.

We explore the combination of above-surface sensing with eye tracking to facilitate concurrent interaction with multiple regions on touch screens. Conventional touch input relies on positional accuracy, thereby requiring tight visual monitoring of one's own motor action. In contrast, above-surface sensing and eye tracking provides information about how user's hands and gaze are distributed across the interface. In these situations we facilitate interaction by 1) showing the visual feedback of the hand hover near user's gaze point and 2) decrease the requisite of positional accuracy by employing gestural information. We contribute input and visual feedback techniques that combine these modalities and demonstrate their use in example applications. A controlled study showed the effectiveness of our techniques for manipulation tasks against conventional touch, while the effectiveness in acquisition tasks depended on the amount of mid-air motion, leading to our conclusion that the techniques can benefit interacting with multiple interface regions.

This paper presents the application of an effective EEG-based brain-computer interface design for binary control in a visually elaborate immersive 3D game. The BCI uses the steady-state visual evoked potential (SSVEP) generated in response to phasereversing checkerboard patterns. Two power-spectrum estimation methods were employed for feature extraction in a series of offline classification tests. Both methods were also implemented during real-time game play. The performance of the BCI was found to be robust to distracting visual stimulation in the game and relatively consistent across six subjects, with 41 of 48 games successfully completed. For the best performing feature extraction method, the average real-time control accuracy across subjects was 89%. The feasibility of obtaining reliable control in such a visually rich environment using SSVEPs is thus demonstrated and the impact of this result is discussed.

Interactions between prefrontal cortex (PFC) and stimulusspecific visual cortical association areas are hypothesized to mediate visual working memory in behaving monkeys. To clarify the roles for homologous regions in humans, event-related fMRI was used to assess neural activity in PFC and fusiform face area (FFA) of subjects performing a delay-recognition task for faces. In both PFC and FFA, activity increased parametrically with memory load during encoding and maintenance of face stimuli, despite quantitative differences in the magnitude of activation. Moreover, timing differences in PFC and FFA activation during memory encoding and retrieval implied a context dependence in the flow of neural information. These results support existing neurophysiological models of visual working memory developed in the nonhuman primate.

Perception and emotion interact, as is borne out by studies of how people recognize emotion from facial expressions. Psychological and neurological research has elucidated the processes, and the brain structures, that participate in facial emotion recognition. Studies have shown that emotional reactions to viewing faces can be very rapid and that these reactions may, in turn, be used to judge the emotion shown in the face. Recent experiments have argued that people actively explore facial expressions in order to recognize the emotion, a mechanism that emphasizes the instrumental nature of social cognition. KEYWORDS—face perception; faces; emotion; amygdala An influential psychological model of face processing argued that early perception (construction of a geometric representation of the face based on its features) led to subsequently separate processing of the identity of the face and of the emotional expression of the face (Bruce & Young, 1986). That model has received considerable support from neuroscience studies suggesting that the separate processes are based on separable neuroanatomical systems: In neuroimaging studies, different parts of the brain are activated in response to emotional expressions or identity changes; and brain damage can result in impairments in recognizing identity but not in recognizing emotional expressions, or the reverse. SOME PROCESSING IS RAPID AND AUTOMATIC AND CAN OCCUR IN THE ABSENCE OF AWARENESS OF THE STIMULI Some responses in the brain to emotional facial expressions are so rapid (less than 100 ms) that they could not plausibly be based on conscious awareness of the stimulus, although the responses, in turn, may contribute to conscious awareness. Evidence comes from studies using event-related potentials, measures of the brain’s electrical activity recorded at the scalp (or, much more rarely, directly from the brain in surgical patients). In those experiments, the responses to many presentations of emotional stimuli are averaged across stimuli, and the changes in electrical potential can be accurately timed in relation to stimulus onset. Evidence has also come from studies in which viewers were shown facial expressions subliminally. Especially salient aspects of faces are most potent in driving nonconscious responses: For instance, subliminal presentation of only the whites of the eyes of fearful faces results in measurable brain activation (Whalen et al., 2004). One specific structure involved in such rapid and automatic neural responses is the amygdala, a structure in the medial temporal lobe that is known to be involved in many aspects of emotion processing. Findings such as these have been of interest to psychologists, as they provide a potential mechanism consistent with two-factor theories of human behavior. For instance, the theory that affect and cognitive judgment are separate processes, and that affect can precede cognitive judgment, receives some support from the neuroscience findings (Zajonc, 1980). The data also add detail to theories of visual consciousness. In one set of studies, emotional facial expressions were shown to neurological patients who, because of their brain damage, were unable to report seeing the stimuli. An individual with damage to the visual cortex had ‘‘blindsight’’ for emotional faces: He could discriminate the emotion shown in faces by guessing, even though he reported no visual experience of seeing the faces. A neuroimaging experiment in this same individual revealed that the amygdala was activated differentially by different emotional faces, despite the absence of conscious visual experience (Morris, deGelder, Weiskrantz, & Dolan, 2001). These and other studies have suggested a distinction between subcortical processing of emotional visual stimuli (e.g., involving the amygdala and brainstem nuclei such as the superior colliculus), which may be independent of conscious vision (Johnson, 2005), and cortical processing, which is usually accompanied by conscious experience. It is interesting to note that amphibians and reptiles have only subcortical visual processing, since they lack a neocortex. One broad interpretation of these observations is thus that the subcortical route for processing emotional stimuli is the more ancient one, and that in Address correspondence to Ralph Adolphs, HSS 228-77, Caltech, Pasadena, CA 91125; e-mail: radolphs@hss.caltech.edu. CURRENT DIRECTIONS IN PSYCHOLOGICAL SCIENCE 222 Volume 15—Number 5 Copyright r 2006 Association for Psychological Science mammals an additional, cortical route has evolved that probably allows more flexible behaviors based on learning and conscious deliberation. A final wrinkle has come from psychological studies of the relationship between attention and emotion. It has been known for some time that emotionally salient stimuli can capture attention, an interaction that makes much adaptive sense (Ohman, Flykt, & Esteves, 2001). But more recent studies have also shown the reverse: that volitionally allocating attention to stimuli can influence their emotional evaluation. Inhibiting attention to visual stimuli that are distractors in a search task, for example, results in devaluation of those stimuli when subjects are asked to rate them on affective dimensions (Raymond, Fenske, & Tavassoli, 2003). Emotions thus represent the value of stimuli—what people approach or avoid, cognitively or behaviorally, volitionally or automatically. FEAR AND DISGUST MAY BE PROCESSED SPECIALLY Several studies have found evidence that the amygdala, the subcortical structure discussed earlier, is disproportionately important for processing facial expressions of fear, although by no means exclusively so. The only other emotion for which a similar neuroanatomical specificity has been reported is disgust. A region of the cortex called the insula represents motor and sensory aspects of that emotional response (Calder, Lawrence, & Young, 2001). In lesion studies, damage to the amygdala is shown to result in impairment in the ability to recognize fear from facial expressions. Similarly, damage to the insula can result in impairment in the ability to recognize disgust from facial expressions. However, questions remain regarding exactly which emotion category or dimension is being encoded in these brain regions, since other emotions are often also variably impaired. Especially informative have been studies in a rare patient, SM, who has total damage to the amygdala on both sides of the brain. The subject can accurately judge age and gender from faces, and she has no difficulty recognizing familiar individuals from their faces. She also has little difficulty recognizing most facial expressions, with the notable exception of fear. When asked to judge the fearfulness of faces, SM is selectively and severely impaired (Adolphs, Tranel, Damasio, & Damasio, 1994). However, other patients with similar damage are impaired on a wider array of emotions; in functional neuroimaging studies, activation of the amygdala is seen in response to several emotions in addition to fear; and even SM’s highly specific impairment depends on the questions she is asked. For example, when asked to classify faces into categories of basic emotions, SM is rather selectively impaired on fear; but when asked to rate how arousing the emotion shown in the face is, she is impaired on all emotions of negative valence (Adolphs, Russell, & Tranel, 1999). These data suggest that, while the amygdala is critical for recognizing fear in faces, its role encompasses a broader, more abstract, and perhaps dimensional rather than categorical aspect of emotions, for which fear is but one instance—an issue I take up again below. MANY BRAIN STRUCTURES ARE INVOLVED Just as there is good evidence that the amygdala does more than solely detect fear, there are of course brain structures other than the amygdala that participate in perceiving emotion from faces. Brain-imaging studies, in particular, have found evidence for a large number of other brain structures that may come into play, depending on the emotion shown in the face and on the demands of the experimental task. One framework that summarizes these data runs as follows. Visual areas in the temporal cortex encode the face’s features and bind them into a global perceptual representation of the face. Subcortical visual areas (such as the superior colliculus, prominent in animals such as frogs) carry out coarser but faster processing of the face. Both routes, the cortical and the subcortical, feed visual information about the face into the amygdala. The amygdala then associates the visual representation of the face with an emotional response in the body, as well as with changes in the operation of other brain structures. For instance, it likely triggers autonomic responses (such as skin-conductance response, the sympathetic autonomic response of sweating on the palms of the hands) to the face, and it also modulates visual attention to the face. Tracing the path from initial perception of the face to recognizing the emotion it expresses is complicated by feedback loops and by multiple pathways that can be engaged. One class of proposals, as influential as they are controversial, argues that the emotional response elicited by the amygdala can in fact be used by the viewer to reconstruct knowledge about the emotion shown in the face. Roughly: if I experience a pang of fear within myself upon seeing a fearful face, I can use the knowledge of my own emotion to infer what the emotional state of the person whose face is shown in the stimulus might be. Broader theories in a similar vein do not restrict themselves to the amygdala or to fear, but more generally propose that we make inferences about other people’s emotional states by simulating within ourselves aspects of those states (Goldman & Sripada, 2005). Emotional contagion and imitation may be the earliest aspects of such a mechanism that can already be seen in infants

We test the possibility of tapping the subconscious mind for face recognition using consumer-grade BCIs. To this end, we performed an experiment whereby subjects were presented with photographs of famous persons with the expectation that about 20% of them would be (consciously) recognized; and since the photos are of famous persons, we expected that subjects would have seen before some of the 80% they didn’t (consciously) recognize. Further, we expected that their subconscious would have recognized some of those in the 80% pool that they had seen before. An exit questionnaire and a set of criteria allowed us to label recognitions as conscious, false, no recognitions, or subconscious recognitions. We analyzed a number of event related potentials training and testing a support vector machine. We found that our method is capable of differentiating between no recognitions and subconscious recognitions with promising accuracy levels, suggesting that tapping the subconscious mind for face recognition is feasible.

We present a method for creating black-and-white illustrations from photographs of human faces. In addition an interactive technique is demonstrated for deforming these black-and-white facial illustrations to create caricatures which highlight and exaggerate representative facial features. We evaluate the effectiveness of the resulting images through psychophysical studies to assess accuracy and speed in both recognition and learning tasks. These studies show that the facial illustrations and caricatures generated using our techniques are as effective as photographs in recognition tasks. For the learning task we find that illustrations are learned two times faster than photographs and caricatures are learned one and a half times faster than photographs. Because our techniques produce images that are effective at communicating complex information, they are useful in a number of potential applications, ranging from entertainment and education to low bandwidth telecommunications and psychology research.

In this paper, an analysis of the effect of partial occlusion on facial expression recognition is investigated. The classification from partially occluded images in one of the six basic facial expressions is performed using a method based on Gabor wavelets texture information extraction, a supervised image decomposition method based on Discriminant Non-negative Matrix Factorization and a shape-based method that exploits the geometrical displacement of certain facial features. We demonstrate how partial occlusion affects the above mentioned methods in the classification of the six basic facial expressions, and indicate the way partial occlusion affects human observers when recognizing facial expressions. An attempt to specify which part of the face (left, right, lower or upper region) contains more discriminant information for each facial expression, is also made and conclusions regarding the pairs of facial expressions misclassifications that each type of occlusion introduces, are drawn. 2008 Published by Elsevier B.V.

People routinely make various trait judgments from facial appearance, and such judgments affect important social outcomes. These judgments are highly correlated with each other, reflecting the fact that valence evaluation permeates trait judgments from faces. Trustworthiness judgments best approximate this evaluation, consistent with evidence about the involvement of the amygdala in the implicit evaluation of face trustworthiness. Based on computer modeling and behavioral experiments, I argue that face evaluation is an extension of functionally adaptive systems for understanding the communicative meaning of emotional expressions. Specifically, in the absence of diagnostic emotional cues, trustworthiness judgments are an attempt to infer behavioral intentions signaling approach/avoidance behaviors. Correspondingly, these judgments are derived from facial features that resemble emotional expressions signaling such behaviors: happiness and anger for the positive and negative ends of the trustworthiness continuum, respectively. The emotion overgeneralization hypothesis can explain highly efficient but not necessarily accurate trait judgments from faces, a pattern that appears puzzling from an evolutionary point of view and also generates novel predictions about brain responses to faces. Specifically, this hypothesis predicts a nonlinear response in the amygdala to face trustworthiness, confirmed in functional magnetic resonance imaging (fMRI) studies, and dissociations between processing of facial identity and face evaluation, confirmed in studies with developmental prosopagnosics. I conclude with some methodological implications for the study of face evaluation, focusing on the advantages of formally modeling representation of faces on social dimensions.


Question: A reversed Yarbus procedure: forecasting the observer's task from eye movement patterns.

This document introduces a versatile algorithm for calculating maximum likelihood estimates from data that is incomplete, applicable across various levels of generality. The theoretical underpinnings that ensure the monotonic increase of the likelihood and the algorithm's convergence are thoroughly explored. The document outlines a multitude of scenarios where this methodology can be applied, encompassing situations with missing values, grouped, censored, or truncated data, finite mixture models, variance component estimation, hyperparameter estimation, iteratively reweighted least squares, and factor analysis. The algorithm, known as the Expectation-Maximization (EM) algorithm, has been widely discussed and is celebrated for its ability to handle incomplete data sets effectively, making it a cornerstone in statistical analysis and machine learning. By iteratively estimating the missing data (the E-step) and then maximizing the likelihood (the M-step), the EM algorithm ensures that the likelihood of the observed data increases with each iteration until convergence is achieved. This approach has proven invaluable in a range of fields, from bioinformatics to econometrics, where data may be incomplete due to various reasons, such as censoring, truncation, or missing values.

To fill the semantic gap between the predictive power of computational saliency models and human behavior, this paper proposes to predict where people look at using spatial-aware object-level cues. While object-level saliency has been recently suggested by psychophysics experiments and shown effective with a few computational models, the spatial relationship between the objects has not yet been explored in this context. We in this work for the first time explicitly model such spatial relationship, as well as leveraging semantic information of an image to enhance object-level saliency modeling. The core computational module is a graphlet-based (i.e., graphlets are moderate-sized connected subgraphs) deep architecture, which hierarchically learns a saliency map from raw image pixels to object-level graphlets (oGLs) and further to spatial-level graphlets (sGLs). Eye tracking data are also used to leverage human experience in saliency prediction. Experimental results demonstrate that the proposed oGLs and sGLs well capture object-level and spatial-level cues relating to saliency, and the resulting saliency model performs competitively compared with the state-of-the-art.

Drawing upon fundamental research in human behavior prediction, recently there has been a research focus on how to predict driver behaviors. In this paper we review the field of driver behavior and intent prediction, with a specific focus on tactical maneuvers, as opposed to operational or strategic maneuvers. The aim of a driver behavior prediction system is to forecast the trajectory of the vehicle prior in real-time, which could allow a Driver Assistance System to compensate for dangerous or uncomfortable circumstances. This review provides insights into the scope of the problem, as well as the inputs, algorithms, performance metrics, and shortcomings in the state-of-the-art systems.

A vision-based real-time driver fatigue detection system is proposed for driving safely. The driver's face is located, from color images captured in a car, by using the characteristic of skin colors. Then, edge detection is used to locate the regions of eyes. In addition to being used as the dynamic templates for eye tracking in the next frame, the obtained eyes' images are also used for fatigue detection in order to generate some warning alarms for driving safety. The system is tested on a Pentium III 550 CPU with 128 MB RAM. The experiment results seem quite encouraging andpromising. The system can reach 20 frames per second for eye tracking, and the average correct rate for eye location and tracking can achieve 99.1% on four test videos. The correct rate for fatigue detection is l00%, but the average precision rate is 88.9% on the test videos.

An important aspect of human perception is anticipation and anticipating which activities will a human do next (and how to do them) in useful for many applications, for example, anticipation enables an assistive robot to plan ahead for reactive responses in the human environments. In this work, we present a constructive approach for generating various possible future human activities by reasoning about the rich spatial-temporal relations through object affordances. We represent each possible future using an anticipatory temporal conditional random field (ATCRF) where we sample the nodes and edges corresponding to future object trajectories and human poses from a generative model. We then represent the distribution over the potential futures using a set of constructed ATCRF particles. In extensive evaluation on CAD-120 human activity RGB-D dataset, for new subjects (not seen in the training set), we obtain an activity anticipation accuracy (defined as whether one of top three predictions actually happened) of 75.4%, 69.2% and 58.1% for an anticipation time of 1, 3 and 10 seconds respectively. 1

Gaze reflects how humans process visual scenes and is therefore increasingly used in computer vision systems. Previous works demonstrated the potential of gaze for object-centric tasks, such as object localization and recognition, but it remains unclear if gaze can also be beneficial for scene-centric tasks, such as image captioning. We present a new perspective on gaze-assisted image captioning by studying the interplay between human gaze and the attention mechanism of deep neural networks. Using a public large-scale gaze dataset, we first assess the relationship between state-of-the-art object and scene recognition models, bottom-up visual saliency, and human gaze. We then propose a novel split attention model for image captioning. Our model integrates human gaze information into an attention-based long short-term memory architecture, and allows the algorithm to allocate attention selectively to both fixated and non-fixated image regions. Through evaluation on the COCO/SALICON datasets we show that our method improves image captioning performance and that gaze can complement machine attention for semantic scene understanding tasks.

We explore the feasibility of measuring learner engagement and classifying the engagement level based on machine learning applied on data from 2D/3D camera sensors and eye trackers in a 1:1 learning setting. Our results are based on nine pilot sessions held in a local high school where we recorded features related to student engagement while consuming educational content. We label the collected data as Engaged or NotEngaged while observing videos of the students and their screens. Based on the collected data, perceptual user features (e.g., body posture, facial points, and gaze) are extracted. We use feature selection and classification methods to produce classifiers that can detect whether a student is engaged or not. Accuracies of up to 85-95% are achieved on the collected dataset. We believe our work pioneers in the successful classification of student engagement based on perceptual user features in a 1:1 authentic learning setting.

Perception and emotion interact, as is borne out by studies of how people recognize emotion from facial expressions. Psychological and neurological research has elucidated the processes, and the brain structures, that participate in facial emotion recognition. Studies have shown that emotional reactions to viewing faces can be very rapid and that these reactions may, in turn, be used to judge the emotion shown in the face. Recent experiments have argued that people actively explore facial expressions in order to recognize the emotion, a mechanism that emphasizes the instrumental nature of social cognition. KEYWORDS—face perception; faces; emotion; amygdala An influential psychological model of face processing argued that early perception (construction of a geometric representation of the face based on its features) led to subsequently separate processing of the identity of the face and of the emotional expression of the face (Bruce & Young, 1986). That model has received considerable support from neuroscience studies suggesting that the separate processes are based on separable neuroanatomical systems: In neuroimaging studies, different parts of the brain are activated in response to emotional expressions or identity changes; and brain damage can result in impairments in recognizing identity but not in recognizing emotional expressions, or the reverse. SOME PROCESSING IS RAPID AND AUTOMATIC AND CAN OCCUR IN THE ABSENCE OF AWARENESS OF THE STIMULI Some responses in the brain to emotional facial expressions are so rapid (less than 100 ms) that they could not plausibly be based on conscious awareness of the stimulus, although the responses, in turn, may contribute to conscious awareness. Evidence comes from studies using event-related potentials, measures of the brain’s electrical activity recorded at the scalp (or, much more rarely, directly from the brain in surgical patients). In those experiments, the responses to many presentations of emotional stimuli are averaged across stimuli, and the changes in electrical potential can be accurately timed in relation to stimulus onset. Evidence has also come from studies in which viewers were shown facial expressions subliminally. Especially salient aspects of faces are most potent in driving nonconscious responses: For instance, subliminal presentation of only the whites of the eyes of fearful faces results in measurable brain activation (Whalen et al., 2004). One specific structure involved in such rapid and automatic neural responses is the amygdala, a structure in the medial temporal lobe that is known to be involved in many aspects of emotion processing. Findings such as these have been of interest to psychologists, as they provide a potential mechanism consistent with two-factor theories of human behavior. For instance, the theory that affect and cognitive judgment are separate processes, and that affect can precede cognitive judgment, receives some support from the neuroscience findings (Zajonc, 1980). The data also add detail to theories of visual consciousness. In one set of studies, emotional facial expressions were shown to neurological patients who, because of their brain damage, were unable to report seeing the stimuli. An individual with damage to the visual cortex had ‘‘blindsight’’ for emotional faces: He could discriminate the emotion shown in faces by guessing, even though he reported no visual experience of seeing the faces. A neuroimaging experiment in this same individual revealed that the amygdala was activated differentially by different emotional faces, despite the absence of conscious visual experience (Morris, deGelder, Weiskrantz, & Dolan, 2001). These and other studies have suggested a distinction between subcortical processing of emotional visual stimuli (e.g., involving the amygdala and brainstem nuclei such as the superior colliculus), which may be independent of conscious vision (Johnson, 2005), and cortical processing, which is usually accompanied by conscious experience. It is interesting to note that amphibians and reptiles have only subcortical visual processing, since they lack a neocortex. One broad interpretation of these observations is thus that the subcortical route for processing emotional stimuli is the more ancient one, and that in Address correspondence to Ralph Adolphs, HSS 228-77, Caltech, Pasadena, CA 91125; e-mail: radolphs@hss.caltech.edu. CURRENT DIRECTIONS IN PSYCHOLOGICAL SCIENCE 222 Volume 15—Number 5 Copyright r 2006 Association for Psychological Science mammals an additional, cortical route has evolved that probably allows more flexible behaviors based on learning and conscious deliberation. A final wrinkle has come from psychological studies of the relationship between attention and emotion. It has been known for some time that emotionally salient stimuli can capture attention, an interaction that makes much adaptive sense (Ohman, Flykt, & Esteves, 2001). But more recent studies have also shown the reverse: that volitionally allocating attention to stimuli can influence their emotional evaluation. Inhibiting attention to visual stimuli that are distractors in a search task, for example, results in devaluation of those stimuli when subjects are asked to rate them on affective dimensions (Raymond, Fenske, & Tavassoli, 2003). Emotions thus represent the value of stimuli—what people approach or avoid, cognitively or behaviorally, volitionally or automatically. FEAR AND DISGUST MAY BE PROCESSED SPECIALLY Several studies have found evidence that the amygdala, the subcortical structure discussed earlier, is disproportionately important for processing facial expressions of fear, although by no means exclusively so. The only other emotion for which a similar neuroanatomical specificity has been reported is disgust. A region of the cortex called the insula represents motor and sensory aspects of that emotional response (Calder, Lawrence, & Young, 2001). In lesion studies, damage to the amygdala is shown to result in impairment in the ability to recognize fear from facial expressions. Similarly, damage to the insula can result in impairment in the ability to recognize disgust from facial expressions. However, questions remain regarding exactly which emotion category or dimension is being encoded in these brain regions, since other emotions are often also variably impaired. Especially informative have been studies in a rare patient, SM, who has total damage to the amygdala on both sides of the brain. The subject can accurately judge age and gender from faces, and she has no difficulty recognizing familiar individuals from their faces. She also has little difficulty recognizing most facial expressions, with the notable exception of fear. When asked to judge the fearfulness of faces, SM is selectively and severely impaired (Adolphs, Tranel, Damasio, & Damasio, 1994). However, other patients with similar damage are impaired on a wider array of emotions; in functional neuroimaging studies, activation of the amygdala is seen in response to several emotions in addition to fear; and even SM’s highly specific impairment depends on the questions she is asked. For example, when asked to classify faces into categories of basic emotions, SM is rather selectively impaired on fear; but when asked to rate how arousing the emotion shown in the face is, she is impaired on all emotions of negative valence (Adolphs, Russell, & Tranel, 1999). These data suggest that, while the amygdala is critical for recognizing fear in faces, its role encompasses a broader, more abstract, and perhaps dimensional rather than categorical aspect of emotions, for which fear is but one instance—an issue I take up again below. MANY BRAIN STRUCTURES ARE INVOLVED Just as there is good evidence that the amygdala does more than solely detect fear, there are of course brain structures other than the amygdala that participate in perceiving emotion from faces. Brain-imaging studies, in particular, have found evidence for a large number of other brain structures that may come into play, depending on the emotion shown in the face and on the demands of the experimental task. One framework that summarizes these data runs as follows. Visual areas in the temporal cortex encode the face’s features and bind them into a global perceptual representation of the face. Subcortical visual areas (such as the superior colliculus, prominent in animals such as frogs) carry out coarser but faster processing of the face. Both routes, the cortical and the subcortical, feed visual information about the face into the amygdala. The amygdala then associates the visual representation of the face with an emotional response in the body, as well as with changes in the operation of other brain structures. For instance, it likely triggers autonomic responses (such as skin-conductance response, the sympathetic autonomic response of sweating on the palms of the hands) to the face, and it also modulates visual attention to the face. Tracing the path from initial perception of the face to recognizing the emotion it expresses is complicated by feedback loops and by multiple pathways that can be engaged. One class of proposals, as influential as they are controversial, argues that the emotional response elicited by the amygdala can in fact be used by the viewer to reconstruct knowledge about the emotion shown in the face. Roughly: if I experience a pang of fear within myself upon seeing a fearful face, I can use the knowledge of my own emotion to infer what the emotional state of the person whose face is shown in the stimulus might be. Broader theories in a similar vein do not restrict themselves to the amygdala or to fear, but more generally propose that we make inferences about other people’s emotional states by simulating within ourselves aspects of those states (Goldman & Sripada, 2005). Emotional contagion and imitation may be the earliest aspects of such a mechanism that can already be seen in infants

Sentiment analysis is crucial for extracting social signals from social media content. Due to the prevalence of images in social media, image sentiment analysis is receiving increasing attention in recent years. However, most existing systems are black-boxes that do not provide insight on how image content invokes sentiment and emotion in the viewers. On the other hand, psychological studies have confirmed that salient objects in an image often invoke emotions. In this work, we investigate more fine-grained and more comprehensive interaction between visual saliency and visual sentiment. In particular, we partition images in several meta-level scene-type dimensions that are relevant to most images, including: open-closed, natural-manmade, indoor-outdoor, and face-noface. Facilitated by state of the art saliency detection algorithm and sentiment classification algorithm, we examine how the sentiment of the salient region(s) in an image relates to the overall sentiment of the image. The experiments on a representative image emotion dataset have shown interesting correlation between saliency and sentiment in different scene types and shed light on the mechanism of visual sentiment evocation.

This paper is conceived as a guide that will describe the very well known Spatial Orienting paradigm, used to explore attentional processes in healthy individuals as well as in people suffering from psychiatric disorders and brain-damaged patients. The paradigm was developed in the late 1970s, and since then, it has been used in thousands of attentional studies. In this review, we attempt to describe, the paradigm for the naïf reader, and explain in detail when is it used, which variables are usually manipulated, how to interpret its results, and how can it be adapted to different populations and methodologies. The main goal of this review is to provide a practical guide to researchers who have never used the paradigm that will help them design their experiments, as a function of their theoretical and experimental needs. We also focus on how to adapt the paradigm to different technologies (such as event-related potentials, functional resonance imaging, or transcranial magnetic stimulation), and to different populations by presenting an example of its use in brain-damaged patients.

The concept of attention as central to human performance extends back to the start of experimental psychology (James 1890), yet even a few years ago, it would not have been possible to outline in even a preliminary form a functional anatomy of the human attentional system. New developments in neuroscience (Hillyard & Picton 1987, Raichle 1983, Wurtz et al 1980) have opened the study of higher cognition to physiological analysis, and have revealed a system of anatomical areas that appear to be basic to the selection of information for focal (conscious) processing. The importance of attention is its unique role in connecting the mental level of description of processes used in cognitive science with the anatomical level common i neuroscience. Sperry (1988, p. 609) describes the central role that mental concepts play in understanding brain function as follows:


Question: Can we grant driving licenses to autonomous vehicles?

The imperative of comprehending the rationale behind the decisions made by machine learning algorithms cannot be overstated, as it significantly impacts system designers and end-users alike. This understanding is crucial for model selection, feature engineering, and fostering trust in predictions, thereby facilitating more intuitive user interfaces. Consequently, the demand for interpretability in machine learning has surged, rekindling interest in interpretable models. In certain scenarios, these models match the accuracy of their non-interpretable counterparts, making them the preferred choice due to their transparency. Even when they fall short in accuracy, interpretable models may still be favored when clarity is of utmost importance. Nonetheless, limiting machine learning to solely interpretable models often imposes significant constraints. This paper advocates for the utilization of model-agnostic approaches to elucidate machine learning predictions. By considering these models as black boxes, these approaches offer enhanced flexibility in selecting models, explanations, and representations, thereby streamlining debugging, comparison, and user interfaces for a wide array of users and models. We also delineate the primary challenges associated with such methods and review a recently proposed model-agnostic explanation technique (LIME) that effectively tackles these challenges. This approach enables a deeper understanding of machine learning models without compromising on the choice of models, thereby broadening the scope for interpretability in complex systems.

Avoiding high computational costs and calibration issues involved in stereo-vision-based algorithms, this paper proposes real-time monocular-vision-based techniques for simultaneous vehicle detection and inter-vehicle distance estimation, in which the performance and robustness of the system remain competitive, even for highly challenging benchmark datasets. This paper develops a collision warning system by detecting vehicles ahead and, by identifying safety distances to assist a distracted driver, prior to occurrence of an imminent crash. We introduce adaptive global Haar-like features for vehicle detection, tail-light segmentation, virtual symmetry detection, intervehicle distance estimation, as well as an efficient single-sensor multifeature fusion technique to enhance the accuracy and robustness of our algorithm. The proposed algorithm is able to detect vehicles ahead at both day or night and also for short- and long-range distances. Experimental results under various weather and lighting conditions (including sunny, rainy, foggy, or snowy) show that the proposed algorithm outperforms state-of-the-art algorithms.

In this paper, we propose an efficient vehicle trajectory prediction framework based on recurrent neural network. Basically, the characteristic of the vehicle's trajectory is different from that of regular moving objects since it is affected by various latent factors including road structure, traffic rules, and driver's intention. Previous state of the art approaches use sophisticated vehicle behavior model describing these factors and derive the complex trajectory prediction algorithm, which requires a system designer to conduct intensive model optimization for practical use. Our approach is data-driven and simple to use in that it learns complex behavior of the vehicles from the massive amount of trajectory data through deep neural network model. The proposed trajectory prediction method employs the recurrent neural network called long short-term memory (LSTM) to analyze the temporal behavior and predict the future coordinate of the surrounding vehicles. The proposed scheme feeds the sequence of vehicles' coordinates obtained from sensor measurements to the LSTM and produces the probabilistic information on the future location of the vehicles over occupancy grid map. The experiments conducted using the data collected from highway driving show that the proposed method can produce reasonably good estimate of future trajectory.

In the near future, humans will be relieved from parking. Major improvements in autonomous driving allow the realization of automated valet parking (AVP). It enables the vehicle to drive to a parking spot and park itself. This paper presents a review of the intelligent vehicles literature on AVP. An overview and analysis of the core components of AVP such as the platforms, sensor setups, maps, localization, perception, environment model, and motion planning is provided. Leveraging the potential of AVP, high density parking (HDP) is reviewed as a future research direction with the capability to either reduce the necessary space for parking by up to 50 % or increase the capacity of future parking facilities. Finally, a synthesized view discussing the remaining challenges in automated valet parking and the technological requirements for high density parking is given.

Cooperative driving behavior is essential for driving in traffic, especially for ramp merging, lane changing or navigating intersections. Autonomous vehicles should also manage these situations by behaving cooperatively and naturally. The challenge of cooperative driving is estimating other vehicles' intentions. In this paper, we present a novel method to estimate other human-driven vehicles' intentions with the aim of achieving a natural and amenable cooperative driving behavior, without using wireless communication. The new approach allows the autonomous vehicle to cooperate with multiple observable merging vehicles on the ramp with a leading vehicle ahead of the autonomous vehicle in the same lane. To avoid calculating trajectories, simplify computation, and take advantage of mature Level-3 components, the new method reacts to merging cars by determining a following target for an off-the-shelf distance keeping module (ACC) which governs speed control of the autonomous vehicle. We train and evaluate the proposed model using real traffic data. Results show that the new approach has a lower collision rate than previous methods and generates more human driver-like behaviors in terms of trajectory similarity and time-to-collision to leading vehicles.

The authors have been conducting research on an autonomous system for loading operation by wheel loader. Experimental results at a field test site using full-size model (length: 6.1m) will be described in this paper. Basic structure of system consists of three sub systems: measuring and modeling of environment, task planning and motion control. The experimental operation includes four cycles of scooping and loading to dump truck. The experimental results prove that the developed system performed the autonomous operation smoothly and completed the mission.

Amidst rapid urban development, sustainable transportation solutions are required to meet the increasing demands for mobility whilst mitigating the potentially negative social, economic, and environmental impacts. This study analyses autonomous vehicles (AVs) as a potential transportation solution for smart and sustainable development. We identified privacy and cybersecurity risks of AVs as crucial to the development of smart and sustainable cities and examined the steps taken by governments around the world to address these risks. We highlight the literature that supports why AVs are essential for smart and sustainable development. We then identify the aspects of privacy and cybersecurity in AVs that are important for smart and sustainable development. Lastly, we review the efforts taken by federal governments in the US, the UK, China, Australia, Japan, Singapore, South Korea, Germany, France, and the EU, and by US state governments to address AV-related privacy and cybersecurity risks in-depth. Overall, the actions taken by governments to address privacy risks are mainly in the form of regulations or voluntary guidelines. To address cybersecurity risks, governments have mostly resorted to regulations that are not specific to AVs and are conducting research and fostering research collaborations with the private sector.

Part I of this paper proposed a development process and a system platform for the development of autonomous cars based on a distributed system architecture. The proposed development methodology enabled the design and development of an autonomous car with benefits such as a reduction in computational complexity, fault-tolerant characteristics, and system modularity. In this paper (Part II), a case study of the proposed development methodology is addressed by showing the implementation process of an autonomous driving system. In order to describe the implementation process intuitively, core autonomous driving algorithms (localization, perception, planning, vehicle control, and system management) are briefly introduced and applied to the implementation of an autonomous driving system. We are able to examine the advantages of a distributed system architecture and the proposed development process by conducting a case study on the autonomous system implementation. The validity of the proposed methodology is proved through the autonomous car A1 that won the 2012 Autonomous Vehicle Competition in Korea with all missions completed.

The automotive industry is moving aggressively in the direction of advanced active safety. Dedicated short-range communication (DSRC) is a key enabling technology for the next generation of communication-based safety applications. One aspect of vehicular safety communication is the routine broadcast of messages among all equipped vehicles. Therefore, channel congestion control and broadcast performance improvement are of particular concern and need to be addressed in the overall protocol design. Furthermore, the explicit multichannel nature of DSRC necessitates a concurrent multichannel operational scheme for safety and non-safety applications. This article provides an overview of DSRC based vehicular safety communications and proposes a coherent set of protocols to address these requirements

The knowledge about lanes and the exact position on the road is fundamental for many advanced driver assistance systems. In this paper, a novel iterative histogram based approach with occupancy grids for the detection of multiple lanes is proposed. In highway scenarios, our approach is highly suitable to determine the correct number of all existing lanes on the road. Additionally, the output of the laserscannner based lane detection is fused with a production-available vision based system. It is shown that both sensor systems perfectly complement each other to increase the robustness of a lane tracking system. The achieved accuracy of the fusion system, the laserscannner and video based system is evaluated with a highly accurate DGPS to investigate the performance with respect to lateral vehicle control applications.

In this paper, we pose a new challenge for AI researchers – to develop intelligent systems that support justified agency. We illustrate this ability with examples and relate it to two more basic topics that are receiving increased attention – agents that explain their decisions and ones that follow societal norms. In each case, we describe the target abilities, consider design alternatives, note some open questions, and review prior research. After this, we return to justified agency, offering a hypothesis about its relation to explanatory and normative behavior. We conclude by proposing testbeds and experiments to evaluate this empirical claim and encouraging other researchers to contribute to this crucial area. 1 Background and Motivation Autonomous artifacts, from self-driving cars to drones to household robots, are becoming more widely adopted, and this trend seems likely to accelerate in the future. The increasing reliance on these devices has raised concerns about our ability to understand their behavior and our capacity to ensure their safety. Before intelligent agents can gain widespred acceptance in society, they must be able to communicate their decision making to humans in ways that convince us they actually share our aims. This challenge involves two distinct but complementary issues. The first is the need for agents to explain the reasons they carried out particular courses of action in terms that we can understand. Langley et al. (2017) have referred to this as explainable agency. The second is the need for assurance that, when agents pursue explicit goals, they will also follow the many implicit rules of society. We will refer to this ability as normative agency. Both of these functions are necessary underpinnings of trustable autonomous systems, and two capabilities are closely intertwined. Consider a scenario in which a person drives a friend with a ruptured appendix to the hospital. The driver exceeds the speed limit, weaves in and out of traffic, runs through red lights, and even drives on a sidewalk, although he is still careful to avoid hitting other cars or losing control on turns. Afterwards, the driver explains that he took such drastic actions because he thought the passenger’s life was in danger, Copyright c © 2019, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. so reaching the hospital in short order had higher priority than being polite to others along the way or obeying traffic laws. Humans can defend their behavior in this manner even when they violate serious societal norms, and we want intelligent agents of the future to exhibit the same ability to justify their decisions and actions on demand. We discuss this challenge in the sections that follow, arguing that such justified agency is an important topic that deserves substantial attention from AI researchers. We first examine the two related topics of explainable agency and normative agency. In each case, we describe the desired abilities, offer illustrative examples, and touch on relevant research. We also consider the space of agent designs, including some open issues that require additional work. After this, we turn to justified agency, arguing that it combines the first two abilities and proposing a hypothesis about what else is required to support it. We close by considering some testbeds and experiments that would let the research community evaluate progress in this critical arena. 2 Explainable Agency People can usually explain their reasons for making decisions and taking actions. When someone purchases a microwave oven, rearranges furniture in a room, or plans a vacation, he can state the choices considered, why he selected one alternative over the others, and even how his decision might have differed in other circumstances. Retrospective reports are seldom perfect, but they often offer important windows into the decision process. As intelligent agents become both more autonomous and more prevalent, it is essential that they support similar abilities. We will say that: • An intelligent system exhibits explainable agency if it can provide, on request, the reasons for its activities. Consider some examples from the realm of autonomous vehicles. If we ask a self-driving car why it followed a given route, it should be able to state that the path had few lights and stop signs while still being reasonably short. More importantly, when we ask the vehicle why it swerved into another car, it should explain that it was the only way to avoid hitting an unexpected jaywalker. Like humans, such agents should have reasons for their actions and they should be able to communicate them to others when asked. Tackling this problem requires that we make design decisions about representations and processes needed to support explainable agency, some of which we will borrow from Langley et al.’s (2017) analysis. One issue concerns what will count as legitimate explanations. Should plausible post hoc rationalizations be acceptable if an agent’s decisionmaking procedures are not interpretable or should we require genuine insights into why the agent took its actions? We argue that only the latter should be viewed as reasonable accounts, which implies that the agents should make decisions in ways that are transparent and easily communicated. There are many well-established methods in the AI arsenal that meet this criterion, but opaque policies induced from large training sets will not suffice. Another design choice involves whether to explain activities in terms of individual actions or higher-level structures. Research in machine learning has emphasized reactive control (e.g., Erwig et al., 2018), which supports the first alternative, whereas AI planning systems make choices about entire sequences of actions. We predict that humans will find plan-oriented explanations more familiar and easier to understand, and thus offer a natural approach to adopt. Even when a plan goes awry during execution, an agent can still give the reasons it decided to change course and how its new plan responded to the surprise. Some frameworks, like hierarchical task networks, also specify plans at multiple levels of abstraction, which would let a system offer more or less detailed explanations, down to individual actions if desired. Any intelligent agent must use information to make decisions about which activities to pursue, and a third issue concerns how to encode this content. One response, widely adopted in AI planning research, relies on symbolic goals, often stated as logical expressions. Another option, popular in game-playing systems, instead uses numeric evaluation functions that comprise sets of weighted features. Each approach has advantages for explainable agency: goals provide explicit end points for chains of actions, while functions show how such plans handle trade offs. Although these typically appear in isolation, they can also be combined. For instance, Langley et al. (2016) describe an agent architecture that associates functions with goals, using their weighted sum to guide planning. Such hybrid frameworks offer one promising approach to building explainable agents. Within this design space, we still need research on a number of open issues about explainable agency: These include extending intelligent systems to: • Generate explanatory content. When deciding on courses of action, an agent must consider different alternatives, evaluate them, and select one of the options to pursue. This should take place during generation of plans and during their execution, producing traces that can be used in later explanations of the agent’s activities. • Store generated content. As it makes these decisions, an agent must cache information about the choices that it considered and the reasons that it favored one over the others, in an episodic memory or similar repository. This requires not just retaining the content about decisions, but also indexing it in ways that support later access. • Retrieve stored content. After it has completed an activity, an agent must be able to retrieve decision traces that are relevant to different types of questions. This requires transforming the queries into cues and using them to access content in episodic memory about alternatives considered, their evaluations, and the final choices made. • Communicate retrieved content. Once it has retrieved episodic traces in response to a question, an agent must identify those aspects most relevant to the query, translate them into an understandable form, and share the answer. This should include no more detail than needed to convey the reasons for making the decision under examination. Research on analogical planning (e.g., Jones and Langley, 2005; Veloso et al., 1995) has addressed issues of storage, indexing, and retrieval, but not for the purpose of self report. Leake (1992) presented a computational theory of explanation, but it focused on accounts of other agents’ behaviors. The AI literature also includes other research relevant to this topic. Early explanation systems recorded inference chains and recounted them when asked to justify their conclusions (Swartout, Paris, and Moore, 1991), with some systems supporting higher-level accounts with meta-level rules (Clancey, 1983), but these did not deal with physical activities. More relevant work comes from Johnson (1994) and van Lent et al. (2004), who developed agents for military mssions that recorded their decisions, offered reasons on request, and anwered counterfactual queries. However, they dealt with knowledge-guided execution rather than agentgenerated plans and, despite linking actions to objectives, did not state why some activities were preferable to others. In more recent work, Briggs and Scheutz (2015) have reported an interactive robot that gives reasons why it cannot carry out some task, drawing on five explanation types, including lack of knowledge and physical ability. Th


Question: Can we consider granting driving licenses to autonomous vehicles?

This paper introduces an innovative safety verification framework for deep neural networks, particularly relevant to the realm of autonomous vehicles. Despite their remarkable success in image classification tasks, deep neural networks have been found to be susceptible to adversarial perturbations—subtle alterations to input images that can lead to misclassification. This vulnerability is particularly concerning when considering the potential deployment of these networks in critical applications such as self-driving car perception modules and end-to-end controllers. To address these safety concerns, we propose a method based on Satisfiability Modulo Theory (SMT) to automatically verify the robustness of feed-forward multi-layer neural networks. Our focus is on common image manipulations, including scratches, changes in camera angle, or variations in lighting conditions. We define the safety of an image classification decision as the invariance of the classification under these manipulations, within a region of images that are close to the original. To enable an exhaustive search of this region, we apply discretization techniques and propagate the analysis through each layer of the network. Unlike existing methods, our framework can provide a guarantee that all potential adversarial examples, if they exist, are identified for a given region and set of manipulations. The discovery of such examples can then facilitate human testing or network refinement. We have implemented our techniques using the Z3 solver and tested them on cutting-edge networks, including those with regularization and deep learning architectures. Our results are compared against existing techniques for adversarial example detection and network robustness estimation.

In the context of autonomous vehicles, this research has profound implications for the safety and reliability of AI systems. Just as human drivers must demonstrate their ability to navigate various road conditions and unexpected scenarios to obtain a license, autonomous vehicles should undergo rigorous safety verification before being granted permission to operate on public roads. The proposed framework could serve as a critical tool in the certification process, ensuring that AI-driven vehicles are capable of handling a wide range of driving conditions without succumbing to adversarial perturbations that could compromise safety. By extending the safety verification techniques developed for image classification to the broader domain of autonomous vehicle decision-making, we can move closer to the day when AI systems are trusted to navigate the complexities of the open road with the same reliability as human drivers.

Development and integration of physiological sensors into automotive applications is gaining importance. Assistance systems which possess knowledge about the driver's cognitive state could increase road safety. In this paper we present a flexible framework that enables the development, evaluation and verification of sensors and algorithms for automotive applications using physiological signals under realistic driving conditions. We have integrated a custom capacitive ECG measurement system into a test car and validated its performance in real world driving tests. During first test runs, the capacitive system achieved a sensitivity of up to 95.5% and a precision rate of up to 92.6%. Our system also records synchronized vehicle dynamics. We discuss the road test measurements which suggest that the driving situation highly impacts the quality of ECG signal. Therefore, information on driving dynamics could be used to improve the precision rate of future capacitive ECG measurement.

This paper addresses the real-time control of autonomous vehicles under a minimum traveling time objective. Control inputs for the vehicle are computed from a nonlinear model predictive control (MPC) scheme. The time-optimal objective is reformulated such that it can be tackled by existing efficient algorithms for real-time nonlinear MPC that build on the generalized Gauss-Newton method. We numerically validate our approach in simulations and present a real-world hardware setup of miniature race cars that is used for an experimental comparison of different approaches.

Autonomous and assisted driving are undoubtedly hot topics in computer vision. However, the driving task is extremely complex and a deep understanding of drivers' behavior is still lacking. Several researchers are now investigating the attention mechanism in order to define computational models for detecting salient and interesting objects in the scene. Nevertheless, most of these models only refer to bottom up visual saliency and are focused on still images. Instead, during the driving experience the temporal nature and peculiarity of the task influence the attention mechanisms, leading to the conclusion that real life driving data is mandatory. In this paper we propose a novel and publicly available dataset acquired during actual driving. Our dataset, composed by more than 500,000 frames, contains drivers' gaze fixations and their temporal integration providing task-specific saliency maps. Geo-referenced locations, driving speed and course complete the set of released data. To the best of our knowledge, this is the first publicly available dataset of this kind and can foster new discussions on better understanding, exploiting and reproducing the driver's attention process in the autonomous and assisted cars of future generations.

As we march down the road of automation in robotics and artificial intelligence, we will need to automate an increasing amount of ethical decision-making in order for our devices to operate independently from us. But automating ethical decision-making raises novel questions for engineers and designers, who will have to make decisions about how to accomplish that task. For example, some ethical decisionmaking involves hard moral cases, which in turn requires user input if we are to respect established norms surrounding autonomy and informed consent. The author considers this and other ethical considerations that accompany the automation of ethical decision-making. He proposes some general ethical requirements that should be taken into account in the design room, and sketches a design tool that can be integrated into the design process to help engineers, designers, ethicists, and policymakers decide how best to automate certain forms of ethical decision-making.

The authors have been conducting research on an autonomous system for loading operation by wheel loader. Experimental results at a field test site using full-size model (length: 6.1m) will be described in this paper. Basic structure of system consists of three sub systems: measuring and modeling of environment, task planning and motion control. The experimental operation includes four cycles of scooping and loading to dump truck. The experimental results prove that the developed system performed the autonomous operation smoothly and completed the mission.

A prerequisite to foster proliferation of automated driving is common system acceptance. However, different users groups (novice, enthusiasts) decline automation, which could be, in turn, problematic for a successful market launch. We see a feasible solution in the combination of the advantages of manual (autonomy) and automated (increased safety) driving. Hence, we've developed the Hotzenplotz interface, combining possibility-driven design with psychological user needs. A simulator study (N=30) was carried-out to assess user experience with subjective criteria (Need Scale, PANAS/-X, HEMA, AttrakDiff) and quantitative measures (driving behavior, HR/HRV) in different conditions. Our results confirm that pure AD is significantly less able to satisfy user needs compared to manual driving and make people feeling bored/out of control. In contrast, the Hotzenplotz interface has proven to reduce the negative effects of AD. Our implication is that drivers should be provided with different control options to secure acceptance and avoid deskilling.

Advances in both semiconductor and automotive industry are today enabling the next generation of vehicles with significant electronics content than ever before. Consumers can now avail vehicle offerings in the form of Electric and Hybrid Electric Vehicles (EV/HEV) that have improved fuel efficiency, provide enhanced driver-passenger comfort and experience through Advance Driver Assistance Systems (ADAS) and car infotainment systems, and more. Increasing electronics, software content, and connectivity drive two consumer concerns — “functional safety” and “security” — to the forefront. In this tutorial, we dissect these concerns from an end application perspective and translate the system level requirements and standards into semiconductor development requirements. We indicate both current and emerging practices, and touch upon areas requiring new or optimal design and electronic design automation (EDA) solutions. While functional safety is the primary focus for deep-dive in this tutorial, we also examine key facets of automotive security which is now emerging as a critical area for further understanding and standardization.

In recent years, many researches on Intelligent Transportation Systems (ITS) have been reported. Automatic License Plate Recognition (ALPR) is one form of ITS technology that not only recognizes and counts vehicles, but distinguishes each as unique by recognizing and recording the license plate’s characters. This paper discusses the main techniques of ALPR. Several open problems are proposed at the end of the paper for future research.

Chris Urmson, Joshua Anhalt, Drew Bagnell, Christopher Baker, Robert Bittner, M. N. Clark, John Dolan, Dave Duggins, Tugrul Galatali, Chris Geyer, Michele Gittleman, Sam Harbaugh, Martial Hebert, Thomas M. Howard, Sascha Kolski, Alonzo Kelly, Maxim Likhachev, Matt McNaughton, Nick Miller, Kevin Peterson, Brian Pilnick, Raj Rajkumar, Paul Rybski, Bryan Salesky, Young-Woo Seo, Sanjiv Singh, Jarrod Snider, Anthony Stentz, William “Red” Whittaker, Ziv Wolkowicki, and Jason Ziglar Carnegie Mellon University Pittsburgh, Pennsylvania 15213 e-mail: curmson@ri.cmu.edu

In this paper, we propose a deep reinforcement learning(DRL) algorithm which combines Deep Deterministic Policy Gradient (DDPG) with expert demonstrations and supervised loss for decision making for autonomous driving. Training DRL agent with supervised learning is adopted to accelerate the exploration process and increase the stability. A supervised loss function is introduced in the algorithm to update the actor networks. In addition, reward construction is combined to make the training process more stable and efficient. The proposed algorithm is applied to a popular autonomous driving simulator called TORCS. The experimental results show that the training efficiency and stability are improved by utilizing our algorithm in autonomous driving.


Question: Can we grant driving licenses to self-driving cars?

A Novel Framework for Autonomous Driving Utilizing Deep Reinforcement Learning
Reinforcement learning, a robust AI methodology, is increasingly recognized for its potential in teaching machines to learn from their environment and improve through trial and error. Despite its theoretical promise, the application of reinforcement learning in autonomous driving has been limited. Inspired by the success of Google DeepMind in learning complex games like Atari and Go, this paper introduces a pioneering framework for autonomous driving based on deep reinforcement learning. This framework is particularly pertinent given the difficulty in formulating autonomous driving as a supervised learning problem due to the intricate interactions with the environment, including other vehicles, pedestrians, and roadworks. Recognizing the novelty of this approach in the autonomous driving domain, we offer a concise overview of deep reinforcement learning and then detail our proposed framework. It features Recurrent Neural Networks (RNNs) for integrating information, enabling the vehicle to handle scenarios with incomplete visibility. The framework also incorporates attention models to concentrate on pertinent information, thereby optimizing computational efficiency for deployment on embedded hardware. The effectiveness of the framework was validated in an open-source 3D car racing simulator, TORCS, demonstrating the learning of autonomous maneuvers in complex road conditions and basic interactions with other vehicles. The framework's integration of DL for feature learning, RNNs for information integration, and attention models for selective focus, positions it as a comprehensive solution for autonomous driving challenges. The success of this approach not only highlights the potential of deep reinforcement learning in autonomous driving but also underscores the importance of combining multiple AI techniques to achieve advanced driving capabilities.

Could We Issue Driving Licenses to Autonomous Vehicles?
The question of whether autonomous vehicles should be granted driving licenses is a critical one, considering the increasing sophistication and prevalence of these vehicles on our roads. The licensure of autonomous vehicles would represent a significant shift in our legal and regulatory frameworks, necessitating a comprehensive understanding of their capabilities, limitations, and the ethical considerations involved. As autonomous vehicles become more advanced, employing complex AI algorithms and deep learning techniques to navigate complex environments, the traditional human-centric approach to driving licensure becomes insufficient. Instead, a new paradigm is needed to assess and certify the driving capabilities of autonomous vehicles. This would involve rigorous testing, not just in controlled environments but also in real-world scenarios, to ensure the vehicles can handle unexpected situations and make ethical decisions in critical moments. The licensure process would also need to consider the continuous learning and adaptation capabilities of autonomous vehicles, ensuring they remain up-to-date with the latest traffic rules and road conditions. Moreover, the licensure framework would have

This paper presents the vision-based road detection system currently installed onto the MOB-LAB land vehicle. Based on a geometrical transform and on a fast morphological processing, the system is capable to detect road markings even in extremely severe shadow conditions on at and structured roads. The use of a special-purpose massively architecture (PAPRICA) allows to achieve a processing rate of about 17 Hz.

There is a strong trend for increasingly sophisticated Advanced Driver Assistance Systems (ADAS) such as Autonomous Emergency Braking (AEB) systems, Lane Keeping Aid (LKA) systems, and indeed autonomous driving. This trend generates a need for online maneuver generation, for which numerous approaches can be found in the large body of work related to path planning and obstacle avoidance. In order to ease the challenge of choosing a method, this paper reports quantitative and qualitative insights about three different path planning methods: a state lattice planner, predictive constraint-based planning, and spline-based search tree. Each method is described, implemented and compared on two specific traffic situations. The paper will not provide a final answer about which method is best. This depends on several factors such as computational constraints and the formulation of maneuver optimality that is appropriate for a given assistance or safety function. Instead, the conclusions will highlight qualitative merits and drawbacks for each method, in order to provide guidance for choosing a method for a specific application.

Recent progress in autonomous and semiautonomous driving has been made possible in part through an assortment of sensors that provide the intelligent agent with an enhanced perception of its surroundings. It has been clear for quite some while now that for intelligent vehicles to function effectively in all situations and conditions, a fusion of different sensor technologies is essential. Consequently, the availability of synchronized multi-sensory data streams are necessary to promote the development of fusion based algorithms for low, mid and high level semantic tasks. In this paper, we provide a comprehensive description of our heavily sensorized, panoramic testbed capable of providing high quality data from a slew of synchronized and calibrated sensors such as cameras, LIDARs, radars, and the IMU/GPS. The vehicle has recorded over 100 hours of real world data for a very diverse set of weather, traffic and daylight conditions. All captured data is accurately calibrated and synchronized using timestamps, and stored safely in high performance servers mounted inside the vehicle itself. Details on the testbed instrumentation, sensor layout, sensor outputs, calibration and synchronization are described in this paper.

Automated driving requires decision making in dynamic and uncertain environments. The uncertainty from the prediction originates from the noisy sensor data and from the fact that the intention of human drivers cannot be directly measured. This problem is formulated as a partially observable Markov decision process (POMDP) with the intended route of the other vehicles as hidden variables. The solution of the POMDP is a policy determining the optimal acceleration of the ego vehicle along a preplanned path. Therefore, the policy is optimized for the most likely future scenarios resulting from an interactive, probabilistic motion model for the other vehicles. Considering possible future measurements of the surrounding cars allows the autonomous car to incorporate the estimated change in future prediction accuracy in the optimal policy. A compact representation results in a low-dimensional state-space. Thus, the problem can be solved online for varying road layouts and number of vehicles. This is done with a point-based solver in an anytime fashion on a continuous state-space. Our evaluation is threefold: At first, the convergence of the algorithm is evaluated and it is shown how the convergence can be improved with an additional search heuristic. Second, we show various planning scenarios to demonstrate how the introduction of different considered uncertainties results in more conservative planning. At the end, we show online simulations for the crossing of complex (unsignalized) intersections. We can demonstrate that our approach performs nearly as good as with full prior information about the intentions of the other vehicles and clearly outperforms reactive approaches.

This paper aims at real-time in-car video analysis to detect and track vehicles ahead for safety, autodriving, and target tracing. This paper describes a comprehensive approach to localizing target vehicles in video under various environmental conditions. The extracted geometry features from the video are continuously projected onto a 1-D profile and are constantly tracked. We rely on temporal information of features and their motion behaviors for vehicle identification, which compensates for the complexity in recognizing vehicle shapes, colors, and types. We probabilistically model the motion in the field of view according to the scene characteristic and the vehicle motion model. The hidden Markov model (HMM) is used to separate target vehicles from the background and track them probabilistically. We have investigated videos of day and night on different types of roads, showing that our approach is robust and effective in dealing with changes in environment and illumination and that real-time processing becomes possible for vehicle-borne cameras.


Question: Can we grant driving licenses to self-driving vehicles?

This report explores the potential security risks associated with the malicious exploitation of artificial intelligence (AI) and outlines strategies for forecasting, prevention, and mitigation. By examining the impact of AI on the digital, physical, and political realms, the report identifies four primary recommendations for AI researchers and relevant stakeholders. Additionally, it highlights several research avenues that could enhance defensive measures, reduce the efficacy of attacks, or increase the difficulty of their execution. The report also engages in a discussion about the long-term balance between attackers and defenders, without reaching a definitive conclusion.

Could We Issue Driving Licenses to Autonomous Vehicles?

**Documents**: The Future of Work: Robots, AI, and Automation
This paper discusses the future of work in the context of the ongoing technological revolution, focusing on the role of robots, AI, and automation. It explores how these technologies are reshaping the job market, the skills required for employment, and the nature of work itself. The paper also addresses the potential societal implications, including income inequality and job displacement, and suggests policy recommendations for governments, businesses, and educators to prepare for and mitigate these challenges.

**Similar Documents**: This paper delves into the evolving landscape of work in the era of rapid technological advancement, with a particular emphasis on the impact of robots, artificial intelligence (AI), and automation. It examines how these innovations are transforming employment patterns, the competencies needed in the workforce, and the fundamental characteristics of work. Moreover, the paper discusses the broader societal consequences, such as the widening income gap and the displacement of jobs, and proposes policy directions for governments, corporations, and educational institutions to anticipate and address these issues.

Intelligent Transportation Systems (ITS) have attracted the attention of researchers and the general public alike as a means to alleviate traffic congestion. Recently, the maturity of wireless technology has enabled a cost-efficient way to achieve ITS by detecting vehicles using Vehicle to Infrastructure (V2I) communications. Traditional ITS algorithms, in most cases, assume that every vehicle is observed, such as by a camera or a loop detector, but a V2I implementation would detect only those vehicles with wireless communications capability. We examine a family of transportation systems, which we will refer to as ‘Partially Detected Intelligent Transportation Systems’. An algorithm that can act well under a small detection rate is highly desirable due to gradual penetration rates of the underlying wireless technologies such as Dedicated Short Range Communications (DSRC) technology. Artificial Intelligence (AI) techniques for Reinforcement Learning (RL) are suitable tools for finding such an algorithm due to utilizing varied inputs and not requiring explicit analytic understanding or modeling of the underlying system dynamics. In this paper, we report a RL algorithm for partially observable ITS based on DSRC. The performance of this system is studied under different car flows, detection rates, and topologies of the road network. Our system is able to efficiently reduce the average waiting time of vehicles at an intersection, even with a low detection rate.

A 10-point plan toward fashioning a proposal to ban some---if not all---lethal autonomous weapons.

Traditionally, the vehicle has been the extension of the man's ambulatory system, docile to the driver's commands. Recent advances in communications, controls and embedded systems have changed this model, paving the way to the Intelligent Vehicle Grid. The car is now a formidable sensor platform, absorbing information from the environment (and from other cars) and feeding it to drivers and infrastructure to assist in safe navigation, pollution control and traffic management. The next step in this evolution is just around the corner: the Internet of Autonomous Vehicles. Pioneered by the Google car, the Internet of Vehicles will be a distributed transport fabric capable to make its own decisions about driving customers to their destinations. Like other important instantiations of the Internet of Things (e.g., the smart building), the Internet of Vehicles will have communications, storage, intelligence, and learning capabilities to anticipate the customers' intentions. The concept that will help transition to the Internet of Vehicles is the Vehicular Cloud, the equivalent of Internet cloud for vehicles, providing all the services required by the autonomous vehicles. In this article, we discuss the evolution from Intelligent Vehicle Grid to Autonomous, Internet-connected Vehicles, and Vehicular Cloud.

This paper aims to highlight vision related tasks centered around “car”, which has been largely neglected by vision community in comparison to other objects. We show that there are still many interesting car-related problems and applications, which are not yet well explored and researched. To facilitate future car-related research, in this paper we present our on-going effort in collecting a large-scale dataset, “CompCars”, that covers not only different car views, but also their different internal and external parts, and rich attributes. Importantly, the dataset is constructed with a cross-modality nature, containing a surveillance-nature set and a web-nature set. We further demonstrate a few important applications exploiting the dataset, namely car model classification, car model verification, and attribute prediction. We also discuss specific challenges of the car-related problems and other potential applications that worth further investigations. The latest dataset can be downloaded at http://mmlab.ie.cuhk.edu.hk/ datasets/comp_cars/index.html.

The IT security of automotive systems is an evolving area of research. To analyse the current situation and the potentially growing tendency of arising threats we performed several practical tests on recent automotive technology. With a focus on automotive systems based on CAN bus technology, this article summarises the results of four selected tests performed on the control systems for the window lift, warning light and airbag control system as well as the central gateway. These results are supplemented in this article by a classification of these four attack scenarios using the established CERT taxonomy and an analysis of underlying security vulnerabilities, and especially, potential safety implications. With respect to the results of these tests, in this article we further discuss two selected countermeasures to address basic weaknesses exploited in our tests. These are adaptations of intrusion detection (discussing three exemplary detection patterns) and IT-forensic measures (proposing proactive measures based on a forensic model). This article discusses both looking at the four attack scenarios introduced before, covering their capabilities and restrictions. While these reactive approaches are short-term measures, which could already be added to today’s automotive IT architecture, long-term concepts also are shortly introduced, which are mainly preventive but will require a major redesign. Beneath a short overview on respective research approaches, we discuss their individual requirements, potential and restrictions. & 2010 Elsevier Ltd. All rights reserved.

This study was aimed at investigating drivers' ability to detect a car ahead decelerating, while doing mobile phone related tasks. Nineteen participants aged between 20 and 29 years, (2000-125000 km driving experience) drove at 80 km/h, 50 m behind a lead car, on a 30 km section of motorway in normal traffic. During each trial the lead car started to decelerate at an average of 0.47 m/s2 while the participant either looked at the car in front (control), continuously dialed series of three random integers on a numeric keypad (divided visual attention), or performed a memory and addition task (non-visual attention). The results indicated that drivers' detection ability was impaired by about 0.5 s in terms of brake reaction time and almost 1 s in terms of time-to-collision, when they were doing the non-visual task whilst driving. This impairment was similar to when the drivers were dividing their visual attention between the road ahead and dialing numbers on the keypad. It was concluded that neither a hands-free option nor a voice controlled interface removes the safety problems associated with the use of mobile phones in a car.

Advances in both semiconductor and automotive industry are today enabling the next generation of vehicles with significant electronics content than ever before. Consumers can now avail vehicle offerings in the form of Electric and Hybrid Electric Vehicles (EV/HEV) that have improved fuel efficiency, provide enhanced driver-passenger comfort and experience through Advance Driver Assistance Systems (ADAS) and car infotainment systems, and more. Increasing electronics, software content, and connectivity drive two consumer concerns — “functional safety” and “security” — to the forefront. In this tutorial, we dissect these concerns from an end application perspective and translate the system level requirements and standards into semiconductor development requirements. We indicate both current and emerging practices, and touch upon areas requiring new or optimal design and electronic design automation (EDA) solutions. While functional safety is the primary focus for deep-dive in this tutorial, we also examine key facets of automotive security which is now emerging as a critical area for further understanding and standardization.

Predicting driver behavior is a key component for Advanced Driver Assistance Systems (ADAS). In this paper, a novel approach based on Support Vector Machine and Bayesian filtering is proposed for online lane change intention prediction. The approach uses the multiclass probabilistic outputs of the Support Vector Machine as an input to the Bayesian filter, and the output of the Bayesian filter is used for the final prediction of lane changes. A lane tracker integrated in a passenger vehicle is used for real-world data collection for the purpose of training and testing. Data from different drivers on different highways were used to evaluate the robustness of the approach. The results demonstrate that the proposed approach is able to predict driver intention to change lanes on average 1.3 seconds in advance, with a maximum prediction horizon of 3.29 seconds.