Question: Functionally interconnected resting-state networks mirror the underlying structural connectivity blueprint of the human brain.

The ebb and flow of disconnection syndromes in neurological understanding.
In a brain characterized by specialized regions interconnected in a complex network, disconnection can lead to significant neurological dysfunction. This concept, foundational to the study of disconnection syndromes, dominated neurological discourse in the 19th century. However, disconnectionism experienced a decline in popularity with the shift away from localized brain theories in the early 20th century. In 1965, an American neurologist reignited interest in disconnection syndromes with a seminal paper titled, 'Disconnexion Syndromes in Animals and Man'. This work served as a pivotal point in the evolution of behavioral neurology, as Norman Geschwind introduced a comprehensive disconnectionist framework that transformed both clinical neurology and the broader neurosciences. Geschwind's theory proposed that disconnection syndromes, characterized by deficits in higher functions, were caused by lesions in white matter or association cortices, which act as intermediaries between primary motor, sensory, and limbic areas. Clinically, this theory provided a valuable tool for linking lesion locations to specific deficits, reviving the practice of single case studies. In the neurosciences, it fostered the development of contemporary theories emphasizing the role of distributed networks and connectionist models in brain function. Geschwind's disconnectionist paradigm dominated clinical neurology for two decades, but by the late 1980s, its influence began to wane as the focus shifted towards specialized functions of association cortex. This led to its integration into more comprehensive models of higher function disorders. By the 1990s, classical disconnection theory received limited attention in neurology textbooks. Modern neuroimaging techniques now allow for direct testing of the classical formulation, expanding its scope to include not only disconnections but also disorders of hyperconnectivity. On the 40th anniversary of Geschwind's groundbreaking publication, we trace the evolution of disconnection theory and adapt its framework to encompass the broad spectrum of higher function disorders seen in both neurology and psychiatry.

The cortical, cerebellar and brainstem BOLD-signal changes have been identified with fMRI in humans during mental imagery of walking. In this study the whole brain activation and deactivation pattern during real locomotion was investigated by [(18)F]-FDG-PET and compared to BOLD-signal changes during imagined locomotion in the same subjects using fMRI. Sixteen healthy subjects were scanned at locomotion and rest with [(18)F]-FDG-PET. In the locomotion paradigm subjects walked at constant velocity for 10 min. Then [(18)F]-FDG was injected intravenously while subjects continued walking for another 10 min. For comparison fMRI was performed in the same subjects during imagined walking. During real and imagined locomotion a basic locomotion network including activations in the frontal cortex, cerebellum, pontomesencephalic tegmentum, parahippocampal, fusiform and occipital gyri, and deactivations in the multisensory vestibular cortices (esp. superior temporal gyrus, inferior parietal lobule) was shown. As a difference, the primary motor and somatosensory cortices were activated during real locomotion as distinct to the supplementary motor cortex and basal ganglia during imagined locomotion. Activations of the brainstem locomotor centers were more prominent in imagined locomotion. In conclusion, basic activation and deactivation patterns of real locomotion correspond to that of imagined locomotion. The differences may be due to distinct patterns of locomotion tested. Contrary to constant velocity real locomotion (10 min) in [(18)F]-FDG-PET, mental imagery of locomotion over repeated 20-s periods includes gait initiation and velocity changes. Real steady-state locomotion seems to use a direct pathway via the primary motor cortex, whereas imagined modulatory locomotion an indirect pathway via a supplementary motor cortex and basal ganglia loop.

Parenting behavior critically shapes human infants' current and future behavior. The parent-infant relationship provides infants with their first social experiences, forming templates of what they can expect from others and how to best meet others' expectations. In this review, we focus on the neurobiology of parenting behavior, including our own functional magnetic resonance imaging (fMRI) brain imaging experiments of parents. We begin with a discussion of background, perspectives and caveats for considering the neurobiology of parent-infant relationships. Then, we discuss aspects of the psychology of parenting that are significantly motivating some of the more basic neuroscience research. Following that, we discuss some of the neurohormones that are important for the regulation of social bonding, and the dysregulation of parenting with cocaine abuse. Then, we review the brain circuitry underlying parenting, proceeding from relevant rodent and nonhuman primate research to human work. Finally, we focus on a study-by-study review of functional neuroimaging studies in humans. Taken together, this research suggests that networks of highly conserved hypothalamic-midbrain-limbic-paralimbic-cortical circuits act in concert to support aspects of parent response to infants, including the emotion, attention, motivation, empathy, decision-making and other thinking that are required to navigate the complexities of parenting. Specifically, infant stimuli activate basal forebrain regions, which regulate brain circuits that handle specific nurturing and caregiving responses and activate the brain's more general circuitry for handling emotions, motivation, attention, and empathy--all of which are crucial for effective parenting. We argue that an integrated understanding of the brain basis of parenting has profound implications for mental health.

Functional magnetic resonance imaging (fMRI) was used to localize brain areas that were active during the observation of actions made by another individual. Object- and non-object-related actions made with different effectors (mouth, hand and foot) were presented. Observation of both object- and non-object-related actions determined a somatotopically organized activation of premotor cortex. The somatotopic pattern was similar to that of the classical motor cortex homunculus. During the observation of object-related actions, an activation, also somatotopically organized, was additionally found in the posterior parietal lobe. Thus, when individuals observe an action, an internal replica of that action is automatically generated in their premotor cortex. In the case of object-related actions, a further object-related analysis is performed in the parietal lobe, as if the subjects were indeed using those objects. These results bring the previous concept of an action observation/execution matching system (mirror system) into a broader perspective: this system is not restricted to the ventral premotor cortex, but involves several somatotopically organized motor circuits.

Recent studies have begun to elucidate the roles played in social cognition by specific neural structures, genes, and neurotransmitter systems. Cortical regions in the temporal lobe participate in perceiving socially relevant stimuli, whereas the amygdala, right somatosensory cortices, orbitofrontal cortices, and cingulate cortices all participate in linking perception of such stimuli to motivation, emotion, and cognition. Open questions remain about the domain-specificity of social cognition, about its overlap with emotion and with communication, and about the methods best suited for its investigation.

Sex differences in visuospatial cognition (VSC) with male advantage are frequently reported in the literature. There is evidence for sexual dimorphisms in the human brain, one of which postulates more gray matter (GM) in females and more white matter (WM) in males relative to total intracranial volume. We investigated the neuroanatomy of VSC independent of general intelligence (g) in sex-separated populations, homogenous in age, education, memory performance, a memory- and brain morphology-related gene, and g. VSC and g were assessed with the Wechsler adult intelligence scale. The influence of g on VSC was removed using a hierarchical factor analysis and the Schmid–Leiman solution. Structural high-resolution magnetic resonance images were acquired and analyzed with voxel-based morphometry. As hypothesized, the clusters of positive correlations between local volumes and VSC performance independent of g were found mainly in parietal areas, but also in pre- and postcentral regions, predominantly in the WM in males, whereas in females these correlations were located in parietal and superior temporal areas, predominantly in the GM. Our results suggest that VSC depends more strongly on parietal WM structures in males and on parietal GM structures in females. This sex difference might have to do with the increased axonal and decreased somatodendritic tissue in males relative to females. Whether such sex-specific implementations of the VSC network can be explained genetically as suggested in investigations into the Turner syndrome or as a result of structural neural plasticity upon different experience and usage remains to be shown.

We used a new theory of the biological basis of the Big Five personality traits to generate hypotheses about the association of each trait with the volume of different brain regions. Controlling for age, sex, and whole-brain volume, results from structural magnetic resonance imaging of 116 healthy adults supported our hypotheses for four of the five traits: Extraversion, Neuroticism, Agreeableness, and Conscientiousness. Extraversion covaried with volume of medial orbitofrontal cortex, a brain region involved in processing reward information. Neuroticism covaried with volume of brain regions associated with threat, punishment, and negative affect. Agreeableness covaried with volume in regions that process information about the intentions and mental states of other individuals. Conscientiousness covaried with volume in lateral prefrontal cortex, a region involved in planning and the voluntary control of behavior. These findings support our biologically based, explanatory model of the Big Five and demonstrate the potential of personality neuroscience (i.e., the systematic study of individual differences in personality using neuroscience methods) as a discipline.

Humans are able to mentally construct an episode when listening to another person's recollection, even though they themselves did not experience the events. However, it is unknown how strongly the neural patterns elicited by mental construction resemble those found in the brain of the individual who experienced the original events. Using fMRI and a verbal communication task, we traced how neural patterns associated with viewing specific scenes in a movie are encoded, recalled, and then transferred to a group of naïve listeners. By comparing neural patterns across the 3 conditions, we report, for the first time, that event-specific neural patterns observed in the default mode network are shared across the encoding, recall, and construction of the same real-life episode. This study uncovers the intimate correspondences between memory encoding and event construction, and highlights the essential role our common language plays in the process of transmitting one's memories to other brains.

The analysis of brain connectivity is a vast field in neuroscience with a frequent use of visual representations and an increasing need for visual analysis tools. Based on an in-depth literature review and interviews with neuroscientists, we explore high-level brain connectivity analysis tasks that need to be supported by dedicated visual analysis tools. A significant example of such a task is the comparison of different connectivity data in the form of weighted graphs. Several approaches have been suggested for graph comparison within information visualization, but the comparison of weighted graphs has not been addressed. We explored the design space of applicable visual representations and present augmented adjacency matrix and node-link visualizations. To assess which representation best support weighted graph comparison tasks, we performed a controlled experiment. Our findings suggest that matrices support these tasks well, outperforming node-link diagrams. These results have significant implications for the design of brain connectivity analysis tools that require weighted graph comparisons. They can also inform the design of visual analysis tools in other domains, e.g. comparison of weighted social networks or biological pathways.

The mirror neuron system (MNS) has been proposed to play an important role in social cognition by providing a neural mechanism by which others' actions, intentions, and emotions can be understood. Here functional magnetic resonance imaging was used to directly examine the relationship between MNS activity and two distinct indicators of social functioning in typically-developing children (aged 10.1 years+/-7 months): empathy and interpersonal competence. Reliable activity in pars opercularis, the frontal component of the MNS, was elicited by observation and imitation of emotional expressions. Importantly, activity in this region (as well as in the anterior insula and amygdala) was significantly and positively correlated with established behavioral measures indexing children's empathic behavior (during both imitation and observation) and interpersonal skills (during imitation only). These findings suggest that simulation mechanisms and the MNS may indeed be relevant to social functioning in everyday life during typical human development.

BACKGROUND
In the present paper, we describe a model of neurovisceral integration in which a set of neural structures involved in cognitive, affective, and autonomic regulation are related to heart rate variability (HRV) and cognitive performance.

METHODS
We detail the pathways involved in the neural regulation of the cardiovascular system and provide pharmacological and neuroimaging data in support of the neural structures linking the central nervous system to HRV in humans. We review a number of studies from our group showing that individual differences in HRV are related to performance on tasks associated with executive function and prefrontal cortical activity. These studies include comparisons of executive- and nonexecutive-function tasks in healthy participants, in both threatening and nonthreatening conditions. In addition, we show that manipulating resting HRV levels is associated with changes in performance on executive-function tasks. We also examine the relationship between HRV and cognitive performance in ecologically valid situations using a police shooting simulation and a naval navigation simulation. Finally, we review our studies in anxiety patients, as well as studies examining psychopathy.

CONCLUSION
These findings in total suggest an important relationship among cognitive performance, HRV, and prefrontal neural function that has important implications for both physical and mental health. Future studies are needed to determine exactly which executive functions are associated with individual differences in HRV in a wider range of situations and populations.


Question: Resting-state networks that are functionally connected mirror the structural connectivity blueprint of the human brain.

This paper explores the application of electroencephalography-correlated functional magnetic resonance imaging (EEG/fMRI) in identifying blood oxygen level-dependent (BOLD) signal alterations linked to spontaneous fluctuations in the alpha rhythm (8-12 Hz) during relaxed wakefulness. Utilizing continuous and simultaneous EEG/fMRI, the study aimed to uncover BOLD signal changes associated with the alpha rhythm, which is the primary EEG pattern observed when an individual is awake but at rest. Ten participants underwent eyes-closed rest inside a 1.5-T magnet resonance (MR) scanner, with 32-channel EEG recordings captured using an MR-compatible EEG system. Functional scanning through echoplanar imaging covered almost the entire cerebrum, with each scan completed every 4 seconds. Post-acquisition software was employed to subtract MRI artifacts, allowing for continuous EEG data during fMRI. Alpha power was calculated over 1-second intervals at various electrode positions through Fast Fourier Transform. The resulting power time course was convolved with a canonical hemodynamic response function, down-sampled, and used to map associated signal changes in the image time series via statistical parametric mapping. Across all studied electrode positions, a robust negative correlation was observed between alpha power and activity in the parietal and frontal cortices. In contrast, positive correlation was scarce and non-systematic. These findings are discussed in the context of contemporary theories on alpha rhythm generation and its functional significance, as well as related neuroimaging results.

Parallel generational tasks for music and language were compared using positron emission tomography. Amateur musicians vocally improvised melodic or linguistic phrases in response to unfamiliar, auditorily presented melodies or phrases. Core areas for generating melodic phrases appeared to be in left Brodmann area (BA) 45, right BA 44, bilateral temporal planum polare, lateral BA 6, and pre-SMA. Core areas for generating sentences seemed to be in bilateral posterior superior and middle temporal cortex (BA 22, 21), left BA 39, bilateral superior frontal (BA 8, 9), left inferior frontal (BA 44, 45), anterior cingulate, and pre-SMA. Direct comparisons of the two tasks revealed activations in nearly identical functional brain areas, including the primary motor cortex, supplementary motor area, Broca's area, anterior insula, primary and secondary auditory cortices, temporal pole, basal ganglia, ventral thalamus, and posterior cerebellum. Most of the differences between melodic and sentential generation were seen in lateralization tendencies, with the language task favouring the left hemisphere. However, many of the activations for each modality were bilateral, and so there was significant overlap. While clarification of this overlapping activity awaits higher-resolution measurements and interventional assessments, plausible accounts for it include component sharing, interleaved representations, and adaptive coding. With these and related findings, we outline a comparative model of shared, parallel, and distinctive features of the neural systems supporting music and language. The model assumes that music and language show parallel combinatoric generativity for complex sound structures (phonology) but distinctly different informational content (semantics).

A new musculoskeltal simulation primitive, the strand, can provide an efficient way of simulating complicated muscle fiber architectures [1]. In this paper we present a new fiber tracking algorithm based on an energy minimizing active curve that is well suited for building these strand type models from diffusion tensor imaging data. We present fiber tracking results for the Brachioradialis muscle in the left forearm of a male subject. The algorithm produces a space filling arrangement of fibers that are uniformly distributed throughout the muscle and aligned with the underlying muscle fiber direction.

Research on how the brain processes music is emerging as a rich and stimulating area of investigation of perception, memory, emotion, and performance. Results emanating from both lesion studies and neuroimaging techniques are reviewed and integrated for each of these musical functions. We focus our attention on the common core of musical abilities shared by musicians and nonmusicians alike. Hence, the effect of musical training on brain plasticity is examined in a separate section, after a review of the available data regarding music playing and reading skills that are typically cultivated by musicians. Finally, we address a currently debated issue regarding the putative existence of music-specific neural networks. Unfortunately, due to scarcity of research on the macrostructure of music organization and on cultural differences, the musical material under focus is at the level of the musical phrase, as typically used in Western popular music.

Visually guided eating, biting and kissing, and avoiding objects moving toward the face and toward which the face moves require prompt, coordinated processing of spatial visual and somatosensory information in order to protect the face and the brain. Single-cell recordings in parietal cortex have identified multisensory neurons with spatially restricted, aligned visual and somatosensory receptive fields, but so far, there has been no evidence for a topographic map in this area. Here we mapped the organization of a multisensory parietal face area in humans by acquiring functional magnetic resonance images while varying the polar angle of facial air puffs and close-up visual stimuli. We found aligned maps of tactile and near-face visual stimuli at the highest level of human association cortex—namely, in the superior part of the postcentral sulcus. We show that this area may code the location of visual stimuli with respect to the face, not with respect to the retina.*NOTE: In the version of this article initially published online, there was an error in the affiliation in the html version. The first affiliation should read Department of Cognitive Science, University of California San Diego, La Jolla, California 92093, USA. The error has been corrected online.

In this article we provide a unifying neural hypothesis on how individuals understand the actions and emotions of others. Our main claim is that the fundamental mechanism at the basis of the experiential understanding of others' actions is the activation of the mirror neuron system. A similar mechanism, but involving the activation of viscero-motor centers, underlies the experiential understanding of the emotions of others.

Cortico-cortical connections through the corpus callosum are a major candidate for mediating bimanual coordination. However, aside from the deficits observed after lesioning this connection, little positive evidence indicates its function in bimanual tasks. In order to address this issue, we simultaneously recorded neuronal activity at multiple sites within the arm area of motor cortex in both hemispheres of awake primates performing different bimanual and unimanual movements. By employing an adapted form of the joint peri-stimulus time histogram technique, we discovered rapid movement-related correlation changes between the local field potentials (LFPs) of the two hemispheres that escaped detection by time-averaged cross-correlation methods. The frequency and amplitude of dynamic modifications in correlations between the hemispheres were similar to those within the same hemisphere. As in previous EEG studies, we found that, on average, correlation decreased during movements. However, a subset of recording site pairs did show transiently increased correlations around movement onset (57% of all pairs and conditions in monkey G, 39% in monkey P). In interhemispheric pairs, these increases were consistently related to the mode of coupling between the two arms. Both the correlations between the movements themselves and the interhemispheric LFP correlation increases were strongest during bimanual symmetric movements, and weaker during bimanual asymmetric and unimanual movements. Increased correlations occurred mainly around movement onset, whilst decreases in correlation dominated during movement execution. The task-specific way in which interhemispheric correlations are modulated is compatible with the notion that interactions between the hemispheres contribute to behavioural coupling between the arms.

This paper introduces NeuroCave, an open-source immersive analytics software tool for exploring and making sense of both structural and functional connectomes describing the interconnectivity between brain regions. The software supports visual analysis tasks for clinical neuroscientists, facilitating explorations of connectome datasets used in comparative group studies. Researchers can enter and leave virtual reality as desired in order to visualize and reason about connectome datasets from a range of different perspectives.

In this study, we investigated blood-flow-related magnetic-resonance (MR) signal changes and the time course underlying short-term motor learning of the dominant right hand in ten piano players (PPs) and 23 non-musicians (NMs), using a complex finger-tapping task. The activation patterns were analyzed for selected regions of interest (ROIs) within the two examined groups and were related to the subjects’ performance. A functional learning profile, based on the regional blood-oxygenation-level-dependent (BOLD) signal changes, was assessed in both groups. All subjects achieved significant increases in tapping frequency during the training session of 35 min in the scanner. PPs, however, performed significantly better than NMs and showed increasing activation in the contralateral primary motor cortex throughout motor learning in the scanner. At the same time, involvement of secondary motor areas, such as bilateral supplementary motor area, premotor, and cerebellar areas, diminished relative to the NMs throughout the training session. Extended activation of primary and secondary motor areas in the initial training stage (7–14 min) and rapid attenuation were the main functional patterns underlying short-term learning in the NM group; attenuation was particularly marked in the primary motor cortices as compared with the PPs. When tapping of the rehearsed sequence was performed with the left hand, transfer effects of motor learning were evident in both groups. Involvement of all relevant motor components was smaller than after initial training with the right hand. Ipsilateral premotor and primary motor contributions, however, showed slight increases of activation, indicating that dominant cortices influence complex sequence learning of the non-dominant hand. In summary, the involvement of primary and secondary motor cortices in motor learning is dependent on experience. Interhemispheric transfer effects are present.

Diffusion-weighted MRI (DW-MRI) has become a popular imaging modality for probing the microstructural properties of white matter and comparing them between populations in vivo. However, the contrast in DW-MRI arises from the microscopic random motion of water molecules in brain tissues, which makes it particularly sensitive to macroscopic head motion. Although this has been known since the introduction of DW-MRI, most studies that use this modality for group comparisons do not report measures of head motion for each group and rely on registration-based correction methods that cannot eliminate the full effects of head motion on the DW-MRI contrast. In this work we use data from children with autism and typically developing children to investigate the effects of head motion on differences in anisotropy and diffusivity measures between groups. We show that group differences in head motion can induce group differences in DW-MRI measures, and that this is the case even when comparing groups that include control subjects only, where no anisotropy or diffusivity differences are expected. We also show that such effects can be more prominent in some white-matter pathways than others, and that they can be ameliorated by including motion as a nuisance regressor in the analyses. Our results demonstrate the importance of taking head motion into account in any population study where one group might exhibit more head motion than the other.

Is there an objective, biological basis for the experience of beauty in art? Or is aesthetic experience entirely subjective? Using fMRI technique, we addressed this question by presenting viewers, naïve to art criticism, with images of masterpieces of Classical and Renaissance sculpture. Employing proportion as the independent variable, we produced two sets of stimuli: one composed of images of original sculptures; the other of a modified version of the same images. The stimuli were presented in three conditions: observation, aesthetic judgment, and proportion judgment. In the observation condition, the viewers were required to observe the images with the same mind-set as if they were in a museum. In the other two conditions they were required to give an aesthetic or proportion judgment on the same images. Two types of analyses were carried out: one which contrasted brain response to the canonical and the modified sculptures, and one which contrasted beautiful vs. ugly sculptures as judged by each volunteer. The most striking result was that the observation of original sculptures, relative to the modified ones, produced activation of the right insula as well as of some lateral and medial cortical areas (lateral occipital gyrus, precuneus and prefrontal areas). The activation of the insula was particularly strong during the observation condition. Most interestingly, when volunteers were required to give an overt aesthetic judgment, the images judged as beautiful selectively activated the right amygdala, relative to those judged as ugly. We conclude that, in observers naïve to art criticism, the sense of beauty is mediated by two non-mutually exclusive processes: one based on a joint activation of sets of cortical neurons, triggered by parameters intrinsic to the stimuli, and the insula (objective beauty); the other based on the activation of the amygdala, driven by one's own emotional experiences (subjective beauty).


Question: An end-to-end warning system for pedestrian collisions utilizing a convolutional neural network with semantic segmentation capabilities.

This paper explores the optimization of feature sets for robust visual object recognition, with a focus on human detection using linear SVM-based techniques. We critically evaluate various edge and gradient-based descriptors and empirically demonstrate that grids of Histograms of Oriented Gradients (HOG) descriptors outperform other feature sets significantly in the context of human detection. Through a detailed analysis, we find that the effectiveness of the HOG descriptor is enhanced by the use of fine-scale gradients, precise orientation binning, relatively coarse spatial binning, and high-quality local contrast normalization applied to overlapping descriptor blocks. These findings are crucial for achieving optimal performance. The proposed approach yields near-perfect separation on the original MIT pedestrian database. To further validate the robustness of our method, we introduce a more challenging dataset that includes over 1800 annotated human images, encompassing a wide range of poses and diverse backgrounds. This comprehensive evaluation underscores the superiority of the HOG descriptor in human detection tasks.

Question: Real-time pedestrian detection using convolutional neural networks for intelligent vehicle systems

Documents: Real-time pedestrian detection with deep learning: A survey
Pedestrian detection is a critical task for intelligent vehicle systems and surveillance applications. In recent years, deep learning techniques, particularly convolutional neural networks (CNNs), have significantly advanced the state-of-the-art in real-time pedestrian detection. This paper provides a comprehensive survey of the latest research in real-time pedestrian detection using deep learning methods, focusing on the contributions of CNNs. We discuss the evolution of CNN architectures specifically designed for pedestrian detection, their performance on various benchmarks, and the challenges they address. Key advancements include the integration of multi-scale feature extraction, context-aware analysis, and real-time processing capabilities. The survey also highlights the impact of large-scale annotated datasets in training CNN models for pedestrian detection and the importance of transfer learning in improving detection accuracy. Finally, we explore emerging trends in real-time pedestrian detection, such as the use of 3D data, fusion of multiple sensor inputs, and the development of specialized hardware for on-device processing, which promise to further enhance the performance and reliability of intelligent vehicle systems.

Similar Documents: This comprehensive review explores the advancements in real-time pedestrian detection utilizing deep learning methodologies, with a particular emphasis on the pivotal role of convolutional neural networks (CNNs). Pedestrian detection is a critical component in enhancing the capabilities of intelligent vehicle systems and surveillance technologies. Over the past decade, deep learning, especially CNNs, has revolutionized real-time pedestrian detection by achieving state-of-the-art performance. The paper del

We perform fast vehicle detection from traffic surveillance cameras. A novel deep learning framework, namely Evolving Boxes, is developed that proposes and refines the object boxes under different feature representations. Specifically, our framework is embedded with a light-weight proposal network to generate initial anchor boxes as well as to early discard unlikely regions; a fine-turning network produces detailed features for these candidate boxes. We show intriguingly that by applying different feature fusion techniques, the initial boxes can be refined for both localization and recognition. We evaluate our network on the recent DETRAC benchmark and obtain a significant improvement over the state-of-the-art Faster RCNN by 9.5% mAP. Further, our network achieves 9–13 FPS detection speed on a moderate commercial GPU.

This work addresses the problem of semantic foggy scene understanding (SFSU). Although extensive research has been performed on image dehazing and on semantic scene understanding with clear-weather images, little attention has been paid to SFSU. Due to the difficulty of collecting and annotating foggy images, we choose to generate synthetic fog on real images that depict clear-weather outdoor scenes, and then leverage these partially synthetic data for SFSU by employing state-of-the-art convolutional neural networks (CNN). In particular, a complete pipeline to add synthetic fog to real, clear-weather images using incomplete depth information is developed. We apply our fog synthesis on the Cityscapes dataset and generate Foggy Cityscapes with 20,550 images. SFSU is tackled in two ways: (1) with typical supervised learning, and (2) with a novel type of semi-supervised learning, which combines (1) with an unsupervised supervision transfer from clear-weather images to their synthetic foggy counterparts. In addition, we carefully study the usefulness of image dehazing for SFSU. For evaluation, we present Foggy Driving, a dataset with 101 real-world images depicting foggy driving scenes, which come with ground truth annotations for semantic segmentation and object detection. Extensive experiments show that (1) supervised learning with our synthetic data significantly improves the performance of state-of-the-art CNN for SFSU on Foggy Driving; (2) our semi-supervised learning strategy further improves performance; and (3) image dehazing marginally advances SFSU with our learning strategy. The datasets, models and code are made publicly available.

This paper describes a method for detecting and recognizing road signs in gray-level and color images acquired by a single camera mounted on a moving vehicle. The method works in three stages. First, the search for the road sign is reduced to a suitable region of the image by using some a priori knowledge on the scene or color clues (when available). Secondly, a geometrical analysis of the edges extracted from the image is carried out, which generates candidates to be circular and triangular signs. Thirdly, a recognition stage tests by cross-correlation techniques each candidate which, if validated, is classi ed according to the data-base of signs. An extensive experimentation has shown that the method is robust against low-level noise corrupting edge detection and contour following, and works for images of cluttered urban streets as well as country roads and highways. A further improvement on the detection and recognition scheme has been obtained by means of temporal integration based on Kalman ltering methods of the extracted information. The proposed approach can be very helpful for the development of a system for driving assistance.

We present a novel and practical deep fully convolutional neural network architecture for semantic pixel-wise segmentation termed SegNet. This core trainable segmentation engine consists of an encoder network, a corresponding decoder network followed by a pixel-wise classification layer. The architecture of the encoder network is topologically identical to the 13 convolutional layers in the VGG16 network [1] . The role of the decoder network is to map the low resolution encoder feature maps to full input resolution feature maps for pixel-wise classification. The novelty of SegNet lies is in the manner in which the decoder upsamples its lower resolution input feature map(s). Specifically, the decoder uses pooling indices computed in the max-pooling step of the corresponding encoder to perform non-linear upsampling. This eliminates the need for learning to upsample. The upsampled maps are sparse and are then convolved with trainable filters to produce dense feature maps. We compare our proposed architecture with the widely adopted FCN [2] and also with the well known DeepLab-LargeFOV [3] , DeconvNet [4] architectures. This comparison reveals the memory versus accuracy trade-off involved in achieving good segmentation performance. SegNet was primarily motivated by scene understanding applications. Hence, it is designed to be efficient both in terms of memory and computational time during inference. It is also significantly smaller in the number of trainable parameters than other competing architectures and can be trained end-to-end using stochastic gradient descent. We also performed a controlled benchmark of SegNet and other architectures on both road scenes and SUN RGB-D indoor scene segmentation tasks. These quantitative assessments show that SegNet provides good performance with competitive inference time and most efficient inference memory-wise as compared to other architectures. We also provide a Caffe implementation of SegNet and a web demo at http://mi.eng.cam.ac.uk/projects/segnet/.

To reduce roadway congestion and improve traffic safety, accurate traffic metrics, such as number of vehicles travelling through lane-ways, are required. Unfortunately most existing infrastructure, such as loop-detectors and many video detectors, do not feasibly provide accurate vehicle counts. Consequently, a novel method is proposed which models vehicle motion using hidden Markov models (HMM). The proposed method represents a specified small region of the roadway as 'empty', 'vehicle entering', 'vehicle inside', and 'vehicle exiting', and then applies a modified Viterbi algorithm to the HMM sequential estimation framework to initialize and track vehicles. Vehicle observations are obtained using an Adaboost trained Haar-like feature detector. When tested on 88 hours of video, from three distinct locations, the proposed method proved to be robust to changes in lighting conditions, moving shadows, and camera motion, and consistently out-performed Multiple Target Tracking (MTT) and Virtual Detection Line(VDL) implementations. The median vehicle count error of the proposed method is lower than MTT and VDL by 28%, and 70% respectively. As future work, this algorithm will be implemented to provide the traffic industry with improved automated vehicle counting, with the intent to eventually provide real-time counts.

For autonomous parking applications to become possible, knowledge about the parking environment is required. Therefore, a real-time algorithm for detecting parked vehicles from radar data is presented. These data are first accumulated in an occupancy grid from which objects are detected by applying techniques borrowed from the computer vision field. Two random forest classifiers are trained to recognize two categories of objects: parallel-parked vehicles and cross-parked vehicles. Performances of the classifiers are evaluated as well as the capacity of the complete system to detect parked vehicles in real world scenarios.

This paper proposes a robust and effective vision-based lane detection approach. First, two binary images are obtained from the region of interest of gray-scale images. The obtained binary images are merged by a novel neighborhood AND operator and then transformed to a bird's eye view (BEV) via inverse perspective mapping. Then, gaussian probability density functions are fit to the left and right regions of a histogram image acquired from the BEV. Finally, a polynomial lane model is estimated from the identified regions. Experimental results show that the proposed method accurately detects lanes in complex situations including worn-out and curved lanes.

This paper introduces a system for estimating the attention of a driver wearing a first person view camera using salient objects to improve gaze estimation. A challenging data set of pedestrians crossing intersections has been captured using Google Glass worn by a driver. A challenge unique to first person view from cars is that the interior of the car can take up a large part of the image. The proposed system automatically filters out the dashboard of the car, along with other parts of the instrumentation. The remaining area is used as a region of interest for a pedestrian detector. Two cameras looking at the driver are used to determine the direction of the driver's gaze, by examining the eye corners and the center of the iris. This coarse gaze estimation is then linked to the detected pedestrians to determine which pedestrian the driver is focused on at any given time.

We aim to detect all instances of a category in an image and, for each instance, mark the pixels that belong to it. We call this task Simultaneous Detection and Segmentation (SDS). Unlike classical bounding box detection, SDS requires a segmentation and not just a box. Unlike classical semantic segmentation, we require individual object instances. We build on recent work that uses convolutional neural networks to classify category-independent region proposals (R-CNN [16]), introducing a novel architecture tailored for SDS. We then use category-specific, topdown figure-ground predictions to refine our bottom-up proposals. We show a 7 point boost (16% relative) over our baselines on SDS, a 5 point boost (10% relative) over state-of-the-art on semantic segmentation, and state-of-the-art performance in object detection. Finally, we provide diagnostic tools that unpack performance and provide directions for future work.

The detection of a traffic light on the road is important for the safety of persons who occupy a vehicle, in a normal vehicles or an autonomous land vehicle. In normal vehicle, a system that helps a driver to perceive the details of traffic signals, necessary to drive, could be critical in a delicate driving manoeuvre (i.e crossing an intersection of roads). Furthermore, traffic lights detection by an autonomous vehicle is a special case of perception, because it is important for the control that the autonomous vehicle must take. Multiples authors have used image processing as a base for achieving traffic light detection. However, the image processing presents a problem regarding conditions for capturing scenes, and therefore, the traffic light detection is affected. For this reason, this paper proposes a method that links the image processing with an estimation state routine formed by Hidden Markov Models (HMM). This method helps to determine the current state of the traffic light detected, based on the obtained states by image processing, aiming to obtain the best performance in the determination of the traffic light states. With the proposed method in this paper, we obtained 90.55% of accuracy in the detection of the traffic light state, versus a 78.54% obtained using solely image processing. The recognition of traffic lights using image processing still has a large dependence on the capture conditions of each frame from the video camera. In this context, the addition of a pre-processing stage before image processing could contribute to improve this aspect, and could provide a better results in determining the traffic light state.


Question: An end-to-end warning system for pedestrian collisions utilizing a convolutional neural network with semantic segmentation capabilities.

This paper details the development of a cutting-edge deep convolutional neural network (CNN) designed to classify the extensive 1.2 million high-resolution images from the ImageNet LSVRC-2010 competition into 1000 distinct categories. The network achieved remarkable results, with top-1 and top-5 error rates of 37.5% and 17.0% respectively, significantly surpassing the previous benchmarks. The CNN, with its 60 million parameters and 650,000 neurons, is structured with five convolutional layers, followed by max-pooling layers in certain cases, and culminates in three fully connected layers with a final 1000-way softmax. To expedite the training process, non-saturating neurons were utilized alongside a highly optimized GPU implementation of the convolution operation. To tackle overfitting in the fully connected layers, a recently introduced regularization technique known as "dropout" was employed, which proved highly effective. A variant of this model was also submitted to the ILSVRC-2012 competition, where it secured the top spot with a top-5 test error rate of 15.3%, outperforming the second-place entry by a substantial margin of 26.2%.

An innovative end-to-end pedestrian collision warning system is presented, leveraging a convolutional neural network (CNN) with semantic segmentation capabilities. This system is designed to process and interpret complex visual scenes in real-time, accurately identifying pedestrians and predicting potential collisions. The CNN at the core of this system is trained on a large dataset of images, enabling it to recognize pedestrians in various environments and under different lighting conditions. Semantic segmentation enhances the system's ability to distinguish pedestrians from the background and other objects, providing a more detailed and accurate understanding of the scene. This approach significantly improves the reliability of pedestrian detection and collision warning, making it a crucial safety feature for autonomous and semi-autonomous vehicles. The system's end-to-end design ensures a seamless integration of perception, decision-making, and action, optimizing response times and enhancing overall system performance.

High-resolution satellite imagery has been increasingly used on remote sensing classification problems. One of the main factors is the availability of this kind of data. Despite the high availability, very little effort has been placed on the zebra crossing classification problem. In this letter, crowdsourcing systems are exploited in order to enable the automatic acquisition and annotation of a large-scale satellite imagery database for crosswalks related tasks. Then, this data set is used to train deep-learning-based models in order to accurately classify satellite images that contain or not contain zebra crossings. A novel data set with more than 240000 images from 3 continents, 9 countries, and more than 20 cities was used in the experiments. The experimental results showed that freely available crowdsourcing data can be used to accurately (97.11%) train robust models to perform crosswalk classification on a global scale.

Future driver assistance systems are likely to use a multisensor approach with heterogeneous sensors for tracking dynamic objects around the vehicle. The quality and type of data available for a data fusion algorithm depends heavily on the sensors detecting an object. This article presents a general framework which allows the use sensor specific advantages while abstracting the specific details of a sensor. Different tracking models are used depending on the current set of sensors detecting the object. A sensor independent algorithm for classifying objects regarding their current and past movement state is presented. The described architecture and algorithms have been successfully implemented in Tartan racingpsilas autonomous vehicle for the urban grand challenge. Results are presented and discussed.

Machine Learning, and in particular Deep Learning, are extremely capable tools for solving problems which are difficult, or intractable to tackle analytically. Application areas include pattern recognition, computer vision, speech and natural language processing. With the automotive industry aiming for increasing amount of automation in driving, the problems to solve become increasingly complex, which appeals to the use of supervised learning methods from Machine Learning and Deep Learning. With this approach, solutions to the problems are learned implicitly from training data, and inspecting their correctness is not possible directly. This presents concerns when the resulting systems are used to support safety-critical functions, as is the case with autonomous driving of automotive vehicles. This thesis studies the safety concerns related to learning systems within autonomous driving and applies a safety monitoring approach to a collision avoidance scenario. Experiments are performed using a simulated environment, with a deep learning system supporting perception for vehicle control, and a safety monitor for collision avoidance. The related operational situations and safety constraints are studied for an autonomous driving function, with potential faults in the learning system introduced and examined. Also, an example is considered for a measure that indicates trustworthiness of the learning system during operation.

For autonomous parking applications to become possible, knowledge about the parking environment is required. Therefore, a real-time algorithm for detecting parked vehicles from radar data is presented. These data are first accumulated in an occupancy grid from which objects are detected by applying techniques borrowed from the computer vision field. Two random forest classifiers are trained to recognize two categories of objects: parallel-parked vehicles and cross-parked vehicles. Performances of the classifiers are evaluated as well as the capacity of the complete system to detect parked vehicles in real world scenarios.

Semantic image segmentation is an essential component of modern autonomous driving systems, as an accurate understanding of the surrounding scene is crucial to navigation and action planning. Current state-of-the-art approaches in semantic image segmentation rely on pre-trained networks that were initially developed for classifying images as a whole. While these networks exhibit outstanding recognition performance (i.e., what is visible?), they lack localization accuracy (i.e., where precisely is something located?). Therefore, additional processing steps have to be performed in order to obtain pixel-accurate segmentation masks at the full image resolution. To alleviate this problem we propose a novel ResNet-like architecture that exhibits strong localization and recognition performance. We combine multi-scale context with pixel-level accuracy by using two processing streams within our network: One stream carries information at the full image resolution, enabling precise adherence to segment boundaries. The other stream undergoes a sequence of pooling operations to obtain robust features for recognition. The two streams are coupled at the full image resolution using residuals. Without additional processing steps and without pre-training, our approach achieves an intersection-over-union score of 71.8% on the Cityscapes dataset.

Human sensing enables intelligent vehicles to provide driver-adaptive support by classifying perceived workload into multiple levels. Objective of this study is to classify driver workload associated with traffic complexity into five levels. We conducted driving experiments in systematically varied traffic complexity levels in a simulator. We recorded driver physiological signals including electrocardiography, electrodermal activity, and electroencephalography. In addition, we integrated driver performance and subjective workload measures. Deep learning based models outperform statistical machine learning methods when dealing with dynamic time-series data with variable sequence lengths. We show that our long short-term memory based recurrent neural network model can classify driver perceived-workload into five classes with an accuracy of 74.5%. Since perceived workload differ between individual drivers for the same traffic situation, our results further highlight the significance of including driver characteristics such as driving style and workload sensitivity to achieve higher classification accuracy.

Many Advanced Driver Assistance Systems (ADAS) have been developed to improve car safety. However, it is still a challenging problem to make autonomous vehicles to drive safely on urban streets such as uncontrolled intersections (without traffic lights) and narrow roads. In this paper, we introduce a decision making system that can assist autonomous vehicles at uncontrolled intersections and narrow roads. We constructed a machine understandable ontology-based Knowledge Base, which contains maps and traffic regulations. The system makes decisions in comply with traffic regulations such as Right-Of-Way rules when it receives a collision warning signal. The decisions are sent to a path planning system to change the route or stop to avoid collisions.

Intelligent agents, such as robots, have to serve a multitude of autonomous functions. Examples are, e.g., collision avoidance, navigation and route planning, active sensing of its environment, or the interaction and non-verbal communication with people in the extended reach space. Here, we focus on the recognition of the action of a human agent based on a biologically inspired visual architecture of analyzing articulated movements. The proposed processing architecture builds upon coarsely segregated streams of sensory processing along different pathways which separately process form and motion information (Layher et al., 2014). Action recognition is performed in an event-based scheme by identifying representations of characteristic pose configurations (key poses) in an image sequence. In line with perceptual studies, key poses are selected unsupervised utilizing a feature-driven criterion which combines extrema in the motion energy with the horizontal and the vertical extendedness of a body shape. Per class representations of key pose frames are learned using a deep convolutional neural network consisting of 15 convolutional layers. The network is trained using the energy-efficient deep neuromorphic networks (Eedn) framework (Esser et al., 2016), which realizes the mapping of the trained synaptic weights onto the IBM Neurosynaptic System platform (Merolla et al., 2014). After the mapping, the trained network achieves real-time capabilities for processing input streams and classify input images at about 1,000 frames per second while the computational stages only consume about 70 mW of energy (without spike transduction). Particularly regarding mobile robotic systems, a low energy profile might be crucial in a variety of application scenarios. Cross-validation results are reported for two different datasets and compared to state-of-the-art action recognition approaches. The results demonstrate, that (I) the presented approach is on par with other key pose based methods described in the literature, which select key pose frames by optimizing classification accuracy, (II) compared to the training on the full set of frames, representations trained on key pose frames result in a higher confidence in class assignments, and (III) key pose representations show promising generalization capabilities in a cross-dataset evaluation.

People counting is a crucial subject in video surveillance application. Factors such as severe occlusions, scene perspective distortions in real application scenario make this task challenging. In this paper, we carefully designed a deep detection framework based on depth information for people counting in crowded environments. Our system performs head detection on depth images collected by an overhead vertical Kinect sensor. To the best of our knowledge, this is the first attempt to use the deep convolutional neural networks on depth images for people counting. We explored the impact of the number and quality of RPN positive anchors on the performance of Faster R-CNN and proposed a solution. Our method is very simple but effective, not only showing promising results but also efficiency as it runs in real-time at a frame rate of about 110 frames per second on a GPU.

In this thesis, the deep learning method convolutional neural networks (CNNs) has been used in an attempt to solve two classification problems, namely traffic sign recognition and Alzheimer’s disease detection. The two datasets used are from the German Traffic Sign Recognition Benchmark (GTSRB) and the Alzheimer’s Disease Neuroimaging Initiative (ADNI). The final test results on the traffic sign dataset generated a classification accuracy of 98.81 %, almost as high as human performance on the same dataset, 98.84 %. Different parameter settings of the selected CNN structure have also been tested in order to see their impact on the classification accuracy. Trying to distinguish between MRI images of healthy brains and brains afflicted with Alzheimer’s disease gained only about 65 % classification accuracy. These results show that the convolutional neural network approach is very promising for classifying traffic signs, but more work needs to be done when working with the more complex problem of detecting Alzheimer’s disease.


Question: An end-to-end pedestrian collision alert system utilizing a convolutional neural network for semantic segmentation.

This paper introduces a pioneering technique for enhancing the training of deep neural networks, specifically addressing the challenge of optimizing very deep architectures. We propose a residual learning framework, designed to simplify the training of networks significantly deeper than those previously utilized. Our approach involves reformulating the layers to learn residual functions relative to the layer inputs, rather than learning functions independently. This modification results in networks that are not only easier to optimize but also benefit from increased depth without suffering from higher complexity. We present extensive experimental evidence demonstrating the effectiveness of our residual networks on the ImageNet dataset, achieving remarkable results with networks as deep as 152 layers, 8 times deeper than VGG nets but with lower complexity. An ensemble of these residual networks attained an error rate of 3.57% on the ImageNet test set, securing the first place in the ILSVRC 2015 classification task. We further explore the performance of our networks on CIFAR-10 with 100 and 1000 layers, showcasing their robustness and versatility. The depth of representations is crucial for various visual recognition tasks, and our deep residual networks provide a 28% relative improvement on the COCO object detection dataset. These networks served as the backbone of our submissions to the ILSVRC & COCO 2015 competitions, where we claimed the top positions in ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation tasks.

This paper introduces a system for estimating the attention of a driver wearing a first person view camera using salient objects to improve gaze estimation. A challenging data set of pedestrians crossing intersections has been captured using Google Glass worn by a driver. A challenge unique to first person view from cars is that the interior of the car can take up a large part of the image. The proposed system automatically filters out the dashboard of the car, along with other parts of the instrumentation. The remaining area is used as a region of interest for a pedestrian detector. Two cameras looking at the driver are used to determine the direction of the driver's gaze, by examining the eye corners and the center of the iris. This coarse gaze estimation is then linked to the detected pedestrians to determine which pedestrian the driver is focused on at any given time.

Recently, autonomous vehicles, particularly self-driving cars, have received significant attention owing to rapid advancements in sensor and computation technologies. In addition to traffic sign recognition, road lane detection is one of the most important factors used in lane departure warning systems and autonomous vehicles for maintaining the safety of semi-autonomous and fully autonomous systems. Unlike traffic signs, road lanes are easily damaged by both internal and external factors such as road quality, occlusion (traffic on the road), weather conditions, and illumination (shadows from objects such as cars, trees, and buildings). Obtaining clear road lane markings for recognition processing is a difficult challenge. Therefore, we propose a method to overcome various illumination problems, particularly severe shadows, by using fuzzy system and line segment detector algorithms to obtain better results for detecting road lanes by a visible light camera sensor. Experimental results from three open databases, Caltech dataset, Santiago Lanes dataset (SLD), and Road Marking dataset, showed that our method outperformed conventional lane detection methods.

In this work we show that Deep Convolutional Neural Networks can outperform humans on the task of boundary detection, as measured on the standard Berkeley Segmentation Dataset. Our detector is fully integrated in the popular Caffe framework and processes a 320x420 image in less than a second. Our contributions consist firstly in combining a careful design of the loss for boundary detection training, a multi-resolution architecture and training with external data to improve the detection accuracy of the current state of the art, from an optimal dataset scale F-measure of 0.780 to 0.808 while human performance is at 0.803. We further improve performance to 0.813 by combining deep learning with grouping, integrating the Normalized Cuts technique within a deep network. We also examine the potential of our boundary detector in conjunction with the higher level tasks of object proposal generation and semantic segmentation for both tasks our detector yields clear improvements over state-of-the-art systems.

Deep neural perception and control networks are likely to be a key component of self-driving vehicles. These models need to be explainable - they should provide easy-tointerpret rationales for their behavior - so that passengers, insurance companies, law enforcement, developers etc., can understand what triggered a particular behavior. Here we explore the use of visual explanations. These explanations take the form of real-time highlighted regions of an image that causally influence the network’s output (steering control). Our approach is two-stage. In the first stage, we use a visual attention model to train a convolution network endto- end from images to steering angle. The attention model highlights image regions that potentially influence the network’s output. Some of these are true influences, but some are spurious. We then apply a causal filtering step to determine which input regions actually influence the output. This produces more succinct visual explanations and more accurately exposes the network’s behavior. We demonstrate the effectiveness of our model on three datasets totaling 16 hours of driving. We first show that training with attention does not degrade the performance of the end-to-end network. Then we show that the network causally cues on a variety of features that are used by humans while driving.

In this paper we propose to solve the problem of Visual Question Answering by using a novel segmentation guided attention based network which we call SegAttendNet. We use image segmentation maps, generated by a Fully Convolutional Deep Neural Network to refine our attention maps and use these refined attention maps to make the model focus on the relevant parts of the image to answer a question. The refined attention maps are used by the LSTM network to learn to produce the answer. We presently train our model on the visual7W dataset and do a category wise evaluation of the 7 question categories. We achieve state of the art results on this dataset and beat the previous benchmark on this dataset by a 1.5% margin improving the question answering accuracy from 54.1% to 55.6% and demonstrate improvements in each of the question categories. We also visualize our generated attention maps and note their improvement over the attention maps generated by the previous best approach.

In this thesis, the deep learning method convolutional neural networks (CNNs) has been used in an attempt to solve two classification problems, namely traffic sign recognition and Alzheimer’s disease detection. The two datasets used are from the German Traffic Sign Recognition Benchmark (GTSRB) and the Alzheimer’s Disease Neuroimaging Initiative (ADNI). The final test results on the traffic sign dataset generated a classification accuracy of 98.81 %, almost as high as human performance on the same dataset, 98.84 %. Different parameter settings of the selected CNN structure have also been tested in order to see their impact on the classification accuracy. Trying to distinguish between MRI images of healthy brains and brains afflicted with Alzheimer’s disease gained only about 65 % classification accuracy. These results show that the convolutional neural network approach is very promising for classifying traffic signs, but more work needs to be done when working with the more complex problem of detecting Alzheimer’s disease.

We propose an approach for exploiting contextual information in semantic image segmentation, and particularly investigate the use of patch-patch context and patch-background context in deep CNNs. We formulate deep structured models by combining CNNs and Conditional Random Fields (CRFs) for learning the patch-patch context between image regions. Specifically, we formulate CNN-based pairwise potential functions to capture semantic correlations between neighboring patches. Efficient piecewise training of the proposed deep structured model is then applied in order to avoid repeated expensive CRF inference during the course of back propagation. For capturing the patch-background context, we show that a network design with traditional multi-scale image inputs and sliding pyramid pooling is very effective for improving performance. We perform comprehensive evaluation of the proposed method. We achieve new state-of-the-art performance on a number of challenging semantic segmentation datasets.

This paper presents a novel approach for video-based person re-identification using multiple Convolutional Neural Networks (CNNs). Unlike previous work, we intend to extract a compact yet discriminative appearance representation from several frames rather than the whole sequence. Specifically, given a video, the representative frames are selected based on the walking profile of consecutive frames. A multiple CNN architecture incorporated with feature pooling is proposed to learn and compile the features of the selected representative frames into a compact description about the pedestrian for identification. Experiments are conducted on benchmark datasets to demonstrate the superiority of the proposed method over existing person re-identification approaches.

In this brief, a convolutional learning system for classification of segmented objects represented in 3-D as point clouds of laser reflections is proposed. Several novelties are discussed: (1) extension of the existing convolutional neural network (CNN) framework to direct processing of 3-D data in a multiview setting which may be helpful for rotation-invariant consideration, (2) improvement of CNN training effectiveness by employing a stochastic meta-descent (SMD) method, and (3) combination of unsupervised and supervised training for enhanced performance of CNN. CNN performance is illustrated on a two-class data set of objects in a segmented outdoor environment.

In this paper, we introduce the DEEP-SEE FACE framework, an assistive device designed to improve cognition, interaction, and communication of visually impaired (VI) people in social encounters. The proposed approach jointly exploits computer vision algorithms (region proposal networks, ATLAS tracking and global, and low-level image descriptors) and deep convolutional neural networks in order to detect, track, and recognize, in real-time, various persons existent in the video streams. The major contribution of the paper concerns a global, fixed-size face representation that takes into the account of various video frames while remaining independent of the length of the image sequence. To this purpose, we introduce an effective weight adaptation scheme that is able to determine the relevance assigned to each face instance, depending on the frame degree of motion/camera blur, scale variation, and compression artifacts. Another relevant contribution involves a hard negative mining stage that helps us differentiating between known and unknown face identities. The experimental results, carried out on a large-scale data set, validate the proposed methodology with an average accuracy and recognition rates superior to 92%. When tested in real life, indoor/outdoor scenarios, the DEEP-SEE FACE prototype proves to be effective and easy to use, allowing the VI people to access visual information during social events.


Question: An end-to-end pedestrian collision warning system utilizing a convolutional neural network for semantic segmentation.

Introducing the UrbanScenes Dataset: A Comprehensive Resource for Semantic Analysis of Complex City Environments

The accurate and detailed understanding of urban street scenes is a critical component for numerous applications, including autonomous driving and smart city initiatives. While large-scale datasets have significantly advanced object detection, particularly in the era of deep learning, there remains a gap in datasets tailored to the intricate and diverse nature of urban environments for semantic analysis. To bridge this gap, we present UrbanScenes, a benchmark suite and extensive dataset designed to foster the development and evaluation of algorithms for pixel-level and instance-level semantic labeling in urban settings.

UrbanScapes is constructed from a vast and varied collection of stereo video sequences captured from the streets of 50 distinct cities. A subset of 5,000 images is meticulously annotated with pixel-level precision, providing a rich ground truth for training and testing. Additionally, 20,000 more images are included with coarser annotations to support methodologies that can leverage large quantities of less refined data. Our dataset surpasses previous efforts in terms of scale, annotation depth, scene diversity, and the complexity of urban environments it encompasses.

Accompanying the dataset is a detailed empirical study that not only characterizes the dataset's attributes but also assesses the performance of several leading-edge approaches against our benchmark. UrbanScenes aims to serve as a pivotal resource for researchers and developers seeking to advance semantic urban scene understanding, offering a comprehensive and nuanced dataset that reflects the multifaceted nature of cityscapes.

An important aspect of collision avoidance and driver assistance systems, as well as autonomous vehicles, is the tracking of vehicle taillights and the detection of alert signals (turns and brakes). In this paper, we present the design and implementation of a robust and computationally lightweight algorithm for a real-time vision system, capable of detecting and tracking vehicle taillights, recognizing common alert signals using a vehicle-mounted embedded smart camera, and counting the cars passing on both sides of the vehicle. The system is low-power and processes scenes entirely on the microprocessor of an embedded smart camera. In contrast to most existing work that addresses either daytime or nighttime detection, the presented system provides the ability to track vehicle taillights and detect alert signals regardless of lighting conditions. The mobile vision system has been tested in actual traffic scenes and the results obtained demonstrate the performance and the lightweight nature of the algorithm.

Many Advanced Driver Assistance Systems (ADAS) have been developed to improve car safety. However, it is still a challenging problem to make autonomous vehicles to drive safely on urban streets such as uncontrolled intersections (without traffic lights) and narrow roads. In this paper, we introduce a decision making system that can assist autonomous vehicles at uncontrolled intersections and narrow roads. We constructed a machine understandable ontology-based Knowledge Base, which contains maps and traffic regulations. The system makes decisions in comply with traffic regulations such as Right-Of-Way rules when it receives a collision warning signal. The decisions are sent to a path planning system to change the route or stop to avoid collisions.

In this paper, a deep learning inspired solution is proposed and developed to enable timeliness and practicality of the pre-existing ground stereo vision guidance system for flxed-wing UAVs' safe landing. Since the ground guidance prototype was restricted within applications due to its untimeliness, eventually the vision-based detection less than 15 fps (frame per second). Under such circumstances, we employ a regression based deep learning algorithm into automatic detection on the flying aircraft in the landing sequential images. The system architecture is upgraded so as to be compatible with the novel deep learning requests, and furthermore, annotated datasets are conducted to support training and testing of the regression-based learning detection algorithm. Experimental results validate that the detection attaches 100 fps or more while the localization accuracy is kept in the same level.

Dempster-Shafer theory provides a sensor fusion framework that autonomously accounts for obstacle occlusion in dynamic, urban environments. However, to discern static and moving obstacles, the Dempster-Shafer approach requires manual tuning of parameters dependent on the situation and sensor types. The proposed methodology utilizes a deep fully convolutional neural network to improve the robust performance of the information fusion algorithm in distinguishing static and moving obstacles from navigable space. The image-like spatial structure of probabilistic occupancy allows a semantic segmentation framework to discern classes for individual grid cells. A subset of the KITTI LIDAR tracking dataset in combination with semantic map data was used for the information fusion task. The probabilistic occupancy grid output of the Dempster-Shafer information fusion algorithm was provided as input to the neural network. The network then learned an offset from the original DST result to improve semantic labeling performance. The proposed framework outperformed the baseline approach in the mean intersection over union metric reaching 0.546 and 0.531 in the validation and test sets respectively. However, little improvement was achieved in discerning moving and static cells due to the limited dataset size. To improve model performance in future work, the dataset will be expanded to facilitate more effective learning, and temporal data will be fed through individual convolutional networks prior to being merged in channels as input to the main network.

Recently, autonomous vehicles, particularly self-driving cars, have received significant attention owing to rapid advancements in sensor and computation technologies. In addition to traffic sign recognition, road lane detection is one of the most important factors used in lane departure warning systems and autonomous vehicles for maintaining the safety of semi-autonomous and fully autonomous systems. Unlike traffic signs, road lanes are easily damaged by both internal and external factors such as road quality, occlusion (traffic on the road), weather conditions, and illumination (shadows from objects such as cars, trees, and buildings). Obtaining clear road lane markings for recognition processing is a difficult challenge. Therefore, we propose a method to overcome various illumination problems, particularly severe shadows, by using fuzzy system and line segment detector algorithms to obtain better results for detecting road lanes by a visible light camera sensor. Experimental results from three open databases, Caltech dataset, Santiago Lanes dataset (SLD), and Road Marking dataset, showed that our method outperformed conventional lane detection methods.

Deep learning has been widely recognized as a promising approach in different computer vision applications. Specifically, one-stage object detector and two-stage object detector are regarded as the most important two groups of Convolutional Neural Network based object detection methods. One-stage object detector could usually outperform two-stage object detector in speed; However, it normally trails in detection accuracy, compared with two-stage object detectors. In this study, focal loss based RetinaNet, which works as one-stage object detector, is utilized to be able to well match the speed of regular one-stage detectors and also defeat two-stage detectors in accuracy, for vehicle detection. State-of-the-art performance result has been showed on the DETRAC vehicle dataset.

A vision-based vehicle guidance system for road vehicles can have three main roles: 1) road detection; 2) obstacle detection; and 3) sign recognition. The first two have been studied for many years and with many good results, but traffi c sign recognition is a less-studied field. Traffi c signs provide drivers with very valuable informatio n about the road, in order to make drivin g safer and easier. We think that traffi c signs must play the same role for autonomous vehicles. They are designed to be easily recognized by human driver s mainly because their color and shapes are very different from natural environments. The algorithm described in this paper takes advantage of these features. I t has two main parts. The first one, for the detection, uses color thresholding to segment the image and shape analysis to detect the signs. The second one, for the classification, uses a neural network. Some results from natural scenes are shown. On the other hand, the algorithm is valid to detect other kinds of marks that would tell the mobile robot to perform some task at that place.

Autonomously searching for hazardous radiation sources requires the ability of the aerial and ground systems to understand the scene they are scouting. In this paper, we present systems, algorithms, and experiments to perform radiation search using unmanned aerial vehicles (UAV) and unmanned ground vehicles (UGV) by employing semantic scene segmentation. The aerial data is used to identify radiological points of interest, generate an orthophoto along with a digital elevation model (DEM) of the scene, and perform semantic segmentation to assign a category (e.g . road, grass) to each pixel in the orthophoto. We perform semantic segmentation by training a model on a dataset of images we collected and annotated, using the model to perform inference on images of the test area unseen to the model, and then refining the results with the DEM to better reason about category predictions at each pixel. We then use all of these outputs to plan a path for a UGV carrying a LiDAR to map the environment and avoid obstacles not present during the flight, and a radiation detector to collect more precise radiation measurements from the ground. Results of the analysis for each scenario tested favorably. We also note that our approach is general and has the potential to work for a variety of different sensing tasks.

This paper describes a LIDAR-based perception system for ground robot mobility, consisting of 3D object detection, classification and tracking. The presented system was demonstrated on-board our autonomous ground vehicle MuCAR-3, enabling it to safely navigate in urban traffic-like scenarios as well as in off-road convoy scenarios. The efficiency of our approach stems from the unique combination of 2D and 3D data processing techniques. Whereas fast segmentation of point clouds into objects is done in a 2 D occupancy grid, classifying the objects is done on raw 3D point clouds. For fast object feature extraction, we advocate the use of statistics of local point cloud properties, captured by histograms over point features. In contrast to most existing work on 3D point cloud classification, where real-time operation is often impossible, this combination allows our system to perform in real-time at 0.1s frame-rate.

In this work we show that Deep Convolutional Neural Networks can outperform humans on the task of boundary detection, as measured on the standard Berkeley Segmentation Dataset. Our detector is fully integrated in the popular Caffe framework and processes a 320x420 image in less than a second. Our contributions consist firstly in combining a careful design of the loss for boundary detection training, a multi-resolution architecture and training with external data to improve the detection accuracy of the current state of the art, from an optimal dataset scale F-measure of 0.780 to 0.808 while human performance is at 0.803. We further improve performance to 0.813 by combining deep learning with grouping, integrating the Normalized Cuts technique within a deep network. We also examine the potential of our boundary detector in conjunction with the higher level tasks of object proposal generation and semantic segmentation for both tasks our detector yields clear improvements over state-of-the-art systems.


Question: An end-to-end pedestrian collision warning system utilizing a convolutional neural network for semantic segmentation.

An Overview of Pedestrian Detection Techniques for Enhanced Driver Assistance Systems

In the realm of advanced driver assistance systems (ADASs), particularly within pedestrian protection systems (PPSs), there has been a surge in research focused on enhancing traffic safety. A critical challenge in PPSs is the development of dependable on-board pedestrian detection mechanisms. The complexity arises from the dynamic and diverse appearance of pedestrians, such as varying attire, size, aspect ratios, and shapes, compounded by the unpredictable nature of the surrounding environment. These factors demand a high level of system robustness that is difficult to achieve. One of the key issues in this research domain is the scarcity of accessible benchmarks and the complexities associated with replicating many of the proposed methodologies, leading to difficulties in comparative analysis. As a result, a straightforward enumeration of proposals does not provide a comprehensive comparative perspective. To address this, we propose a systematic approach to survey the literature. We dissect the pedestrian detection process from images into distinct processing stages, each with specific functions. Subsequently, the various proposed methods are meticulously analyzed and categorized according to these processing stages, facilitating a comparative analysis. Concludingly, we delve into critical topics, with a focus on the future requirements and challenges that lie ahead in this field.

Crowd counting usually addressed by density estimation becomes an increasingly important topic in computer vision due to its widespread applications in video surveillance, urban planning, and intelligence gathering. However, it is essentially a challenging task because of the greatly varied sizes of objects, coupled with severe occlusions and vague appearance of extremely small individuals. Existing methods heavily rely on multi-column learning architectures to extract multi-scale features, which however suffer from heavy computational cost, especially undesired for crowd counting. In this paper, we propose the single-column counting network (SCNet) for efficient crowd counting without relying on multi-column networks. SCNet consists of residual fusion modules (RFMs) for multi-scale feature extraction, a pyramid pooling module (PPM) for information fusion, and a sub-pixel convolutional module (SPCM) followed by a bilinear upsampling layer for resolution recovery. Those proposed modules enable our SCNet to fully capture multi-scale features in a compact single-column architecture and estimate high-resolution density map in an efficient way. In addition, we provide a principled paradigm for density map generation and data augmentation for training, which shows further improved performance. Extensive experiments on three benchmark datasets show that our SCNet delivers new state-of-the-art performance and surpasses previous methods by large margins, which demonstrates the great effectiveness of SCNet as a singlecolumn network for crowd counting. c © 2018. The copyright of this document resides with its authors. It may be distributed unchanged freely in print or electronic forms. *Equal contribution 2 WANG et al.: IN DEFENSE OF SINGLE-COLUMN NETWORKS FOR CROWD COUNTING

The semantic segmentation of High Resolution Remote Sensing (HRRS) images is the fundamental research area of the earth observation. Convolutional Neural Network (CNN), which has achieved superior performance in computer vision task, is also useful for semantic segmentation of HRRS images. In this work, focal loss is used instead of cross-entropy loss in training of CNN to handle the imbalance in training data. To evaluate the effect of focal loss, we train SegNet and FCN with focal loss and confirm improvement in accuracy in ISPRS 2D Semantic Labeling Contest dataset, especially when $\gamma$ is 0.5 in SegNet.

Visual object tracking is challenging as target objects often undergo significant appearance changes caused by deformation, abrupt motion, background clutter and occlusion. In this paper, we exploit features extracted from deep convolutional neural networks trained on object recognition datasets to improve tracking accuracy and robustness. The outputs of the last convolutional layers encode the semantic information of targets and such representations are robust to significant appearance variations. However, their spatial resolution is too coarse to precisely localize targets. In contrast, earlier convolutional layers provide more precise localization but are less invariant to appearance changes. We interpret the hierarchies of convolutional layers as a nonlinear counterpart of an image pyramid representation and exploit these multiple levels of abstraction for visual tracking. Specifically, we adaptively learn correlation filters on each convolutional layer to encode the target appearance. We hierarchically infer the maximum response of each layer to locate targets. Extensive experimental results on a largescale benchmark dataset show that the proposed algorithm performs favorably against state-of-the-art methods.

In this paper we address the problem of semantic labeling of indoor scenes on RGB-D data. With the availability of RGB-D cameras, it is expected that additional depth measurement will improve the accuracy. Here we investigate a solution how to incorporate complementary depth information into a semantic segmentation framework by making use of convolutional neural networks (CNNs). Recently encoder-decoder type fully convolutional CNN architectures have achieved a great success in the field of semantic segmentation. Motivated by this observation we propose an encoder-decoder type network, where the encoder part is composed of two branches of networks that simultaneously extract features from RGB and depth images and fuse depth features into the RGB feature maps as the network goes deeper. Comprehensive experimental evaluations demonstrate that the proposed fusion-based architecture achieves competitive results with the state-of-the-art methods on the challenging SUN RGB-D benchmark obtaining 76.27% global accuracy, 48.30% average class accuracy and 37.29% average intersectionover-union score.

This article proposes a system that fuses radar and monocular vision sensor data in order to detect and classify on-road obstacles, like cars or not cars (other obstacles). The obstacle detection process and classification is divided into three stages, the first consist in reading radar signals and capturing the camera data, the second stage is the data fusion, and the third step is the classify the obstacles, aiming to differentiate the obstacles types identified by the radar and confirmed by the computer vision. In the detection task it is important to locate, measure, and rank the obstacles to be able to take adequate decisions and actions (e.g. Generate alerts, autonomously control the vehicle path), so the correct classification of the obstacle type and position, is very important, also avoiding false positives and/or false negatives in the classification task.

This paper introduces a system for estimating the attention of a driver wearing a first person view camera using salient objects to improve gaze estimation. A challenging data set of pedestrians crossing intersections has been captured using Google Glass worn by a driver. A challenge unique to first person view from cars is that the interior of the car can take up a large part of the image. The proposed system automatically filters out the dashboard of the car, along with other parts of the instrumentation. The remaining area is used as a region of interest for a pedestrian detector. Two cameras looking at the driver are used to determine the direction of the driver's gaze, by examining the eye corners and the center of the iris. This coarse gaze estimation is then linked to the detected pedestrians to determine which pedestrian the driver is focused on at any given time.

Vehicles of higher automation levels require the creation of situation awareness. One important aspect of this situation awareness is an understanding of the current risk of a driving situation. In this work, we present a novel approach for the dynamic risk assessment of driving situations based on images of a front stereo camera using deep learning. To this end, we trained a deep neural network with recorded monocular images, disparity maps and a risk metric for diverse traffic scenes. Our approach can be used to create the aforementioned situation awareness of vehicles of higher automation levels and can serve as a heterogeneous channel to systems based on radar or lidar sensors that are used traditionally for the calculation of risk metrics.

This paper proposes a robust and effective vision-based lane detection approach. First, two binary images are obtained from the region of interest of gray-scale images. The obtained binary images are merged by a novel neighborhood AND operator and then transformed to a bird's eye view (BEV) via inverse perspective mapping. Then, gaussian probability density functions are fit to the left and right regions of a histogram image acquired from the BEV. Finally, a polynomial lane model is estimated from the identified regions. Experimental results show that the proposed method accurately detects lanes in complex situations including worn-out and curved lanes.

Reliable traffic light detection is one crucial key component for autonomous driving in urban areas. This includes the extraction of direction arrows contained within the traffic lights as an autonomous car will need this information for selecting the traffic light corresponding to its current lane. Current state of the art traffic light detection systems are not able to provide such information. Within this work we present a hierarchical traffic light detection algorithm, which is able to detect traffic lights and determine their state and contained direction information within one CNN forward pass. This Hierarchical DeepTLR (HDTLR) outperforms current state of the art traffic light detection algorithms in state aware detection and can detect traffic lights with direction information down to a size of 4 pixel in width at a frequency of 12 frames per second.

Video-based traffic sign detection, tracking, and recognition is one of the important components for the intelligent transport systems. Extensive research has shown that pretty good performance can be obtained on public data sets by various state-of-the-art approaches, especially the deep learning methods. However, deep learning methods require extensive computing resources. In addition, these approaches mostly concentrate on single image detection and recognition task, which is not applicable in real-world applications. Different from previous research, we introduce a unified incremental computational framework for traffic sign detection, tracking, and recognition task using the mono-camera mounted on a moving vehicle under non-stationary environments. The main contributions of this paper are threefold: 1) to enhance detection performance by utilizing the contextual information, this paper innovatively utilizes the spatial distribution prior of the traffic signs; 2) to improve the tracking performance and localization accuracy under non-stationary environments, a new efficient incremental framework containing off-line detector, online detector, and motion model predictor together is designed for traffic sign detection and tracking simultaneously; and 3) to get a more stable classification output, a scale-based intra-frame fusion method is proposed. We evaluate our method on two public data sets and the performance has shown that the proposed system can obtain results comparable with the deep learning method with less computing resource in a near-real-time manner.