Question: DCAN: Dual Channel-Wise Alignment Networks for unsupervised adaptation of scene data.

This paper presents Dual Channel-Wise Alignment Networks (DCAN), a novel method for unsupervised scene adaptation in neural networks. Inspired by the effectiveness of regularization techniques, DCAN introduces a mechanism to enhance the adaptation of features across different domains. In contrast to traditional methods that often focus on aligning global or local features, DCAN innovatively aligns features at the channel-wise level, enabling a more nuanced and precise adaptation. The proposed method not only aligns features between source and target domains but also maintains the discriminative power of the features. This is achieved through a dual alignment strategy: one that aligns the channel-wise features and another that ensures the aligned features are discriminative. DCAN is evaluated on various unsupervised scene adaptation tasks, demonstrating its superiority over existing methods. Through extensive experiments, DCAN shows state-of-the-art performance on several benchmarks, highlighting its effectiveness in unsupervised scene adaptation. By leveraging the channel-wise alignment, DCAN achieves robust adaptation across different scenes without requiring labeled data from the target domain.

In this paper, we make a multiview extension of the spectral rotation technique raised in single view spectral clustering research. Since spectral rotation is closely related to the Procrustes Analysis for points matching, we point out that classical Procrustes Average approach can be used for multiview clustering. Besides, we show that direct applying Procrustes Average (PA) in multiview tasks may not be optimal theoretically and empirically, since it does not take the clustering capacity differences of different views into consideration. Other than that, we propose an Adaptively Weighted Procrustes (AWP) approach to overcome the aforementioned deficiency. Our new AWP weights views with their clustering capacities and forms a weighted Procrustes Average problem accordingly. The optimization algorithm to solve the new model is computational complexity analyzed and convergence guaranteed. Experiments on five real-world datasets demonstrate the effectiveness and efficiency of the new models.

Over the last years, object recognition has become a more and more active field of research in robotics. An important problem in object recognition is the need for sufficient labeled training data to learn good classifiers. In this paper we show how to significantly reduce the need for manually labeled training data by leveraging data sets available on the World Wide Web. Specifically, we show how to use objects from Google’s 3D Warehouse to train classifiers for 3D laser scans collected by a robot navigating through urban environments. In order to deal with the different characteristics of the web data and the real robot data, we additionally use a small set of labeled 3D laser scans and perform domain adaptation. Our experiments demonstrate that additional data taken from the 3D Warehouse along with our domain adaptation greatly improves the classification accuracy on real laser scans.

We consider the semi-supervised multi-class classification problem of learning from sparse labelled and abundant unlabelled training data. To address this problem, existing semi-supervised deep learning methods often rely on the up-to-date “network-in-training” to formulate the semi-supervised learning objective. This ignores both the discriminative feature representation and the model inference uncertainty revealed by the network in the preceding learning iterations, referred to as the memory of model learning. In this work, we propose a novel Memory-Assisted Deep Neural Network (MA-DNN) capable of exploiting the memory of model learning to enable semi-supervised learning. Specifically, we introduce a memory mechanism into the network training process as an assimilation-accommodation interaction between the network and an external memory module. Experiments demonstrate the advantages of the proposed MA-DNN model over the state-of-the-art semisupervised deep learning methods on three image classification benchmark datasets: SVHN, CIFAR10, and CIFAR100.

We show how to train a fully convolutional neural network to perform inverse rendering from a single, uncontrolled image. The network takes an RGB image as input, regresses albedo and normal maps from which we compute lighting coefficients. Our network is trained using large uncontrolled image collections without ground truth. By incorporating a differentiable renderer, our network can learn from self-supervision. Since the problem is ill-posed we introduce additional supervision: 1. We learn a statistical natural illumination prior, 2. Our key insight is to perform offline multiview stereo (MVS) on images containing rich illumination variation. From the MVS pose and depth maps, we can cross project between overlapping views such that Siamese training can be used to ensure consistent estimation of photometric invariants. MVS depth also provides direct coarse supervision for normal map estimation. We believe this is the first attempt to use MVS supervision for learning inverse rendering.

In this paper, we raise important issues on scalability and the required degree of supervision of existing Mahalanobis metric learning methods. Often rather tedious optimization procedures are applied that become computationally intractable on a large scale. Further, if one considers the constantly growing amount of data it is often infeasible to specify fully supervised labels for all data points. Instead, it is easier to specify labels in form of equivalence constraints. We introduce a simple though effective strategy to learn a distance metric from equivalence constraints, based on a statistical inference perspective. In contrast to existing methods we do not rely on complex optimization problems requiring computationally expensive iterations. Hence, our method is orders of magnitudes faster than comparable methods. Results on a variety of challenging benchmarks with rather diverse nature demonstrate the power of our method. These include faces in unconstrained environments, matching before unseen object instances and person re-identification across spatially disjoint cameras. In the latter two benchmarks we clearly outperform the state-of-the-art.

We take a new approach to computing dense scene flow between a pair of consecutive RGB-D frames. We exploit the availability of depth data by seeking correspondences with respect to patches specified not as the pixels inside square windows, but as the 3D points that are the inliers of spheres in world space. Our primary contribution is to show that by reasoning in terms of such patches under 6 DoF rigid body motions in 3D, we succeed in obtaining compelling results at displacements large and small without relying on either of two simplifying assumptions that pervade much of the earlier literature: brightness constancy or local surface planarity. As a consequence of our approach, our output is a dense field of 3D rigid body motions, in contrast to the 3D translations that are the norm in scene flow. Reasoning in our manner additionally allows us to carry out occlusion handling using a 6 DoF consistency check for the flow computed in both directions and a patchwise silhouette check to help reason about alignments in occlusion areas, and to promote smoothness of the flow fields using an intuitive local rigidity prior. We carry out our optimization in two steps, obtaining a first correspondence field using an adaptation of PatchMatch, and subsequently using alpha-expansion to jointly handle occlusions and perform regularization. We show attractive flow results on challenging synthetic and real-world scenes that push the practical limits of the aforementioned assumptions.

Recent approaches on visual scene understanding attempt to build a scene graph – a computational representation of objects and their pairwise relationships. Such rich semantic representation is very appealing, yet difficult to obtain from a single image, especially when considering complex spatial arrangements in the scene. Differently, an image sequence conveys useful information using the multi-view geometric relations arising from camera motions. Indeed, object relationships are naturally related to the 3D scene structure. To this end, this paper proposes a system that first computes the geometrical location of objects in a generic scene and then efficiently constructs scene graphs from video by embedding such geometrical reasoning. Such compelling representation is obtained using a new model where geometric and visual features are merged using an RNN framework. We report results on a dataset we created for the task of 3D scene graph generation in multiple views.

In this paper we propose a convolutional neural network (CNN), which allows to identify corresponding patches of very high resolution (VHR) optical and SAR imagery of complex urban scenes. Instead of a siamese architecture as conventionally used in CNNs designed for image matching, we resort to a pseudo-siamese configuration with no interconnection between the two streams for SAR and optical imagery. The network is trained with automatically generated training data and does not resort to any hand-crafted features. First evaluations show that the network is able to predict corresponding patches with high accuracy, thus indicating great potential for further development to a generalized multi-sensor matching procedure.

Although it is typically expected that using a large amount of labeled training data would lead to improve performance in deep learning, it is generally difficult to obtain such DataBase (DB). In competitions such as the Detection and Classification of Acoustic Scenes and Events (DCASE) challenge Task 1, participants are constrained to use a relatively small DB as a rule, which is similar to the aforementioned issue. To improve Acoustic Scene Classification (ASC) performance without employing additional DB, this paper proposes to use Generative Adversarial Networks (GAN) based method for generating additional training DB. Since it is not clear whether every sample generated by GAN would have equal impact in classification performance, this paper proposes to use Support Vector Machine (SVM) hyper plane for each class as reference for selecting samples, which have class discriminative information. Based on the crossvalidated experiments on development DB, the usage of the generated features could improve ASC performance.

High-resolution remote sensing (HRRS) scene classification plays an important role in numerous applications. During the past few decades, a lot of remarkable efforts have been made to develop various methods for HRRS scene classification. In this paper, focusing on the problems of complex context relationship and large differences of object scale in HRRS scene images, we propose a deep CNN-based scene classification method, which not only enables to enhance the ability of spatial representation, but adaptively recalibrates channel-wise feature responses to suppress useless feature channels. We evaluated the proposed method on a publicly large-scale dataset with several state-of-the-art convolutional neural network (CNN) models. The experimental results demonstrate that the proposed method is effective to extract high-level category features for HRRS scene classification.


Question: A visible band image segmentation method using shadow-based rooftop analysis.

This paper presents an innovative method for urban area and building detection in high-resolution satellite images, leveraging Scale-Invariant Feature Transform (SIFT) keypoints and graph theory. The extraction of urban boundaries and building locations from these images is a laborious task for human experts, making automated techniques a desirable solution. However, traditional image processing and pattern recognition methods are not always effective. To address this, we propose a technique that combines SIFT with graph theoretical concepts. SIFT keypoints are adept at identifying objects under diverse imaging conditions, yet they are not sufficient on their own for urban and building detection. To overcome this, we formulate the problem within the framework of graph theory. Each keypoint is represented as a vertex in the graph, with edges determined by the spatial distance and intensity values between vertices. Utilizing this formulation, we introduce a novel multiple subgraph matching technique to delineate urban areas. Subsequently, a new graph cut method is applied to isolate individual buildings within these urban zones. Our method is evaluated using a comprehensive test set comprised of 1-meter resolution panchromatic Ikonos imagery. Extensive testing reveals highly promising outcomes in the automated detection of urban areas and buildings.

Based on a multi-step process, an automatic hair segmentation method is created and tested on a database of 115 manually segmented hair images. By extracting various information components from an image, including background color, face position, hair color, skin color, and skin mask, a heuristic-based method is created for the detection and segmentation of hair that can detect hair with an accuracy of approximately 75% and with a false hair overestimation error of 34%. Furthermore, it is shown that downsampling the image down to a face width of 25px results in a 73% reduction in computation time with insignificant change in detection accuracy.

Image segmentation is one of the basic concepts widely used in each and every fields of image processing. The entire process of the proposed work for image segmentation comprises of 3 phases: Threshold generation with dynamic Modified Region Growing phase (DMRG), texture feature generation phase and region merging phase. by dynamically changing two thresholds, the given input image can be performed as DMRG, in which the cuckoo search optimization algorithm helps to optimize the two thresholds in modified region growing. after obtaining the region growth segmented image, the edges are detected with edge detection algorithm. In the second phase, the texture feature is extracted using entropy based operation from the input image. In region merging phase, the results obtained from the texture feature generation phase is combined with the results of DMRG phase and similar regions are merged by using a distance comparison between regions. The proposed work is implemented using Mat lab platform with several medical images. the performance of the proposed work is evaluated using the metrics sensitivity, specificity and accuracy. the results show that this proposed work provides very good accuracy for the segmentation process in images.

A novel approach is proposed in this paper for automatic forest fire detection from video. Based on 3D point cloud of the collected sample fire pixels, Gaussian mixture model is built and helps segment some possible flame regions in single image. Then the new specific flame pattern is defined for forest, and three types of fire colors are labeled accordingly. With 11 static features including color distributions, texture parameters and shape roundness, the static SVM classifier is trained and filters the segmented results. Using defined overlapping degree and varying degree, the remained candidate regions are matched among consecutive frames. Subsequently the variations of color, texture, roundness, area, contour are computed, then the average and the mean square deviation of them are obtained. Together with the flickering frequency from temporal wavelet based Fourier descriptors analysis of flame contour, 27 dynamic features are used to train the dynamic SVM classifier, which is applied for final decision. Our approach has been tested with dozens of video clips, and it can detect forest fire while recognize the fire like objects, such as red house, bright light and flying flag. Except for the acceptable accuracy, our detection algorithm performs in real time, which proves its value for computer vision based forest fire surveillance.

Image registration is an essential step in the process of image fusion, environment surveillance and change detection. Finding correct feature matches during the registration process proves to be difficult, especially for remote sensing images with large background variations (e.g., images taken pre and post an earthquake or flood). Traditional registration methods based on local intensity probably cannot maintain steady performances, as differences are significant in the same area of the corresponding images, and ground control points are not always available in many disaster images. In this paper, an automatic image registration method based on the line segments on the main shape contours (e.g., coastal lines, long roads and mountain ridges) is proposed for remote sensing images with large background variations because the main shape contours can hold relatively more invariant information. First, a line segment detector called EDLines (Edge Drawing Lines), which was proposed by Akinlar et al. in 2011, is used to extract line segments from two corresponding images, and a line validation step is performed to remove meaningless and fragmented line segments. Then, a novel line segment descriptor with a new histogram binning strategy, which is robust to global geometrical distortions, is generated for each line segment based on the geometrical relationships,including both the locations and orientations of theremaining line segments relative to it. As a result of the invariance of the main shape contours, correct line segment matches will have similar descriptors and can be obtained by cross-matching among the descriptors. Finally, a spatial consistency measure is used to remove incorrect matches, and transformation parameters between the reference and sensed images can be figured out. Experiments with images from different types of satellite datasets, such as Landsat7, QuickBird, WorldView, and so on, demonstrate that the proposed algorithm is automatic, fast (4 ms faster than the second fastest method, i.e., the rotationand scale-invariant shape context) and can achieve a recall of 79.7%, a precision of 89.1% and a root mean square error (RMSE) of 1.0 pixels on average for remote sensing images with large background variations.

A reliable human skin detection method that is adaptable to different human skin colors and illumination conditions is essential for better human skin segmentation. Even though different human skin-color detection solutions have been successfully applied, they are prone to false skin detection and are not able to cope with the variety of human skin colors across different ethnic. Moreover, existing methods require high computational cost. In this paper, we propose a novel human skin detection approach that combines a smoothed 2-D histogram and Gaussian model, for automatic human skin detection in color image(s). In our approach, an eye detector is used to refine the skin model for a specific person. The proposed approach reduces computational costs as no training is required, and it improves the accuracy of skin detection despite wide variation in ethnicity and illumination. To the best of our knowledge, this is the first method to employ fusion strategy for this purpose. Qualitative and quantitative results on three standard public datasets and a comparison with state-of-the-art methods have shown the effectiveness and robustness of the proposed approach.

Hand segmentation for hand-object interaction is a necessary preprocessing step in many applications such as augmented reality, medical application, and human-robot interaction. However, typical methods are based on color information which is not robust to objects with skin color, skin pigment difference, and light condition variations. Thus, we propose hand segmentation method for hand-object interaction using only a depth map. It is challenging because of the small depth difference between a hand and objects during an interaction. To overcome this challenge, we propose the two-stage random decision forest (RDF) method consisting of detecting hands and segmenting hands. To validate the proposed method, we demonstrate results on the publicly available dataset of hand segmentation for hand-object interaction. The proposed method achieves high accuracy in short processing time comparing to the other state-of-the-art methods.

A method is presented for automated segmentation of vessels in two-dimensional color images of the retina. This method can be used in computer analyses of retinal images, e.g., in automated screening for diabetic retinopathy. The system is based on extraction of image ridges, which coincide approximately with vessel centerlines. The ridges are used to compose primitives in the form of line elements. With the line elements an image is partitioned into patches by assigning each image pixel to the closest line element. Every line element constitutes a local coordinate frame for its corresponding patch. For every pixel, feature vectors are computed that make use of properties of the patches and the line elements. The feature vectors are classified using a kNN-classifier and sequential forward feature selection. The algorithm was tested on a database consisting of 40 manually labeled images. The method achieves an area under the receiver operating characteristic curve of 0.952. The method is compared with two recently published rule-based methods of Hoover et al. and Jiang et al. . The results show that our method is significantly better than the two rule-based methods (p<0.01). The accuracy of our method is 0.944 versus 0.947 for a second observer.

A method is described which quantifies the speed and direction of several moving objects in a sequence of digital images. A relationship between the time variation of intensity, the spatial gradient, and velocity has been developed which allows the determination of motion using clustering techniques. This paper describes these relationships, the clustering technique, and provides examples of the technique on real images containing several moving objects.

This paper proposes a robust motion detection method for illumination variations which uses histogram of oriented gradients. The detection process is divided into two phases: coarse detection and refinement. In the coarse detection phase, first, a texture-based background model is built which implements a group of adaptive histograms of oriented gradients; then, by comparing the histogram of oriented gradients of each pixel between current frame and background model, a foreground is segmented based on texture feature which is not susceptible to illumination variations, whereas some missing foreground regions exist; finally, the result based on texture is optimized by combining the pixel-wise detection result produced by Gaussian Mixture Model (GMM) algorithm, which greatly improves the detection performance by incorporating efficient morphological operations. In the refinement phase, the above detection result is refined based on the distinction in color feature to eliminate errors like shadows, noises, redundant contour, etc. Experimental results show the effectiveness and robustness of our approach in detecting moving objects in varying illumination conditions.

To facilitate scene understanding and robot navigation in a modern urban area, we design a multilayer feature graph (MFG) based on two views from an on-board camera. The nodes of an MFG are features such as scale invariant feature transformation (SIFT) feature points, line segments, lines, and planes while edges of the MFG represent different geometric relationships such as adjacency, parallelism, collinearity, and coplanarity. MFG also connects the features in two views and the corresponding 3D coordinate system. Building on SIFT feature points and line segments, MFG is constructed using feature fusion which incrementally, iteratively, and extensively verifies the aforementioned geometric relationships using random sample consensus (RANSAC) framework. Physical experiments show that MFG can be successfully constructed in urban area and the construction method is demonstrated to be very robust in identifying feature correspondence.


Question: Rooftop segmentation using shadow-based techniques in visible band imagery.

In this work, we explore a novel technique for rooftop segmentation in visible band images, drawing inspiration from the "GrabCut" algorithm, which is renowned for interactive foreground extraction using iterated graph cuts. Unlike traditional methods that often struggle with precise boundary detection, "GrabCut" offers a more refined and interactive approach to segmenting objects, particularly in complex scenes. By employing graph cuts, the algorithm iteratively refines the segmentation, allowing for the accurate delineation of rooftops within the images. This technique is particularly advantageous in scenarios where shadows and varying lighting conditions can pose challenges to standard segmentation methods. Through user-provided initial estimates or rough boundaries, "GrabCut" iteratively improves the segmentation by minimizing an energy function that considers both color and spatial information. This results in a more accurate and detailed segmentation of rooftops, making it a valuable tool in various applications such as urban planning, disaster response, and environmental monitoring. The iterative nature of "GrabCut" ensures that even subtle features and boundaries of rooftops are captured, enhancing the overall quality and reliability of the segmentation process.

Photometric stereo is one technique for 3D shape determination that has been implemented in a variety of experimental settings and that has produced consistently good results. The idea is to use intensity values recorded from multiple images obtained from the same viewpoint but under different conditions of illumination. The resulting radiometric constraint makes it possible to obtain local estimates of both surface orientation and surface curvature without requiring global smoothness assumptions and without requiring prior image segmentation. This paper moves photometric stereo one step closer to practical viability by describing an experimental setting in which surface gradient estimation is achieved on full frame video data at near video frame rates (i.e., 15Hz). The implementation uses commercially available hardware. Reflectance is modeled empirically using measurements obtained from a calibration sphere. Estimation of the gradient, (p, q), requires only simple table lookup. Curvature estimation uses, in addition, the reflectance map, R(p, q). The required lookup table and reflectance maps are derived during calibration. Because reflectance is modeled empirically, no prior physical model of the reflectance characteristics of the objects to be analyzed is assumed. At the same time, if a good physical model is available, it can be retrofit to the method for implementation purposes. Photometric stereo is subject to error in the presence of cast shadows and interreflection. No purely local technique can succeed since these phenomena are inherently non-local. Nevertheless, this paper demonstrates that one can exploit the redundancy in three light source photometric stereo to, in most cases, locally detect the presence of cast shadows and interreflection. Detection is facilitated by explicitly including a local confidence estimate in the lookup table used for gradient estimation.

We formulate edge detection as statistical inference. This statistical edge detection is data driven, unlike standard methods for edge detection which are model based. For any set of edge detection filters (implementing local edge cues) we use pre-segmented images to learn the probability distributions of filter responses conditioned on whether they are evaluated on or off an edge. Edge detection is formulated as a discrimination task specified by a likelihood ratio test on the filter responses. This approach emphasizes the necessity of modeling the image background (the off-edges). We represent the conditional probability distributions non-parametrically and learn them on two different datasets of 100 (Sowerby) and 50 (South Florida) images. Multiple edges cues, including chrominance and multiple-scale, are combined by using their joint distributions. Hence this cue combination is optimal in the statistical sense. We evaluate the effectiveness of different visual cues using the Chernoff information and Receiver Operator Characteristic (ROC) curves. This shows that our approach gives quantitatively better results than the Canny edge detector when the image background contains

Traditionally, the indoor scene classification problem has been approached from a 2D image recognition point of view. In most visual scene classification systems, a descriptor for the input image is generated to obtain a suitable representation that includes features related to color, shape or spatial information. Techniques based on the use of a spatial pyramid have proven to be adequate to perform this step. In the past years, on the other hand, 3D sensors have become widely available, which allows to include new information sources to the framework previously described. In this work we rely on RGB-D data to extend the spatial pyramid approach, aimed at building descriptors that can lead to a more robust representation against changing lighting conditions. The proposed descriptors are evaluated on the RobotVision@ImageCLEF-2013 benchmark dataset, remarkably outperforming state-of-the-art 3D local and global descriptors.

This is the first in a pair of papers in which we present image-processing-based procedures for the measurement of fluvial gravels. The spatial and temporal resolution of surface grain-size characterization is constrained by the time-consuming and costly nature of traditional measurement techniques. Several groups have developed image-processing-based procedures, but none have demonstrated the transferability of these techniques between sites with different lithological, clast form and textural characteristics. Here we focus on image-processing procedures for identifying and measuring image objects (i.e. grains); the second paper examines the application of such procedures to the measurement of fluvially deposited gravels. Four image-segmentation procedures are developed, each having several internal parameters, giving a total of 416 permutations. These are executed on 39 images from three field sites at which the clasts have contrasting physical properties. The performance of each procedure is evaluated against a sample of manually digitized grains in the same images, by comparing three derived statistics. The results demonstrate that it is relatively straightforward to develop procedures that satisfactorily identify objects in any single image or a set of images with similar sedimentary characteristics. However, the optimal procedure is that which gives consistently good results across sites with dissimilar sedimentary characteristics. We show that neighborhood-based operations are the most powerful, and a morphological bottom-hat transform with a double threshold is optimal. It is demonstrated that its performance approaches that of the procedures giving the best results for individual sites. Overall, it out-performs previously published, or improvements to previously published, methods.

Precise object segmentation in image data is a fundamental problem with various applications, including 3D object reconstruction. We present an efficient algorithm to automatically segment a static foreground object from highly cluttered background in light fields. A key insight and contribution of our article is that a significant increase of the available input data can enable the design of novel, highly efficient approaches. In particular, the central idea of our method is to exploit high spatio-angular sampling on the order of thousands of input frames, for example, captured as a hand-held video, such that new structures are revealed due to the increased coherence in the data. We first show how purely local gradient information contained in slices of such a dense light field can be combined with information about the camera trajectory to make efficient estimates of the foreground and background. These estimates are then propagated to textureless regions using edge-aware filtering in the epipolar volume. Finally, we enforce global consistency in a gathering step to derive a precise object segmentation in both 2D and 3D space, which captures fine geometric details even in very cluttered scenes. The design of each of these steps is motivated by efficiency and scalability, allowing us to handle large, real-world video datasets on a standard desktop computer. We demonstrate how the results of our method can be used for considerably improving the speed and quality of image-based 3D reconstruction algorithms, and we compare our results to state-of-the-art segmentation and multiview stereo methods.

Knowing where people live is a fundamental component of many decision making processes such as urban development, infectious disease containment, evacuation planning, risk management, conservation planning, and more. While bottom-up, survey driven censuses can provide a comprehensive view into the population landscape of a country, they are expensive to realize, are infrequently performed, and only provide population counts over broad areas. Population disaggregation techniques and population projection methods individually address these shortcomings, but also have shortcomings of their own. To jointly answer the questions of "where do people live" and "how many people live there," we propose a deep learning model for creating high-resolution population estimations from satellite imagery. Specifically, we train convolutional neural networks to predict population in the USA at a 0.01°x0.01° resolution grid from 1-year composite Landsat imagery. We validate these models in two ways: quantitatively, by comparing our model's grid cell estimates aggregated at a county-level to several US Census county-level population projections, and qualitatively, by directly interpreting the model's predictions in terms of the satellite image inputs. We find that aggregating our model's estimates gives comparable results to the Census county-level population projections and that the predictions made by our model can be directly interpreted, which give it advantages over traditional population disaggregation methods. In general, our model is an example of how machine learning techniques can be an effective tool for extracting information from inherently unstructured, remotely sensed data to provide effective solutions to social problems.

This paper proposes a deep learning-based efficient and compact solution for road scene segmentation problem, named deep residual coalesced convolutional network (RCC-Net). Initially, the RCC-Net performs dimensionality reduction to compress and extract relevant features, from which it is subsequently delivered to the encoder. The encoder adopts the residual network style for efficient model size. In the core of each residual network, three different convolutional layers are simultaneously coalesced for obtaining broader information. The decoder is then altered to upsample the encoder for pixel-wise mapping from the input images to the segmented output. Experimental results reveal the efficacy of the proposed network over the state-of-the-art methods and its capability to be deployed in an average system.

In this paper a novel reconstruction method is presented that uses the topological relationship of detected image features to create a highly abstract but semantically rich 3D model of the reconstructed scenes. In the first step, a combined image-based reconstruction of points and lines is performed based on the current state of art structure from motion methods. Subsequently, connected planar threedimensional structures are reconstructed by a novel method that uses the topological relationships between the detected image features. The reconstructed 3D models enable a simple extraction of geometric shapes, such as rectangles, in the scene.

Satellite image classification process involves grouping the image pixel values into meaningful categories. Several satellite image classification methods and techniques are available. Satellite image classification methods can be broadly classified into three categories 1) automatic 2) manual and 3) hybrid. All three methods have their own advantages and disadvantages. Majority of the satellite image classification methods fall under first category. Satellite image classification needs selection of appropriate classification method based on the requirements. The current research work is a study on satellite image classification methods and techniques. The research work also compares various researcher's comparative results on satellite image classification methods.

Sky detection plays an essential role in various computer vision applications. Most existing sky detection approaches, being trained on ideal dataset, may lose efficacy when facing unfavorable conditions like the effects of weather and lighting conditions. In this paper, a novel algorithm for sky detection in hazy images is proposed from the perspective of probing the density of haze. We address the problem by an image segmentation and a region-level classification. To characterize the sky of hazy scenes, we unprecedentedly introduce several haze-relevant features that reflect the perceptual hazy density and the scene depth. Based on these features, the sky is separated by two imbalance SVM classifiers and a similarity measurement. Moreover, a sky dataset (named HazySky) with 500 annotated hazy images is built for model training and performance evaluation. To evaluate the performance of our method, we conducted extensive experiments both on our HazySky dataset and the SkyFinder dataset. The results demonstrate that our method performs better on the detection accuracy than previous methods, not only under hazy scenes, but also under other weather conditions.


Question: Rooftop segmentation using shadow-based techniques in visible band imagery.

This paper presents an innovative method for automatically identifying buildings with irregular shapes in complex landscapes from single very high resolution (VHR) optical satellite images. The initial step involves leveraging shadow evidence to pinpoint potential building areas. To this end, a novel fuzzy landscape generation technique is introduced to model the spatial relationship between buildings and their shadows in various directions. After collecting all landscapes, a refinement process is implemented to filter out landscapes that might result from non-building elements. The final identification of building regions is achieved through the GrabCut segmentation technique. In this work, the input parameters required by the GrabCut algorithm are automatically derived from the previously identified shadow and landscape regions, thereby enabling a fully automated and efficient detection of buildings. Comprehensive testing on 20 diverse sites extracted from QuickBird and Geoeye-1 VHR images demonstrates the effectiveness of the proposed method in accurately detecting buildings of various shapes and sizes in intricate environments. The results also indicate that, even under challenging environmental and lighting conditions, the proposed approach can still achieve satisfactory building detection outcomes.

In this paper, we propose an active contour algorithm for object detection in vectorvalued images (such as RGB or multispectral). The model is an extension of the scalar Chan–Vese algorithm to the vector-valued case [1]. The model minimizes a Mumford–Shah functional over the length of the contour, plus the sum of the fitting error over each component of the vector-valued image. Like the Chan–Vese model, our vector-valued model can detect edges both with or without gradient. We show examples where our model detects vector-valued objects which are undetectable in any scalar representation. For instance, objects with different missing parts in different channels are completely detected (such as occlusion). Also, in color images, objects which are invisible in each channel or in intensity can be detected by our algorithm. Finally, the model is robust with respect to noise, requiring no a priori denoising step. C © 2000 Academic Press

We present a new approach for tracking roads from satellite images, and thereby illustrate a general computational strategy (\active testing") for tracking 1D structures and other recognition tasks in computer vision. Our approach is related to recent work in active vision on \where to look next" and motivated by the \divide-and-conquer" strategy of parlor games such as \Twenty Questions." We choose \tests" (matched lters for short road segments) one at a time in order to remove as much uncertainty as possible about the \true hypothesis" (road position) given the results of the previous tests. The tests are chosen on-line based on a statistical model for the joint distribution of tests and hypotheses. The problem of minimizing uncertainty (measured by entropy) is formulated in simple and explicit analytical terms. To execute this entropy testing rule we then alternate between data collection and optimization: at each iteration new image data are examined and a new entropy minimization problem is solved (exactly), resulting in a new image location to inspect, and so forth. We report experiments using panchromatic SPOT satellite imagery with a ground resolution of ten meters: given a starting point and starting direction, we are able to rapidly track highways in southern France over distances on the order of one hundred kilometers without manual intervention. Une m ethode active pour le suivi de routes dans des images satellitaires R esum e : Nous pr esentons une approche originale pour suivre automatiquement une route dans une image de la Terre acquise par satellite. Ce faisant, nous illustrons une m ethodologie plus g en erale pour le probl eme du suivi de structures lin eiques et autres dans des images. Notre approche est li ee a des travaux r ecents en vision active concernant les strat egies de perception ainsi qu'' a la m ethode algorithmique "diviser pour r egner" dans le jeu des vingt questions. Nous choisissons des questions (d etecteur local pour la pr esence d'une route) s equentiel-lement, de mani ere a r eduire le plus possible l'incertitude sur l'inconnue (la localisation de la route dans l'image). Ce choix est bas e sur un mod ele statistique pour la distribution jointe de l'inconnue et des r eponses aux questions. Le probl eme de minimisation de l'incer-titude (mesur ee par l'entropie de C. Shannon) est formul e de mani ere analytique. Ex ecuter cette strat egie consiste donc a alterner entre …

This paper outlines how light Unmanned Aerial Vehicles (UAV) can be used in remote sensing for precision farming. It focuses on the combination of simple digital photographic cameras with spectral filters, designed to provide multispectral images in the visible and near-infrared domains. In 2005, these instruments were fitted to powered glider and parachute, and flown at six dates staggered over the crop season. We monitored ten varieties of wheat, grown in trial micro-plots in the South-West of France. For each date, we acquired multiple views in four spectral bands corresponding to blue, green, red, and near-infrared. We then performed accurate corrections of image vignetting, geometric distortions, and radiometric bidirectional effects. Afterwards, we derived for each experimental micro-plot several vegetation indexes relevant for vegetation analyses. Finally, we sought relationships between these indexes and field-measured biophysical parameters, both generic and date-specific. Therefore, we established a robust and stable generic relationship between, in one hand, leaf area index and NDVI and, in the other hand, nitrogen uptake and GNDVI. Due to a high amount of noise in the data, it was not possible to obtain a more accurate model for each date independently. A validation protocol showed that we could expect a precision level of 15% in the biophysical parameters estimation while using these relationships.

This paper proposes an automated system for recognizing plant species based on leaf images. Plant leaf images corresponding to three plant types, are analyzed using three different shape modelling techniques, the first two based on the Moments-Invariant (M-I) model and the Centroid-Radii (C-R) model and the third based on a proposed technique of Binary-Superposition (B-S). For the M-I model the first four central normalized moments have been considered. For the C-R model an edge detector has been used to identify the boundary of the leaf shape and 36 radii at 10 degree angular separation have been used to build the shape vector. The proposed approach consists of comparing binary versions of the leaf images through superposition and using the sum of non-zero pixel values of the resultant as the feature vector. The data set for experimentations consists of 180 images divided into training and testing sets and comparison between them is done using Manhattan, Euclidean and intersection norms. Accuracies obtained using the proposed technique is seen to be an improvement over the M-I and C-R based techniques, and comparable to the best figures reported in extant literature.

In this paper, we propose a novel algorithm that infers the 3D layout of building facades from a single 2D image of an urban scene. Different from existing methods that only yield coarse orientation labels or qualitative block approximations, our algorithm quantitatively reconstructs building facades in 3D space using a set of planes mutually related by 3D geometric constraints. Each plane is characterized by a continuous orientation vector and a depth distribution. An optimal solution is reached through inter-planar interactions. Due to the quantitative and plane-based nature of our geometric reasoning, our model is more expressive and informative than existing approaches. Experiments show that our method compares competitively with the state of the art on both 2D and 3D measures, while yielding a richer interpretation of the 3D scene behind the image.

Object detection in very high resolution (VHR) optical remote sensing images is one of the most fundamental but challenging problems in the field of remote sensing image analysis. As object detection is usually carried out in feature space, effective feature representation is very important to construct a high-performance object detection system. During the last decades, a great deal of effort has been made to develop various feature representations for the detection of different types of objects. Among various features developed for visual object detection, the histogram of oriented gradients (HOG) feature is maybe one of the most popular features that has been successfully applied to computer vision community. However, although the HOG feature has achieved great success in nature scene images, it is problematic to directly use it for object detection in optical remote sensing images because it is difficult to effectively handle the problem of object rotation variations. To explore a possible solution to the problem, this paper proposes a novel method to learn rotation-invariant HOG (RIHOG) features for object detection in optical remote sensing images. This is achieved by learning a rotation-invariant transformation model via optimizing a new objective function, which constrains the training samples before and after rotation to share the similar features to achieve rotation-invariance. In the experiments, we evaluate the proposed method on a publicly available 10-class VHR geospatial object detection dataset and comprehensive comparisons with state-of-the-arts demonstrate the effectiveness the proposed method.

This paper presents a set of segmentation methods for various types of 3D point clouds. Segmentation of dense 3D data (e.g. Riegl scans) is optimised via a simple yet efficient voxelisation of the space. Prior ground extraction is empirically shown to significantly improve segmentation performance. Segmentation of sparse 3D data (e.g. Velodyne scans) is addressed using ground models of non-constant resolution either providing a continuous probabilistic surface or a terrain mesh built from the structure of a range image, both representations providing close to real-time performance. All the algorithms are tested on several hand labeled data sets using two novel metrics for segmentation evaluation.

A vision-based vehicle guidance system for road vehicles can have three main roles: 1) road detection; 2) obstacle detection; and 3) sign recognition. The first two have been studied for many years and with many good results, but traffi c sign recognition is a less-studied field. Traffi c signs provide drivers with very valuable informatio n about the road, in order to make drivin g safer and easier. We think that traffi c signs must play the same role for autonomous vehicles. They are designed to be easily recognized by human driver s mainly because their color and shapes are very different from natural environments. The algorithm described in this paper takes advantage of these features. I t has two main parts. The first one, for the detection, uses color thresholding to segment the image and shape analysis to detect the signs. The second one, for the classification, uses a neural network. Some results from natural scenes are shown. On the other hand, the algorithm is valid to detect other kinds of marks that would tell the mobile robot to perform some task at that place.

Recent advances in spectral-spatial classification of hyperspectral images are presented in this paper. Several techniques are investigated for combining both spatial and spectral information. Spatial information is extracted at the object (set of pixels) level rather than at the conventional pixel level. Mathematical morphology is first used to derive the morphological profile of the image, which includes characteristics about the size, orientation, and contrast of the spatial structures present in the image. Then, the morphological neighborhood is defined and used to derive additional features for classification. Classification is performed with support vector machines (SVMs) using the available spectral information and the extracted spatial information. Spatial postprocessing is next investigated to build more homogeneous and spatially consistent thematic maps. To that end, three presegmentation techniques are applied to define regions that are used to regularize the preliminary pixel-wise thematic map. Finally, a multiple-classifier (MC) system is defined to produce relevant markers that are exploited to segment the hyperspectral image with the minimum spanning forest algorithm. Experimental results conducted on three real hyperspectral images with different spatial and spectral resolutions and corresponding to various contexts are presented. They highlight the importance of spectral-spatial strategies for the accurate classification of hyperspectral images and validate the proposed methods.

To increase autonomous ground vehicle (AGV) safety and efficiency on outdoor terrains the vehicle's control system should have settings for individual terrain surfaces. A first step in such a terrain-dependent control system is classification of the surface upon which the AGV is traversing. This paper considers vision-based terrain surface classification for the path directly in front of the vehicle (≪ 1 m). Most visionbased terrain classification has focused on terrain traversability and not on terrain surface classification. The few approaches to classifying traversable terrain surfaces, with the exception of the use of infrared cameras to classify mud, have relied on stand-alone cameras that are designed for daytime use and are not expected to perform well in the dark. In contrast, this research uses a laser stripe-based structured light sensor, which uses a laser in conjunction with a camera, and hence can work at night. Also, unlike most previous results, the classification here does not rely on color since color changes with illumination and weather, and certain terrains have multiple colors (e.g., sand may be red or white). Instead, it relies only on spatial relationships, specifically spatial frequency response and texture, which captures spatial relationships between different gray levels. Terrain surface classification using each of these features separately is conducted by using a probabilistic neural network. Experimental results based on classifying four outdoor terrains demonstrate the effectiveness of the proposed methods.


Question: Rooftop segmentation using shadow-based techniques in visible band imagery.

This research introduces an automated procedure for constructing 3D building models from airborne LiDAR (Light Detection and Ranging) data acquired over urban environments. The proposed methodology builds upon the conventional framework for building reconstruction but introduces several innovations to improve both the efficiency and the reliability of the process. Key advancements include: 1) the development of a new vegetation detection technique that leverages differential geometry attributes and unbalanced Support Vector Machines (SVM); 2) following the segmentation of roof patches, a rapid boundary extraction method is implemented to generate topologically accurate, watertight boundaries; 3) rather than imposing assumptions on the angles between roof boundary lines, a data-driven approach is employed to learn the primary orientations of these boundaries, which are then utilized in the creation of building footprints. Additionally, the flexibility of our algorithm is showcased through its capability to handle non-flat object patterns with minimal user input. The efficacy and precision of our method are substantiated through experimental outcomes on a variety of urban datasets. However, in the context of rooftop segmentation in visible band images, this paper does not directly address the specific challenges associated with shadow-based rooftop segmentation, which is the focus of the query. Instead, it highlights advancements in building modeling from airborne LiDAR data, which may indirectly contribute to the broader field of urban scene understanding and analysis.

Background subtraction is a method typically used to segment moving regions in image sequences taken from a static camera by comparing each new frame to a model of the scene background. We present a novel non-parametric background model and a background subtraction approach. The model can handle situations where the background of the scene is cluttered and not completely static but contains small motions such as tree branches and bushes. The model estimates the probability of observing pixel intensity values based on a sample of intensity values for each pixel. The model adapts quickly to changes in the scene which enables very sensitive detection of moving targets. We also show how the model can use color information to suppress detection of shadows. The implementation of the model runs in real-time for both gray level and color imagery. Evaluation shows that this approach achieves very sensitive detection with very low false alarm rates.

Automated visual assessment of plant condition, specifically foliage wilting, reflectance and growth parameters, using machine vision has potential use as input for real-time variable-rate irrigation and fertigation systems in precision agriculture. This paper reviews the research literature for both outdoor and indoor applications of machine vision of plants, which reveals that different environments necessitate varying levels of complexity in both apparatus and nature of plant measurement which can be achieved. Deployment of systems to the field environment in precision agriculture applications presents the challenge of overcoming image variation caused by the diurnal and seasonal variation of sunlight. From the literature reviewed, it is argued that augmenting a monocular RGB vision system with additional sensing techniques potentially reduces image analysis complexity while enhancing system robustness to environmental variables. Therefore, machine vision systems with a foundation in optical and lighting design may potentially expedite the transition from laboratory and research prototype to robust field tool.

Texture is one of the important characteristics used in identifying objects or regions of interest in an image, whether the image be a photomicrograph, an aerial photograph, or a satellite image. This paper describes some easily computable textural features based on graytone spatial dependancies, and illustrates their application in categoryidentification tasks of three different kinds of image data: photomicrographs of five kinds of sandstones, 1:20 000 panchromatic aerial photographs of eight land-use categories, and Earth Resources Technology Satellite (ERTS) multispecial imagery containing seven land-use categories. We use two kinds of decision rules: one for which the decision regions are convex polyhedra (a piecewise linear decision rule), and one for which the decision regions are rectangular parallelpipeds (a min-max decision rule). In each experiment the data set was divided into two parts, a training set and a test set. Test set identification accuracy is 89 percent for the photomicrographs, 82 percent for the aerial photographic imagery, and 83 percent for the satellite imagery. These results indicate that the easily computable textural features probably have a general applicability for a wide variety of image-classification applications.

Semantic segmentation has emerged as a mainstream method in very-high-resolution remote sensing land-use/land-cover applications. In this paper, we first review the state-of-the-art semantic segmentation models in both computer vision and remote sensing fields. Subsequently, we introduce two semantic segmentation frameworks: SNFCN and SDFCN, both of which contain deep fully convolutional networks with shortcut blocks. We adopt an overlay strategy as the postprocessing method. Based on our frameworks, we conducted experiments on two online ISPRS datasets: Vaihingen and Potsdam. The results indicate that our frameworks achieve higher overall accuracy than the classic FCN-8s and SegNet models. In addition, our postprocessing method can increase the overall accuracy by about 1%–2% and help to eliminate “salt and pepper” phenomena and block effects.

Despite of the success of convolutional neural networks for semantic image segmentation, CNNs cannot be used for many applications due to limited computational resources. Even efficient approaches based on random forests are not efficient enough for real-time performance in some cases. In this work, we propose an approach based on superpixels and label propagation that reduces the runtime of a random forest approach by factor 192 while increasing the segmentation accuracy.

In this paper we present a novel method for shadow removal in single images. For each shadow region we use a trained classifier to identify a neighboring lit region of the same material. Given a pair of lit-shadow regions we perform a region relighting transformation based on histogram matching of luminance values between the shadow region and the lit region. Then, we adjust the CIELAB a and b channels of the shadow region by adding constant offsets based on the difference of the median shadow and lit pixel values. We demonstrate that our approach produces results that outperform the state of the art by evaluating our method using a publicly available benchmark dataset.

Detection of traversable road regions, positive and negative obstacles, and water hazards is a fundamental task for autonomous driving vehicles, especially in off-road environment. This paper proposes an efficient method, called Lidar-histogram. It can be used to integrate the detection of traversable road regions, obstacles and water hazards into one single framework. The weak assumption of the Lidar-histogram is that a decent-sized area in front of the vehicle is flat. The Lidar-histogram is derived from an efficient organized map of Lidar point cloud, called Lidar-imagery, to index, describe and store Lidar data. The organized point-cloud map can be easily obtained by indexing the original unordered 3D point cloud to a Lidar-specific 2D coordinate system. In the Lidar-histogram representation, the 3D traversable road plane in front of vehicle can be projected as a straight line segment, and the positive and negative obstacles are projected above and below the line segment, respectively. In this way, the problem of detecting traversable road and obstacles is converted into a simple linear classification task in 2D space. Experiments have been conducted in different kinds of off-road and urban scenes, and we have obtained very promising results.

Threshold based segmentation methods can effectively separate objects from the background. The conventional Otsu's method can achieve good result only when the histogram of an image has two distinct peaks. In this paper, an iterative thresholding based on 2D improved Otsu method using a novel threshold value recognition function is proposed which is used to find the optimum threshold value in different types of histograms and separate into two classes. For the two classes separated by the above threshold value, mean values are computed. Based on the threshold value and the two mean values, the histogram of an image is divided iteratively into three regions, namely, the Foreground (FG) region, Background (BG) region and To-Be-Processed (TBP) region. The pixels greater than the larger mean belongs to FG region, the pixel which are lesser than the smaller mean belongs to BG region. The pixel that lies between the two mean values belongs to the TBP region and only this region is processed at next iteration, the FG and BG region are not processed further. Again for that TBP region, the novel threshold value recognition function is used to find a new threshold value and then the two mean values are computed to separate into three classes, namely, the FG, BG and TBP region. Then the new TBP region is processed in a similar manner iteratively until the two mean values computed at successive iterations are not changed. Finally all the previously determined foreground regions are combined separately and the background regions are combined separately to give the segmented image. The experimental results show that the proposed method performs well in detecting the objects and has better anti-noise performance compared with the existing methods.

In this paper, we address the problem of detecting pedestrians in crowded real-world scenes with severe overlaps. Our basic premise is that this problem is too difficult for any type of model or feature alone. Instead, we present an algorithm that integrates evidence in multiple iterations and from different sources. The core part of our method is the combination of local and global cues via probabilistic top-down segmentation. Altogether, this approach allows examining and comparing object hypotheses with high precision down to the pixel level. Qualitative and quantitative results on a large data set confirm that our method is able to reliably detect pedestrians in crowded scenes, even when they overlap and partially occlude each other. In addition, the flexible nature of our approach allows it to operate on very small training sets.

Based on a multi-step process, an automatic hair segmentation method is created and tested on a database of 115 manually segmented hair images. By extracting various information components from an image, including background color, face position, hair color, skin color, and skin mask, a heuristic-based method is created for the detection and segmentation of hair that can detect hair with an accuracy of approximately 75% and with a false hair overestimation error of 34%. Furthermore, it is shown that downsampling the image down to a face width of 25px results in a 73% reduction in computation time with insignificant change in detection accuracy.


Question: A visible band image processing technique for shadow-based rooftop segmentation.

This paper introduces an innovative method for identifying small objects in high-resolution panchromatic satellite images obtained from the Modular Optoelectronic Multispectral Scanner (MOMS-02). The proposed technique integrates supervised shape classification with unsupervised image segmentation within an iterative framework, enabling a targeted search for distinct object shapes. Unlike conventional approaches, this method optimizes the detection process by focusing on specific object characteristics, thereby enhancing the accuracy and efficiency of small object identification in satellite imagery.

A novel approach is proposed in this paper for automatic forest fire detection from video. Based on 3D point cloud of the collected sample fire pixels, Gaussian mixture model is built and helps segment some possible flame regions in single image. Then the new specific flame pattern is defined for forest, and three types of fire colors are labeled accordingly. With 11 static features including color distributions, texture parameters and shape roundness, the static SVM classifier is trained and filters the segmented results. Using defined overlapping degree and varying degree, the remained candidate regions are matched among consecutive frames. Subsequently the variations of color, texture, roundness, area, contour are computed, then the average and the mean square deviation of them are obtained. Together with the flickering frequency from temporal wavelet based Fourier descriptors analysis of flame contour, 27 dynamic features are used to train the dynamic SVM classifier, which is applied for final decision. Our approach has been tested with dozens of video clips, and it can detect forest fire while recognize the fire like objects, such as red house, bright light and flying flag. Except for the acceptable accuracy, our detection algorithm performs in real time, which proves its value for computer vision based forest fire surveillance.

This paper describes an algorithm to detect the road lane boundary for a driver who changes lane as ‘access road’, ‘passing lane’, ‘approach ramp’, etc. The proposed algorithm starts with binarization and labeling for the image. The binary image is generated by adaptive threshold. The binary image is labeled recursively to blobs. From each blob, the algorithm analyses principal axes calculated by Eigen value and vector. Those are based on the covariance matrix of pixel distribution for the blobs. Then it shows intersection points by the combination of each long principal axis. Among intersection points, the highest density cluster is selected to a dominant vanishing point as the parallel road lines. In experiments of road line detection from the urban road image, the proposed method shows the effectiveness of lane boundary detection in the rate of more than 94.3%.

An effective lane-detection algorithm is a fundamental component of an advanced driver assistant system, as it provides important information that supports driving safety. The challenges faced by the lane detection and tracking algorithm include the lack of clarity of lane markings, poor visibility due to bad weather, illumination and light reflection, shadows, and dense road-based instructions. In this paper, a robust and real-time vision-based lane detection algorithm with an efficient region of interest is proposed to reduce the high noise level and the calculation time. The proposed algorithm also processes a gradient cue and a color cue together and a line clustering with scan-line tests to verify the characteristics of the lane markings. It removes any false lane markings and tracks the real lane markings using the accumulated statistical data. The experiment results show that the proposed algorithm gives accurate results and fulfills the real-time operation requirement on embedded systems with low computing power.

In this paper, we propose an improved image dehazing algorithm using dark channel prior and Multi-Scale Retinex. Main improvement lies in automatic and fast acquisition of transmission map of the scene. We implement the Multi-scale Retinex algorithm on the luminance component in YCbCr space, obtain a pseudo transmission map whose function is similar to the transmission map in original approach. Combining with the haze image model and the dark channel prior, we can recover a high quality haze-free image. Compared with the original method, our algorithm has two main advantages: (i) no user interaction is needed, and (ii) restoring the image much faster while maintaining comparable dehazing performance.

In this paper we present a perception system for agriculture robotics that enables an unmanned ground vehicle (UGV) equipped with a multi spectral camera to automatically perform the crop/weed detection and classification tasks in real-time. Our approach exploits a pipeline that includes two different convolutional neural networks (CNNs) applied to the input RGB+near infra-red (NIR) images. A lightweight CNN is used to perform a fast and robust, pixelwise, binary image segmentation, in order to extract the pixels that represent projections of 3D points that belong to green vegetation. A deeper CNN is then used to classify the extracted pixels between the crop and weed classes. A further important contribution of this work is a novel unsupervised dataset summarization algorithm that automatically selects from a large dataset the most informative subsets that better describe the original one. This enables to streamline and speed-up the manual dataset labeling process, otherwise extremely time consuming, while preserving good classification performances. Experiments performed on different datasets taken from a real farm robot confirm the effectiveness of our approach.

In image sequence processing kalman filtering is used for an adaptive background estimation, in order to separate the foreground from the background. The presented work is an approach which takes into account that changing illumination should be considered in the background estimation, and should not be detected as foreground. The new approach assumes a stationary CCD cameras with fixed focal length and considers non-rigid objects moving non-continuously like human bodies. Furthermore, statistic based methods are used to overcome the problems caused by shadow borders and the adaptation, when the background is covered by the foreground.

This is the first in a pair of papers in which we present image-processing-based procedures for the measurement of fluvial gravels. The spatial and temporal resolution of surface grain-size characterization is constrained by the time-consuming and costly nature of traditional measurement techniques. Several groups have developed image-processing-based procedures, but none have demonstrated the transferability of these techniques between sites with different lithological, clast form and textural characteristics. Here we focus on image-processing procedures for identifying and measuring image objects (i.e. grains); the second paper examines the application of such procedures to the measurement of fluvially deposited gravels. Four image-segmentation procedures are developed, each having several internal parameters, giving a total of 416 permutations. These are executed on 39 images from three field sites at which the clasts have contrasting physical properties. The performance of each procedure is evaluated against a sample of manually digitized grains in the same images, by comparing three derived statistics. The results demonstrate that it is relatively straightforward to develop procedures that satisfactorily identify objects in any single image or a set of images with similar sedimentary characteristics. However, the optimal procedure is that which gives consistently good results across sites with dissimilar sedimentary characteristics. We show that neighborhood-based operations are the most powerful, and a morphological bottom-hat transform with a double threshold is optimal. It is demonstrated that its performance approaches that of the procedures giving the best results for individual sites. Overall, it out-performs previously published, or improvements to previously published, methods.

Aerial scene classification, which aims to automatically label an aerial image with a specific semantic category, is a fundamental problem for understanding high-resolution remote sensing imagery. In recent years, it has become an active task in the remote sensing area, and numerous algorithms have been proposed for this task, including many machine learning and data-driven approaches. However, the existing data sets for aerial scene classification, such as UC-Merced data set and WHU-RS19, contain relatively small sizes, and the results on them are already saturated. This largely limits the development of scene classification algorithms. This paper describes the Aerial Image data set (AID): a large-scale data set for aerial scene classification. The goal of AID is to advance the state of the arts in scene classification of remote sensing images. For creating AID, we collect and annotate more than 10000 aerial scene images. In addition, a comprehensive review of the existing aerial scene classification techniques as well as recent widely used deep learning methods is given. Finally, we provide a performance analysis of typical aerial scene classification and deep learning approaches on AID, which can be served as the baseline results on this benchmark.

We use the AdaBoost algorithm to classify 3D aerial lidar scattered height data into four categories: road, grass, buildings, and trees. To do so we use five features: height, height variation, normal variation, lidar return intensity, and image intensity. We also use only lidar-derived features to organize the data into three classes (the road and grass classes are merged). We apply and test our results using ten regions taken from lidar data collected over an area of approximately eight square miles, obtaining higher than 92% accuracy. We also apply our classifier to our entire dataset, and present visual classification results both with and without uncertainty. We implement and experiment with several variations within the AdaBoost family of algorithms. We observe that our results are robust and stable over all the various tests and algorithmic variations. We also investigate features and values that are most critical in distinguishing between the classes. This insight is important in extending the results from one geographic region to another.

In this paper, we propose a novel explicit image filter called guided filter. Derived from a local linear model, the guided filter computes the filtering output by considering the content of a guidance image, which can be the input image itself or another different image. The guided filter can be used as an edge-preserving smoothing operator like the popular bilateral filter [1], but it has better behaviors near edges. The guided filter is also a more generic concept beyond smoothing: It can transfer the structures of the guidance image to the filtering output, enabling new filtering applications like dehazing and guided feathering. Moreover, the guided filter naturally has a fast and nonapproximate linear time algorithm, regardless of the kernel size and the intensity range. Currently, it is one of the fastest edge-preserving filters. Experiments show that the guided filter is both effective and efficient in a great variety of computer vision and computer graphics applications, including edge-aware smoothing, detail enhancement, HDR compression, image matting/feathering, dehazing, joint upsampling, etc.


Question: Learning supervised distance metrics by maximizing the Jeffrey divergence.

This paper delves into the case of n unity-variance random variables x1, x2, ..., x_n governed by a joint probability density w(x1, x2, ..., x_n), where the density is contingent upon the (normalized) cross-covariances p_(ij) = E[(x_i - \mu_i)(x_j - \mu_j)]. We establish that a condition (*) holds for an "arbitrary" function f(x1, x2, ..., x_n) of n variables if and only if the underlying density w(x1, x2, ..., x_n) corresponds to the standard n-dimensional Gaussian density for correlated random variables. This finding represents an extended form of Price's theorem, characterized by: 1) the condition (*) encompassing Price's original condition; 2) a proof method that circumvents the reliance on Laplace integral expansions; and 3) the exclusion of conditions involving derivatives with respect to diagonal terms p_(ii), thereby maintaining the unity variance assumption. The manuscript was received on February 10, 1966, and revised on May 2, 1966. The author is affiliated with the Ordnance Research Laboratory at Pennsylvania State University, State College, Pennsylvania. Price's theorem, along with its various extensions ([1]-[4]), has been instrumental in determining output correlations between zero-memory nonlinearities subjected to jointly Gaussian inputs. Initially, the theorem addressed n jointly normal random variables, x1, x2, ..., x_n, with respective means \mu_1, \mu_2, ..., \mu_n, and an nth-order joint probability density P(z_1, z_2, ..., z_n) as given by equation (1). Here, |M| is the determinant of J = [p_(ij)], p_(ij) = E[(x_i - \mu_i)(x_j - \mu_j)] = \rho_(ij) \sigma_i \sigma_j, where \rho_(ij) is the correlation coefficient of x_i and x_j, and |M|_ij is the cofactor of p_(ij) in J. As stated in [1], the theorem asserts: "Let there be n zero-memory nonlinear devices specified by the input-output relationship f_i(x), i = 1, 2, ..., n. Let each x_i be the single input to a corresponding f_i(x). Authorized licensed use limited to: West Virginia University. Downloaded on February 26,

Matching people across nonoverlapping camera views at different locations and different times, known as person reidentification, is both a hard and important problem for associating behavior of people observed in a large distributed space over a prolonged period of time. Person reidentification is fundamentally challenging because of the large visual appearance changes caused by variations in view angle, lighting, background clutter, and occlusion. To address these challenges, most previous approaches aim to model and extract distinctive and reliable visual features. However, seeking an optimal and robust similarity measure that quantifies a wide range of features against realistic viewing conditions from a distance is still an open and unsolved problem for person reidentification. In this paper, we formulate person reidentification as a relative distance comparison (RDC) learning problem in order to learn the optimal similarity measure between a pair of person images. This approach avoids treating all features indiscriminately and does not assume the existence of some universally distinctive and reliable features. To that end, a novel relative distance comparison model is introduced. The model is formulated to maximize the likelihood of a pair of true matches having a relatively smaller distance than that of a wrong match pair in a soft discriminant manner. Moreover, in order to maintain the tractability of the model in large scale learning, we further develop an ensemble RDC model. Extensive experiments on three publicly available benchmarking datasets are carried out to demonstrate the clear superiority of the proposed RDC models over related popular person reidentification techniques. The results also show that the new RDC models are more robust against visual appearance changes and less susceptible to model overfitting compared to other related existing models.

In this paper we propose and evaluate an algorithm that learns a similarity measure for comparing never seen objects. The measure is learned from pairs of training images labeled "same" or "different". This is far less informative than the commonly used individual image labels (e.g., "car model X"), but it is cheaper to obtain. The proposed algorithm learns the characteristic differences between local descriptors sampled from pairs of "same" and "different" images. These differences are vector quantized by an ensemble of extremely randomized binary trees, and the similarity measure is computed from the quantized differences. The extremely randomized trees are fast to learn, robust due to the redundant information they carry and they have been proved to be very good clusterers. Furthermore, the trees efficiently combine different feature types (SIFT and geometry). We evaluate our innovative similarity measure on four very different datasets and consistently outperform the state-of-the-art competitive approaches.

Discovering the structure inherent in a set of patterns is a fundamental aim of statistical inference or learning. One fruitful approach is to build a parameterized stochastic generative model, independent draws from which are likely to produce the patterns. For all but the simplest generative models, each pattern can be generated in exponentially many ways. It is thus intractable to adjust the parameters to maximize the probability of the observed patterns. We describe a way of finessing this combinatorial explosion by maximizing an easily computed lower bound on the probability of the observations. Our method can be viewed as a form of hierarchical self-supervised learning that may relate to the function of bottom-up and top-down cortical processing pathways.

We propose a fast algorithm, EMD-L1, for computing the Earth Mover’s Distance (EMD) between a pair of histograms. Compared to the original formulation, EMD-L1 has a largely simplified structure. The number of unknown variables in EMD-L1 is O(N) that is significantly less than O(N) of the original EMD for a histogram with N bins. In addition, the number of constraints is reduced by half and the objective function is also simplified. We prove that the EMD-L1 is formally equivalent to the original EMD with L1 ground distance without approximation. Exploiting the L1 metric structure, an efficient tree-based algorithm is designed to solve the EMD-L1 computation. An empirical study demonstrates that the new algorithm has the time complexity of O(N), which is much faster than previously reported algorithms with super-cubic complexities. The proposed algorithm thus allows the EMD to be applied for comparing histogram-based features, which is practically impossible with previous algorithms. We conducted experiments for shape recognition and interest point matching. EMD-L1 is applied to compare shape contexts on the widely tested MPEG7 shape dataset and SIFT image descriptors on a set of images with large deformation, illumination change and heavy noise. The results show that our EMD-L1based solutions outperform previously reported state-of-the-art features and distance measures in solving the two tasks.

This paper presents a supervised deep hashing approach that constructs binary hash codes from labeled data for large-scale image search. We assume that semantic labels are governed by a set of latent attributes in which each attribute can be on or off, and classification relies on these attributes. Based on this assumption, our approach, dubbed supervised semanticspreserving deep hashing (SSDH), constructs hash functions as a latent layer in a deep network in which binary codes are learned by the optimization of an objective function defined over classification error and other desirable properties of hash codes. With this design, SSDH has a nice property that classification and retrieval are unified in a single learning model, and the learned binary codes not only preserve the semantic similarity between images but also are efficient for image search. Moreover, SSDH performs joint learning of image representations, hash codes, and classification in a pointwised manner and thus is naturally scalable to large-scale datasets. SSDH is simple and can be easily realized by a slight modification of an existing deep architecture for classification; yet it is effective and outperforms other unsupervised and supervised hashing approaches on several benchmarks and one large dataset comprising more than 1 million images.

The distribution information of data points in two classes as the structural information is inserted into the classifiers to improve their generalization performance. Recently many algorithms such as S-TWSVM has used this information to construct two nonparallel hyperplanes which each one lies as close as possible to one class and being far away from the other. It is well known that different classes have different data distribution in real world problems, thus the covariance matrices of these classes are not the same. In these situations, the Mahalanobis is often more popular than Euclidean as a measure of distance. In this paper, in addition to apply the idea of S-TWSVM, the classical Euclidean distance is replaced by Mahalanobis distance which leads to simultaneously consider the covariance matrices of the two classes. By this modification, the orientation information in two classes can be better exploited than S-TWSVM. The experiments indicate our proposed algorithm is often superior to other learning algorithms in terms of generalization performance.

Transductive Adversarial Networks (TAN) is a novel domain-adaptation machine learning framework that is designed for learning a conditional probability distribution on unlabelled input data in a target domain, while also only having access to: (1) easily obtained labelled data from a related source domain, which may have a different conditional probability distribution than the target domain, and (2) a marginalised prior distribution on the labels for the target domain. TAN leverages a fully adversarial training procedure and a unique generator/encoder architecture which approximates the transductive combination of the available sourceand target-domain data. A benefit of TAN is that it allows the distance between the sourceand target-domain label-vector marginal probability distributions to be greater than 0 (i.e. different tasks across the source and target domains) whereas other domain-adaptation algorithms require this distance to equal 0 (i.e. a single task across the source and target domains). TAN can, however, still handle the latter case and is a more generalised approach to this case. Another benefit of TAN is that due to being a fully adversarial algorithm, it has the potential to accurately approximate highly complex distributions. Theoretical analysis demonstrates the viability of the TAN framework.

Learning can be made more efficient if we can actively select particularly salient data points. Within a Bayesian learning framework, objective functions are discussed that measure the expected informativeness of candidate measurements. Three alternative specifications of what we want to gain information about lead to three different criteria for data selection. All these criteria depend on the assumption that the hypothesis space is correct, which may prove to be their main weakness.

Scientists working with large volumes of high-dimensional data, such as global climate patterns, stellar spectra, or human gene distributions, regularly confront the problem of dimensionality reduction: finding meaningful low-dimensional structures hidden in their high-dimensional observations. The human brain confronts the same problem in everyday perception, extracting from its high-dimensional sensory inputs-30,000 auditory nerve fibers or 10(6) optic nerve fibers-a manageably small number of perceptually relevant features. Here we describe an approach to solving dimensionality reduction problems that uses easily measured local metric information to learn the underlying global geometry of a data set. Unlike classical techniques such as principal component analysis (PCA) and multidimensional scaling (MDS), our approach is capable of discovering the nonlinear degrees of freedom that underlie complex natural observations, such as human handwriting or images of a face under different viewing conditions. In contrast to previous algorithms for nonlinear dimensionality reduction, ours efficiently computes a globally optimal solution, and, for an important class of data manifolds, is guaranteed to converge asymptotically to the true structure.

We consider the semi-supervised learning problem, where a decision rule is to be learned from labeled and unlabeled data. In this framework, we motivate minimum entropy regularization, which enables to incorporate unlabeled data in the standard supervised learning. Our approach includes other approaches to the semi-supervised problem as particular or limiting cases. A series of experiments illustrates that the proposed solutions benefit from unlabeled data. The method challenges mixture models when the data are sampled from the distribution class spanned by the generative model. The performances are definitely in favor of minimum entropy regularization when generative models are misspecified, and the weighting of unlabeled data provides robustness to the violation of the “cluster assumption”. Finally, we also illustrate that the method can also be far superior to manifold learning in high dimension spaces.