Question: Chapter 1: Fundamentals of Synthetic Aperture Radar Imaging.

This paper chronicles the development of synthetic aperture radar (SAR) systems, from their inception to their pivotal role in the Earth Observing System (EOS) SAR. The National Aeronautics and Space Administration's (NASA) spaceborne imaging radar program has undergone significant transformation, primarily through the Seasat and Shuttle Imaging Radar (SIR) projects, culminating in a multifrequency, multipolarization SAR system. This advanced system is capable of monitoring the Earth's surface with remarkable imaging geometries, resolutions, and swaths. A critical factor in this evolution has been the capability to digitally process and calibrate the data, which has enabled the development of algorithms for operational extraction of geophysical and biophysical information about the Earth's surface using SAR technology.

The paper traces the evolution of spaceborne imaging radar from Seasat SAR, through the SIR-A, SIR-B, and SIR-CI X-SAR missions, leading up to the EOS SAR, slated for launch as part of the EOS in the late 1990s. It also provides an overview of planned international missions, which could establish a continuous active microwave capability in space as early as 1991. The paper discusses the importance of airborne systems in algorithm development and long-term calibration of spaceborne data. Additionally, it offers a brief summary of planetary missions that employ SAR, comparing their imaging capabilities with those utilized on Earth. This comprehensive overview highlights the significant advancements in SAR technology and its critical role in Earth observation and planetary exploration.

Ship classification is the key step in maritime surveillance using synthetic aperture radar (SAR) imagery. In this letter, we develop a new ship classification method in TerraSAR-X images based on sparse representation in feature space, in which the sparse representation classification (SRC) method is exploited. In particular, to describe the ship more accurately and to reduce the dimension of the dictionary in SRC, we propose to employ a representative feature vector to construct the dictionary instead of utilizing the image pixels directly. By testing on a ship data set collected from TerraSAR-X images, we show that the proposed method is superior to traditional methods such as the template matching (TM), K-nearest neighbor (K-NN), Bayes and Support Vector Machines (SVM).

In this paper, radar cross section (RCS) measurement for human subjects and vehicles in 77 GHz automotive frequency modulated continuous wave (FMCW) radar system is presented. In this system, it is impossible to utilize conventional RCS definition due to high frequency band and the modulation technique used. Therefore, we introduce a new parameter called pseudo-RCS that can replace the conventional RCS. Then, we conduct actual experiment in the road to measure the RCS of the human subjects and the vehicles. In the measurement, data of four human subjects and four different kinds of vehicles are recorded with the 77 GHz FMCW radar. From the actual measured data, we find RCS distributions of the human subjects and the vehicles. For the human subjects, the RCS values are distributed following the Nakagami distribution. On the other hand, the log-normal distribution is well fit for the RCS values in the case of the vehicle.

It is shown that the cyclostationarity attribute, as it is reflected in the periodicities of (second-order) moments of the signal, can be interpreted in terms of the property that allows generation of spectral lines from the signal by putting it through a (quadratic) nonlinear transformation. The fundamental link between the spectral-line generation property and the statistical property called spectral correlation, which corresponds to the correlation that exists between the random fluctuations of components of the signal residing in distinct spectral bands, is explained. The effects on the spectral-correlation characteristics of some basic signal processing operations, such as filtering, product modulation, and time sampling, are examined. It is shown how to use these results to derive the spectral-correlation characteristics for various types of man-made signals. Some ways of exploiting the inherent spectral redundancy associated with spectral correlation to perform various signal processing tasks involving detection and estimation of highly corrupted man-made signals are described.<<ETX>>

We have developed a method for generating arbitrary radiation patterns from antennas on complex objects. The object is coated with an artificial impedance surface consisting of a lattice of sub-wavelength metal patches on a grounded dielectric substrate. The effective surface impedance depends on the size of the patches, and can be varied as a function of position. Using holography, the surface impedance is designed to generate any desired radiation pattern from currents in the surface. With this technique we can create antennas with novel properties such as radiation toward angles that would otherwise be shadowed

The Multilook Cross Correlation (MLCC) is one of the most reliable algorithms used for Doppler ambiguity number estimation of the Doppler centroid parameter. However, the existing MLCC algorithm is only suitable for low contrast scenes. In high contrast scenes, the estimated result is not reliable, and the error is unacceptable. Besides, the Doppler centroid estimation processing time is long and can only be used in offline processing. In this paper, we introduce a modified MLCC algorithm that has better sensitivity which is suitable not only for low contrast scenes, but also for high contrast scenes. In addition, the modified MLCC algorithm can be implemented on parallel signal processing units for better time efficiency. Experiments with RADARSAT-1 data show that the modified algorithm works well in both high and low contrast scenes.

We consider the problem of automatic parameter selection in regularization-based radar image formation techniques. It has previously been shown that non-quadratic regularization produces feature-enhanced radar images; can yield superresolution; is robust to uncertain or limited data; and can generate enhanced images in non-conventional data collection scenarios such as sparse aperture imaging. However, this regularized imaging framework involves some hyper-parameters, whose choice is crucial because that directly affects the characteristics of the reconstruction. Hence there is interest in developing methods for automatic parameter choice. We investigate Stein’s unbiased risk estimator (SURE) and generalized cross-validation (GCV) for automatic selection of hyper-parameters in regularized radar imaging. We present experimental results based on the Air Force Research Laboratory (AFRL) “Backhoe Data Dome,” to demonstrate and discuss the effectiveness of these methods.

Autonomous driving requires precise detection of the environment and correct classification of all targets. W-band radar technology provides high accuracy object detection in all weather conditions, both day and night. Target classification can be improved by evaluating polarimetric scattering information, where it is in particular promising to work with cicularly polarized waves. Both circular polarizations, left-handed circular (LHC) and right-handed circular (RHC), are necessary with a high cross polarization discrimination (XPD). A particularly high XPD can be achieved by hollow waveguide technology. For that purpose, a radar-frontend with corrugated horn antennas has been designed, manufactured and tested. LHC and RHC data is then entered into an algorithm which takes into account switched transmitter technology, fast-ramp frequency-modulated continuous-wave (FMCW) and digital beamforming. Polarimetric radar images are generated, allowing for discrimination and classification of different targets.

This paper describes a computational approach to edge detection. The success of the approach depends on the definition of a comprehensive set of goals for the computation of edge points. These goals must be precise enough to delimit the desired behavior of the detector while making minimal assumptions about the form of the solution. We define detection and localization criteria for a class of edges, and present mathematical forms for these criteria as functionals on the operator impulse response. A third criterion is then added to ensure that the detector has only one response to a single edge. We use the criteria in numerical optimization to derive detectors for several common image features, including step edges. On specializing the analysis to step edges, we find that there is a natural uncertainty principle between detection and localization performance, which are the two main goals. With this principle we derive a single operator shape which is optimal at any scale. The optimal detector has a simple approximate implementation in which edges are marked at maxima in gradient magnitude of a Gaussian-smoothed image. We extend this simple detector using operators of several widths to cope with different signal-to-noise ratios in the image. We present a general method, called feature synthesis, for the fine-to-coarse integration of information from operators at different scales. Finally we show that step edge detector performance improves considerably as the operator point spread function is extended along the edge.

OF THE DISSERTATION Statistical Nested Sensor Array Signal Processing by Keyong Han Doctor of Philosophy in Electrical Engineering Washington University in St. Louis, 2015 Professor Arye Nehorai, Chair Source number detection and direction-of-arrival (DOA) estimation are two major applications of sensor arrays. Both applications are often confined to the use of uniform linear arrays (ULAs), which is expensive and difficult to yield wide aperture. Besides, a ULA with N scalar sensors can resolve at most N − 1 sources. On the other hand, a systematic approach was recently proposed to achieve O(N) degrees of freedom (DOFs) using O(N) sensors based on a nested array, which is obtained by combining two or more ULAs with successively increased spacing. This dissertation will focus on a fundamental study of statistical signal processing of nested arrays. Five important topics are discussed, extending the existing nested-array strategies to more practical scenarios. Novel signal models and algorithms are proposed. First, based on the linear nested array, we consider the problem for wideband Gaussian sources. To employ the nested array to the wideband case, we propose effective strategies to apply nested-array processing to each frequency component, and combine all the spectral xi information of various frequencies to conduct the detection and estimation. We then consider the practical scenario with distributed sources, which considers the spreading phenomenon of sources. Next, we investigate the self-calibration problem for perturbed nested arrays, for which existing works require certain modeling assumptions, for example, an exactly known array geometry, including the sensor gain and phase. We propose corresponding robust algorithms to estimate both the model errors and the DOAs. The partial Toeplitz structure of the covariance matrix is employed to estimate the gain errors, and the sparse total least squares is used to deal with the phase error issue. We further propose a new class of nested vector-sensor arrays which is capable of significantly increasing the DOFs. This is not a simple extension of the nested scalar-sensor array. Both the signal model and the signal processing strategies are developed in the multidimensional sense. Based on the analytical results, we consider two main applications: electromagnetic (EM) vector sensors and acoustic vector sensors. Last but not least, in order to make full use of the available limited valuable data, we propose a novel strategy, which is inspired by the jackknifing resampling method. Exploiting numerous iterations of subsets of the whole data set, this strategy greatly improves the results of the existing source number detection and DOA estimation methods.

Frequency diverse array (FDA) is an emerging technology, the hybrid of FDA and multiple-input-multiple-output (FDA-MIMO) under monostatic scenarios has received much attention in recent years. However, little work have been done for bistatic FDA-MIMO radar. In this paper, we investigate strategies on estimating direction-of-departure (DOD), direction-of-arrival and range for bistatic FDA-MIMO radar. Our strategies have two aspects. First, nonlinear frequency increments including both subarray and random modes are employed to overcome the problem that the DOD and range parameters of FDA transmitting steering vectors are coupled. Second, in order to reduce the computational complexity associated with the 3-D spectral peak searching algorithms, estimation of signal parameters via rotational invariance technique and parallel factor algorithms with their corresponding phase ambiguity resolving methods, are proposed for subarray and random modes, respectively. Both of the two algorithms perform well while the range parameter of targets satisfy a range constraint criterion. This criterion can also be used for designing frequency increments of bistatic FDA-MIMO radar. Additionally, the Cramér-Rao bound of bistatic FDA-MIMO radar and the algorithm performance analysis consist of identifiability and complexity are derived. All the proposed methods are verified by both theoretical analysis and numerical simulations. And satisfactory results are achieved.


Question: Chapter one: An overview of synthetic aperture radar principles.

This study presents a detailed coseismic displacement field map for the Landers, California earthquake of June 28, 1992, derived from data obtained by an orbiting high-resolution radar system. The results achieved are notably more precise than those from earlier space-based studies and match the accuracy of conventional field survey techniques. Utilizing data from the ERS 1 synthetic aperture radar (SAR) instrument, acquired in April, July, and August 1992, a comprehensive and high-resolution displacement map is generated for a vast area. The SAR data accurately measures the motion in the direction of the radar line of sight, with centimeter-level precision, for each 30-meter resolution cell within an image spanning 113 km by 90 km. The coseismic displacement contour map produced reveals a lobed pattern that aligns with theoretical models of the earthquake's displacement field. Intriguing fine-scale displacement tiling observed in regions several kilometers away from the fault is attributed to local surface fracturing. When compared with Global Positioning System (GPS) and electronic distance measurement (EDM) survey data, a high correlation of 0.96 is achieved, demonstrating the radar measurements' capability to extend point measurements from traditional methods into an area map format. The methodology employed is (1) more automated, (2) more accurate, and (3) better substantiated than prior differential radar interferometry applications. Given that it relies solely on remotely sensed satellite data without the need for additional ancillary information, this technique is ideal for global seismic monitoring and analysis.

**Question**: A Direct Search Method to solve Economic Dispatch Problem with Valve-Point Effect

**Documents**: A hybrid of genetic algorithm and particle swarm optimization for recurrent network design
An evolutionary recurrent network which automates the design of recurrent neural/fuzzy networks using a new evolutionary learning algorithm is proposed in this paper. This new evolutionary learning algorithm is based on a hybrid of genetic algorithm (GA) and particle swarm optimization (PSO), and is thus called HGAPSO. In HGAPSO, individuals in a new generation are created, not only by crossover and mutation operation as in GA, but also by PSO. The concept of elite strategy is adopted in HGAPSO, where the upper-half of the best-performing individuals in a population are regarded as elites. However, instead of being reproduced directly to the next generation, these elites are first enhanced. The group constituted by the elites is regarded as a swarm, and each elite corresponds to a particle

This paper presents an overview of the radar sensor technology used in adaptive cruise control (ACC) applications. The performance tradeoffs of the different modulation schemes and scanning antenna types used by these radar sensors are discussed. The test requirements of these millimeter wave radar sensors during production, installation on a vehicle, and during after-market service are also discussed. Different test methods used for characterizing and aligning these radar sensors are presented.

Light detection and ranging (lidar) is a versatile light-based remote sensing technology that recently has been the subject of great attention. It has shown up in a number of media venues and has even led to public debate about the engineering choices of a well-known electric car company, Tesla Motors [1]. While this article is not going to enter the fray, it will provide some background on lidar and discuss its strong connection to power electronics technologies.

This paper introduces a novel 77 GHz FMCW automotive long range radar (LRR) system concept. High resolution direction of arrival (DOA) estimation is an important requirement for the application in automotive safety systems. The challenges in system design regarding low cost and superresolution signal processing are discussed. Dominant interferences to the MUSIC DOA estimator are amplitude and phase mismatches due to inhomogeneous antenna patterns. System simulation results deliver design guidelines for the required signal-to-noise ratio (SNR) and the antenna design. Road traffic measurements with a demonstrator system show superior DOA resolution and demonstrate the feasibility of the design goals.

Pulse compression radar is used in a great number of applications. Excellent range resolution and high electronic counter-countermeasures performance is achieved by wideband long pulses, which spread out the transmitted energy in frequency and time. By using a random noise waveform, the range ambiguity is suppressed as well. In most applications, the random signal is transmitted directly from a noise-generating microwave source. A sine wave, which is phase or frequency modulated by random noise, is an alternative, and in this paper, the ambiguity function and the statistical characteristics of the correlation output for the latter configuration are further analyzed. Range resolution is then improved because the noise bandwidth of the modulated carrier is wider than that of the modulating signal, and the range sidelobes are also further suppressed. Random biphase modulation gives a 4-dB (/spl pi//sup 2//4) improvement, but much higher sidelobe suppression could be achieved using continuous phase/frequency modulation. Due to the randomness of the waveform, the output correlation integral is accompanied by a noise floor, which limits the possible sidelobe suppression as determined by the time-bandwidth product. In synthetic aperture radar (SAR) applications with distributed targets, this product should be large compared with the number of resolution elements inside the antenna main beam. The advantages of low range sidelobes and enhanced range resolution make frequency/phase-modulated noise radar attractive for many applications, including SAR mapping, surveillance, altimetry, and scatterometry. Computer algorithms for reference signal delay and compression are discussed as replacements for the classical delay line implementation.

For application in a short range ground based surveillance radar a combination between frequency modulated continuous wave (FMCW) transmit signals and a receive antenna array system is considered in this paper. The target echo signal is directly down converted by the instantaneous transmit frequency. The target range R will be estimated based on the measured frequency shift fB between transmit and receive signal. Due to an extremely short chirp duration Tchirp, the target radial velocity υτ has only a very small influence to the measured frequency shift fB. Therefore the radial velocity υτ will not be measured inside a single FMCW chirp but in a sequence of chirp signals and inside each individual range gate. Finally, the target azimuth angle is calculated utilizing the receive antenna array and applying a digital beamforming scheme. Furthermore, in order to unambiguously measure even high radial velocities, a variable chirp duration is proposed on a dwell to dwell basis.

This paper proposed a design scheme of wide band monopulse antenna system for the passive radar seeker (PRS) of anti-radiation missile (ARM). To save the installation space of the PRS, the conical conformal log periodic antennas (LPDA) are employed to form the monopulse antenna system. The specific implementation scheme of PRS based on the conically conformal monopulse antennas was designed and analyzed. A practical monopulse antenna system composed of two log periodic antennas was designed and fabricated over the operating frequency range of 1GHz to 8GHz. A loaded resistor with impedance equal to 50Ω is used to reduce the return loss at low frequency. The experimental results demonstrate that acceptable wide band impedance matching and port isolation for each fabricated antenna was achieved. Furthermore, the wide band sum and difference radiation patterns were formed, which validates the proposed conformal monopulse antenna design scheme for the PRS application.

An approach has been developed that involves the fit of a combination of three simple scattering mechanisms to polarimetric SAR observations. The mechanisms are canopy scatter from a cloud of randomly oriented dipoles, evenor double-bounce scatter from a pair of orthogonal surfaces with different dielectric constants and Bragg scatter from a moderately rough surface. This composite scattering model is used to describe the polarimetric backscatter from naturally occurring scatterers. The model is shown to describe the behavior of polarimetric backscatter from tropical rain forests quite well by applying it to data from NASA/Jet Propulsion Laboratory’s (JPL’s) airborne polarimetric synthetic aperture radar (AIRSAR) system. The model fit allows clear discrimination between flooded and nonflooded forest and between forested and deforested areas, for example. The model is also shown to be usable as a predictive tool to estimate the effects of forest inundation and disturbance on the fully polarimetric radar signature. An advantage of this model fit approach is that the scattering contributions from the three basic scattering mechanisms can be estimated for clusters of pixels in polarimetric SAR images. Furthermore, it is shown that the contributions of the three scattering mechanisms to the HH, HV, and VV backscatter can be calculated from the model fit. Finally, this model fit approach is justified as a simplification of more complicated scattering models, which require many inputs to solve the forward scattering problem.

We present a new spotlight SAR data-focusing algorithm based on a two-step processing strategy that combines the advantages of two commonly adopted processing approaches: the efficiency of SPECAN algorithms and the precision of stripmap focusing techniques. The first step of the proposed algorithm implements a linear and space-invariant azimuth filtering that is carried out via a deramping-based technique representing a simplified version of the SPECAN approach. This operation allows us to perform a bulk azimuth raw data compression and to achieve a pixel spacing smaller than (or equal to) the expected azimuth resolution of the fully focused image. Thus, the azimuth spectral folding phenomenon, typically affecting the spotlight data, is overcome, and the space-variant characteristics of the stripmap system transfer function are preserved. Accordingly, the residual and precise focusing of the SAR data is achieved by applying a conventional stripmap processing procedure requiring a minor modification and implemented in the frequency domain. The extension of the proposed technique to the case of high bandwidth transmitted chirp signals is also discussed. Experiments carried out on real and simulated data confirm the validity of the presented approach, which is mainly focused on spaceborne systems.

It is shown that the cyclostationarity attribute, as it is reflected in the periodicities of (second-order) moments of the signal, can be interpreted in terms of the property that allows generation of spectral lines from the signal by putting it through a (quadratic) nonlinear transformation. The fundamental link between the spectral-line generation property and the statistical property called spectral correlation, which corresponds to the correlation that exists between the random fluctuations of components of the signal residing in distinct spectral bands, is explained. The effects on the spectral-correlation characteristics of some basic signal processing operations, such as filtering, product modulation, and time sampling, are examined. It is shown how to use these results to derive the spectral-correlation characteristics for various types of man-made signals. Some ways of exploiting the inherent spectral redundancy associated with spectral correlation to perform various signal processing tasks involving detection and estimation of highly corrupted man-made signals are described.<<ETX>>

Marc Rioux National Research Council, Canada Institute for Information Technology Visual Information Technology Ottawa, Canada, K1A 0R6 Abstract. We review some usual laser range finding techniques for industrial applications. After outlining the basic principles of triangulation and time of flight [pulsed, phase-shift and frequency modulated continuous wave (FMCW)], we discuss their respective fundamental limitations. Selected examples of traditional and new applications are also briefly presented. © 2001 Society of Photo-Optical Instrumentation Engineers. [DOI: 10.1117/1.1330700]


Question: Chapter one: An overview of synthetic aperture radar principles.

The Precision of Topographic Representations from ERS-1 Interferometric Radar Data

Interferometric radar technology offers a high-resolution method for creating digital elevation models, providing detailed topographic maps of various surfaces. This paper discusses the analysis of data gathered by the synthetic aperture radar (SAR) instrument on the ERS-1 satellite, which orbits the Earth on consecutive paths. Utilizing a single satellite in a repeatable orbit is a cost-effective and streamlined approach, enabling the observation of surface changes through the correlation of radar signals over time.

The SAR data have been processed to generate correlation maps and digital elevation models. These correlation maps reveal a notable decrease in temporal correlation over time, though the rate of decline is not consistently linear, likely influenced by environmental conditions. When high correlation persists between passes, the creation of digital elevation models becomes feasible.

Our examination of expected noise in ERS-1 interferometric data, collected over Alaska and the southwestern United States, suggests that elevation maps with relative errors below 5 meters RMS can be achieved in certain areas. However, uncertainties in satellite orbit necessitate the use of tie points to minimize absolute height errors to a similar level. Our findings indicate that approximately six tie points per 40 x 40 km scene, with 5 m RMS height accuracy, are sufficient to maintain systematic map height errors below 5 m RMS.

The ERS-1 SAR system's performance for topographic applications is valuable for regional and local research, but improvements can be made to address temporal decorrelation errors and enhance absolute height accuracy. Adjusting the orbit repeat period and incorporating advanced orbit determination techniques could refine the system's capabilities, meeting the majority of requirements for a global mapping mission, albeit not all.

RFIDs are emerging as a vital component of the Internet of Things. In 2012, billions of RFIDs have been deployed to locate equipment, track drugs, tag retail goods, etc. Current RFID systems, however, can only identify whether a tagged object is within radio range (which could be up to tens of meters), but cannot pinpoint its exact location. Past proposals for addressing this limitation rely on a line-of-sight model and hence perform poorly when faced with multipath effects or non-line-of-sight, which are typical in real-world deployments. This paper introduces the first fine-grained RFID positioning system that is robust to multipath and non-line-of-sight scenarios. Unlike past work, which considers multipath as detrimental, our design exploits multipath to accurately locate RFIDs. The intuition underlying our design is that nearby RFIDs experience a similar multipath environment (e.g., reflectors in the environment) and thus exhibit similar multipath profiles. We capture and extract these multipath profiles by using a synthetic aperture radar (SAR) created via antenna motion. We then adapt dynamic time warping (DTW) techniques to pinpoint a tag's location. We built a prototype of our design using USRP software radios. Results from a deployment of 200 commercial RFIDs in our university library demonstrate that the new design can locate misplaced books with a median accuracy of 11~cm.

A revolutionary type of imaging device, known as a silicon retina or event-based sensor, has recently been developed and is gaining in popularity in the field of artificial vision systems. These devices are inspired by a biological retina and operate in a significantly different way to traditional CCD-based imaging sensors. While a CCD produces frames of pixel intensities, an event-based sensor produces a continuous stream of events, each of which is generated when a pixel detects a change in log light intensity. These pixels operate asynchronously and independently, producing an event-based output with high temporal resolution. There are also no fixed exposure times, allowing these devices to offer a very high dynamic range independently for each pixel. Additionally, these devices offer high-speed, low power operation and a sparse spatiotemporal output. As a consequence, the data from these sensors must be interpreted in a significantly different way to traditional imaging sensors and this paper explores the advantages this technology provides for space imaging. The applicability and capabilities of event-based sensors for SSA applications are demonstrated through telescope field trials. Trial results have confirmed that the devices are capable of observing resident space objects from LEO through to GEO orbital regimes. Significantly, observations of RSOs were made during both day-time and nighttime (terminator) conditions without modification to the camera or optics. The event based sensor’s ability to image stars and satellites during day-time hours offers a dramatic capability increase for terrestrial optical sensors. This paper shows the field testing and validation of two different architectures of event-based imaging sensors. An eventbased sensor’s asynchronous output has an intrinsically low data-rate. In addition to low-bandwidth communications requirements, the low weight, low-power and high-speed make them ideally suitable to meeting the demanding challenges required by space-based SSA systems. Results from these experiments and the systems developed highlight the applicability of event-based sensors to ground and space-based SSA tasks.

Frequency diverse array (FDA) is an emerging technology, the hybrid of FDA and multiple-input-multiple-output (FDA-MIMO) under monostatic scenarios has received much attention in recent years. However, little work have been done for bistatic FDA-MIMO radar. In this paper, we investigate strategies on estimating direction-of-departure (DOD), direction-of-arrival and range for bistatic FDA-MIMO radar. Our strategies have two aspects. First, nonlinear frequency increments including both subarray and random modes are employed to overcome the problem that the DOD and range parameters of FDA transmitting steering vectors are coupled. Second, in order to reduce the computational complexity associated with the 3-D spectral peak searching algorithms, estimation of signal parameters via rotational invariance technique and parallel factor algorithms with their corresponding phase ambiguity resolving methods, are proposed for subarray and random modes, respectively. Both of the two algorithms perform well while the range parameter of targets satisfy a range constraint criterion. This criterion can also be used for designing frequency increments of bistatic FDA-MIMO radar. Additionally, the Cramér-Rao bound of bistatic FDA-MIMO radar and the algorithm performance analysis consist of identifiability and complexity are derived. All the proposed methods are verified by both theoretical analysis and numerical simulations. And satisfactory results are achieved.

With the award of one of the first multiyear procurements to be authorized by the Congress, the Department of Defense celebrates the tenth anniversary of a major Joint-Service / Agency program that promises to revolutionize the art and science of navigation. This contract, totaling 1.17 billion dollars, will result in delivery by the end of this decade of 28 satellites that are to become the heart of the NAVSTAR Global Positioning System (GPS). This paper traces the history of this program from its inception, identifying and discussing the major technologies that made it all possible. In so doing, the concept of operations of the entire system, including the satellites, their associated ground-based control network, and the family of user equipment being developed and tested to meet the broad scope of Department of Defence positioning and navigation requirements is introduced.

Segmentation of Synthetic Aperture Radar (SAR) images has several uses, but it is a difficult task due to a number of properties related to SAR images. In this article we show how Convolutional Neural Networks (CNNs) can easily be trained for SAR image segmentation with good results. Besides this contribution we also suggest a new way to do pixel wise annotation of SAR images that replaces a human expert manual segmentation process, which is both slow and troublesome. Our method for annotation relies on 3D CAD models of objects and scene, and converts these to labels for all pixels in a SAR image. Our algorithms are evaluated on the Moving and Stationary Target Acquisition and Recognition (MSTAR) dataset which was released by the Defence Advanced Research Projects Agency during the 1990s. The method is not restricted to the type of targets imaged in MSTAR but can easily be extended to any SAR data where prior information about scene geometries can be estimated.

Takeo Kanade, Robert T. Collins, and Alan J. Lipton Carnegie Mellon University, Pittsburgh, PA e-mail: fkanade,rcollins,ajlg@cs.cmu.edu homepage: http://www.cs.cmu.edu/ vsam P. Anandan, Peter Burt, and Lambert Wixson David Sarno Research Center, Princeton, NJ e-mail: fpanandan,pburt,lwixsong@sarno .com Abstract Carnegie Mellon University (CMU) and the David Sarno Research Center (Sarno ) have begun a joint, integrated feasibility demonstration in the area of Video Surveillance and Monitoring (VSAM). The objective is to develop a cooperative, multi-sensor video surveillance system that provides continuous coverage over large battle eld areas. Image Understanding (IU) technologies will be developed to: 1) coordinate multiple sensors to seamlessly track moving targets over an extended area, 2) actively control sensor and platform parameters to track multiple moving targets, 3) integrate multisensor output with collateral data to maintain an evolving, scene-level representation of all targets and platforms, and 4) monitor the scene for unusual \trigger" events and activities. These technologies will be integrated into an experimental testbed to support evaluation, data collection, and demonstration of other VSAM technologies developed within the DARPA IU community.

Automotive radar sensors will be used in many future applications to increase comfort and safety. Adaptive cruise control (ACC), parking aid, collision warning or avoidance and pre crash are some automotive applications, which are based on high performance radar sensors. This paper describes results gained by an European Commission funded research project and a new radar network based on four individual monostatic 77 GHz FMCW radar sensors. Each sensor measures target range and velocity simultaneously and extremely accurate even in multiple target situations with a maximum range of 40 m. The azimuth angle is estimated by multilateration techniques in a radar network architecture. The system design and the signal processing of this network are described in detail in this paper. Due to the high range resolution performance of each radar sensor ordinary targets cannot be considered any longer as point but as extended targets. System capability, performance and reliability can be improved essentially if the single sensor FMCW signal processing, the target position processing in the radar network and the tracking scheme are jointly optimized.

For application in a short range ground based surveillance radar a combination between frequency modulated continuous wave (FMCW) transmit signals and a receive antenna array system is considered in this paper. The target echo signal is directly down converted by the instantaneous transmit frequency. The target range R will be estimated based on the measured frequency shift fB between transmit and receive signal. Due to an extremely short chirp duration Tchirp, the target radial velocity υτ has only a very small influence to the measured frequency shift fB. Therefore the radial velocity υτ will not be measured inside a single FMCW chirp but in a sequence of chirp signals and inside each individual range gate. Finally, the target azimuth angle is calculated utilizing the receive antenna array and applying a digital beamforming scheme. Furthermore, in order to unambiguously measure even high radial velocities, a variable chirp duration is proposed on a dwell to dwell basis.

Human Vision system is the most complex and accurate system. In order to extract better features about Synthetic Aperture Radar (SAR) targets, a SAR automatic target recognition (ATR) algorithm based on human visual cortical system is proposed. This algorithm contains three stages: (1) Image preprocessing (we use a Kuan filter to do the enhancement and an adaptive Intersecting Cortical Model (ICM) to do the segmentation) (2) Feature extraction using a sparse autoencoder. (3) Classification using a softmax regression classifier. Experiment result of MSTAR public data shows a better performance of recognition.

Many types of GNSS antennas have been developed in recent years to make them suitable for different applications. Since their designs and performance requirements vary depending on their application, they have been grouped into six different categories in this book: 1. The design and performance of each type of antenna will be described here and throughout the rest of the book. This chapter discusses the design of the first two varieties: FRPAs and high-gain directional antennas. Figure 2.1 shows a representative sample of these antennas. FRPAs are the most popular and widely used of all GNSS antennas. There are many different types of these antennas; hence they will need two chapters—this chapter and Chapter 3—to fully cover all the important types. This chapter will discuss the following FRPA designs: microstrip patch antennas, which are the most ubiquitous of all GNSS antennas, and quadrifilar helix antennas (QHAs), which are also popular for handheld receivers and crossed " drooping " bow-type dipoles, which provide relatively wider bandwidths and good circular polarization. Spiral antennas such as the Archimedean spiral antenna will be discussed in Chapter 3 under multiband antennas since they are ultrawideband and can cover the entire GNSS band from 1.1 to 1.6 GHz. The conical spiral antenna is also a high-gain di


Question: Principles of synthetic aperture radar, as outlined in Chapter 1.

Satellite Radar Interferometry (SRI) as a Tool for Observing Ice Sheet Dynamics: A Case Study of an Antarctic Ice Stream

Satellite Radar Interferometry (SRI) serves as a highly effective technique for monitoring the velocities of ice streams and pinpointing their grounding-line positions, both of which are crucial indicators of ice sheet reactions to climate change or internal instabilities. This method boasts detection capabilities for vertical and horizontal motions as low as approximately 1.5 millimeters and 4 millimeters, respectively, in the radar beam direction. Notably, SRI enables the mapping of grounding lines, the boundary where ice transitions from grounded to floating, with a resolution of roughly 0.5 kilometers. A case study focusing on the Rutford Ice Stream in Antarctica demonstrates the alignment of SRI-derived velocities and grounding line positions with previous ground-based observations. The synergistic application of SRI in conjunction with other satellite methodologies holds promise for advancing our comprehension of ice stream mechanics and facilitating the prediction of ice sheet dynamics.

This paper addresses the problem of multisensor data fusion for target tracking using a Doppler radar with range rate measurements and an Electro-Optic (EO) camera. We present three fusion architectures, named FA1-FA3, to perform data fusion using the above mentioned sensors. FA1 and FA2 are distributed fusion architectures employing the information matrix fusion method with dynamic feedback. In FA1, radar and camera pseudo measurements are formed that allow us to make use of a linear Kalman Filter (KF) for the radar local filter and an Extended Kalman Filter (EKF) for the EO camera local filter. In FA2, the radar and camera measurements are used directly and therefore the system comprises two EKFs. FA3 is a centralised architecture where the data fusion is performed by way of the measurement fusion method. The final contribution of this paper is a performance comparison of these sensor data fusion techniques when making use of range rate measurements. In order to evaluate the performance of the fusion architectures, Monte Carlo simulations are performed and two filter metrics are presented: an absolute metric - the root mean squared error (RMSE) and a performance metric - the average normalised estimation error squared (ANEES). The results show that the fusion architectures presented are accurate, stable and credible.

We introduce space–time block coding , a new paradigm for communication over Rayleigh fading channels using multiple transmit antennas. Data is encoded using a space–time block code and the encoded data is split inton streams which are simultaneously transmitted usingn transmit antennas. The received signal at each receive antenna is a linear superposition of the n transmitted signals perturbed by noise. Maximumlikelihood decoding is achieved in a simple way through decoupling of the signals transmitted from different antennas rather than joint detection. This uses the orthogonal structure of the space–time block code and gives a maximum-likelihood decoding algorithm which is based only on linear processing at the receiver. Space–time block codes are designed to achieve the maximum diversity order for a given number of transmit and receive antennas subject to the constraint of having a simple decoding algorithm. The classical mathematical framework of orthogonal designs is applied to construct space–time block codes. It is shown that space–time block codes constructed in this way only exist for few sporadic values ofn. Subsequently, a generalization of orthogonal designs is shown to provide space–time block codes for both real and complex constellations for any number of transmit antennas. These codes achieve the maximum possible transmission rate for any number of transmit antennas using any arbitrary real constellation such as PAM. For an arbitrary complex constellation such as PSK and QAM, space–time block codes are designed that achieve1=2 of the maximum possible transmission rate for any number of transmit antennas. For the specific cases of two, three, and four transmit antennas, space–time block codes are designed that achieve, respectively, all, 3=4, and 3=4 of maximum possible transmission rate using arbitrary complex constellations. The best tradeoff between the decoding delay and the number of transmit antennas is also computed and it is shown that many of the codes presented here are optimal in this sense as well.

Next generation RF sensor modules for multifunction active electronically steered antenna (AESA) systems will need a combination of different operating modes, such as radar, electronic warfare (EW) functionalities and communications/datalinks within the same antenna frontend. They typically operate in C-Band, X-Band and Ku-Band and imply a bandwidth requirement of more than 10 GHz. For the realisation of modern active electronically steered antennas, the transmit/receive (T/R) modules have to match strict geometry demands. A major challenge for these future multifunction RF sensor modules is dictated by the half-wavelength antenna grid spacing, that limits the physical channel width to < 12 mm or even less, depending on the highest frequency of operation with accordant beam pointing requirements. A promising solution to overcome these geometry demands is the reduction of the total monolithic microwave integrated circuit (MMIC) chip area, achieved by integrating individual RF functionalities, which are commonly achieved through individual integrated circuits (ICs), into new multifunctional (MFC) MMICs. Various concepts, some of them already implemented, towards next generation RF sensor modules will be discussed and explained in this work.

In this work the realization of a 77-GHz frequency-division multiple-access-based frequency-modulated continuous-wave multiple-input multiple-output radar with four transceiver channels in conjunction with a non-uniform linear antenna array is presented. The radar system, consisting of an RF-frontend utilizes transceiver chips with integrated inphase/quadrature-modulators to generate the frequency shifted transmit signals and a field programmable gate array-based baseband board, is used for test measurements to verify the system performance and to demonstrate the beamforming capability as well as the accuracy of the digital-beamforming method.

A coarray-based aperture synthesis scheme using subarrays and postdata acquisition beamforming is presented for through-the-wall wideband microwave imaging applications. The wall causes wave refraction and a change in the propagation speed, both effects alter the travel time between the transmitter, the target, and the receiver. Coherent combining of the pulse waveforms emitted by the different transmitters and incident at the receivers through reflections from targets and clutter requires incorporation of wall effects into the beamformer design. Simulation results verifying the proposed synthetic aperture technique for a through-the-wall imaging (TWI) system are presented. The impact of the wall ambiguities or incorrect estimates of the wall parameters, such as thickness and dielectric constant, on performance is considered.

The radar capable to detect and perform diagnostics of a state of the human being behind obstacles and in conditions of bad visibility could be used for the manifold applications. The potential consumers of the radar are: rescue services, antiterrorist detachments and law-enforcement bodies. It is possible also to use this device in medicine. In this paper, the experiments with using of continuouswave subsurface radar are described. The recorded oscillograms and their frequency spectrums for heartbeat, respiration and articulation of a man, which is taking place behind an obstacle (wall), are presented.

MIMO (multiple-input multiple-output) radar refers to an architecture that employs multiple, spatially distributed transmitters and receivers. While, in a general sense, MIMO radar can be viewed as a type of multistatic radar, the separate nomenclature suggests unique features that set MIMO radar apart from the multistatic radar literature and that have a close relation to MIMO communications. This article reviews some recent work on MIMO radar with widely separated antennas. Widely separated transmit/receive antennas capture the spatial diversity of the target's radar cross section (RCS). Unique features of MIMO radar are explained and illustrated by examples. It is shown that with noncoherent processing, a target's RCS spatial variations can be exploited to obtain a diversity gain for target detection and for estimation of various parameters, such as angle of arrival and Doppler. For target location, it is shown that coherent processing can provide a resolution far exceeding that supported by the radar's waveform.

In this work a multimode-beamforming 77-GHz frequency-modulated continuous-wave radar system is presented. Four transceiver chips with integrated inphase/quadrature modulators in the transmit path are used in order to simultaneously realize a short-range frequency-division multiple-access (FDMA) multiple-input multiple-output (MIMO) and a long-range transmit phased-array (PA) radar system with the same antennas. It combines the high angular resolution of FDMA MIMO radars and the high-gain and steerable beam of PA transmit antennas. Several measurements were carried out to show the potential benefits of using this concept for a linear antenna array with four antennas and methods of digital beamforming in the receive path.

In this paper we propose a convolutional neural network (CNN), which allows to identify corresponding patches of very high resolution (VHR) optical and SAR imagery of complex urban scenes. Instead of a siamese architecture as conventionally used in CNNs designed for image matching, we resort to a pseudo-siamese configuration with no interconnection between the two streams for SAR and optical imagery. The network is trained with automatically generated training data and does not resort to any hand-crafted features. First evaluations show that the network is able to predict corresponding patches with high accuracy, thus indicating great potential for further development to a generalized multi-sensor matching procedure.


Question: Recognition of events and actions using 3D thermal depth sensing.

In the context of Alvey Project MMI149, our objective is to employ computer vision to decipher the complex 3D environment, where the variety of objects within the visual scenes is too extensive for conventional top-down recognition methods to be effective. Our goal is to achieve a comprehensive understanding of natural landscapes, encompassing elements such as roads, buildings, trees, and shrubbery, as exemplified by the two scenes depicted in Figure 1. To tackle this challenge, we are developing a computer vision system that relies on the analysis of motion in monocular image sequences captured by a mobile camera. By identifying and tracking key image features, we can construct representations that closely mirror their 3D equivalents, facilitating a deeper comprehension of the dynamic scenes. This approach enables us to extract meaningful information from the visual data, even in environments with a high degree of object diversity, by leveraging the temporal consistency of features across multiple frames.

Previous work on muscle activity sensing has leveraged specialized sensors such as electromyography and force sensitive resistors. While these sensors show great potential for detecting finger/hand gestures, they require additional hardware that adds to the cost and user discomfort. Past research has utilized sensors on commercial devices, focusing on recognizing gross hand gestures. In this work we present Serendipity, a new technique for recognizing unremarkable and fine-motor finger gestures using integrated motion sensors (accelerometer and gyroscope) in off-the-shelf smartwatches. Our system demonstrates the potential to distinguish 5 fine-motor gestures like pinching, tapping and rubbing fingers with an average f1-score of 87%. Our work is the first to explore the feasibility of using solely motion sensors on everyday wearable devices to detect fine-grained gestures. This promising technology can be deployed today on current smartwatches and has the potential to be applied to cross-device interactions, or as a tool for research in fields involving finger and hand motion.

With the rising of AR/VR technology and miniaturization of mobile devices, gesture is becoming an increasingly popular means of interacting with smart devices. Some pioneer ultrasound based human gesture recognition systems have been proposed. They mostly rely on low resolution Doppler Effect, and hence focus on whole hand motion and cannot deal with minor finger motions. In this paper, we present UltraGesture, a Channel Impulse Response (CIR) based ultrasonic finger motion perception and recognition system. CIR measurements can provide with 7 mm resolution, rendering it sufficient for minor finger motion recognition. UltraGesture encapsulates CIR measurements into an image, and builds a Convolutional Neural Network model to classify these images into different categories, which corresponding to distinct gestures. Our system runs on commercial speakers and microphones that already exist on most mobile devices without hardware modification. Our results show that UltraGesture achieves an average accuracy of greater than 97% for 12 gestures including finger click and rotation.

In principle, activity recognition can be exploited to great societal benefits, especially in real-life, human centric applications such as elder care and healthcare. This article focused on recognizing simple human activities. Recognizing complex activities remains a challenging and active area of research and the nature of human activities poses different challenges. Human activity understanding encompasses activity recognition and activity pattern discovery. The first focuses on accurate detection of human activities based on a predefined activity model. An activity pattern discovery researcher builds a pervasive system first and then analyzes the sensor data to discover activity patterns.

Detecting multimedia events in web videos is an emerging hot research area in the fields of multimedia and computer vision. In this paper, we introduce the core methods and technologies of the framework we developed recently for our Event Labeling through Analytic Media Processing (E-LAMP) system to deal with different aspects of the overall problem of event detection. More specifically, we have developed efficient methods for feature extraction so that we are able to handle large collections of video data with thousands of hours of videos. Second, we represent the extracted raw features in a spatial bag-of-words model with more effective tilings such that the spatial layout information of different features and different events can be better captured, thus the overall detection performance can be improved. Third, different from widely used early and late fusion schemes, a novel algorithm is developed to learn a more robust and discriminative intermediate feature representation from multiple features so that better event models can be built upon it. Finally, to tackle the additional challenge of event detection with only very few positive exemplars, we have developed a novel algorithm which is able to effectively adapt the knowledge learnt from auxiliary sources to assist the event detection. Both our empirical results and the official evaluation results on TRECVID MED’11 and MED’12 demonstrate the excellent performance of the integration of these ideas.

I ncreasingly powerful integrated circuits are making an entire range of new applications possible. Complementary metal-oxide semiconductor (CMOS) sensors, for example, have made the digital camera a commonplace consumer item. These light-sensitive chips, positioned where film would normally be, capture images as reusable digital files that users can upload to their computer, manipulate with software, and distribute electronically. Recent technological advances are enabling a new generation of smart cameras that represent a quantum leap in sophistication. While today's digital cameras capture images, smart cameras capture high-level descriptions of the scene and analyze what they see. These devices could support a wide variety of applications including human and animal detection, surveillance, motion analysis, and facial identification. Video processing has an insatiable demand for real-time performance. Fortunately, Moore's law provides an increasing pool of available computing power to apply to real-time analysis. Smart cameras leverage very large-scale integration (VLSI) to provide such analysis in a low-cost, low-power system with substantial memory. Moving well beyond pixel processing and compression, these systems run a wide range of algorithms to extract meaning from streaming video. Because they push the design space in so many dimensions, smart cameras are a leading-edge application for embedded system research. has developed a first-generation smart camera system that can detect people and analyze their movement in real time. Although there are many approaches to real-time video analysis, we chose to focus initially on human gesture recognition—identifying whether a subject is walking, standing, waving his arms, and so on. Because much work remains to be done on this problem, we sought to design an embedded system that can incorporate future algorithms as well as use those we created exclusively for this application. As Figure 1 shows, our algorithms use both low-level and high-level processing. The low-level component identifies different body parts and categorizes their movement in simple terms. The high-level component, which is application-dependent, uses this information to recognize each body part's action and the person's overall activity based on scenario parameters. The system captures images from the video input, which can be either uncompressed or compressed (MPEG and motion JPEG), and applies four different algorithms to detect and identify human body parts. Region extraction. The first algorithm transforms the pixels of an image, like that shown in Figure 2a, into an M × N bitmap and eliminates the background. It then detects the body part's skin area using a YUV …

This paper proposes an algorithm for fall detection using a ceiling-mounted 3D depth camera. The lying pose is separated from common daily activities by a k-NN classifier, which was trained on features expressing head-floor distance, person area and shape's major length to width. In order to distinguish between intentional lying postures and accidental falls the algorithm also employs motion between static postures. The experimental validation of the algorithm was conducted on realistic depth image sequences of daily activities and simulated falls. It was evaluated on more than 45000 depth images and gave 0% error. To reduce the processing overload an accelerometer was used to indicate the potential impact of the person and to start an analysis of depth images.

The use of AV features alone is insufficient to induce high-level semantics. This article proposes a framework that utilizes both internal AV features and various types of external information sources for event detection in team sports video. Three schemes are also proposed to tackle the asynchronism between the fusion of AV and external information. The framework is extensible as it can provide increasing functionalities given more detailed external information and domain knowledge. By demonstrating its effectiveness on soccer and American football, we believe that with the availability of appropriate domain knowledge, the framework is applicable to other team sports.

The aim of this paper is to address recognition of natural human actions in diverse and realistic video settings. This challenging but important subject has mostly been ignored in the past due to several problems one of which is the lack of realistic and annotated video datasets. Our first contribution is to address this limitation and to investigate the use of movie scripts for automatic annotation of human actions in videos. We evaluate alternative methods for action retrieval from scripts and show benefits of a text-based classifier. Using the retrieved action samples for visual learning, we next turn to the problem of action classification in video. We present a new method for video classification that builds upon and extends several recent ideas including local space-time features, space-time pyramids and multi-channel non-linear SVMs. The method is shown to improve state-of-the-art results on the standard KTH action dataset by achieving 91.8% accuracy. Given the inherent problem of noisy labels in automatic annotation, we particularly investigate and show high tolerance of our method to annotation errors in the training set. We finally apply the method to learning and classifying challenging action classes in movies and show promising results.

This paper presents an unsupervised deep learning framework that derives spatio-temporal features for human-robot interaction. The respective models extract high-level features from low-level ones through a hierarchical network, viz. the Hierarchical Temporal Memory (HTM), providing at the same time a solution to the curse of dimensionality in shallow techniques. The presented work incorporates the tensor-based framework within the operation of the nodes and, thus, enhances the feature derivation procedure. This is due to the fact that tensors allow the preservation of the initial data format and their respective correlation and, moreover, attain more compact representations. The computational nodes form spatial and temporal groups by exploiting the multilinear algebra, subsequently express the samples according to those groups in terms of proximity. This generic framework may be applied in a diverse of visual data, whilst it has been examined on sequences of color and depth images, exhibiting remarkable performance.

Recent introduction of low-cost 3D sensing and affordable immersive virtual reality have lowered the barriers for creating and maintaining 3D virtual worlds. In this paper, we propose a way to combine these technologies with discrete-event simulation to improve the use of simulation in decision making in manufacturing. This work will describe how feedback is possible from real world systems directly into a simulation model to guide smart behaviors. Technologies included in the research include feedback from RGBD images of shop floor motion and human interaction within full immersive virtual reality that includes the latest headset technologies.


Question: Recognition of events and actions using 3D thermal depth sensing.

Skeletal Quads: A Novel Approach for Human Action Recognition from 3D Depth Data

Advancements in human motion analysis have enabled the extraction of human skeleton structures from single depth images, facilitating a new era of action recognition. In this context, we introduce a unique local skeleton descriptor that captures the relative positions of joint quadruples. This descriptor, coined as the skeletal quad, undergoes a similarity normalization process, resulting in a compact, 6D representation that is invariant to viewpoint changes. To describe the skeletal quads within a (sub)action, we employ a Fisher kernel representation. A Gaussian mixture model is trained on the dataset to encode the generation of any set of quads using its Fisher vector. The action description is enhanced by a multi-level representation of Fisher vectors, which preserves the approximate sequence of sub-actions within each action sequence. This method enables efficient classification using linear support vector machines (SVMs). The efficacy of the proposed action representation is validated on popular datasets, namely MSRAction3D and HDM05. Comparative analysis reveals that our method surpasses current state-of-the-art algorithms that solely rely on joint information, while it performs competitively against methods that integrate additional cues with joints.

We describe a technique that automatically generates plausible depth maps from videos using non-parametric depth sampling. We demonstrate our technique in cases where past methods fail (non-translating cameras and dynamic scenes). Our technique is applicable to single images as well as videos. For videos, we use local motion cues to improve the inferred depth maps, while optical flow is used to ensure temporal depth consistency. For training and evaluation, we use a Kinect-based system to collect a large data set containing stereoscopic videos with known depths. We show that our depth estimation technique outperforms the state-of-the-art on benchmark databases. Our technique can be used to automatically convert a monoscopic video into stereo for 3D visualization, and we demonstrate this through a variety of visually pleasing results for indoor and outdoor scenes, including results from the feature film Charade.

In this work, a system for recognizing activities in the home setting using a set of small and simple state-change sensors is introduced. The sensors are designed to be “tape on and forget” devices that can be quickly and ubiquitously installed in home environments. The proposed sensing system presents an alternative to sensors that are sometimes perceived as invasive, such as cameras and microphones. Unlike prior work, the system has been deployed in multiple residential environments with non-researcher occupants. Preliminary results on a small dataset show that it is possible to recognize activities of interest to medical professionals such as toileting, bathing, and grooming with detection accuracies ranging from 25% to 89% depending on the evaluation criteria used .

Activity recognition fits within the bigger framework of context awareness. In this paper, we report on our efforts to recognize user activity from accelerometer data. Activity recognition is formulated as a classification problem. Performance of base-level classifiers and meta-level classifiers is compared. Plurality Voting is found to perform consistently well across different settings.

Recent approaches in robot perception follow the insight that perception is facilitated by interaction with the environment. These approaches are subsumed under the term Interactive Perception (IP). This view of perception provides the following benefits. First, interaction with the environment creates a rich sensory signal that would otherwise not be present. Second, knowledge of the regularity in the combined space of sensory data and action parameters facilitates the prediction and interpretation of the sensory signal. In this survey, we postulate this as a principle for robot perception and collect evidence in its support by analyzing and categorizing existing work in this area. We also provide an overview of the most important applications of IP. We close this survey by discussing remaining open questions. With this survey, we hope to help define the field of Interactive Perception and to provide a valuable resource for future research.

In last decade lot of efforts had been made by research community to create sign language recognition system which provide a medium of communication for differently-abled people and their machine translations help others having trouble in understanding such sign languages. Computer vision and machine learning can be collectively applied to create such systems. In this paper, we present a sign language recognition system which makes use of depth images that were captured using a Microsoft Kinect® camera. Using computer vision algorithms, we develop a characteristic depth and motion profile for each sign language gesture. The feature matrix thus generated was trained using a multi-class SVM classifier and the final results were compared with existing techniques. The dataset used is of sign language gestures for the digits 0-9.

As robots begin to collaborate with humans in everyday workspaces, they will need to understand the functions of tools and their parts. To cut an apple or hammer a nail, robots need to not just know the tool's name, but they must localize the tool's parts and identify their functions. Intuitively, the geometry of a part is closely related to its possible functions, or its affordances. Therefore, we propose two approaches for learning affordances from local shape and geometry primitives: 1) superpixel based hierarchical matching pursuit (S-HMP); and 2) structured random forests (SRF). Moreover, since a part can be used in many ways, we introduce a large RGB-Depth dataset where tool parts are labeled with multiple affordances and their relative rankings. With ranked affordances, we evaluate the proposed methods on 3 cluttered scenes and over 105 kitchen, workshop and garden tools, using ranked correlation and a weighted F-measure score [26]. Experimental results over sequences containing clutter, occlusions, and viewpoint changes show that the approaches return precise predictions that could be used by a robot. S-HMP achieves high accuracy but at a significant computational cost, while SRF provides slightly less accurate predictions but in real-time. Finally, we validate the effectiveness of our approaches on the Cornell Grasping Dataset [25] for detecting graspable regions, and achieve state-of-the-art performance.

We propose Action-Reaction Learning as an approach for analyzing and synthesizing human behaviour. This paradigm uncovers causal mappings between past and future events or between an action and its reaction by observing time sequences. We apply this method to analyze human interaction and to subsequently synthesize human behaviour. Using a time series of perceptual measurements, a system automatically uncovers a mapping between gestures from one human participant (an action) and a subsequent gesture (a reaction) from another participant. A probabilistic model is trained from data of the human interaction using a novel estimation technique, Conditional Expectation Maximization (CEM). The system drives a graphical interactive character which probabilistically predicts the most likely response to the user's behaviour and performs it interactively. Thus, after analyzing human interaction in a pair of participants, the system is able to replace one of them and interact with a single remaining user.

Activity recognition is an important subject in different areas of research, like e-health and ubiquitous computing. In this paper we give an overview of recent work in the field of activity recognition from both body mounted sensors and ambient sensors. We discuss some of the differences among approaches and present guidelines for choosing a suitable approach under various circumstances.

Recently, Wi-Fi has caught tremendous attention for its ubiquity. Motivated by Wi-Fi’s low cost and privacy preservation, researchers have been putting lots of investigation into its potential on action recognition and even person identification. In this paper, for bringing a new modality for multimedia community, we offer an comprehensive overview on these two topics in Wi-Fi. Also, through looking at these two topics from an unprecedented perspective, we could achieve generality instead of designing specific ad-hoc features for each scenario. Observing the great resemblance of Channel State Information (CSI, a finegrained information captured from the received Wi-Fi signal) to texture, we propose a brand-new framework based on computer vision methods. To minimize the effect of location dependency embedded in CSI, we propose a novel de-noising method based on Singular Value Decomposition (SVD) to eliminate the background energy and effectively extract the channel information of signals reflected by human bodies. From the experiments conducted, we demonstrate the feasibility and efficacy of the proposed methods. Also, we conclude the factors that would affect the performance and highlight a few potential issues that require further deliberations.

In this paper we introduce a 3-dimensional (3D) SIFT descriptor for video or 3D imagery such as MRI data. We also show how this new descriptor is able to better represent the 3D nature of video data in the application of action recognition. This paper will show how 3D SIFT is able to outperform previously used description methods in an elegant and efficient manner. We use a bag of words approach to represent videos, and present a method to discover relationships between spatio-temporal words in order to better describe the video data.


Question: Recognition of events and actions using 3D thermal depth sensing.

This document introduces a novel technique for pedestrian detection and tracking using a solitary night-vision camera mounted on a vehicle. Addressing the inherent variability in human appearance on the road, a dual-stage detection and tracking method is proposed. The detection process is executed using a support vector machine (SVM) with pedestrians' candidates normalized for size, while the tracking stage integrates Kalman filter prediction and mean shift tracking. The detection phase is enhanced through the integration of a road-detection module, which offers crucial data for the validation of pedestrian candidates. Various experimental evaluations, such as comparisons between grayscale SVM recognition and binary SVM recognition, and between entire-body detection and upper-body detection, have been conducted to demonstrate the practicality and effectiveness of our proposed approach.

The method's robustness in dealing with the dynamic nature of pedestrian appearances is a key focus, utilizing size-normalized candidates in the SVM for improved detection accuracy. The tracking phase's synergy between Kalman filter prediction and mean shift tracking ensures continuous and accurate monitoring of detected pedestrians. Moreover, the road-detection module's provision of essential information for pedestrian validation significantly enhances the reliability of the detection process. The experimental evaluations serve to highlight the method's superiority over alternative approaches, illustrating its potential for real-world application in night-time pedestrian detection and tracking scenarios.

Human activity recognition is an important area of ubiquitous computing. Most current researches in activity recognition mainly focus on simple activities, e.g., sitting, running, walking, and standing. Compared with simple activities, complex activities are more complicated with high-level semantics, e.g., working, commuting, and having a meal. This paper presents a hierarchical model to recognize complex activities as mixtures of simple activities and multiple actions. We generate the components of complex activities using a clustering algorithm, represent and recognize complex activities by applying a topic model on these components. It is a data-driven method that can retain effective information for representing and recognizing complex activities. In addition, acceleration and physiological signals are fused in classifier level to ensure the overall performance of complex activity recognition. The results of experiments show that our method has ability to represent and recognize complex activities effectively.

A key aspect of pervasive computing is using computers and sensor networks to effectively and unobtrusively infer users' behavior in their environment. This includes inferring which activity users are performing, how they're performing it, and its current stage. Recognizing and recording activities of daily living is a significant problem in elder care. A new paradigm for ADL inferencing leverages radio-frequency-identification technology, data mining, and a probabilistic inference engine to recognize ADLs, based on the objects people use. We propose an approach that addresses these challenges and shows promise in automating some types of ADL monitoring. Our key observation is that the sequence of objects a person uses while performing an ADL robustly characterizes both the ADL's identity and the quality of its execution. So, we have developed Proactive Activity Toolkit (PROACT).

Anticipating actions and objects before they start or appear is a difficult problem in computer vision with several real-world applications. This task is challenging partly because it requires leveraging extensive knowledge of the world that is difficult to write down. We believe that a promising resource for efficiently learning this knowledge is through readily available unlabeled video. We present a framework that capitalizes on temporal structure in unlabeled video to learn to anticipate human actions and objects. The key idea behind our approach is that we can train deep networks to predict the visual representation of images in the future. Visual representations are a promising prediction target because they encode images at a higher semantic level than pixels yet are automatic to compute. We then apply recognition algorithms on our predicted representation to anticipate objects and actions. We experimentally validate this idea on two datasets, anticipating actions one second in the future and objects five seconds in the future.

This paper proposes an algorithm for fall detection using a ceiling-mounted 3D depth camera. The lying pose is separated from common daily activities by a k-NN classifier, which was trained on features expressing head-floor distance, person area and shape's major length to width. In order to distinguish between intentional lying postures and accidental falls the algorithm also employs motion between static postures. The experimental validation of the algorithm was conducted on realistic depth image sequences of daily activities and simulated falls. It was evaluated on more than 45000 depth images and gave 0% error. To reduce the processing overload an accelerometer was used to indicate the potential impact of the person and to start an analysis of depth images.

This paper contains a survey on aspects of visual event computing. We start by presenting events and their classifications, and continue with discussing the problem of capturing events in terms of photographs, videos, etc, as well as the methodologies for event storing and retrieving. Later, we review an extensive set of papers taken from well-known conferences and journals in multiple disciplines. We analyze events, and summarize the procedure of visual event actions. We introduce each component of a visual event computing system, and its computational aspects, we discuss the progress of each component and review its overall status. Finally, we suggest future research trends in event computing and hope to introduce a comprehensive profile of visual event computing to readers.

This paper describes the primary and secondary driving task together with Human Machine Interface (HMI) trends and issues which are driving automotive user interface designers to consider hand gesture recognition as a realistic alternative for user controls. A number of hand gesture recognition technologies and applications for Human Vehicle Interaction (HVI) are also discussed including a summary of current automotive hand gesture recognition research.

Human Activity Recognition (HAR) refers to the problem of automatically identifying human activities by learning from data collected either from the individual or from the environment surrounding that individual. The form of the data might be inertial measurements, physiological signals from devices like wearable sensors or those such as image, audio and video data collected from environmental devices/sensors. HAR has been a well studied problem for its potential capability to mine human behavioral patterns and for monitoring human activity for security and health purposes.

We propose a visual event recognition framework for consumer videos by leveraging a large amount of loosely labeled web videos (e.g., from YouTube). Observing that consumer videos generally contain large intraclass variations within the same type of events, we first propose a new method, called Aligned Space-Time Pyramid Matching (ASTPM), to measure the distance between any two video clips. Second, we propose a new transfer learning method, referred to as Adaptive Multiple Kernel Learning (A-MKL), in order to 1) fuse the information from multiple pyramid levels and features (i.e., space-time features and static SIFT features) and 2) cope with the considerable variation in feature distributions between videos from two domains (i.e., web video domain and consumer video domain). For each pyramid level and each type of local features, we first train a set of SVM classifiers based on the combined training set from two domains by using multiple base kernels from different kernel types and parameters, which are then fused with equal weights to obtain a prelearned average classifier. In A-MKL, for each event class we learn an adapted target classifier based on multiple base kernels and the prelearned average classifiers from this event class or all the event classes by minimizing both the structural risk functional and the mismatch between data distributions of two domains. Extensive experiments demonstrate the effectiveness of our proposed framework that requires only a small number of labeled consumer videos by leveraging web data. We also conduct an in-depth investigation on various aspects of the proposed method A-MKL, such as the analysis on the combination coefficients on the prelearned classifiers, the convergence of the learning algorithm, and the performance variation by using different proportions of labeled consumer videos. Moreover, we show that A-MKL using the prelearned classifiers from all the event classes leads to better performance when compared with A-MKL using the prelearned classifiers only from each individual event class.

In this work1, we present a method to represent a video with a sequence of words, and learn the temporal sequencing of such words as the key information for predicting and recognizing human actions. We leverage core concepts from the Natural Language Processing (NLP) literature used in sentence classification to solve the problems of action prediction and action recognition. Each frame is converted into a word that is represented as a vector using the Bag of Visual Words (BoW) encoding method. The words are then combined into a sentence to represent the video, as a sentence. The sequence of words in different actions are learned with a simple but effective Temporal Convolutional Neural Network (T-CNN) that captures the temporal sequencing of information in a video sentence. We demonstrate that a key characteristic of the proposed method is its low-latency, i.e. its ability to predict an action accurately with a partial sequence (sentence). Experiments on two datasets, UCF101 and HMDB51 show that the method on average reaches 95% of its accuracy within half the video frames. Results, also demonstrate that our method achieves compatible state-of-the-art performance in action recognition (i.e. at the completion of the sentence) in addition to action prediction.

Gesture-based interaction, as a natural way for human-computer interaction, has a wide range of applications in ubiquitous computing environment. This paper presents an acceleration-based gesture recognition approach, called FDSVM (Frame-based Descriptor and multi-class SVM), which needs only a wearable 3-dimensional accelerometer. With FDSVM, firstly, the acceleration data of a gesture is collected and represented by a frame-based descriptor, to extract the discriminative information. Then a SVM-based multi-class gesture classifier is built for recognition in the nonlinear gesture feature space. Extensive experimental results on a data set with 3360 gesture samples of 12 gestures over weeks demonstrate that the proposed FDSVM approach significantly outperforms other four methods: DTW, Naïve Bayes, C4.5 and HMM. In the user-dependent case, FDSVM achieves the recognition rate of 99.38% for the 4 direction gestures and 95.21% for all the 12 gestures. In the user-independent case, it obtains the recognition rate of 98.93% for 4 gestures and 89.29% for 12 gestures. Compared to other accelerometer-based gesture recognition approaches reported in literature FDSVM gives the best resulrs for both user-dependent and user-independent cases.


Question: Recognition of events and actions utilizing thermal sensing and 3D depth perception.

This document explores the application of Support Vector Machines (SVMs) in the field of computer vision, specifically in the context of face detection. SVMs, developed by V. Vapnik and his team at AT&T Bell Labs, represent a novel method for training classifiers such as polynomial, neural network, or Radial Basis Functions. The decision boundaries are determined by solving a quadratic programming problem with linear constraints. This optimization challenge becomes particularly complex due to the quadratic form's density and the exponential growth of memory requirements with the increase in data points. To address these issues, a decomposition algorithm is introduced, ensuring global optimality and enabling the training of SVMs on extensive datasets. The algorithm iteratively solves sub-problems while evaluating optimality conditions, which are utilized to refine iterative values and determine the algorithm's termination point. Experimental outcomes from the SVM implementation are presented, showcasing its viability in tackling a face detection problem involving a dataset of 50,000 points. Similarly, in the realm of event and action recognition from thermal and 3D depth sensing, SVMs can be applied to process and analyze the complex data. By employing a decomposition algorithm tailored to handle the large volumes of data generated by thermal and 3D depth sensors, SVMs can effectively classify and recognize various events and actions. This method ensures optimal performance and scalability, making it a promising approach for real-time monitoring and analysis in applications such as surveillance, healthcare monitoring, and environmental studies.

Understanding the activities of human from videos is demanding task in Computer Vision. Identifying the actions being accomplished by the human in the video sequence automatically and tagging their actions is the prime functionality of intelligent video systems. The goal of activity recognition is to identify the actions and objectives of one or more objects from a series of examination on the action of object and their environmental condition. The major applications of Human Activity Recognition varies from Content-based Video Analytics, Robotics, Human-Computer Interaction, Human fall detection, Ambient Intelligence, Visual Surveillance, Video Indexing etc. This paper collectively summarizes and deciphers the various methodologies, challenges and issues of Human Activity Recognition systems. Variants of Human Activity Recognition systems such as Human Object Interactions and Human-Human Interactions are also explored. Various benchmarking datasets and their properties are being explored. The Experimental Evaluation of various papers are analyzed efficiently with the various performance metrics like Precision, Recall, and Accuracy.

We introduce a new dataset, Human3.6M, of 3.6 Million accurate 3D Human poses, acquired by recording the performance of 5 female and 6 male subjects, under 4 different viewpoints, for training realistic human sensing systems and for evaluating the next generation of human pose estimation models and algorithms. Besides increasing the size of the datasets in the current state-of-the-art by several orders of magnitude, we also aim to complement such datasets with a diverse set of motions and poses encountered as part of typical human activities (taking photos, talking on the phone, posing, greeting, eating, etc.), with additional synchronized image, human motion capture, and time of flight (depth) data, and with accurate 3D body scans of all the subject actors involved. We also provide controlled mixed reality evaluation scenarios where 3D human models are animated using motion capture and inserted using correct 3D geometry, in complex real environments, viewed with moving cameras, and under occlusion. Finally, we provide a set of large-scale statistical models and detailed evaluation baselines for the dataset illustrating its diversity and the scope for improvement by future work in the research community. Our experiments show that our best large-scale model can leverage our full training set to obtain a 20% improvement in performance compared to a training set of the scale of the largest existing public dataset for this problem. Yet the potential for improvement by leveraging higher capacity, more complex models with our large dataset, is substantially vaster and should stimulate future research. The dataset together with code for the associated large-scale learning models, features, visualization tools, as well as the evaluation server, is available online at http://vision.imar.ro/human3.6m.

This paper describes a mapping system that acquires 3D object models of man-made indoor environments such as kitchens. The system segments and geometrically reconstructs cabinets with doors, tables, drawers, and shelves, objects that are important for robots retrieving and manipulating objects in these environments. The system also acquires models of objects of daily use such glasses, plates, and ingredients. The models enable the recognition of the objects in cluttered scenes and the classification of newly encountered objects. Key technical contributions include (1) a robust, accurate, and efficient algorithm for constructing complete object models from 3D point clouds constituting partial object views, (2) feature-based recognition procedures for cabinets, tables, and other task-relevant furniture objects, and (3) automatic inference of object instance and class signatures for objects of daily use that enable robots to reliably recognize the objects in cluttered and real task contexts. We present results from the sensor-based mapping of a real kitchen.

Enabling touch gesture sensing on all surfaces of the mobile device, not limited to the touchscreen area, leads to new user interaction experiences. In this paper, we propose VSkin, a system that supports fine-grained gesture-sensing on the back of mobile devices based on acoustic signals. VSkin utilizes both the structure-borne sounds, i.e., sounds propagating through the structure of the device, and the air-borne sounds, i.e., sounds propagating through the air, to sense finger tapping and movements. By measuring both the amplitude and the phase of each path of sound signals, VSkin detects tapping events with an accuracy of 99.65% and captures finger movements with an accuracy of 3.59mm.

This paper introduces our initial investigation on the problem of providing a semi-autonomous robot collaborator with anticipative capabilities to predict human actions. Anticipative robot behavior is a desired characteristic of robot collaborators that lead to fluid, proactive interactions. We are particularly interested in improving reactive methods that rely on human action recognition to activate the corresponding robot action. Action recognition invariably causes delay in the robot’s response, and the goal of our method is to eliminate this delay by predicting the next human action. Prediction is achieved by using a lookup table containing variations of assembly sequences, previously demonstrated by different users. The method uses the nearest neighbor sequence in the table that matches the actual sequence of human actions. At the movement level, our method uses a probabilistic representation of interaction primitives to generate robot trajectories. The method is demonstrated using a 7 degree-offreedom lightweight arm equipped with a 5-finger hand on an assembly task consisting of 17 steps.

We propose to combine recent Convolutional Neural Networks (CNN) models with depth imaging to obtain a reliable and fast multi-person pose estimation algorithm applicable to Human Robot Interaction (HRI) scenarios. Our hypothesis is that depth images contain less structures and are easier to process than RGB images while keeping the required information for human detection and pose inference, thus allowing the use of simpler networks for the task. Our contributions are threefold. (i) we propose a fast and efficient network based on residual blocks (called RPM) for body landmark localization from depth images; (ii) we created a public dataset DIH comprising more than 170k synthetic images of human bodies with various shapes and viewpoints as well as real (annotated) data for evaluation; (iii) we show that our model trained on synthetic data from scratch can perform well on real data, obtaining similar results to larger models initialized with pre-trained networks. It thus provides a good trade-off between performance and computation. Experiments on real data demonstrate the validity of our approach.

Traffic safety is a severe problem around the world. Many road accidents are normally related with the driver's unsafe driving behavior, e.g. eating while driving. In this work, we propose a vision-based solution to recognize the driver's behavior based on convolutional neural networks. Specifically, given an image, skin-like regions are extracted by Gaussian Mixture Model, which are passed to a deep convolutional neural networks model, namely R*CNN, to generate action labels. The skin-like regions are able to provide abundant semantic information with sufficient discriminative capability. Also, R*CNN is able to select the most informative regions from candidates to facilitate the final action recognition. We tested the proposed methods on Southeast University Driving-posture Dataset and achieve mean Average Precision(mAP) of 97.76% on the dataset which prove the proposed method is effective in drivers's action recognition.

While intelligent environments are often cited as a reason for doing work on visual person-tracking, really making an intelligent environment exposes many realworld problems in visual tracking that must be solved to make the technology practical. In the context of our EasyLiving project in intelligent environments, we created a practical person-tracking system that solves most of the real-world problems. It uses two sets of color stereo cameras for tracking multiple people during live demonstrations in a living room. The stereo images are used for locating people, and the color images are used for maintaining their identities. The system runs quickly enough to make the room feel responsive, and it tracks multiple people standing, walking, sitting, occluding, and entering and leaving the

Human-centered computing is an emerging research field that aims to understand human behavior and integrate users and their social context with computer systems. One of the most recent, challenging and appealing applications in this framework consists in sensing human body motion using smartphones to gather context information about people actions. In this context, we describe in this work an Activity Recognition database, built from the recordings of 30 subjects doing Activities of Daily Living (ADL) while carrying a waist-mounted smartphone with embedded inertial sensors, which is released to public domain on a well-known on-line repository. Results, obtained on the dataset by exploiting a multiclass Support Vector Machine (SVM), are also acknowledged.

This paper summarizes recent activities carried out for the development of an innovative anthropomorphic robotic hand called the DEXMART Hand. The main goal of this research is to face the problems that affect current robotic hands by introducing suitable design solutions aimed at achieving simplification and cost reduction while possibly enhancing robustness and performance. While certain aspects of the DEXMART Hand development have been presented in previous papers, this paper is the first to give a comprehensive description of the final hand version and its use to replicate humanlike grasping. In this paper, particular emphasis is placed on the kinematics of the fingers and of the thumb, the wrist architecture, the dimensioning of the actuation system, and the final implementation of the position, force and tactile sensors. The paper focuses also on how these solutions have been integrated into the mechanical structure of this innovative robotic hand to enable precise force and displacement control of the whole system. Another important aspect is the lack of suitable control tools that severely limits the development of robotic hand applications. To address this issue, a new method for the observation of human hand behavior during interaction with common day-to-day objects by means of a 3D computer vision system is presented in this work together with a strategy for mapping human hand postures to the robotic hand. A simple control strategy based on postural synergies has been used to reduce the complexity of the grasp planning problem. As a preliminary evaluation of the DEXMART Hand’s capabilities, this approach has been adopted in this paper to simplify and speed up the transfer of human actions to the robotic hand, showing its effectiveness in reproducing human-like grasping.