Question: A review of methods for modeling and indexing events within multimedia content.

Exploring Causal Connections through Econometric Analysis and Cross-Spectral Techniques

This document delves into the intricate process of examining causal relationships through the application of econometric models and cross-spectral methods. It highlights the importance of understanding these relationships in the context of economic and financial data analysis. The econometric models, renowned for their ability to capture complex relationships between variables, are complemented by cross-spectral techniques, which offer a unique perspective on the interaction between time series data. This approach not only enhances the accuracy of predictions but also deepens our understanding of the underlying mechanisms that drive economic phenomena. The document emphasizes the need for users to adhere to the terms and conditions set by JSTOR, which stipulate that content may only be used for personal, non-commercial purposes unless prior permission has been obtained from the publisher. This ensures that the intellectual property rights of the authors and publishers are respected, fostering a sustainable environment for academic research and discourse. For any inquiries regarding the use of this work beyond personal use, users are encouraged to contact the publisher directly for permission and further guidance.

Our objective is to index people in a TV-Content. In this context, because of multi-face shots and non-speaking face shots, it is difficult to determine which face is speaking. There is no guaranteed synchronization between sequences of a person’s appearance and sequences of his or her speech. In this work, we want to separate talking and non-talking faces by detecting lip motion. We propose a method to detect the lip motion by measuring the degree of disorder of pixel directions around the lip. Results of experiments on a TV-Show database show that a high correct classification rate can be achieved by the proposed method.

Retrospective news event detection (RED) is defined as the discovery of previously unidentified events in historical news corpus. Although both the contents and time information of news articles are helpful to RED, most researches focus on the utilization of the contents of news articles. Few research works have been carried out on finding better usages of time information. In this paper, we do some explorations on both directions based on the following two characteristics of news articles. On the one hand, news articles are always aroused by events; on the other hand, similar articles reporting the same event often redundantly appear on many news sources. The former hints a generative model of news articles, and the latter provides data enriched environments to perform RED. With consideration of these characteristics, we propose a probabilistic model to incorporate both content and time information in a unified framework. This model gives new representations of both news articles and news events. Furthermore, based on this approach, we build an interactive RED system, HISCOVERY, which provides additional functions to present events, Photo Story and Chronicle.

We report on a four year academic research project to build a natural language processing platform in support of a large media company. The Computable News platform processes news stories, producing a layer of structured data that can be used to build rich applications. We describe the underlying platform and the research tasks that we explored building it. The platform supports a wide range of prototype applications designed to support different newsroom functions. We hope that this qualitative review provides some insight into the challenges involved in this type of project.

This paper presents strategies and lessons learned from the use of natural language annotations to facilitate question answering in the START information access system.

In the absence of a rich dataset, sports video analysis remains a challenging task. Shot analysis, event detection, and player tracking are important aspects of soccer game research, however currently available datasets focus mainly on single-type annotations, and have several limitations when applied to full video analysis; these include a lack of training samples, temporal annotations, and rich labels of different types. In this paper, we focus on broadcast soccer videos and present a comprehensive dataset for analysis. In its current version, the dataset contains 222 broadcast soccer videos, totaling 170 video hours. The dataset covers three annotation types: (1) A shot boundary with two shot transition types, and shot type annotations with five shot types; (2) Event annotations with 11 event labels, and a story annotation with 15 story labels at coarser granularity; and (3) bounding boxes of the players under analysis in a subset of 19908 video frames. We hope that the introduction of this dataset will enable the research community to develop soccer video analysis techniques.

1. ABSTRACT Animated agents based either on real video, cartoon-style drawings or even model-based 3D graphics offer great promise for computer-based presentations as they make presentations more lively and appealing and allow for the emulation of conversation styles known from human-human communication. In this paper, we describe a life-like interface agent which presents multimedia material to the user following the directives of a script. The overall behavior of the presentation agent is partly determined by such a script, and partly by the agent's self-behavior. In our approach, the agent's behavior is defined in a declarative specification language. Behavior specifications are used to automatically generate a control module for an agent display system. The first part of the paper describes the generation process which involves AI planning and a two-step compilation. Since the manual creation of presentation scripts is tedious and error-prone, we also address the automated generation of presentation scripts which may be forwarded to the interface agent. The second part of the paper presents an approach for multimedia presentation design which combines hierarchical planning with temporal reasoning.

Exponential growth of social media consumption demands eective user interests-social contents modeling for more personalized recommendation and social media summarization. However, due to the heterogeneous nature of social contents, traditional approaches lack the ability of capturing the hidden semantic correlations across these multi-modal data, which leads to semantic gaps between social content understanding and user interests. To eectively bridge the semantic gaps, we propose a novel deep learning framework for user interests-social contents modeling. We rst mine and parse data, i.e. textual content, visual content, social context and social relation, from heterogeneous social media feeds. en, we design a two-branch network to map the social contents and users into a same latent space. Particularly, the network is trained by a largemargin objective that combines a cross-instance distance constraint with a within-instance semantic-preserving constraint in an endto-end manner. At last, a Deep Semantic-Preserving Embedding (D-Sempre) is learned, and the ranking results can be given by calculating distances between social contents and users. To demonstrate the eectiveness of D-Sempre in user interests-social contents modeling, we construct a Twier dataset and conduct extensive experiments on it. As a result, D-Sempre eectively integrates the multimodal data from heterogeneous social media feeds and captures the hidden semantic correlations between users’ interests and social contents.

0957-4174/$ see front matter 2008 Elsevier Ltd. A doi:10.1016/j.eswa.2008.09.042 q Part of the content of this paper has been publi International Conference on Multimedia, 2005. * Corresponding author. Tel.: +886 2 29393091x67 E-mail address: mkshan@cs.nccu.edu.tw (M.-K. Sh With the growth of digital music, the development of music recommendation is helpful for users to pick desirable music pieces from a huge repository of music. The existing music recommendation approaches are based on a user’s preference on music. However, sometimes, it might better meet users’ requirement to recommend music pieces according to emotions. In this paper, we propose a novel framework for emotion-based music recommendation. The core of the recommendation framework is the construction of the music emotion model by affinity discovery from film music, which plays an important role in conveying emotions in film. We investigate the music feature extraction and propose the Music Affinity Graph and Music Affinity Graph-Plus algorithms for the construction of music emotion model. Experimental result shows the proposed emotion-based music recommendation achieves 85% accuracy in average. 2008 Elsevier Ltd. All rights reserved.

Currently most approaches to retrieving textual materials from scienti c databases depend on a lexical match between words in users requests and those in or assigned to documents in a database Because of the tremendous diversity in the words people use to describe the same document lexical methods are necessarily incomplete and imprecise Using the singular value decomposition SVD one can take advantage of the implicit higher order structure in the association of terms with documents by determining the SVD of large sparse term by document matrices Terms and documents represented by of the largest singular vectors are then matched against user queries We call this retrieval method Latent Semantic Indexing LSI because the subspace represents important associative relationships between terms and documents that are not evident in individual documents LSI is a completely automatic yet intelligent indexing method widely applicable and a promising way to improve users access to many kinds of textual materials or to documents and services for which textual descriptions are available A survey of the computational requirements for managing LSI encoded databases as well as current and future applications of LSI is presented

The goal of TDT Topic Detection and Tracking is to develop aut omatic methods of identifying topically related stories wit hin a stream of news media. We describe approaches for both detection and tr cking based on the well-known idf -weighted cosine coefficient similarity metric. The surprising outcome of this research is th at we achieved very competitive results for tracking using a very simple method of feature selection, without word stemming and with ou a score normalization scheme. The detection task results wer e not as encouraging though we attribute this more to the clustering algorithm than the underlying similarity metric. 1. The Tracking Task The goal of the topic tracking task for TDT2 is to identify new s stories on a particular event defined by a small number ( Nt) of positive training examples and a greater number of negative examples . All stories in the news stream subsequent to the final positive ex ample are to be classified as on-topic if they pertain to the event or off-topic if they do not. Although the task is similar to IR routing and fi ltering tasks, the definition of event leads to at least one significan t difference. An event is defined as an occurrence at a given place and t ime covered by the news media. Stories are on-topic if they cover th event itself or any outcome (strictly-defined in [2]) of the e vent. By this definition, all stories prior to the occurrence are offt pic, which contrary to the IR tasks mentioned, theoretically provides for unlimited off-topic training material (assuming retrospective corpora are available). We expected to be able to take advantage of these unlimited negative examples but in our final implementation did so only to the extent that we used a retrospective corpus to improve t erm statistics of our database. 1.1. idf-Weighted Cosine Coefficient As the basis for our approach we used the idf -weighted cosine coefficient described in [1] often referred to as tf idf . Using this metric, the tracking task becomes two-fold. Firstly, choosing an op timal set of features to represent topics, i.e. feature selection. Th e approach must choose features from a single story as well as from multi ple stories (forNt > 1). Secondly, determining a threshold (potentially one per topic) which optimizes the miss and false alarm proba bilities for a particular cost function, effectively normalizing th e similarity scores across topics. The cosine coefficient is a document similarity metric which has been investigated extensively. Here documents (and querie s) ar represented as vectors in an n-dimensional space, where n is the number of unique terms in the database. The coefficients of th e vector for a given document are the term frequencies ( tf ) for that dimension. The resulting vectors are extremely sparse and t ypically high frequency words (mostly closed class) are ignored. The cosine of the angle between two vectors is an indication of vector si milarity and is equal to the dot-product of the vectors normalized by the product of the vector lengths. cos( ) = ~ A ~ B k ~ Akk ~ Bk tf idf (term frequency times inverse document frequency) weighting is an ad-hoc modification to the cosine coefficient calcul tion which weights words according to their usefulness in discriminating documents. Words that appear in few documents are more usefu l than words that appear in many documents. This is captured in the equation for the inverse document frequency of a word: idf(w) = log10 N df(w) Wheredf(w) is the number of documents in a collection which contain wordw andN is the total number of documents in the collection. For our implementation we weighted only the topic vector by idf and left the story vector under test unchanged. This allows u s to calculate and fix anidf -scaled topic vector immediately after training on the last positive example story for a topic. The resulting calculation for the similarity measure becomes: sim(a; b) = Pnw=1 tfa(w) tfb(w) idf(w) pPnw=1 tf2 a (w) pPnw=1 tf2 b (w) 1.2. UPENN System Attributes To facilitate testing, the stories were loaded into a simple document processing system. Once in the system, stories are proc ssed in chronological order testing all topics simultaneously w ith a single pass over the data 1 at a rate of approximately 6000 stories per minute on a Pentium 266 MHz machine. The system tokenizer delimits on white space and punctuation (and discards it), col lapses case, but provides no stemming. A list of 179 stop words consi sti g almost entirely of close classed words was also employed. In order to improve word statistics, particularly for the beginning of the test set, we prepended a retrospective corpus (the TDT Pilot Data [3]) of approximately 16 thousand stories. 1In accordance with the evaluation specification for this pro ject [2] no information is shared across topics. 1.3. Feature Selection Thechoice as well asnumber of features (words) used to represent a topic has a direct effect on the trade-off between miss and f lse alarm probabilities. We investigated four methods of produ cing lists of features sorted by their effectiveness in discriminatin g a topic. This then allowed us to easily vary the number of those featur s for the topic vectors 2. 1. Keep all features except those words belonging to the stop word list. 2. Relative to training stories, sort words by document coun t, keepn most frequent. This approach has the advantage of finding those words which are common across training stories, an d therefore are more general to the topic area, but has the disa dvantage of extending poorly from the Nt = 16 case to the Nt = 1 case. 3. For each story, sort by word count ( tf ), keepn most frequent. While this approach tends to ignore low count words which occur in multiple training documents, it generalizes well f rom theNt = 16 to theNt = 1 case. 4. As a variant on the previous method we tried adding to the initial n features using a simple greedy algorithm. Against a database containing all stories up to and including the Nt-th training story, we queried the database with the n f atures plus the next most frequent term. If the separation of on-topic an d off-topic stories increased, we kept the term, if not we igno red it and tested the next term in the list. We defined separation as the difference between the average on-topic scores and th e average of the 20 highest scoring off-topic documents. Of the feature selection methods we tried the forth one yield ed the best results across varying values of Nt, although only slightly better than the much simpler third method. Occam’s Razor prompt ed us to omit this complication from the algorithm. The DET curves 3 in Figure 1. show the effect of varying the number of features (o btained from method 3) on the miss and false alarm probabilities. The upper right most curve results from choosing the single most fr equent feature. Downward to the left, in order are the curves for 5, 1 0, 50, 150 and 300 features. After examining similar plots from the pilot, training4, and development-test data sets, we set the number of features for our system to 50. It can be seen that there is limited benefit in adding features after this point. 1.4. Normalization / Threshold Selection With a method of feature selection in place, a threshold for t he similarity score must be determined above which stories will be d e med on-topic, and below which they will not. Since each topic is r epresented by its own unique vector it cannot be expected that the same threshold value will be optimal across all topics unless the scores are normalized. We tried two approaches for normalizing the topic similarity scores. For the first approach we calculated the similarity of a rando m sample of several hundred off-topic documents in order to estim a e an 2We did not employ feature selection on the story under test bu t used the text in entirety. 3See [5] for detailed description of DET curves. 4The first two month period of TDT2 data is called the training s et, not to be confused with training data. 1 2 5 10 20 40 60 80 90 .01 .02 .05 .1 .2 .5 1 2 5 10 20 40 60 80 90 M is s pr ob ab ili ty ( in % ) False Alarms probability (in %) random performance num_features=1 num_features=5 num_features=10 num_features=50 num_features=150 num_features=300 Figure 1: DET curve for varying number of features. (Nt=4, TD T2 evaluation data set, newswire and ASR transcripts) average off-topic score relative to the topic vector. The no rmalized score is then a function of the average on-topic 5 and off-topic scores and the off-topic standard deviation 6. The second approach looked at only thehighest scoringoff-topic stories returned from a query of the topic vector against a retrospective database with th e score normalized in a similar fashion to the first approach. Both attempts reduced the story-weighted miss probability by approximately 10 percent at low false alarm probability relat ive. However, this results was achieved at the expense of higher miss probability at higher false alarm probability, and a higher cost a t the operating point defined by the cost function for the task 7. Ctrack = Cmiss P (miss) Ptopic +Cfa P (fa) (1 Ptopic) where Cmiss = 1: (the cost of a miss) Cfa = 1: (the cost of a false alarm) Ptopic = 0:02: (thea priori probability of a story being on a given topic was chosen based on the TDT2 training topics and traini ng corpus.) Because of the less optimal trade-off between error probabi lities at the point defined by the cost function, we choose to ignore nor malization and look directly at cost as a function of a single thr es old value across all topics. We plotted tf idf score against story and topic-weighted cost for the training and development-test data sets. As our global threshold we averaged the scores at which story and topic-weighted cost were minimized. This is depicted in figu re 2. Figure 3 shows the same curves for the evaluation data set. Th e threshold resulting from the traini


Question: A review of methods for modeling and indexing events in multimedia content.

This paper presents a novel approach for the semantic analysis of soccer video footage, utilizing dynamic Bayesian networks (DBNs) and Bayesian networks (BNs) to decode the high-level semantics embedded within low-level image features. The system, tailored specifically for soccer game videos, demonstrates its effectiveness in recognizing key events such as goals, corner kicks, penalty kicks, and card incidents. By adopting a frame-based analysis technique, the proposed method differs significantly from traditional shot-based approaches. For each frame, it not only determines the current semantics of the event nodes but also deciphers the status of hidden nodes. A significant advancement is the automatic generation of BNs and DBNs through a training process, eliminating the need for ad hoc determinations. Additionally, the introduction of a temporal intervening network enhances the accuracy of the semantic output, representing a notable improvement over previous methods. This comprehensive framework, combining low-level image analysis with high-level semantic interpretation, showcases a robust and adaptable solution for the semantic analysis of soccer videos.

In natural-language discourse, related events tend to appear near each other to describe a larger scenario. Such structures can be formalized by the notion of a frame (a.k.a. template), which comprises a set of related events and prototypical participants and event transitions. Identifying frames is a prerequisite for information extraction and natural language generation, and is usually done manually. Methods for inducing frames have been proposed recently, but they typically use ad hoc procedures and are difficult to diagnose or extend. In this paper, we propose the first probabilistic approach to frame induction, which incorporates frames, events, participants as latent topics and learns those frame and event transitions that best explain the text. The number of frames is inferred by a novel application of a split-merge method from syntactic parsing. In end-to-end evaluations from text to induced frames and extracted facts, our method produced state-of-the-art results while substantially reducing engineering effort.

Online social and news media generate rich and timely information about real-world events of all kinds. However, the huge amount of data available, along with the breadth of the user base, requires a substantial effort of information filtering to successfully drill down to relevant topics and events. Trending topic detection is therefore a fundamental building block to monitor and summarize information originating from social sources. There are a wide variety of methods and variables and they greatly affect the quality of results. We compare six topic detection methods on three Twitter datasets related to major events, which differ in their time scale and topic churn rate. We observe how the nature of the event considered, the volume of activity over time, the sampling procedure and the pre-processing of the data all greatly affect the quality of detected topics, which also depends on the type of detection method used. We find that standard natural language processing techniques can perform well for social streams on very focused topics, but novel techniques designed to mine the temporal distribution of concepts are needed to handle more heterogeneous streams containing multiple stories evolving in parallel. One of the novel topic detection methods we propose, based on -grams cooccurrence and topic ranking, consistently achieves the best performance across all these conditions, thus being more reliable than other state-of-the-art techniques.

Among the various types of spatio-temporal queries, the most common ones involve window queries in time. In particular, timestamp (or timeslice) queries retrieve all objects that intersect a window at a specific timestamp. Interval queries include multiple (usually consecutive) timestamps. Although several indexes have been developed for either type, currently there does not exist a structure that can efficiently process both query types. This is a significant problem due to the fundamental importance of these queries in any spatio-temporal system that deals with historical information retrieval. Our paper addresses the problem by proposing the MV3R-tree, a structure that utilizes the concepts of multi-version B-trees and 3D-Rtrees. Extensive experimentation proves that MV3R-trees compare favorably with specialized structures aimed at timestamp and interval window queries, both in terms of time and space requirements.

We propose a demonstration of Cayuga, a complex event monitoring system for high speed data streams. Our demonstration will show Cayuga applied to monitoring Web feeds; the demo will illustrate the expressiveness of the Cayuga query language, the scalability of its query processing engine to high stream rates, and a visualization of the internals of the query processing engine.

In this paper, we consider the problem of fast and efficient indexing techniques for sequences evolving in non-Euclidean spaces. This problem has several applications in the areas of human activity analysis, where there is a need to perform fast search, and recognition in very high dimensional spaces. The problem is made more challenging when representations such as landmarks, contours, and human skeletons etc. are naturally studied in a non-Euclidean setting where even simple operations are much more computationally intensive than their Euclidean counterparts. We propose a geometry and data adaptive symbolic framework that is shown to enable the deployment of fast and accurate algorithms for activity recognition, dynamic texture recognition, motif discovery. Toward this end, we present generalizations of key concepts of piece-wise aggregation and symbolic approximation for the case of non-Euclidean manifolds. We show that one can replace expensive geodesic computations with much faster symbolic computations with little loss of accuracy in activity recognition and discovery applications. The framework is general enough to work across both Euclidean and non-Euclidean spaces, depending on appropriate feature representations without compromising on the ultra-low bandwidth, high speed and high accuracy. The proposed methods are ideally suited for real-time systems and low complexity scenarios.

Problem statement: Financial time series were usually large in size, unstructured and of high dimensionality. Since, the illustration of fin a cial time series shape was typically characterize d by a few number of important points. These important p oints moved in zigzag directions which could form technical patterns. However, these important p oints exhibited in different resolutions and diffic ult to determine. Approach: In this study, we proposed novel methods of financ i l time series indexing by considering their zigzag movement. The methods c onsist of two major algorithms: first, the identification of important points, namely the Zigz ag-Perceptually Important Points (ZIPs) identification method and next, the indexing method namely Zigzag based M-ary Tree (ZM-Tree) to structure and organize the important points. Results: The errors of the tree building and retrieving compared to the original time series increased when t important points increased. The dimensionality reduction using ZM-Tree based on tree pruning and n umber of retrieved points techniques performed better when the number of important points increase d. Conclusion: Our proposed techniques illustrated mostly acceptable performance in tree op rations and dimensionality reduction comparing to existing similar technique like Specialize Binar y Tree (SB-Tree).

The Latent Dirichlet Allocation topic model of Blei, Ng, & Jordan (2003) is well-established as an effective approach to recovering meaningful topics of conversation from a set of documents. However, a useful analysis of user-generated content is concerned not only with the recovery of topics from a static data set, but with the evolution of topics over time. We employ a compound topic model (CTM) to track topics across two distinct data sets (i.e. past and present) and to visualize trends in topics over time; we evaluate several metrics for detecting a change in the distribution of topics within a time-window; and we illustrate how our approach discovers emerging conversation topics related to current events in real data sets.

In this paper, we present an algorithm for extracting keywords representing the asserted main point in a document, without relying on external devices such as natural language processing tools or a document corpus. Our algorithm KeyGraph is based on the segmentation of a graph, representing the co-occurrence between terms in a document, into clusters. Each cluster corresponds to a concept on which author's idea is based, and top ranked terms by a statistic based on each term's relationship to these clusters are selected as keywords. This strategy comes from considering that a document is constructed like a building for expressing new ideas based on traditional concepts. The experimental results show that thus extracted terms match author's point quite accurately, even though KeyGraph does not use each term's average frequency in a corpus, i.e., KeyGraph is a contentsensitive, domain independent device of indexing.

We introduce a novel coreference resolution system that models entities and events jointly. Our iterative method cautiously constructs clusters of entity and event mentions using linear regression to model cluster merge operations. As clusters are built, information flows between entity and event clusters through features that model semantic role dependencies. Our system handles nominal and verbal events as well as entities, and our joint formulation allows information from event coreference to help entity coreference, and vice versa. In a cross-document domain with comparable documents, joint coreference resolution performs significantly better (over 3 CoNLL F1 points) than two strong baselines that resolve entities and events separately.

Neural Tensor Network for Learning Event Embeddings Event Representation E = (O1, P, O2, T), where P is the action, O1 is the actor, O2 is the object and T is the timestamp (T is mainly used for aligning stock data with news data). For example, the event “Sep 3, 2013 Microsoft agrees to buy Nokia’s mobile phone business for $7.2 billion.” is modeled as: (Actor = Microsoft, Action = buy, Object = Nokia’s mobile phone business, Time = Sep 3, 2013) Event Embedding


Question: 3D ActionSLAM: A wearable system for person tracking across multiple floors.

This paper introduces SignalSLAM, an innovative approach to indoor localization and mapping that integrates a variety of radio frequency (RF) signals, including WiFi, Bluetooth, Long-Term Evolution (LTE), and magnetic field readings. Traditionally, indoor positioning systems have relied on Received Signal Strength (RSS) measurements from WiFi networks, coupled with pre-established signal fingerprint maps. However, the advent of 4G LTE small cells, with their short-range but detailed Reference Signal Received Power (RSRP) data, offers a new dimension to localization technology. The proposed method, SignalSLAM, aggregates multiple RF signal sources to create comprehensive, multimodal signal maps for localization purposes or for optimizing network deployment. The core of SignalSLAM is Simultaneous Localization and Mapping (SLAM), a technique designed to construct a map of observations without prior knowledge of the observer's location. This concept has been previously extended to incorporate WiFi signal strength in WiFi-SLAM. Building upon WiFi-SLAM, other localization algorithms have emerged that utilize inertial motion sensors in conjunction with known maps of either WiFi RSS or magnetic field strength. Our study harnesses all available measurements from a standard smartphone, pooling data from multiple users navigating freely within a building. This includes timestamped WiFi and Bluetooth RSS, 4G LTE RSRP, magnetic field magnitude, GPS coordinates when outdoors, Near-Field Communication (NFC) tags at designated landmarks, and pedestrian dead reckoning based on inertial sensor data. We determine the users' positions through a modified version of Graph-SLAM, optimizing user poses using a combination of absolute location and pairwise constraints that factor in multimodal signal similarity. We showcase the ability to accurately pinpoint user locations and concurrently generate detailed signal maps for each WiFi access point and 4G LTE small cell, achieved "from the pocket." Lastly, we evaluate the localization performance using select single modalities, such as WiFi alone, and the WiFi signal maps we have created.

We tackle the problem of globally localizing a camera-equipped micro aerial vehicle flying within urban environments for which a Google Street View image database exists. To avoid the caveats of current image-search algorithms in case of severe viewpoint changes between the query and the database images, we propose to generate virtual views of the scene, which exploit the air-ground geometry of the system. To limit the computational complexity of the algorithm, we rely on a histogram-voting scheme to select the best putative image correspondences. The proposed approach is tested on a 2 km image dataset captured with a small quadroctopter flying in the streets of Zurich. The success of our approach shows that our new air-ground matching algorithm can robustly handle extreme changes in viewpoint, illumination, perceptual aliasing, and over-season variations, thus, outperforming conventional visual place-recognition approaches.

Loop closure detection systems for monocular SLAM come in three broad categories: (i) map-to-map, (ii) image-to-image and (iii) image-to-map. In this paper, we have chosen an implementation of each and performed experiments allowing the three approaches to be compared. The sequences used include both indoor and outdoor environments and single and multiple loop trajectories. © 2009 Elsevier B.V. All rights reserved.

[3] Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Leichter, D.K.C.R.I., Wei, A.V.Y., Krupka, D.F.P.K.E., Fitzgibbon, A., Izadi, S.: Accurate, robust, and flexible real-time hand tracking. In: CHI (2015) [4] Oberweger, M., Wohlhart, P., Lepetit, V.: Training a feedback loop for hand pose estimation. In: ICCV (2015) [18] Oberweger, M., Wohlhart, P., Lepetit, V.: Hands deep in deep learning for hand pose estimation. arXiv preprint arXiv:1502.06807 (2015) [19] Tang, D., Taylor, J., Kohli, P., Keskin, C., Kim, T.K., Shotton, J.: Opening the black box: Hierarchical sampling optimization for estimating human hand pose. In: ICCV (2015) * Indicates equal contribution. Results T θ

Ego-pose estimation, i.e., estimating a person’s 3D pose with a single wearable camera, has many potential applications in activity monitoring. For these applications, both accurate and physically plausible estimates are desired, with the latter often overlooked by existing work. Traditional computer vision-based approaches using temporal smoothing only take into account the kinematics of the motion without considering the physics that underlies the dynamics of motion, which leads to pose estimates that are physically invalid. Motivated by this, we propose a novel control-based approach to model human motion with physics simulation and use imitation learning to learn a videoconditioned control policy for ego-pose estimation. Our imitation learning framework allows us to perform domain adaption to transfer our policy trained on simulation data to real-world data. Our experiments with real egocentric videos show that our method can estimate both accurate and physically plausible 3D ego-pose sequences without observing the cameras wearer’s body.

Hand gestures are an important type of natural language used in many research areas such as human-computer interaction and computer vision. Hand gestures recognition requires the prior determination of the hand position through detection and tracking. One of the most efficient strategies for hand tracking is to use 2D visual information such as color and shape. However, visual-sensor-based hand tracking methods are very sensitive when tracking is performed under variable light conditions. Also, as hand movements are made in 3D space, the recognition performance of hand gestures using 2D information is inherently limited. In this article, we propose a novel real-time 3D hand tracking method in depth space using a 3D depth sensor and employing Kalman filter. We detect hand candidates using motion clusters and predefined wave motion, and track hand locations using Kalman filter. To verify the effectiveness of the proposed method, we compare the performance of the proposed method with the visual-based method. Experimental results show that the performance of the proposed method out performs visual-based method.

This paper presents the on-going design and implementation of a robust inertial sensor based Simultaneous Localisation And Mapping (SLAM) algorithm for an Unmanned Aerial Vehicle (UAV) using bearing-only observations. A single colour vision camera is used to observe the terrain from which image points corresponding to features in the environment are extracted. The SLAM algorithm estimates the complete 6-DoF motion of the UAV along with the three-dimensional position of the features in the environment. An Extended Kalman Filter (EKF) approach is used where a technique of delayed initialisation is performed to initialise the 3D position of features from bearing-only observations. Data association is achieved using a multi-hypothesis innovation gate based on the spatial uncertainty of each feature. Results are presented by running the algorithm off-line using inertial sensor and vision data collected during a flight test of a UAV.

The ability to simultaneously localize a robot and accurately map its surroundings is considered by many to be a key prerequisite of truly autonomous robots. However, few approaches to this problem scale up to handle the very large number of landmarks present in real environments. Kalman filter-based algorithms, for example, require time quadratic in the number of landmarks to incorporate each sensor observation. This paper presents FastSLAM, an algorithm that recursively estimates the full posterior distribution over robot pose and landmark locations, yet scales logarithmically with the number of landmarks in the map. This algorithm is based on an exact factorization of the posterior into a product of conditional landmark distributions and a distribution over robot paths. The algorithm has been run successfully on as many as 50,000 landmarks, environments far beyond the reach of previous approaches. Experimental results demonstrate the advantages and limitations of the FastSLAM algorithm on both simulated and real-world data.

Accurate 3-D human body pose tracking from a monocular video stream is important for a number of applications. We describe a novel hierarchical approach for tracking human pose that uses edge-based features during the coarse stage and later other features for global optimization. At first, humans are detected by motion and tracked by fitting an ellipse in the image. Then, body components are found using edge features and used to estimate the 2D positions of the body joints accurately. This helps to bootstrap the estimation of 3D pose using a sampling-based search method in the last stage. We present experiment results with sequences of different realistic scenes to illustrate the performance of the method.

In this paper, a robust, real-time object tracking approach capable of dealing with multiple symmetric and non-symmetric objects in a real-time requirement setting is proposed. The approach relies only on depth data to track multiple objects in a dynamic environment and uses random-forest based learning to deal with problems like object occlusion, motion-blur due to camera motion and clutter. We show that the relation between object motion and the corresponding change in its 3D point cloud data can be learned using only 6 random forests. A framework that unites object pose estimation and object pose tracking to efficiently track multiple objects in 3D space is presented. The approach is robust in tracking objects even in presence of motion blur that causes noisy depth data and is capable of real-time performance with 1.8ms per frame. The experimental evaluations demonstrate the performance of the approach against robustness, accuracy and speed and compare the approach with the state of the art. A publicly available dataset with real-world data is also provided for future benchmarking.

Designing and simulating realistic clothing is challenging. Previous methods addressing the capture of clothing from 3D scans have been limited to single garments and simple motions, lack detail, or require specialized texture patterns. Here we address the problem of capturing regular clothing on fully dressed people in motion. People typically wear multiple pieces of clothing at a time. To estimate the shape of such clothing, track it over time, and render it believably, each garment must be segmented from the others and the body. Our ClothCap approach uses a new multi-part 3D model of clothed bodies, automatically segments each piece of clothing, estimates the minimally clothed body shape and pose under the clothing, and tracks the 3D deformations of the clothing over time. We estimate the garments and their motion from 4D scans; that is, high-resolution 3D scans of the subject in motion at 60 fps. ClothCap is able to capture a clothed person in motion, extract their clothing, and retarget the clothing to new body shapes; this provides a step towards virtual try-on.


Question: 3D ActionSLAM: A wearable system for person tracking in environments with multiple floors.

This document provides an overview of the simultaneous localization and mapping (SLAM) problem and the fundamental techniques used to tackle it, highlighting significant implementations and practical demonstrations. Despite ongoing challenges, particularly in intricate outdoor settings, the core SLAM methodology is now a recognized and integral component of robotics research. A subsequent section of the tutorial delves into more contemporary research efforts aimed at resolving outstanding issues in SLAM, encompassing computational efficiency, feature representation, and data association strategies. This work underscores the evolution of SLAM techniques in wearable technology, such as 3D ActionSLAM, which enables accurate person tracking in multi-floor environments, showcasing advancements in real-world applications of SLAM methodologies.

Designing and simulating realistic clothing is challenging. Previous methods addressing the capture of clothing from 3D scans have been limited to single garments and simple motions, lack detail, or require specialized texture patterns. Here we address the problem of capturing regular clothing on fully dressed people in motion. People typically wear multiple pieces of clothing at a time. To estimate the shape of such clothing, track it over time, and render it believably, each garment must be segmented from the others and the body. Our ClothCap approach uses a new multi-part 3D model of clothed bodies, automatically segments each piece of clothing, estimates the minimally clothed body shape and pose under the clothing, and tracks the 3D deformations of the clothing over time. We estimate the garments and their motion from 4D scans; that is, high-resolution 3D scans of the subject in motion at 60 fps. ClothCap is able to capture a clothed person in motion, extract their clothing, and retarget the clothing to new body shapes; this provides a step towards virtual try-on.

We present an open-source marker-based localization system intended as a low-cost easy-to-deploy solution for aerial and swarm robotics. The main advantage of the presented method is its high computational efficiency, which allows its deployment on small robots with limited computational resources. Even on low-end computers, the core component of the system can detect and estimate 3D positions of hundreds of black and white markers at the maximum frame-rate of standard cameras. The method is robust to changing lighting conditions and achieves accuracy in the order of millimeters to centimeters. Due to its reliability, simplicity of use and availability as an open-source ROS module (http://purl.org/robotics/whycon), the system is now used in a number of aerial robotics projects where fast and precise relative localization is required.

In this paper, we present a novel approach for human action recognition with histograms of 3D joint locations (HOJ3D) as a compact representation of postures. We extract the 3D skeletal joint locations from Kinect depth maps using Shotton et al.'s method [6]. The HOJ3D computed from the action depth sequences are reprojected using LDA and then clustered into k posture visual words, which represent the prototypical poses of actions. The temporal evolutions of those visual words are modeled by discrete hidden Markov models (HMMs). In addition, due to the design of our spherical coordinate system and the robust 3D skeleton estimation from Kinect, our method demonstrates significant view invariance on our 3D action dataset. Our dataset is composed of 200 3D sequences of 10 indoor activities performed by 10 individuals in varied views. Our method is real-time and achieves superior results on the challenging 3D action dataset. We also tested our algorithm on the MSR Action 3D dataset and our algorithm outperforms Li et al. [25] on most of the cases.

We propose UnLoc, an unsupervised indoor localization scheme that bypasses the need for war-driving. Our key observation is that certain locations in an indoor environment present identifiable signatures on one or more sensing dimensions. An elevator, for instance, imposes a distinct pattern on a smartphone's accelerometer; a corridor-corner may overhear a unique set of WiFi access points; a specific spot may experience an unusual magnetic fluctuation. We hypothesize that these kind of signatures naturally exist in the environment, and can be envisioned as internal landmarks of a building. Mobile devices that "sense" these landmarks can recalibrate their locations, while dead-reckoning schemes can track them between landmarks. Results from 3 different indoor settings, including a shopping mall, demonstrate median location errors of 1:69m. War-driving is not necessary, neither are floorplans the system simultaneously computes the locations of users and landmarks, in a manner that they converge reasonably quickly. We believe this is an unconventional approach to indoor localization, holding promise for real-world deployment.

In the last years several direct (i.e. featureless) monocular SLAM approaches have appeared showing impressive semi-dense or dense scene reconstructions. These works have questioned the need of features, in which consolidated SLAM techniques of the last decade were based. In this paper we present a novel feature-based monocular SLAM system that is more robust, gives more accurate camera poses, and obtains comparable or better semi-dense reconstructions than the current state of the art. Our semi-dense mapping operates over keyframes, optimized by local bundle adjustment, allowing to obtain accurate triangulations from wide baselines. Our novel method to search correspondences, the measurement fusion and the inter-keyframe depth consistency tests allow to obtain clean reconstructions with very few outliers. Against the current trend in direct SLAM, our experiments show that by decoupling the semi-dense reconstruction from the trajectory computation, the results obtained are better. This opens the discussion on the benefits of features even if a semi-dense reconstruction is desired.

We present a new real-time hand tracking system based on a single depth camera. The system can accurately reconstruct complex hand poses across a variety of subjects. It also allows for robust tracking, rapidly recovering from any temporary failures. Most uniquely, our tracker is highly flexible, dramatically improving upon previous approaches which have focused on front-facing close-range scenarios. This flexibility opens up new possibilities for human-computer interaction with examples including tracking at distances from tens of centimeters through to several meters (for controlling the TV at a distance), supporting tracking using a moving depth camera (for mobile scenarios), and arbitrary camera placements (for VR headsets). These features are achieved through a new pipeline that combines a multi-layered discriminative reinitialization strategy for per-frame pose estimation, followed by a generative model-fitting stage. We provide extensive technical details and a detailed qualitative and quantitative analysis.

This paper is about the efficient generation of dense, colored models of city-scale environments from range data and in particular, stereo cameras. Better maps make for better understanding; better understanding leads to better robots, but this comes at a cost. The computational and memory requirements of large dense models can be prohibitive. We provide the theory and the system needed to create cityscale dense reconstructions. To do so, we apply a regularizer over a compressed 3D data structure while dealing with the complex boundary conditions this induces during the data-fusion stage. We show that only with these considerations can we swiftly create neat, large, “well behaved” reconstructions. We evaluate our system using the KITTI dataset and provide statistics for the metric errors in all surfaces created compared to those measured with 3D laser. Our regularizer reduces the median error by 40% in 3.4 km of dense reconstructions with a median accuracy of 6 cm. For subjective analysis, we provide a qualitative review of 6.1 km of our dense reconstructions in an attached video. These are the largest dense reconstructions from a single passive camera we are aware of in the literature. Video: https://youtu.be/FRmF7mH86EQ

Human activity understanding with 3D/depth sensors has received increasing attention in multimedia processing and interactions. This work targets on developing a novel deep model for automatic activity recognition from RGB-D videos. We represent each human activity as an ensemble of cubic-like video segments, and learn to discover the temporal structures for a category of activities, i.e. how the activities to be decomposed in terms of classification. Our model can be regarded as a structured deep architecture, as it extends the convolutional neural networks (CNNs) by incorporating structure alternatives. Specifically, we build the network consisting of 3D convolutions and max-pooling operators over the video segments, and introduce the latent variables in each convolutional layer manipulating the activation of neurons. Our model thus advances existing approaches in two aspects: (i) it acts directly on the raw inputs (grayscale-depth data) to conduct recognition instead of relying on hand-crafted features, and (ii) the model structure can be dynamically adjusted accounting for the temporal variations of human activities, i.e. the network configuration is allowed to be partially activated during inference. For model training, we propose an EM-type optimization method that iteratively (i) discovers the latent structure by determining the decomposed actions for each training example, and (ii) learns the network parameters by using the back-propagation algorithm. Our approach is validated in challenging scenarios, and outperforms state-of-the-art methods. A large human activity database of RGB-D videos is presented in addition.

Augmented Reality (AR) based navigation has emerged as an impressive, yet seamless way of guiding users in unknown environments. Its quality of experience depends on many factors, including the accuracy of camera pose estimation, response delay, and energy consumption. In this paper, we present SeeNav - a seamless and energy-efficient AR navigation system for indoor environments. SeeNav combines image-based localization and inertial tracking to provide an accurate and robust camera pose estimation. As vision processing is much more compute intensive than the processing of inertial sensor data, SeeNav offloads the former one from resource-constrained mobile devices to a cloud to improve tracking performance and reduce power consumption. More than that, SeeNav implements a context-aware task scheduling algorithm that further minimizes energy consumption while maintaining the accuracy of camera pose estimation. Our experimental results, including a user study, show that SeeNav provides seamless navigation experience and reduces the overall energy consumption by 21.56% with context-aware task scheduling.

In this paper, we propose 3DLoc, which performs 3-dimensional localization on the tagged objects by using the RFID tag arrays. 3DLoc deploys three arrays of RFID tags on three mutually orthogonal surfaces of each object. When performing 3D localization, 3DLoc continuously moves the RFID antenna and scans the tagged objects in a 2-dimensional space right in front of the tagged objects. It then estimates the object's 3D position according to the phases from the tag arrays. By referring to the fixed layout of the tag array, we use Angle of Arrival-based schemes to accurately estimate the tagged objects' orientation and 3D coordinates in the 3D space. To suppress the localization errors caused by the multipath effect, we use the linear relationship of the AoA parameters to remove the unexpected outliers from the estimated results. We have implemented a prototype system and evaluated the actual performance in the real complex environment. The experimental results show that 3DLoc achieves the mean accuracy of 10cm in free space and 15.3cm in the multipath environment for the tagged object.


Question: 3D ActionSLAM: A system for tracking individuals in multi-level environments using wearable technology.

Addressing the Simultaneous Localization and Mapping (SLAM) Challenge: Achieving Wearable Person Tracking in Multi-Floor Settings

The core inquiry of the simultaneous localization and mapping (SLAM) problem pertains to the feasibility of an autonomous vehicle initiating in an uncharted location within an unfamiliar environment, progressively constructing a map of this terrain while concurrently utilizing this map to ascertain its absolute position. Building upon the estimation-theoretic underpinnings established in prior works [1], [2], [3], this paper substantiates the solvability of the SLAM problem. Initially, the intrinsic structure of the SLAM problem is elucidated. Subsequently, a proof is presented demonstrating the monotonous convergence of the estimated map towards a relative map with absolute certainty. It is further elucidated that the precision of the map and the vehicle's location ultimately attains a lower limit dictated solely by the initial uncertainty of the vehicle's position. These findings collectively affirm the capability of an autonomous vehicle to commence in an uncharted location within an unexplored environment and, utilizing solely relative observations, progressively construct a flawless map of the surroundings while concurrently computing a bounded estimate of its location. The paper also details a comprehensive implementation of the SLAM algorithm on a vehicle navigating an outdoor environment, employing millimeter-wave (MMW) radar for relative map observations. This implementation serves to illustrate the handling of pivotal issues such as map management and data association in a practical setting. The outcomes are cross-validated with absolute locations of the map landmarks obtained through surveying. Ultimately, the paper delves into a range of critical issues raised by the resolution of the SLAM problem, encompassing sub-optimal map-building algorithms and map management strategies.

In this paper we describe a method that estimates the motion of a calibrated camera (settled on an experimental vehicle) and the tridimensional geometry of the environment. The only data used is a video input. In fact, interest points are tracked and matched between frames at video rate. Robust estimates of the camera motion are computed in real-time, key-frames are selected and permit the features 3D reconstruction. The algorithm is particularly appropriate to the reconstruction of long images sequences thanks to the introduction of a fast and local bundle adjustment method that ensures both good accuracy and consistency of the estimated camera poses along the sequence. It also largely reduces computational complexity compared to a global bundle adjustment. Experiments on real data were carried out to evaluate speed and robustness of the method for a sequence of about one kilometer long. Results are also compared to the ground truth measured with a differential GPS.

This paper investigates the differences between 3D level designs in 21 popular games. We have developed a framework to analyze 3D level designs based on patterns extracted from level designers and game play sessions. We then use these patterns to analyze several game play sessions. Results of this analysis reveal methods by which designers push and pull players through levels. We discuss an analysis of these patterns in terms of three level affordance configurations, combat, environmental resistance, and mixed goals in 21 different games using one walkthrough play session per game. By looking at the variety of games, we can further explore the level similarities and differences between games.

We present a new real-time hand tracking system based on a single depth camera. The system can accurately reconstruct complex hand poses across a variety of subjects. It also allows for robust tracking, rapidly recovering from any temporary failures. Most uniquely, our tracker is highly flexible, dramatically improving upon previous approaches which have focused on front-facing close-range scenarios. This flexibility opens up new possibilities for human-computer interaction with examples including tracking at distances from tens of centimeters through to several meters (for controlling the TV at a distance), supporting tracking using a moving depth camera (for mobile scenarios), and arbitrary camera placements (for VR headsets). These features are achieved through a new pipeline that combines a multi-layered discriminative reinitialization strategy for per-frame pose estimation, followed by a generative model-fitting stage. We provide extensive technical details and a detailed qualitative and quantitative analysis.

In this paper we explore the combination of a latest generation mobile device and a micro quadrotor platform to perform indoor autonomous navigation for the purpose of autonomous photography. We use the Yellowstone tablet from Google's Tango project [1], equipped with onboard, fully integrated sensing platform and with significant computational capability. To the best of our knowledge we are the first to exploit the Google's Tango tablet as source of pose estimate to control the quadrotor's motion. Using the tablet's onboard camera the system is able to detect people and generate a desired pose that the quadrotor will have to reach in order to take a well framed picture of the detected subject. The experimental results and live video demonstrate the capabilities of the autonomous flying robot photographer using the system described throughout this manuscript.

We present an open-source marker-based localization system intended as a low-cost easy-to-deploy solution for aerial and swarm robotics. The main advantage of the presented method is its high computational efficiency, which allows its deployment on small robots with limited computational resources. Even on low-end computers, the core component of the system can detect and estimate 3D positions of hundreds of black and white markers at the maximum frame-rate of standard cameras. The method is robust to changing lighting conditions and achieves accuracy in the order of millimeters to centimeters. Due to its reliability, simplicity of use and availability as an open-source ROS module (http://purl.org/robotics/whycon), the system is now used in a number of aerial robotics projects where fast and precise relative localization is required.

The present study describes the development of a triaxial accelerometer (TA) and a portable data processing unit for the assessment of daily physical activity. The TA is composed of three orthogonally mounted uniaxial piezoresistive accelerometers and can be used to register accelerations covering the amplitude and frequency ranges of human body acceleration. Interinstrument and test-retest experiments showed that the offset and the sensitivity of the TA were equal for each measurement direction and remained constant on two measurement days. Transverse sensitivity was significantly different for each measurement direction, but did not influence accelerometer output (<3% of the sensitivity along the main axis). The data unit enables the on-line processing of accelerometer output to a reliable estimator of physical activity over eight-day periods. Preliminary evaluation of the system in 13 male subjects during standardized activities in the laboratory demonstrated a significant relationship between accelerometer output and energy expenditure due to physical activity, the standard reference for physical activity (r=0.89). Shortcomings of the system are its low sensitivity to sedentary activities and the inability to register static exercise. The validity of the system for the assessment of normal daily physical activity and specific activities outside the laboratory should be studied in free-living subjects.

Smartphone pedometry offers the possibility of ubiquitous health monitoring, context awareness and indoor location tracking through Pedestrian Dead Reckoning (PDR) systems. However, there is currently no detailed understanding of how well pedometry works when applied to smartphones in typical, unconstrained use.
 This paper evaluates common walk detection (WD) and step counting (SC) algorithms applied to smartphone sensor data. Using a large dataset (27 people, 130 walks, 6 smartphone placements) optimal algorithm parameters are provided and applied to the data. The results favour the use of standard deviation thresholding (WD) and windowed peak detection (SC) with error rates of less than 3%. Of the six different placements, only the back trouser pocket is found to degrade the step counting performance significantly, resulting in undercounting for many algorithms.

Both simultaneous localization and mapping (SLAM) and detection and tracking of moving objects (DTMO) play key roles in robotics and automation. For certain constrained environments, SLAM and DTMO are becoming solved problems. But for robots working outdoors, and at high speeds, SLAM and DTMO are still incomplete. In earlier works, SLAM and DTMO are treated as two separate problems. In fact, they can be complementary to one another. In this paper, we present a new method to integrate SLAM and DTMO to solve both problems simultaneously for both indoor and outdoor applications. The results of experiments carried out with CMU Navlab8 and Navlab11 vehicles with the maximum speed of 45 mph in crowded urban and suburban areas verify the described work.

Three-dimensional hand gesture recognition has attracted increasing research interests in computer vision, pattern recognition, and human-computer interaction. The emerging depth sensors greatly inspired various hand gesture recognition approaches and applications, which were severely limited in the 2D domain with conventional cameras. This paper presents a survey of some recent works on hand gesture recognition using 3D depth sensors. We first review the commercial depth sensors and public data sets that are widely used in this field. Then, we review the state-of-the-art research for 3D hand gesture recognition in four aspects: 1) 3D hand modeling; 2) static hand gesture recognition; 3) hand trajectory gesture recognition; and 4) continuous hand gesture recognition. While the emphasis is on 3D hand gesture recognition approaches, the related applications and typical systems are also briefly summarized for practitioners.

In this paper a novel evaluation framework for measuring the performance of real-time action recognition methods is presented. The evaluation framework will extend the time-based event detection metric to model multiple distinct action classes. The proposed metric provides more accurate indications of the performance of action recognition algorithms for games and other similar applications since it takes into consideration restrictions related to time and consecutive repetitions. Furthermore, a new dataset, G3D for real-time action recognition in gaming containing synchronised video, depth and skeleton data is provided. Our results indicate the need of an advanced metric especially designed for games and other similar real-time applications.


Question: 3D ActionSLAM: a wearable system for person tracking in multi-level indoor spaces.

This paper investigates the efficacy of common walk detection (WD) and step counting (SC) algorithms when applied to smartphone sensor data in unconstrained environments, addressing the gap in understanding their performance under typical usage scenarios. With the growing relevance of smartphone pedometry for health monitoring, context awareness, and indoor location tracking through Pedestrian Dead Reckoning (PDR) systems, it is crucial to assess the accuracy of these algorithms. Utilizing an extensive dataset comprising 27 participants, 130 walks, and six different smartphone placements, the study optimizes algorithm parameters and applies them to the collected data. The findings highlight the superiority of standard deviation thresholding for walk detection and windowed peak detection for step counting, both demonstrating error rates below 3%. Among the six placements evaluated, the back trouser pocket emerges as the most challenging, leading to significant undercounting in several algorithms. This research underscores the importance of considering smartphone placement in the design and deployment of pedometry systems, particularly for applications requiring high accuracy in step counting.

Accurate recognition and tracking of human activities is an important goal of ubiquitous computing. Recent advances in the development of multi-modal wearable sensors enable us to gather rich datasets of human activities. However, the problem of automatically identifying the most useful features for modeling such activities remains largely unsolved. In this paper we present a hybrid approach to recognizing activities, which combines boosting to discriminatively select useful features and learn an ensemble of static classifiers to recognize different activities, with hidden Markov models (HMMs) to capture the temporal regularities and smoothness of activities. We tested the activity recognition system using over 12 hours of wearable-sensor data collected by volunteers in natural unconstrained environments. The models succeeded in identifying a small set of maximally informative features, and were able identify ten different human activities with an accuracy of 95%.

Monitoring of physical activities is a growing field with potential applications such as lifecare and healthcare. Accelerometry shows promise in providing an inexpensive but effective means of long-term activity monitoring of elderly patients. However, even for the same physical activity the output of any body-worn Triaxial Accelerometer (TA) varies at different positions of a subject's body, resulting in a high within-class variance. Thus almost all existing TA-based human activity recognition systems require firm attachment of TA to a specific body part, making them impractical for long-term activity monitoring during unsupervised free living. Therefore, we present a novel hierarchical recognition model that can recognize human activities independent of TA's position along a human body. The proposed model minimizes the high within-class variance significantly and allows subjects to carry TA freely in any pocket without attaching it firmly to a body-part. We validated our model using six daily physical activities: resting (sit/stand), walking, walk-upstairs, walk-downstairs, running, and cycling. Activity data is collected from four most probable body positions of TA: chest pocket, front trousers pocket, rear trousers pocket, and inner jacket pocket. The average accuracy of about 95% illustrates the effectiveness of the proposed method.

Autonomous microhelicopters will soon play a major role in tasks like search and rescue, environment monitoring, security surveillance, and inspection. If they are further realized in small scale, they can also be used in narrow outdoor and indoor environments and represent only a limited risk for people. However, for such operations, navigating based only on global positioning system (GPS) information is not sufficient. Fully autonomous operation in cities or other dense environments requires microhelicopters to fly at low altitudes, where GPS signals are often shadowed, or indoors and to actively explore unknown environments while avoiding collisions and creating maps. This involves a number of challenges on all levels of helicopter design, perception, actuation, control, and navigation, which still have to be solved. The Swarm of Micro Flying Robots (SFLY) project was a European Union-funded project with the goal of creating a swarm of vision-controlled microaerial vehicles (MAVs) capable of autonomous navigation, three-dimensional (3-D) mapping, and optimal surveillance coverage in GPS-denied environments. The SFLY MAVs do not rely on remote control, radio beacons, or motion-capture systems but can fly all by themselves using only a single onboard camera and an inertial measurement unit (IMU). This article describes the technical challenges that have been faced and the results achieved from hardware design and embedded programming to vision-based navigation and mapping, with an overview of how all the modules work and how they have been integrated into the final system. Code, data sets, and videos are publicly available to the robotics community. Experimental results demonstrating three MAVs navigating autonomously in an unknown GPS-denied environment and performing 3-D mapping and optimal surveillance coverage are presented.

Estimating the ground plane is often one of the first steps in geometric reasoning processes as it offers easily accessible context knowledge. Especially unconstrained platforms that capture video from egocentric viewpoints can benefit from such knowledge in various ways. A key requirement here is keeping orientation, which can be greatly achieved by keeping track of the ground. We present an approach to keep track of the ground plane in cluttered inner-urban environments using stereo vision in real-time. We fuse a planar model fit in low-resolution disparity data with the direction of the vertical vanishing point. Our experiments show how this effectively decreases the error of plane attitude estimation compared to classic least-squares fitting and allows to track the plane with camera configurations in which the ground is not visible. We evaluate the approach using ground-truth from an inertial measurement unit and demonstrate long-term stability on a dataset of challenging inner city scenes.

The use of wearable cameras makes it possible to record life logging egocentric videos. Browsing such long unstructured videos is time consuming and tedious. Segmentation into meaningful chapters is an important first step towards adding structure to egocentric videos, enabling efficient browsing, indexing and summarization of the long videos. Two sources of information for video segmentation are (i) the motion of the camera wearer, and (ii) the objects and activities recorded in the video. In this paper we address the motion cues for video segmentation. Motion based segmentation is especially difficult in egocentric videos when the camera is constantly moving due to natural head movement of the wearer. We propose a robust temporal segmentation of egocentric videos into a hierarchy of motion classes using a new Cumulative Displacement Curves. Unlike instantaneous motion vectors, segmentation using integrated motion vectors performs well even in dynamic and crowded scenes. No assumptions are made on the underlying scene structure and the method works in indoor as well as outdoor situations. We demonstrate the effectiveness of our approach using publicly available videos as well as choreographed videos. We also suggest an approach to detect the fixation of wearer's gaze in the walking portion of the egocentric videos.

This paper proposes an algorithm for fall detection using a ceiling-mounted 3D depth camera. The lying pose is separated from common daily activities by a k-NN classifier, which was trained on features expressing head-floor distance, person area and shape's major length to width. In order to distinguish between intentional lying postures and accidental falls the algorithm also employs motion between static postures. The experimental validation of the algorithm was conducted on realistic depth image sequences of daily activities and simulated falls. It was evaluated on more than 45000 depth images and gave 0% error. To reduce the processing overload an accelerometer was used to indicate the potential impact of the person and to start an analysis of depth images.

Understanding natural human activity involves not only identifying the action being performed, but also locating the semantic elements of the scene and describing the person's interaction with them. We present a system that is able to recognize complex, fine-grained human actions involving the manipulation of objects in realistic action sequences. Our method takes advantage of recent advances in sensors and pose trackers in learning an action model that draws on successful discriminative techniques while explicitly modeling both pose trajectories and object manipulations. By combining these elements in a single model, we are able to simultaneously recognize actions and track the location and manipulation of objects. To showcase this ability, we introduce a novel Cooking Action Dataset that contains video, depth readings, and pose tracks from a Kinect sensor. We show that our model outperforms existing state of the art techniques on this dataset as well as the VISINT dataset with only video sequences.

This paper presents a novel algorithm that utilizes a 2D floorplan to align panorama RGBD scans. While effective panorama RGBD alignment techniques exist, such a system requires extremely dense RGBD image sampling. Our approach can significantly reduce the number of necessary scans with the aid of a floorplan image. We formulate a novel Markov Random Field inference problem as a scan placement over the floorplan, as opposed to the conventional scan-to-scan alignment. The technical contributions lie in multi-modal image correspondence cues (between scans and schematic floorplan) as well as a novel coverage potential avoiding an inherent stacking bias. The proposed approach has been evaluated on five challenging large indoor spaces. To the best of our knowledge, we present the first effective system that utilizes a 2D floorplan image for building-scale 3D pointcloud alignment. The source code and the data are shared with the community to further enhance indoor mapping research.

Inertial navigation based on micro-electromechanical system (MEMS) inertial measurement units (IMUs) has attracted numerous researchers due to its high reliability and independence. The heading estimation, as one of the most important parts of inertial navigation, has been a research focus in this field. Heading estimation using magnetometers is perturbed by magnetic disturbances, such as indoor concrete structures and electronic equipment. The MEMS gyroscope is also used for heading estimation. However, the accuracy of gyroscope is unreliable with time. In this paper, a wearable multi-sensor system has been designed to obtain the high-accuracy indoor heading estimation, according to a quaternion-based unscented Kalman filter (UKF) algorithm. The proposed multi-sensor system including one three-axis accelerometer, three single-axis gyroscopes, one three-axis magnetometer and one microprocessor minimizes the size and cost. The wearable multi-sensor system was fixed on waist of pedestrian and the quadrotor unmanned aerial vehicle (UAV) for heading estimation experiments in our college building. The results show that the mean heading estimation errors are less 10° and 5° to multi-sensor system fixed on waist of pedestrian and the quadrotor UAV, respectively, compared to the reference path.

This paper focuses on motion estimation using inertial measurements and observations of naturally occurring point features. To date, this task has primarily been addressed using filtering methods, which track the system state starting from known initial conditions. However, when no prior knowledge of the initial system state is available, (e.g., at the onset of the system's operation), the existing approaches are not applicable. To address this problem, in this work we present algorithms for computing the system's observable quantities (platform attitude and velocity, feature positions, and IMU-camera calibration) directly from the sensor measurements, without any prior knowledge. A key contribution of this work is a convex-optimization based algorithm for computing the rotation matrix between the camera and IMU. We show that once this rotation matrix has been computed, all remaining quantities can be determined by solving a quadratically constrained least-squares problem. To increase their accuracy, the initial estimates are refined by an iterative maximum-likelihood estimator.


Question: 3D ActionSLAM: a method for tracking individuals in multi-level settings with wearable technology.

This study delves into the optimization of activity recognition in wearable computing environments, focusing on the pivotal role of human actions as a core component of contextual data. Activities like walking, standing, and sitting can be accurately identified from information gathered by body-mounted acceleration sensors. Contrary to conventional methods that apply a uniform set of features for all activities, we advocate for a tailored feature selection approach for each specific activity. Through a detailed analysis of features derived from real-world datasets, we elucidate the impact of feature choice and the duration of the computation window on the recognition accuracy of diverse activities. Our findings culminate in a practical guide recommending optimal features and window lengths for a range of standard activities, thereby enhancing the precision of activity recognition in wearable computing scenarios.

JackIn is a new human-human communication framework for connecting two or more people. With first-person view video streaming from a person (called Body) wearing a transparent head-mounted display and a head-mounted camera, the other person (called Ghost) participates in the shared first-person view. With JackIn, people's activities can be shared and assistance or guidance can be given through other peoples expertise. This can be applied to daily activities such as cooking lessons, shopping navigation, education in craft-work or electrical work, and sharing experiences of sporting and live events. For a better viewing experience with frist-person view, we developed the out-of-body view in which first-person images are integrated to construct a scene around a Body, and a Ghost can virtually control the viewpoint to look around the space surrounding the Body. We also developed a tele-pointing gesture interface. We conducted an experiment to evaluate how effective this framework is and found that Ghosts can understand the spatial situation of the Body.

The massive diffusion of smartphones, the growing interest in wearable devices and the Internet of Things, and the exponential rise of location based services (LBSs) have made the problem of localization and navigation inside buildings one of the most important technological challenges of recent years. Indoor positioning systems have a huge market in the retail sector and contextual advertising; in addition, they can be fundamental to increasing the quality of life for citizens if deployed inside public buildings such as hospitals, airports, and museums. Sometimes, in emergency situations, they can make the difference between life and death. Various approaches have been proposed in the literature. Recently, thanks to the high performance of smartphones’ cameras, marker-less and marker-based computer vision approaches have been investigated. In a previous paper, we proposed a technique for indoor localization and navigation using both Bluetooth low energy (BLE) and a 2D visual marker system deployed into the floor. In this paper, we presented a qualitative performance evaluation of three 2D visual markers, Vuforia, ArUco marker, and AprilTag, which are suitable for real-time applications. Our analysis focused on specific case study of visual markers placed onto the tiles, to improve the efficiency of our indoor localization and navigation approach by choosing the best visual marker system.

We propose a new method for the localization of a Micro Aerial Vehicle (MAV) with respect to a ground robot. We solve the problem of registering the 3D maps computed by the robots using different sensors: a dense 3D reconstruction from the MAV monocular camera is aligned with the map computed from the depth sensor on the ground robot. Once aligned, the dense reconstruction from the MAV is used to augment the map computed by the ground robot, by extending it with the information conveyed by the aerial views. The overall approach is novel, as it builds on recent developments in live dense reconstruction from moving cameras to address the problem of air-ground localization. The core of our contribution is constituted by a novel algorithm integrating dense reconstructions from monocular views, Monte Carlo localization, and an iterative pose refinement. In spite of the radically different vantage points from which the maps are acquired, the proposed method achieves high accuracy whereas appearance-based, state-of-the-art approaches fail. Experimental validation in indoor and outdoor scenarios reported an accuracy in position estimation of 0.08 meters and real time performance. This demonstrates that our new approach effectively overcomes the limitations imposed by the difference in sensors and vantage points that negatively affect previous techniques relying on matching visual features.

We present a new real-time hand tracking system based on a single depth camera. The system can accurately reconstruct complex hand poses across a variety of subjects. It also allows for robust tracking, rapidly recovering from any temporary failures. Most uniquely, our tracker is highly flexible, dramatically improving upon previous approaches which have focused on front-facing close-range scenarios. This flexibility opens up new possibilities for human-computer interaction with examples including tracking at distances from tens of centimeters through to several meters (for controlling the TV at a distance), supporting tracking using a moving depth camera (for mobile scenarios), and arbitrary camera placements (for VR headsets). These features are achieved through a new pipeline that combines a multi-layered discriminative reinitialization strategy for per-frame pose estimation, followed by a generative model-fitting stage. We provide extensive technical details and a detailed qualitative and quantitative analysis.

Activity recognition fits within the bigger framework of context awareness. In this paper, we report on our efforts to recognize user activity from accelerometer data. Activity recognition is formulated as a classification problem. Performance of base-level classifiers and meta-level classifiers is compared. Plurality Voting is found to perform consistently well across different settings.

This paper presents a tracking system for ego-motion estimation which fuses vision and inertial measurements using EKF and UKF (Extended and Unscented Kalman Filters), where a comparison of their performance has been done. It also considers the multi-rate nature of the sensors: inertial sensing is sampled at a fast sampling frequency while the sampling frequency of vision is lower. the proposed approach uses a constant linear acceleration model and constant angular velocity model based on quaternions, which yields a non-linear model for states and a linear model in measurement equations. Results show that a significant improvement is obtained on the estimation when fusing both measurements with respect to just vision or just inertial measurements. It is also shown that the proposed system can estimate fast-motions even when vision system fails. Moreover, a study of the influence of the noise covariance is also performed, which aims to select their appropriate values at the tuning process. The setup is an end-effector mounted camera, which allow us to pre-define basic rotational and translational motions for validating results. KEY WORDS—vision and inertial, multi-rate systems, sensor fusion, tracking

We propose UnLoc, an unsupervised indoor localization scheme that bypasses the need for war-driving. Our key observation is that certain locations in an indoor environment present identifiable signatures on one or more sensing dimensions. An elevator, for instance, imposes a distinct pattern on a smartphone's accelerometer; a corridor-corner may overhear a unique set of WiFi access points; a specific spot may experience an unusual magnetic fluctuation. We hypothesize that these kind of signatures naturally exist in the environment, and can be envisioned as internal landmarks of a building. Mobile devices that "sense" these landmarks can recalibrate their locations, while dead-reckoning schemes can track them between landmarks. Results from 3 different indoor settings, including a shopping mall, demonstrate median location errors of 1:69m. War-driving is not necessary, neither are floorplans the system simultaneously computes the locations of users and landmarks, in a manner that they converge reasonably quickly. We believe this is an unconventional approach to indoor localization, holding promise for real-world deployment.

Inspired by the emerging WiFi-based applications, in this paper, we leverage ubiquitous WiFi signals and propose a device-free step counting system, called WiStep. Based on the multipath propagation model, when a person is walking, her torso and limbs move at different speeds, which modulates wireless signals to the propagation paths with different lengths and thus introduces different frequency components into the received Channel State Information (CSI). To count walking steps, we first utilize time-frequency analysis techniques to segment and recognize the walking movement, and then dynamically select the sensitive subcarriers with largest amplitude variances from multiple CSI streams. Wavelet decomposition is applied to extract the detail coefficients corresponding to the frequencies induced by feet or legs, and compress the data so as to improve computing speed. Short-time energy of the coefficients is then calculated as the metric for step counting. Finally, we combine the results derived from the selected subcarriers to produce a reliable step count estimation. In contrast to counting steps based on the torso frequency analysis, WiStep can count the steps of in-place walking even when the person's torso speed is null. We implement WiStep on commodity WiFi devices in two different indoor scenarios, and various influence factors are taken into consideration when evaluating the performance of WiStep. The experimental results demonstrate that WiStep can realize overall step counting accuracies of 90.2% and 87.59% respectively in these two scenarios, and it is resilient to the change of scenarios.

This paper proposes an algorithm for fall detection using a ceiling-mounted 3D depth camera. The lying pose is separated from common daily activities by a k-NN classifier, which was trained on features expressing head-floor distance, person area and shape's major length to width. In order to distinguish between intentional lying postures and accidental falls the algorithm also employs motion between static postures. The experimental validation of the algorithm was conducted on realistic depth image sequences of daily activities and simulated falls. It was evaluated on more than 45000 depth images and gave 0% error. To reduce the processing overload an accelerometer was used to indicate the potential impact of the person and to start an analysis of depth images.

In this work, a system for recognizing activities in the home setting using a set of small and simple state-change sensors is introduced. The sensors are designed to be “tape on and forget” devices that can be quickly and ubiquitously installed in home environments. The proposed sensing system presents an alternative to sensors that are sometimes perceived as invasive, such as cameras and microphones. Unlike prior work, the system has been deployed in multiple residential environments with non-researcher occupants. Preliminary results on a small dataset show that it is possible to recognize activities of interest to medical professionals such as toileting, bathing, and grooming with detection accuracies ranging from 25% to 89% depending on the evaluation criteria used .