Question: A mobile robot's localization and map construction utilizing an RFID sensor fusion framework.

This paper presents an innovative approach to mobile robot localization in indoor environments using Monte Carlo Localization (MCL). Reliable position estimation is critical for mobile robotics, and probabilistic methods, particularly those based on particle-based density representations, offer promising solutions to the localization challenge. Traditional methods, however, often struggle with limitations related to the representation of probability densities over the robot's state space. The MCL method, as introduced herein, addresses these issues by maintaining a set of samples randomly drawn from the probability density, enabling the representation of complex distributions. Experimental results demonstrate that MCL is capable of efficiently localizing a mobile robot, even without prior knowledge of its initial position. This method outperforms previous grid-based techniques in terms of speed, accuracy, and memory usage, making it a significant advancement in the field of mobile robot localization. Additionally, the integration of RFID sensor fusion systems could further enhance the accuracy and robustness of MCL, offering a promising direction for future research in mobile robot navigation and map-building.

In this paper the possibilities that Matlab provides to design, implementation and monitoring programs of autonomous navigation for mobile robots, on both simulated and real platforms, through its new toolbox for robotics will be explored. Robotics System Toolbox has established itself as a solid tool of integration between Matlab and robot operating under ROS environment. It will be studied the tools available in this toolbox that allow achieving the connection between the two platforms, in addition to the generation of algorithms of location, path planning, mapping and autonomous navigation.

The emergence of Mobile Mapping Systems (MMS) has set a marked paradigm in the photogrammetric and mapping community that has not only facilitated comprehensive 3D mapping of different environments but has also paved way for new aspects of applied research in this direction. Out of the many essential blocks that make these MMS a viable tool for mapping, the positioning and orientation module is considered to be a crucial yet an expensive component. The integration of such a module with mapping sensors has allowed for the extensive implementation of such systems to provide high-quality maps. However, while such systems do not lack in system robustness and performance in general, the deployment of these systems is restricted to applications and environments where a consistent availability of GNSS signals is assured. Extending these MMS to GNSS-denied areas, such as indoor environments, is therefore quite challenging and necessitates the development of an alternative module that can act as a viable substitute to GNSS/INS for system operation without having to resort to an exhaustive modification of the same to function in GNSS-denied locations. In this research, such a case has been considered for the implementation of an indoor MMS using an Unmanned Ground Vehicle (UGV) and a 3D laser scanner for the task of generating high density maps of GNSS-denied indoor areas. To mitigate the absence of GNSS data, this paper proposes a Pseudo-GNSS/INS module integrated framework which utilizes probabilistic Simultaneous Localization and Mapping (SLAM) techniques to estimate the platform pose and heading from 3D laser scanner data. This proposed framework has been implemented based on three major notions: (i) using geometric methods for sparse point cloud extraction to carry out real-time SLAM, (ii) generating position data and geo-referencing signals from these real-time SLAM pose estimates, and (iii) carrying out the entire operation through use of a single 3D mapping sensor. The final geo-referenced point cloud can then be generated through post-processing by the Iterative Closest Projected Point (ICPP) registration technique which also diminishes the effect of sensor measurement noise. The implementation, performance and results of the proposed MMS framework for an indoor mapping system have been presented in this paper that demonstrate the ability of this Pseudo-GNSS/INS framework to operate flexibly in GNSS-denied areas.

Wireless Sensor network (WSN) is an emerging technology and has great potential to be employed in critical situations. The development of wireless sensor networks was originally motivated by military applications like battlefield surveillance. However, Wireless Sensor Networks are also used in many areas such as Industrial, Civilian, Health, Habitat Monitoring, Environmental, Military, Home and Office application areas. Detection and tracking of targets (eg. animal, vehicle) as it moves through a sensor network has become an increasingly important application for sensor networks. The key advantage of WSN is that the network can be deployed on the fly and can operate unattended, without the need for any pre-existing infrastructure and with little maintenance. The system will estimate and track the target based on the spatial differences of the target signal strength detected by the sensors at different locations. Magnetic and acoustic sensors and the signals captured by these sensors are of present interest in the study. The system is made up of three components for detecting and tracking the moving objects. The first component consists of inexpensive off-the shelf wireless sensor devices, such as MicaZ motes, capable of measuring acoustic and magnetic signals generated by vehicles. The second component is responsible for the data aggregation. The third component of the system is responsible for data fusion algorithms. This paper inspects the sensors available in the market and its strengths and weakness and also some of the vehicle detection and tracking algorithms and their classification. This work focuses the overview of each algorithm for detection and tracking and compares them based on evaluation parameters.

Occupancy grid maps provide a reliable vehicle environmental model and usually process data from range finding sensors. Object grid maps additionally contain information about the classes of objects, which is crucial for applications like autonomous driving. Unfortunately, they lack the precision of occupancy grid maps, since they mostly process classification results from camera data by projecting the corresponding images onto the ground plane. This paper proposes a modular framework to create precise object grid maps. The presented algorithm creates classical occupancy grid maps and object grid maps. In a combination step, it transforms both maps into the same frame of discernment based on the Dempster-Shafer theory of evidence. This allows fusing the maps to one object grid map, which contains valuable object information and at the same time benefits from the precision of the occupancy grid map.

For applications such as augmented reality, autonomous driving, self-localization/camera pose estimation and scene parsing are crucial technologies. In this paper, we propose a unified framework to tackle these two problems simultaneously. The uniqueness of our design is a sensor fusion scheme which integrates camera videos, motion sensors (GPS/IMU), and a 3D semantic map in order to achieve robustness and efficiency of the system. Specifically, we first have an initial coarse camera pose obtained from consumer-grade GPS/IMU, based on which a label map can be rendered from the 3D semantic map. Then, the rendered label map and the RGB image are jointly fed into a pose CNN, yielding a corrected camera pose. In addition, to incorporate temporal information, a multi-layer recurrent neural network (RNN) is further deployed improve the pose accuracy. Finally, based on the pose from RNN, we render a new label map, which is fed together with the RGB image into a segment CNN which produces perpixel semantic label. In order to validate our approach, we build a dataset with registered 3D point clouds and video camera images. Both the point clouds and the images are semantically-labeled. Each video frame has ground truth pose from highly accurate motion sensors. We show that practically, pose estimation solely relying on images like PoseNet [25] may fail due to street view confusion, and it is important to fuse multiple sensors. Finally, various ablation studies are performed, which demonstrate the effectiveness of the proposed system. In particular, we show that scene parsing and pose estimation are mutually beneficial to achieve a more robust and accurate system.

In the supply chain, a warehouse is an essential component for linking the chain partners. It is necessary to allocate warehouse resources efficiently and effectively to enhance the productivity and reduce the operation costs of the warehouse. Therefore, warehouse management systems (WMSs) have been developed for handling warehouse resources and monitoring warehouse operations. However, it is difficult to update daily operations of inventory level, locations of forklifts and stock keeping units (SKUs) in realtime by using the bar-code-based or manual-based warehouse management systems. In this paper, RFID technology is adopted to facilitate the collection and sharing of data in a warehouse. Tests are performed for evaluating the reading performance of both the active and passive RFID apparatus. With the help of the testing results, the efficient radio frequency cover ranges of the readers are examined for formulating a radio frequency identification case-based logistics resource management system (R-LRMS). The capabilities of R-LRMS are demonstrated in GSL Limited. Three objectives are achieved: (i) a simplification of RFID adoption procedure, (ii) an improvement in the visibility of warehouse operations and (iii) an enhancement of the productivity of the warehouse. The successful case example proved the feasibility of R-LRMS in real working practice. 2008 Elsevier Ltd. All rights reserved.

In this paper indoor localization system based on the RF power measurements of the Received Signal Strength (RSS) in WLAN environment is presented. Today, the most viable solution for localization is the RSS fingerprinting based approach, where in order to establish a relationship between RSS values and location, different machine learning approaches are used. The advantage of this approach based on WLAN technology is that it does not need new infrastructure (it reuses already and widely deployed equipment), and the RSS measurement is part of the normal operating mode of wireless equipment. We derive the Cramér-Rao Lower Bound (CRLB) of localization accuracy for RSS measurements. In analysis of the bound we give insight in localization performance and deployment issues of a localization system, which could help designing an efficient localization system. To compare different machine learning approaches we developed a localization system based on an artificial neural network, k-nearest neighbors, probabilistic method based on the Gaussian kernel and the histogram method. We tested the developed system in real world WLAN indoor environment, where realistic RSS measurements were collected. Experimental comparison of the results has been investigated and average location estimation error of around 2 meters was obtained.

Odometry-like localization solutions can be built upon Light Detection And Ranging (LIDAR) sensors, by sequentially registering the point clouds acquired along a robot trajectory. Yet such solutions inherently drift over time: we propose an approach that adds a graphical model layer on top of such LIDAR odometry layer, using the Iterative Closest Points (ICP) algorithm for registration. Reference frames called keyframes are defined along the robot trajectory, and ICP results are used to build a pose graph, that in turn is used to solve an optimization problem that provides updates for the keyframes upon loop closing, enabling the correction of the path of the robot and of the map of the environment. We present in details the configuration used to register data from the Velodyne High Definition LIDAR (HDL), and a strategy to build local maps upon which current frames are registered, either when discovering new areas or revisiting previously mapped areas. Experiments show that it is possible to build the graph using data from ICP and that the loop closings in the graph level reduces the overall drift of the system.

Knowledge of a person’s whereabouts in the home is key to context-aware applications, but many people do not want to carry or wear a tag or mobile device in the home. Therefore, many tracking systems are now using so-called weak biometrics such as height, weight, and width. In this paper, we propose to use body shape as a weak biometric, differentiating people based on features such as head size, shoulder size, or torso size. The basic idea is to scan the body with a radar sensor and to compute the reflection profile: the amount of energy that reflects back from each part of the body. Many people have different body shapes even if they have the same height, weight, or width, which makes body shape a stronger biometric. We built a proof-of-concept system called FormaTrack to test this approach, and evaluate using eight participants of varying height and weight. We collected over 2800 observations while capturing a wide range of factors such as clothing, hats, shoes, and backpacks. Results show that FormaTrack can achieve a precision, recall, direction and identity accuracy (over all possible groups of 2 people) of 100%, 99.86%, 99.7% and 95.3% respectively. Results indicate that FormaTrack can achieve over 99% tracking accuracy with 2 people in a home with 5 or more rooms.

RSSI-based indoor localization is getting much attention. Thanks to a number of researchers, the localization accuracy has already reached a sufficient level. However, it is still not easy-to-use technology because of its heavy installation cost. When an indoor localization system is installed, it needs to collect RSSI data for training classifiers. Existing techniques need to collect enough data at each location. This is why the installation cost is very heavy. We propose a technique to gather data efficiently by using machine learning techniques. Our proposed algorithm is based on multi-task learning and Bayesian optimization. This algorithm can remove the need to collect data of all location labels and select location labels to acquire new data efficiently. We verify this algorithm by using a Wi-Fi RSSI dataset collected in a building. The empirical results suggest that the algorithm is superior to an existing algorithm applying single-task learning and Active Class Selection.


Question: A mobile robot's localization and map construction utilizing an RFID sensor fusion framework.

This document introduces an innovative algorithm for mobile robot localization, known as Monte Carlo Localization (MCL). MCL is a variant of Markov localization, a class of probabilistic methods that have recently achieved significant practical success in the field. However, earlier methods were either computationally intensive (such as grid-based techniques that model the state space using high-resolution 3D grids), or they had to adopt a very low resolution to manage computational demands. Our proposed algorithm, MCL, is both computationally efficient and capable of representing complex distributions with a high degree of accuracy. MCL utilizes sampling-based techniques to approximate probability distributions, allocating computational resources to areas of high importance. The number of samples is dynamically adjusted based on the current situation, ensuring that large sample sets are only utilized when necessary. Experimental results demonstrate that MCL offers enhanced precision while significantly reducing computational requirements by an order of magnitude compared to previous methods, making it much simpler to implement and deploy.

Radio-frequency identification (RFID) is a wireless data capturing technique that utilizes radio frequency (RF) waves for automatic identification of objects. RFID relies on RF waves for data transmission between the data carrying device, called the RFID tag, and the interrogator.

The massive diffusion of smartphones, the growing interest in wearable devices and the Internet of Things, and the exponential rise of location based services (LBSs) have made the problem of localization and navigation inside buildings one of the most important technological challenges of recent years. Indoor positioning systems have a huge market in the retail sector and contextual advertising; in addition, they can be fundamental to increasing the quality of life for citizens if deployed inside public buildings such as hospitals, airports, and museums. Sometimes, in emergency situations, they can make the difference between life and death. Various approaches have been proposed in the literature. Recently, thanks to the high performance of smartphones’ cameras, marker-less and marker-based computer vision approaches have been investigated. In a previous paper, we proposed a technique for indoor localization and navigation using both Bluetooth low energy (BLE) and a 2D visual marker system deployed into the floor. In this paper, we presented a qualitative performance evaluation of three 2D visual markers, Vuforia, ArUco marker, and AprilTag, which are suitable for real-time applications. Our analysis focused on specific case study of visual markers placed onto the tiles, to improve the efficiency of our indoor localization and navigation approach by choosing the best visual marker system.

We present Travi-Navi - a vision-guided navigation system that enables a self-motivated user to easily bootstrap and deploy indoor navigation services, without comprehensive indoor localization systems or even the availability of floor maps. Travi-Navi records high quality images during the course of a guider's walk on the navigation paths, collects a rich set of sensor readings, and packs them into a navigation trace. The followers track the navigation trace, get prompt visual instructions and image tips, and receive alerts when they deviate from the correct paths. Travi-Navi also finds the most efficient shortcuts whenever possible. We encounter and solve several challenges, including robust tracking, shortcut identification, and high quality image capture while walking. We implement Travi-Navi and conduct extensive experiments. The evaluation results show that Travi-Navi can track and navigate users with timely instructions, typically within a 4-step offset, and detect deviation events within 9 steps.

Indoor localization has long been a goal of pervasive computing research. In this paper, we explore the possibility of determining user's location based on the camera images received from a smart phone. In our system, the smart phone is worn by the user as a pendant and images are periodically captured and transmitted over GPRS to a web server. The web server returns the location of the user by comparing the received images with images stored in a database. We tested our system inside the Computer Science department building. Preliminary results show that user's location can be determined correctly with more than 80% probability of success. As opposed to earlier solutions for indoor localization, this approach does not have any infrastructure requirements. The only cost is that of building an image database.

With the rapid progress in the development of wireless technology, Global Positioning System (GPS) based vehicle navigation systems are being widely deployed in automobiles to serve the location-based needs of users and for efficient traffic management. An essential process in vehicle navigation is to map match the position obtained from GPS (or/and other sensors) on a road network map. This process of map matching in turn helps in mitigating errors from navigation solution. GPS based vehicle navigation systems have difficulties in tracking vehicles in urban canyons due to poor satellite availability. High Sensitivity GPS (HS GPS) receivers can alleviate this problem by acquiring and tracking weak signals (and increasing the availability), but at the cost of high measurement noise and errors due to multipath and cross correlation. Position and velocity results in such conditions are typically biased and have unknown distributions. Thus filtering and other statistical methods are difficult to implement. Soft computing has replaced classical computing on many fronts where uncertainties are difficult to model. Fuzzy logic, based on fuzzy reasoning concepts, is one of the most widely used soft computational methods. In many circumstances, it can take noisy, imprecise input, to yield crisp (i.e. numerically accurate) output. Fuzzy logic can be applied effectively to map match the output from a HS GPS receiver in urban canyons because of its inherent tolerance to imprecise inputs. This paper describes a map matching algorithm based on fuzzy logic. The input of the system comes from a SiRF HS XTrac GPS receiver and a low cost gyro (Murata ENV-05G). The results show an improvement in tracking the vehicle in urban canyon conditions.

In this paper, we developed an image-based indoor localization system using omnidirectional panoramic images to which location information is added. By the combination of the robust image matching by PCA-SIFT and fast nearest neighbor search algorithm based on Locality Sensitive Hashing (LSH), our system can estimate users' positions with high accuracy and in a short time. To improve the precision, we introduced the "confidence" of the image matching results. We conducted experiments at the Railway Museum and we obtained 426 omnidirectional panoramic reference images and 1067 supplemental images for image matching. Experimental results using 126 test images demonstrated that the location detection accuracy is above 90% with about 2.2s of processing time.

This paper surveys the developments of the last 20 years in the area of vision for mobile robot navigation. Two major components of the paper deal with indoor navigation and outdoor navigation. For each component, we have further subdivided our treatment of the subject on the basis of structured and unstructured environments. For indoor robots in structured environments, we have dealt separately with the cases of geometrical and topological models of space. For unstructured environments, we have discussed the cases of navigation using optical flows, using methods from the appearance-based paradigm, and by recognition of specific objects in the environment.

Localization is one of the key technologies in wireless sensor networks (WSNs), since it provides fundamental support for many location-aware protocols and applications. Constraints on cost and power consumption make it infeasible to equip each sensor node in the network with a global position system (GPS) unit, especially for large-scale WSNs. A promising method to localize unknown nodes is to use mobile anchor nodes (MANs), which are equipped with GPS units moving among unknown nodes and periodically broadcasting their current locations to help nearby unknown nodes with localization. A considerable body of research has addressed the mobile anchor node assisted localization (MANAL) problem. However, to the best of our knowledge, no updated surveys on MAAL reflecting recent advances in the field have been presented in the past few years. This survey presents a review of the most successful MANAL algorithms, focusing on the achievements made in the past decade, and aims to become a starting point for researchers who are initiating their endeavors in MANAL research field. In addition, we seek to present a comprehensive review of the recent breakthroughs in the field, providing links to the most interesting and successful advances in this research field.

This thesis presents WiTrack, a system that tracks the 3D motion of a user from the radio signals reflected off her body. It works even if the person is occluded from the WiTrack device or in a different room. WiTrack does not require the user to carry any wireless device, yet its accuracy exceeds current RF localization systems, which require the user to hold a transceiver. Empirical measurements with a WiTrack prototype show that, on average, it localizes the center of a human body to within a median of 10 to 13 cm in the x and y dimensions, and 21 cm in the z dimension. It also provides coarse tracking of body parts, identifying the direction of a pointing hand with a median of 11.20. WiTrack bridges a gap between RF-based localization systems which locate a user through walls and occlusions, and human-computer interaction systems like Kinect, which can track a user without instrumenting her body, but require the user to stay within the direct line of sight of the device. Thesis Supervisor: Dina Katabi Title: Professor of Computer Science and Engineering

Large scale, long-term, distributed mapping is a core challenge to modern field robotics. Using the sensory output of multiple robots and fusing it in an efficient way enables the creation of globally accurate and consistent metric maps. To combine data from multiple agents into a global map, most existing approaches use a central entity that collects and manages the information from all agents. Often, the raw sensor data of one robot needs to be made available to processing algorithms on other agents due to the lack of computational resources on that robot. Unfortunately, network latency and low bandwidth in the field limit the generality of such an approach and make multi-robot map building a tedious task. In this paper, we present a distributed and decentralized back-end for concurrent and consistent robotic mapping. We propose a set of novel approaches that reduce the bandwidth usage and increase the effectiveness of inter-robot communication for distributed mapping. Instead of locking access to the map during operations, we define a version control system which allows concurrent and consistent access to the map data. Updates to the map are then shared asynchronously with agents which previously registered notifications. A technique for data lookup is provided by state-of-the-art algorithms from distributed computing. We validate our approach on real-world datasets and demonstrate the effectiveness of the proposed algorithms.


Question: A mobile robot's localization and map construction utilizing an RFID sensor fusion framework.

This paper presents a novel algorithm, Condensation—Conditional Density Propagation, specifically designed for robust visual tracking in environments plagued by dense visual clutter. The tracking of curves under such conditions presents a significant challenge due to the complexity and variability of the visual scene. Traditional methods, such as Kalman filtering, fall short in this context because they are based on unimodal Gaussian densities, which cannot adequately represent the multiple alternative hypotheses that arise in such scenarios. To overcome these limitations, the Condensation algorithm adopts a technique known as "factored sampling," previously utilized in the interpretation of static images. This method represents the probability distribution of potential interpretations through a randomly generated set of samples. By integrating learned dynamical models with visual observations, Condensation is capable of propagating this random set over time, enabling the algorithm to track agile motion with remarkable robustness. Despite its reliance on stochastic methods, the algorithm is designed to operate in near real-time, ensuring its practicality for real-world applications.

In this paper the possibilities that Matlab provides to design, implementation and monitoring programs of autonomous navigation for mobile robots, on both simulated and real platforms, through its new toolbox for robotics will be explored. Robotics System Toolbox has established itself as a solid tool of integration between Matlab and robot operating under ROS environment. It will be studied the tools available in this toolbox that allow achieving the connection between the two platforms, in addition to the generation of algorithms of location, path planning, mapping and autonomous navigation.

In an attempt to find an accurate, environmentally robust, and fast process for RFID distance estimations, a method was developed called RSSI Informed Phase. This method uses both phase and RSSI measurements to estimate the distance, and potentially location, of a moving RFID tag. RSSI is initially used to find an approximate reader to tag separation distance. This distance value is applied to find an approximate slope of the phase angle vs. frequency curve. The estimated slope can inform the phase distance calculation, meaning fewer reads are required to find the actual phase angle vs. frequency slope. The reduction in the number of necessary reads accelerates the localization process and makes this method more robust for dynamic environments.

The mission of tracking a pedestrian is valuable for many applications including walking distance estimation for the purpose of pervasive healthcare, museum and shopping mall guides, and locating emergency responders. In this paper, we show how accurate and ubiquitous tracking of a pedestrian can be performed using only the inertial sensors embedded in his/her mobile phone. Our work depends on performing dead reckoning to track the user's movement. The main challenge that needs to be addressed is handling the noise of the low cost low quality inertial sensors in cell phones. Our proposed system combines two novel contributions: a novel step count estimation technique and a gait-based accurate variable step size detection algorithm. The step count estimation technique is based on a lightweight finite state machine approach that leverages orientation-independent features. In order to capture the varying stride length of the user, based on his changing gait, we employ a multi-class hierarchical Support Vector Machine classifier. Combining the estimated number of steps with the an accurate estimate of the individual stride length, we achieve ubiquitous and accurate tracking of a person in indoor environments. We implement our system on different Android-based phones and compare it to the state-of-the-art techniques in indoor and outdoor testbeds with arbitrary phone orientation. Our results in two different testbeds show that we can provide an accurate step count estimation with an error of 5.72%. In addition, our gait type classifier has an accuracy of 97.74%. This leads to a combined tracking error of 6.9% while depending only on the inertial sensors and turning off the GPS sensor completely. This highlights the ability of the system to provide ubiquitous, accurate, and energy efficient tracking.

Magnetic anomaly detection (MAD) is an important problem in applications ranging from geological surveillance to military reconnaissance. MAD sensors detect local disturbances in the magnetic field, which can be used to detect the existence of and to estimate the position of buried, hidden, or submerged objects, such as ore deposits or mines. These sensors may experience false positive and false negative detections and, without prior knowledge of the targets, can only determine proximity to a target. The uncertainty in the sensors, coupled with a lack of knowledge of even the existence of targets, makes the estimation and control problems challenging. We utilize a hierarchical decomposition of the environment, coupled with an estimation algorithm based on random finite sets, to determine the number of and the locations of targets in the environment. The small team of robots follow the gradient of mutual information between the estimated set of targets and the future measurements, locally maximizing the rate of information gain. We present experimental results of a team of quadrotor micro aerial vehicles discovering and localizing an unknown number of permanent magnets.

Nowadays, smartphones have become indispensable to everyone, with more and more built-in location-based applications to enrich our daily life. In the last decade, fingerprinting based on RSS has become a research focus in indoor localization, due to its minimum hardware requirement and satisfiable positioning accuracy. However, its time-consuming and labor-intensive site survey is a big hurdle for practical deployments. Fingerprint crowdsourcing has recently been promoted to relieve the burden of site survey by allowing common users to contribute to fingerprint collection in a participatory sensing manner. For its promising commitment, new challenges arise to practice fingerprint crowdsourcing. This article first identifies two main challenging issues, fingerprint annotation and device diversity, and then reviews the state of the art of fingerprint crowdsourcing-based indoor localization systems, comparing their approaches to cope with the two challenges. We then propose a new indoor subarea localization scheme via fingerprint crowdsourcing, clustering, and matching, which first constructs subarea fingerprints from crowdsourced RSS measurements and relates them to indoor layouts. We also propose a new online localization algorithm to deal with the device diversity issue. Our experiment results show that in a typical indoor scenario, the proposed scheme can achieve a 95 percent hit rate to correctly locate a smartphone in its subarea.

In many mobile wireless applications such as the automated driving of cars, formation flying of unmanned air vehicles, and source localization or target tracking with wireless sensor networks, it is more important to know the precise relative locations of nodes than their absolute coordinates. GPS, the most ubiquitous localization system available, generally provides only absolute coordinates. Furthermore, low-cost receivers can exhibit tens of meters of error or worse in challenging RF environments. This paper presents an approach that uses GPS to derive relative location information for multiple receivers. Nodes in a network share their raw satellite measurements and use this data to track the relative motions of neighboring nodes as opposed to computing their own absolute coordinates. The system has been implemented using a network of Android phones equipped with a custom Bluetooth headset and integrated GPS chip to provide raw measurement data. Our evaluation shows that centimeter-scale tracking accuracy at an update rate of 1 Hz is possible under various conditions with the presented technique. This is more than an order of magnitude more accurate than simply taking the difference of reported absolute node coordinates or other simplistic approaches due to the presence of uncorrelated measurement errors.

Devices with integrated Wi-Fi chips broadcast beacons for network connection management purposes. Such information can be captured with inexpensive monitors and used to extract user behavior. To understand the behavior of visitors, we deployed our passive monitoring system---CrowdProbe, in a multi-floor museum for six months. We used a Hidden Markov Models (HMM) based trajectory inference algorithm to infer crowd movement using more than 1.7 million opportunistically obtained probe request frames.
 However, as more devices adopt schemes to randomize their MAC addresses in the passive probe session to protect user privacy, it becomes more difficult to track crowd and understand their behavior. In this paper, we try to make use of historical transition probability to reason about the movement of those randomized devices with spatial and temporal constraints. With CrowdProbe, we are able to achieve sufficient accuracy to understand the movement of visitors carrying devices with randomized MAC addresses.

Location fingerprinting in wireless LAN (WLAN) positioning has received much attention recently. One of the key issues of this technique is generating the database of ‘fingerprints’. The conventional method does not utilise the spatial correlation of measurements sampled at adjacent reference points (RPs), and the ‘training’ process is not an easy task. A new method based on kriging is presented in this paper. An experiment shows that the new method can not only achieve more accurate estimation, but can also greatly reduce the workload and save training time. This can make the fingerprinting technique more flexible and easier to implement. Binghao Li, Yufei Wang, Andrew Dempster, Chris Rizos are with the School of Surveying and Spatial Information System, University of New South Wales, Sydney, Australia Hyung Keun Lee is with the School of Avionics and Telecommunication, Hankuk Aviation University, Kyunggi-do, Korea

Over the last few years, the convincing forward steps in the development of Internet of Things (IoT)-enabling solutions are spurring the advent of novel and fascinating applications. Among others, mainly radio frequency identification (RFID), wireless sensor network (WSN), and smart mobile technologies are leading this evolutionary trend. In the wake of this tendency, this paper proposes a novel, IoT-aware, smart architecture for automatic monitoring and tracking of patients, personnel, and biomedical devices within hospitals and nursing institutes. Staying true to the IoT vision, we propose a smart hospital system (SHS), which relies on different, yet complementary, technologies, specifically RFID, WSN, and smart mobile, interoperating with each other through a Constrained Application Protocol (CoAP)/IPv6 over low-power wireless personal area network (6LoWPAN)/representational state transfer (REST) network infrastructure. The SHS is able to collect, in real time, both environmental conditions and patients' physiological parameters via an ultra-low-power hybrid sensing network (HSN) composed of 6LoWPAN nodes integrating UHF RFID functionalities. Sensed data are delivered to a control center where an advanced monitoring application (MA) makes them easily accessible by both local and remote users via a REST web service. The simple proof of concept implemented to validate the proposed SHS has highlighted a number of key capabilities and aspects of novelty, which represent a significant step forward compared to the actual state of the art.

This paper presents an integrated approach to exploration, mapping, and localization. Our algorithm uses a highly efficient Rao-Blackwellized particle filter to represent the posterior about maps and poses. It applies a decision-theoretic framework which simultaneously considers the uncertainty in the map and in the pose of the vehicle to evaluate potential actions. Thereby, it trades off the cost of executing an action with the expected information gain and takes into account possible sensor measurements gathered along the path taken by the robot. We furthermore describe how to utilize the properties of the Rao-Blackwellization to efficiently compute the expected information gain. We present experimental results obtained in the real world and in simulation to demonstrate the effectiveness of our approach.


Question: A mobile robot's localization and map construction utilizing an RFID sensor fusion framework.

This paper introduces a novel Bayesian Landmark Learning (BaLL) algorithm designed to enhance the localization capabilities of mobile robots in indoor settings. Current localization techniques often fall short in terms of adaptability, self-sufficiency, and optimality, as they necessitate human intervention to determine the most relevant sensor data components for localization purposes, such as identifying appropriate landmarks. BaLL addresses these limitations by empowering robots to autonomously discover the most effective features or landmarks for localization and to train artificial neural networks for their precise extraction from sensor data. A comprehensive Bayesian analysis is provided, which not only supports the rationale for evaluating and selecting features optimally but also guides the training of neural networks to approximate the ideal solution. Through a series of controlled experiments, BaLL demonstrates superior performance compared to two contemporary methods in the domain of mobile robot localization, showcasing its potential to revolutionize the way robots navigate and operate in complex indoor environments.

While WiFi-based indoor localization is attractive, the need for a significant degree of pre-deployment effort is a key challenge. In this paper, indoor localization with no pre-deployment effort in an indoor space, such as an office building corridor, with WiFi coverage but no a priori knowledge of the placement of the access points(APs) is implemented for mobile devices. WiFi Received Signal Strength(RSS) in the considered environment is used to build radio maps using WiFi fingerprinting approach. An offline RSS fingerprint of the environment is compared with an online RSS measurement to estimate the location of a user. Two architectures are developed based on this localization algorithm. The first one involves a client-server approach where the localization algorithm runs on the server whereas the second one is a standalone architecture and the algorithm runs on the SD card of the mobile device. Experimental results in the considered environment validate the approach for both architectures.

In an attempt to find an accurate, environmentally robust, and fast process for RFID distance estimations, a method was developed called RSSI Informed Phase. This method uses both phase and RSSI measurements to estimate the distance, and potentially location, of a moving RFID tag. RSSI is initially used to find an approximate reader to tag separation distance. This distance value is applied to find an approximate slope of the phase angle vs. frequency curve. The estimated slope can inform the phase distance calculation, meaning fewer reads are required to find the actual phase angle vs. frequency slope. The reduction in the number of necessary reads accelerates the localization process and makes this method more robust for dynamic environments.

This paper describes a geometrically constrained Extended Kalman Filter (EKF) framework for a line feature based SLAM, which is applicable to a rectangular indoor environment. Its focus is on how to handle sparse and noisy sensor data, such as PSD infrared sensors with limited range and limited number, in order to develop a low-cost navigation system. It has been applied to a vacuum cleaning robot in our research. In order to meet the real-time objective with low computing power, we develop an efficient line feature extraction algorithm based upon an iterative end point fit (IEPF) technique assisted by our constrained version of the Hough transform. It uses a geometric constraint that every line is orthogonal or parallel to each other because in a general indoor setting, most furniture and walls satisfy this constraint. By adding this constraint to the measurement model of EKF, we build a geometrically constrained EKF framework which can estimate line feature positions more accurately as well as allow their covariance matrices to converge more rapidly when compared to the case of an unconstrained EKF. The experimental results demonstrate the accuracy and robustness to the presence of sensor noise and errors in an actual indoor environment.

Robot navigation in the environment with obstacles is still a challenging problem. In this paper, the navigation problems with wheeled mobile robots (WMRs) are reviewed, the navigation mechanism of WMRs is analyzed in detail, the methods of solving the sub problems such as mapping, localization and path planning which all both related to robot navigation are summarized and the advantages and disadvantages of the existing methods are expounded. Especially in the agricultural field, the precise navigation of robots in the complex agricultural environment is the prerequisite for the completion of various tasks. This paper is aimed at the special complexity of the agricultural environment, prospected the application of the solution to the navigation problem of WMRs in agricultural engineering, put forward the research direction to solve the problems of precise navigation in agricultural environments.

This thesis presents WiTrack, a system that tracks the 3D motion of a user from the radio signals reflected off her body. It works even if the person is occluded from the WiTrack device or in a different room. WiTrack does not require the user to carry any wireless device, yet its accuracy exceeds current RF localization systems, which require the user to hold a transceiver. Empirical measurements with a WiTrack prototype show that, on average, it localizes the center of a human body to within a median of 10 to 13 cm in the x and y dimensions, and 21 cm in the z dimension. It also provides coarse tracking of body parts, identifying the direction of a pointing hand with a median of 11.20. WiTrack bridges a gap between RF-based localization systems which locate a user through walls and occlusions, and human-computer interaction systems like Kinect, which can track a user without instrumenting her body, but require the user to stay within the direct line of sight of the device. Thesis Supervisor: Dina Katabi Title: Professor of Computer Science and Engineering

There are lots of local positioning systems available for indoor applications, partly even for outdoor use. Many of them rely on a rather clean environment which does not interfere in their measurements too much. Industrial environments, in- or outdoors, often do not meet these requirements. The positioning applications here are surrounded by dirt, dust and vibrations. The number of local positioning systems being able to perform properly here at all times is limited to such not dependent on optical or sonic measurements. Local secondary radar systems proved to be a reliable source of distance and position measurements, even in harsh environments. This paper examines a few use cases and depicts the performance of local positioning radar systems there.

This paper presents a new approach to the cooperative localization problem, namely distributed multi-robot localization. A group of M robots is viewed as a single system composed of robots that carry, in general, di erent sensors and have di erent positioning capabilities. A single Kalman lter is formulated to estimate the position and orientation of all the members of the group. This centralized schema is capable of fusing information provided by the sensors distributed on the individual robots while accommodating independencies and interdependencies among the collected data. In order to allow for distributed processing, the equations of the centralized Kalman lter are treated so that this lter can be decomposed into M modi ed Kalman lters each running on a separate robot. The distributed localization algorithm is applied to a group of 3 robots and the improvement in localization accuracy is presented.

We present Travi-Navi - a vision-guided navigation system that enables a self-motivated user to easily bootstrap and deploy indoor navigation services, without comprehensive indoor localization systems or even the availability of floor maps. Travi-Navi records high quality images during the course of a guider's walk on the navigation paths, collects a rich set of sensor readings, and packs them into a navigation trace. The followers track the navigation trace, get prompt visual instructions and image tips, and receive alerts when they deviate from the correct paths. Travi-Navi also finds the most efficient shortcuts whenever possible. We encounter and solve several challenges, including robust tracking, shortcut identification, and high quality image capture while walking. We implement Travi-Navi and conduct extensive experiments. The evaluation results show that Travi-Navi can track and navigate users with timely instructions, typically within a 4-step offset, and detect deviation events within 9 steps.

This paper presents the design and analysis of a computer vision-aided motion sensing algorithm for wheeled mobile robot's indoor navigation. The algorithm is realized using two vision cameras attached on a wheeled mobile robot. The first camera is positioned at front-looking direction while the second camera is positioned at downward-looking direction. An algorithm is developed to process the images acquired from the cameras to yield the mobile robot's positions and orientations. The proposed algorithm is implemented on a wheeled mobile robot for real-world effectiveness testing. Results are compared and shown the accuracy of the proposed algorithm. At the end of the paper, an artificial landmark approach is introduced to improve the navigation efficiency. Future work involved implementing the proposed artificial landmark for indoor navigation applications with minimized accumulated errors.

Odometry-like localization solutions can be built upon Light Detection And Ranging (LIDAR) sensors, by sequentially registering the point clouds acquired along a robot trajectory. Yet such solutions inherently drift over time: we propose an approach that adds a graphical model layer on top of such LIDAR odometry layer, using the Iterative Closest Points (ICP) algorithm for registration. Reference frames called keyframes are defined along the robot trajectory, and ICP results are used to build a pose graph, that in turn is used to solve an optimization problem that provides updates for the keyframes upon loop closing, enabling the correction of the path of the robot and of the map of the environment. We present in details the configuration used to register data from the Velodyne High Definition LIDAR (HDL), and a strategy to build local maps upon which current frames are registered, either when discovering new areas or revisiting previously mapped areas. Experiments show that it is possible to build the graph using data from ICP and that the loop closings in the graph level reduces the overall drift of the system.


Question: A mobile robot's localization and map construction utilizing an RFID sensor fusion framework.

This paper provides an in-depth tutorial on particle filters for addressing nonlinear and non-Gaussian Bayesian tracking challenges in real-time. As the need for accurate modeling of complex physical systems grows across various fields, the inclusion of nonlinearity and non-Gaussian characteristics becomes imperative. Moreover, the capability to process incoming data immediately is essential, not only to manage storage costs but also to ensure swift adaptation to evolving signal dynamics. In this context, we present an overview of Bayesian algorithms, with a particular emphasis on particle filters, designed for tackling nonlinear and non-Gaussian tracking scenarios. Particle filters, as a form of sequential Monte Carlo method, employ point mass representations of probability densities, applicable to any state-space model, thereby offering a generalization of conventional Kalman filtering techniques. The paper introduces and examines various particle filter variants, including SIR, ASIR, and RPF, all of which are encapsulated within the sequential importance sampling (SIS) framework. These methods are contrasted and compared with the traditional Extended Kalman Filter (EKF) through a practical example, demonstrating their utility and effectiveness in real-world applications.

This paper presents an approach to using a large team of UAVs to find radio frequency (RF) emitting targets in a large area. Small, inexpensive UAVs that can collectively and rapidly determine the approximate location of intermittently broadcasting and mobile RF emitters have a range of applications in both military, e.g., for finding SAM batteries, and civilian, e.g., for finding lost hikers, domains. Received Signal Strength Indicator (RSSI) sensors on board the UAVs measure the strength of RF signals across a range of frequencies. The signals, although noisy and ambiguous due to structural noise, e.g., multipath effects, overlapping signals and sensor noise, allow estimates to be made of emitter locations. Generating a probability distribution over emitter locations requires integrating multiple signals from different UAVs into a Bayesian filter, hence requiring cooperation between the UAVs. Once likely target locations are identified, EO-camera equipped UAVs must be tasked to provide a video stream of the area to allow a user to identify the emitter.

Nowadays the use of smart mobile devices and the accompanying needs for emerging services relying on indoor location-based services (LBS) for mobile devices are rapidly increasing. For more accurate location tracking using Bluetooth Low Energy (BLE), this paper proposes a novel trilateration-based algorithm and presents experimental results that demonstrate its effectiveness.

The ability to simultaneously localize a robot and accurately map its surroundings is considered by many to be a key prerequisite of truly autonomous robots. However, few approaches to this problem scale up to handle the very large number of landmarks present in real environments. Kalman filter-based algorithms, for example, require time quadratic in the number of landmarks to incorporate each sensor observation. This paper presents FastSLAM, an algorithm that recursively estimates the full posterior distribution over robot pose and landmark locations, yet scales logarithmically with the number of landmarks in the map. This algorithm is based on an exact factorization of the posterior into a product of conditional landmark distributions and a distribution over robot paths. The algorithm has been run successfully on as many as 50,000 landmarks, environments far beyond the reach of previous approaches. Experimental results demonstrate the advantages and limitations of the FastSLAM algorithm on both simulated and real-world data.

This paper describes the simultaneous localization and mapping (SLAM) problem and the essential methods for solving the SLAM problem and summarizes key implementations and demonstrations of the method. While there are still many practical issues to overcome, especially in more complex outdoor environments, the general SLAM method is now a well understood and established part of robotics. Another part of the tutorial summarized more recent works in addressing some of the remaining issues in SLAM, including computation, feature representation, and data association

This paper presents the design, implementation, and evaluation of Cricket, a location-support system for in-building, mobile, location-dependent applications. It allows applications running on mobile and static nodes to learn their physical location by using listeners that hear and analyze information from beacons spread throughout the building. Cricket is the result of several design goals, including user privacy, decentralized administration, network heterogeneity, and low cost. Rather than explicitly tracking user location, Cricket helps devices learn where they are and lets them decide whom to advertise this information to; it does not rely on any centralized management or control and there is no explicit coordination between beacons; it provides information to devices regardless of their type of network connectivity; and each Cricket device is made from off-the-shelf components and costs less than U.S. $10. We describe the randomized algorithm used by beacons to transmit information, the use of concurrent radio and ultrasonic signals to infer distance, the listener inference algorithms to overcome multipath and interference, and practical beacon configuration and positioning techniques that improve accuracy. Our experience with Cricket shows that several location-dependent applications such as in-building active maps and device control can be developed with little effort or manual configuration.

Odometry-like localization solutions can be built upon Light Detection And Ranging (LIDAR) sensors, by sequentially registering the point clouds acquired along a robot trajectory. Yet such solutions inherently drift over time: we propose an approach that adds a graphical model layer on top of such LIDAR odometry layer, using the Iterative Closest Points (ICP) algorithm for registration. Reference frames called keyframes are defined along the robot trajectory, and ICP results are used to build a pose graph, that in turn is used to solve an optimization problem that provides updates for the keyframes upon loop closing, enabling the correction of the path of the robot and of the map of the environment. We present in details the configuration used to register data from the Velodyne High Definition LIDAR (HDL), and a strategy to build local maps upon which current frames are registered, either when discovering new areas or revisiting previously mapped areas. Experiments show that it is possible to build the graph using data from ICP and that the loop closings in the graph level reduces the overall drift of the system.

LoRa-based (Long Range Communication based) localization systems make use of RSSI (Received Signal Strength Indicator) to locate an object properly in an outdoor environment for IoT (Internet of Things) applications in smart environment and urban networking. The accuracy of localization is highly degraded, however, by noisy environments (e.g., electronic interference, blocking). To address this issue, this paper proposes two new localization algorithms to improve localization accuracy. Furthermore, the paper develops a winner selection strategy to select a winner from the traditional localization algorithm and the two new algorithms to minimize localization errors. Our simulations and real experiments found that the two new algorithms can significantly improve localization accuracy. Additionally, the winner selection strategy effectively selects a winner from the three algorithms to improve localization accuracy further. Using our proposed algorithms in some real experiments, the localization error is only a few meters in a real measurement field longer than 100 m.

The sheer volume of data generated by depth cameras provides a challenge to process in real time, in particular when used for indoor mobile robot localization and navigation. We introduce the Fast Sampling Plane Filtering (FSPF) algorithm to reduce the volume of the 3D point cloud by sampling points from the depth image, and classifying local grouped sets of points as belonging to planes in 3D (the “plane filtered” points) or points that do not correspond to planes within a specified error margin (the “outlier” points). We then introduce a localization algorithm based on an observation model that down-projects the plane filtered points on to 2D, and assigns correspondences for each point to lines in the 2D map. The full sampled point cloud (consisting of both plane filtered as well as outlier points) is processed for obstacle avoidance for autonomous navigation. All our algorithms process only the depth information, and do not require additional RGB data. The FSPF, localization and obstacle avoidance algorithms run in real time at full camera frame rates (30Hz) with low CPU requirements (16%). We provide experimental results demonstrating the effectiveness of our approach for indoor mobile robot localization and navigation. We further compare the accuracy and robustness in localization using depth cameras with FSPF vs. alternative approaches that simulate laser rangefinder scans from the 3D data.

Build accurate DNN models requires training on large labeled, context specific datasets, especially those matching the target scenario. We believe advances in wireless localization, working in unison with cameras, can produce automated annotation of targets on images and videos captured in the wild. Using pedestrian and vehicle detection as examples, we demonstrate the feasibility, benefits, and challenges of an automatic image annotation system. Our work calls for new technical development on passive localization, mobile data analytics, and error-resilient ML models, as well as design issues in user privacy policies.

The growing commercial interest in indoor location-based services (ILBS) has spurred recent development of many indoor positioning techniques. Due to the absence of Global Positioning System (GPS) signal, many other signals have been proposed for indoor usage. Among them, Wi-Fi (802.11) emerges as a promising one due to the pervasive deployment of wireless LANs (WLANs). In particular, Wi-Fi fingerprinting has been attracting much attention recently because it does not require line-of-sight measurement of access points (APs) and achieves high applicability in complex indoor environment. This survey overviews recent advances on two major areas of Wi-Fi fingerprint localization: advanced localization techniques and efficient system deployment. Regarding advanced techniques to localize users, we present how to make use of temporal or spatial signal patterns, user collaboration, and motion sensors. Regarding efficient system deployment, we discuss recent advances on reducing offline labor-intensive survey, adapting to fingerprint changes, calibrating heterogeneous devices for signal collection, and achieving energy efficiency for smartphones. We study and compare the approaches through our deployment experiences, and discuss some future directions.


Question: Acquiring actionable representations through the use of goal-conditioned policies.

Introducing Image-Driven Control: A Locally Linear Latent Dynamics Framework
We unveil Image-Driven Control (IDC), a novel approach for modeling and controlling complex non-linear dynamical systems using raw pixel images as inputs. IDC is anchored in a sophisticated deep generative model, specifically a variant of the variational autoencoder family, which is meticulously engineered to produce image sequences from a latent space characterized by locally linear dynamics. This design is directly influenced by an optimal control framework applied in the latent space, enabling IDC to accurately forecast long-term image sequences and demonstrate impressive performance across a wide spectrum of intricate control tasks.

Representation learning is a fundamental problem in natural language processing. This paper studies how to learn a structured representation for text classification. Unlike most existing representation models that either use no structure or rely on pre-specified structures, we propose a reinforcement learning (RL) method to learn sentence representation by discovering optimized structures automatically. We demonstrate two attempts to build structured representation: Information Distilled LSTM (ID-LSTM) and Hierarchically Structured LSTM (HS-LSTM). ID-LSTM selects only important, taskrelevant words, and HS-LSTM discovers phrase structures in a sentence. Structure discovery in the two representation models is formulated as a sequential decision problem: current decision of structure discovery affects following decisions, which can be addressed by policy gradient RL. Results show that our method can learn task-friendly representations by identifying important words or task-relevant structures without explicit structure annotations, and thus yields competitive performance.

Multi-agent systems are rapidly finding applications in a variety of domains, including robotics, distributed control, telecommunications, economics. Many tasks arising in these domains require that the agents learn behaviors online. A significant part of the research on multi-agent learning concerns reinforcement learning techniques. However, due to different viewpoints on central issues, such as the formal statement of the learning goal, a large number of different methods and approaches have been introduced. In this paper we aim to present an integrated survey of the field. First, the issue of the multi-agent learning goal is discussed, after which a representative selection of algorithms is reviewed. Finally, open issues are identified and future research directions are outlined. Keywords—multi-agent systems, reinforcement learning, game theory, distributed control

Reinforcement learning offers a promising methodology for developing skills for simulated characters, but typically requires working with sparse hand-crafted features. Building on recent progress in deep reinforcement learning (DeepRL), we introduce a mixture of actor-critic experts (MACE) approach that learns terrain-adaptive dynamic locomotion skills using high-dimensional state and terrain descriptions as input, and parameterized leaps or steps as output actions. MACE learns more quickly than a single actor-critic approach and results in actor-critic experts that exhibit specialization. Additional elements of our solution that contribute towards efficient learning include Boltzmann exploration and the use of initial actor biases to encourage specialization. Results are demonstrated for multiple planar characters and terrain classes.

In this paper, we propose the Quantile Option Architecture (QUOTA) for exploration based on recent advances in distributional reinforcement learning (RL). In QUOTA, decision making is based on quantiles of a value distribution, not only the mean. QUOTA provides a new dimension for exploration via making use of both optimism and pessimism of a value distribution. We demonstrate the performance advantage of QUOTA in both challenging video games and physical robot simulators.

Interpreting decision making logic in demonstration videos is key to collaborating with and mimicking humans. To empower machines with this ability, we propose a neural program synthesizer that is able to explicitly synthesize underlying programs from behaviorally diverse and visually complicated demonstration videos. We introduce a summarizer module as part of our model to improve the network’s ability to integrate multiple demonstrations varying in behavior. We also employ a multi-task objective to encourage the model to learn meaningful intermediate representations for end-to-end training. We show that our model is able to reliably synthesize underlying programs as well as capture diverse behaviors exhibited in demonstrations. The code is available at https://shaohua0116.github.io/demo2program.

In standard passive imitation learning, the goal is to learn a target policy by passively observing full execution trajectories of it. Unfortunately, generating such trajectories can require substantial expert effort and be impractical in some cases. In this paper, we consider active imitation learning with the goal of reducing this effort by querying the expert about the desired action at individual states, which are selected based on answers to past queries and the learner’s interactions with an environment simulator. We introduce a new approach based on reducing active imitation learning to i.i.d. active learning, which can leverage progress in the i.i.d. setting. Our first contribution, is to analyze reductions for both non-stationary and stationary policies, showing that the label complexity (number of queries) of active imitation learning can be substantially less than passive learning. Our second contribution, is to introduce a practical algorithm inspired by the reductions, which is shown to be highly effective in four test domains compared to a number of alternatives.

Though successful in high-dimensional domains, deep reinforcement learning exhibits high sample complexity and suffers from stability issues as reported by researchers and practitioners in the field. These problems hinder the application of such algorithms in real-world and safety-critical scenarios. In this paper, we take steps towards stable and efficient reinforcement learning by following a model-based approach that is known to reduce agent-environment interactions. Namely, our method augments deep Q-networks (DQNs) with model predictions for transitions, rewards, and termination flags. Having the model at hand, we then conduct a rigorous theoretical study of our algorithm and show, for the first time, convergence to a stationary point. En route, we provide a counterexample showing that ’vanilla’ DQNs can diverge confirming practitioners’ and researchers’ experiences. Our proof is novel in its own right and can be extended to other forms of deep reinforcement learning. In particular, we believe exploiting the relation between reinforcement (with deep function approximators) and online learning can serve as a recipe for future proofs in the domain. Finally, we validate our theoretical results in 20 games from the Atari benchmark. Our results show that following the proposed model-based learning approach not only ensures convergence but leads to a reduction in sample complexity and superior performance. Introduction Model-free deep reinforcement learning methods have recently demonstrated impressive performance on a range of complex learning tasks (Hessel et al. 2018; Lillicrap et al. 2017; Jaderberg et al. 2017). Deep Q-Networks (DQNs) (Mnih et al. 2015), in particular, stand out as a versatile and effective tool for a wide range of applications. DQNs offer an end-to-end solution for many reinforcement learning problems by generalizing tabular Q-learning to highdimensional input spaces. Unfortunately, DQNs still suffer from a number of important drawbacks that hinder wide-spread adoption. There is a general lack of theoretical understanding to guide the use of deep Q-learning algorithms. Combining non-linear function approximation with off-policy reinforcement learning is known to lead to possible divergence issues, even in very simple tasks (Boyan and Preliminary work. Moore 1995). Despite improvements introduced by deep Qnetworks (Mnih et al. 2015), Q-learning approaches based on deep neural networks still exhibit occasional instability during the learning process, see for example (van Hasselt, Guez, and Silver 2016). In this paper, we aim to address the instability of current deep RL approaches by augmenting model-free value learning with a predictive model. We propose a hybrid model and value learning objective, which stabilises the learning process and reduces sample complexity when compared to model-free algorithms. We attribute this reduction to the richer feedback signal provided by our method compared to standard deep Q-networks. In particular, our model provides feedback at every time step based on the current prediction error, which in turn eliminates one source of sample inefficiency caused by sparse rewards. These conclusions are also in accordance with previous research conducted on linear function approximation in reinforcement learning (Parr et al. 2008; Sutton et al. 2012; Song et al. 2016). While linear function approximation results provide a motivation for model-based stabilisation, such theories fail to generalise to deep non-linear architectures (Song et al. 2016). To close this gap in the literature and demonstrate stability of our method, we prove convergence in the general deep RL setting with deep value function approximators. Theoretically analysing deep RL algorithms, however, is challenging due to nonlinear functional dependencies and objective non-convexities prohibiting convergence to globally optimal policies. As such, the best one would hope for is a stationary point (e.g., a first-order one) that guarantees vanishing gradients. Even understanding gradient behavior for deep networks in RL can be difficult due to explorationbased policies and replay memory considerations. To alleviate some of these problems, we map deep reinforcement learning to a mathematical framework explicitly designed for understanding the exploration/exploitation trade-off. Namely, we formalise the problem as an online learning game the agent plays against an adversary (i.e., the environment). Such a link allows us to study a more general problem combining notions of regret, reinforcement learning, and optimisation. Given such a link, we prove that for any > 0, regret vanishes as O ( T 1−2 lnT ) with T being the total number of rounds. This, in turn, guarantees convergence to a stationary point. ar X iv :1 80 9. 01 90 6v 1 [ cs .L G ] 6 S ep 2 01 8 Guided by the above theoretical results, we validate that our algorithm leads to faster learning by conducting experiments on 20 Atari games. Due to the high computational and budgeting demands, we chose a benchmark that is closest to our setting. Concretely, we picked DQNs as our template algorithm to improve on both theoretically and empirically. It is worth noting that extending our theoretical results to more complex RL algorithms is an orthogonal future direction. Background: Reinforcement Learning We consider a discrete time, infinite horizon, discounted Markov Decision Process (MDP) 〈 S,A, P , P , γ 〉 . Here S denotes the state space, A the set of possible actions, P (R) : S × A × R → [0, 1] is the reward function, P (S) : S × A × S → [0, 1] is the state transition function, and γ ∈ (0, 1) is a discount factor. An agent, being in state St ∈ S at time step t, executes an action at ∈ A sampled from its policy π(at|St), a conditional probability distribution where π : S × A → [0, 1]. The agent’s action elicits from the environment a reward signal rt ∈ R indicating instantaneous reward, a terminal flag ft ∈ {0, 1} indicating a terminal event that restarts the environment, and a transition to a successor state St+1 ∈ S. We assume that the sets S, A, and R are discrete. The reward rt is sampled from the conditional probability distribution P (rt|St,at). Similarly, ft ∼ P (F (ft|St,at) with P (F ) : S × A × {0, 1} → [0, 1], where a terminal event (ft = 1) restarts the environment according to some initial state distribution S0 ∼ P (S) 0 (S0). The state transition to a successor state is determined by a stochastic state transition function according to St+1 ∼ P (St+1|St,at). The agent’s goal is to maximise future cumulative reward E P (S) 0 ,π,P (R),P (F ),P (S) [ ∞ ∑ t=0 ( t ∏ t′=0 (1− ft′−1) ) γrt ] with respect to the policy π. An important quantity in RL are Q-values Q(St,at), which are defined as the expected future cumulative reward value when executing action at in state St and subsequently following policy π. Qvalues enable us to conveniently phrase the RL problem as maxπ ∑ S p (S) ∑ a π(a|S)Q(S,a), where p(S) is the stationary state distribution obtained when executing π in the environment (starting from S0 ∼ P (S) 0 (S0)). Deep Q-Networks: Value-based reinforcement learning approaches identify optimal Q-values directly using parametric function approximators Qθ(S,a), where θ ∈ Θ represents the parameters (Watkins 1989; Busoniu et al. 2010). Optimal Q-value estimates then correspond to an optimal policy π(a|S) = δa,argmaxa′Qθ(S,a′). Deep Q-networks (Mnih et al. 2015) learn a deep neural network based Qvalue approximation by performing stochastic gradient descent on the following training objective: L(θ) = ES,a,r,f,S′ [( r + (1− f)γmax a′ Qθ−(S ′,a′) −Qθ(S,a) )2] . (1) The expectation ranges over transitions S,a, r, f,S′ sampled from an experience replay memory (S′ denotes the state at the next time step). Use of this replay memory, together with the use of a separate target network Qθ− (with different parameters θ− ∈ Θ) for calculating the bootstrap values maxa′ Qθ−(S ′,a′), helps stabilise the learning process. Background: Online Learning In this paper, we employ a special form of regret minimisation games that we briefly review here. A regret minimisation game is a triple 〈Θ,F , T 〉, where Θ is a non-empty decision set, F is the set of moves of the adversary which contains bounded convex functions from R to R and T is the total number of rounds. The game commences in rounds, where at round j = 1, . . . , T , the agent chooses a prediction θj ∈ Θ and the adversary a loss function Lj ∈ F . At the end of the round, the adversary reveals its choice and the agent suffers a loss Lj(θj). In this paper, we are concerned with the full-information case where the agent can observe the complete loss function Lj at the end of each round. The goal of the game is for the agent to make successive predictions to minimise cumulative regret defined as:

In this paper, we propose a learning algorithm that speeds up the search in task and motion planning problems. Our algorithm proposes solutions to three different challenges that arise in learning to improve planning efficiency: what to predict, how to represent a planning problem instance, and how to transfer knowledge from one problem instance to another. We propose a method that predicts constraints on the search space based on a generic representation of a planning problem instance, called score space, where we represent a problem instance in terms of performance of a set of solutions attempted so far. Using this representation, we transfer knowledge, in the form of constraints, from previous problems based on the similarity in score space. We design a sequential algorithm that efficiently predicts these constraints, and evaluate it in three different challenging task and motion planning problems. Results indicate that our approach perform orders of magnitudes faster than an unguided planner.

Adversarial examples have been shown to exist for a variety of deep learning architectures. Deep reinforcement learning has shown promising results on training agent policies directly on raw inputs such as image pixels. In this paper we present a novel study into adversarial attacks on deep reinforcement learning polices. We compare the effectiveness of the attacks using adversarial examples vs. random noise. We present a novel method for reducing the number of times adversarial examples need to be injected for a successful attack, based on the value function. We further explore how re-training on random noise and FGSM perturbations affects the resilience against adversarial examples.

Exploration in an unknown environment is the core functionality for mobile robots. Learning-based exploration methods, including convolutional neural networks, provide excellent strategies without human-designed logic for the feature extraction [1]. But the conventional supervised learning algorithms cost lots of efforts on the labeling work of datasets inevitably. Scenes not included in the training set are mostly unrecognized either. We propose a deep reinforcement learning method for the exploration of mobile robots in an indoor environment with the depth information from an RGB-D sensor only. Based on the Deep Q-Network framework [2], the raw depth image is taken as the only input to estimate the Q values corresponding to all moving commands. The training of the network weights is end-to-end. In arbitrarily constructed simulation environments, we show that the robot can be quickly adapted to unfamiliar scenes without any man-made labeling. Besides, through analysis of receptive fields of feature representations, deep reinforcement learning motivates the convolutional networks to estimate the traversability of the scenes. The test results are compared with the exploration strategies separately based on deep learning [1] or reinforcement learning [3]. Even trained only in the simulated environment, experimental results in real-world environment demonstrate that the cognitive ability of robot controller is dramatically improved compared with the supervised method. We believe it is the first time that raw sensor information is used to build cognitive exploration strategy for mobile robots through end-to-end deep reinforcement learning.


Question: Acquiring actionable representations through the use of goal-conditioned policies.

This paper details our approach to training a large-scale, deep convolutional neural network for classifying high-resolution images from the ImageNet LSVRC-2010 dataset into 1000 distinct categories. Our model achieved top-1 and top-5 error rates of 37.5% and 17.0%, respectively, on the test data, significantly outperforming the previous state-of-the-art results. The neural network, boasting 60 million parameters and 650,000 neurons, is composed of five convolutional layers, several of which are followed by max-pooling layers, and three fully connected layers culminating in a 1000-way softmax classification. To expedite the training process, we utilized non-saturating neurons and an efficient GPU implementation of convolution operations. To tackle overfitting in the fully connected layers, we implemented a novel regularization technique known as "dropout," which proved highly effective. We also submitted a variation of this model to the ILSVRC-2012 competition, where it secured the top position with a test error rate of 15.3%, markedly lower than the 26.2% achieved by the runner-up. This demonstrates the robustness and superiority of our deep convolutional neural network in image classification tasks.

Providing Reinforcement Learning agents with expert advice can dramatically improve various aspects of learning. Prior work has developed teaching protocols that enable agents to learn efficiently in complex environments; many of these methods tailor the teacher’s guidance to agents with a particular representation or underlying learning scheme, offering effective but specialized teaching procedures. In this work, we explore protocol programs, an agent-agnostic schema for Human-in-the-Loop Reinforcement Learning. Our goal is to incorporate the beneficial properties of a human teacher into Reinforcement Learning without making strong assumptions about the inner workings of the agent. We show how to represent existing approaches such as action pruning, reward shaping, and training in simulation as special cases of our schema and conduct preliminary experiments on simple domains.

In this work, we propose a novel robot learning framework called Neural Task Programming (NTP), which bridges the idea of few-shot learning from demonstration and neural program induction. NTP takes as input a task specification (e.g., video demonstration of a task) and recursively decomposes it into finer sub-task specifications. These specifications are fed to a hierarchical neural program, where bottom-level programs are callable subroutines that interact with the environment. We validate our method in three robot manipulation tasks. NTP achieves strong generalization across sequential tasks that exhibit hierarchal and compositional structures. The experimental results show that NTP learns to generalize well towards unseen tasks with increasing lengths, variable topologies, and changing objectives.stanfordvl.github.io/ntp/.

The ability to use a 2D map to navigate a complex 3D environment is quite remarkable, and even difficult for many humans. Localization and navigation is also an important problem in domains such as robotics, and has recently become a focus of the deep reinforcement learning community. In this paper we teach a reinforcement learning agent to read a map in order to find the shortest way out of a random maze it has never seen before. Our system combines several state-of-theart methods such as A3C and incorporates novel elements such as a recurrent localization cell. Our agent learns to localize itself based on 3D first person images and an approximate orientation angle. The agent generalizes well to bigger mazes, showing that it learned useful localization and naviga-

Reinforcement learning offers one of the most general framework to take traditional robotics towards true autonomy and versatility. However, applying reinforcement learning to high dimensional movement systems like humanoid robots remains an unsolved problem. In this paper, we discuss different approaches of reinforcement learning in terms of their applicability in humanoid robotics. Methods can be coarsely classified into three different categories, i.e., greedy methods, ‘vanilla’ policy gradient methods, and natural gradient methods. We discuss that greedy methods are not likely to scale into the domain humanoid robotics as they are problematic when used with function approximation. ‘Vanilla’ policy gradient methods on the other hand have been successfully applied on real-world robots including at least one humanoid robot [3]. We demonstrate that these methods can be significantly improved using the natural policy gradient instead of the regular policy gradient. A derivation of the natural policy gradient is provided, proving that the average policy gradient of Kakade [10] is indeed the true natural gradient. A general algorithm for estimating the natural gradient, the Natural Actor-Critic algorithm, is introduced. This algorithm converges to the nearest local minimum of the cost function with respect to the Fisher information metric under suitable conditions. The algorithm outperforms non-natural policy gradients by far in a cart-pole balancing evaluation, and for learning nonlinear dynamic motor primitives for humanoid robot control. It offers a promising route for the development of reinforcement learning for truly high-dimensionally continuous state-action systems.

Existing multi-objective reinforcement learning (MORL) algorithms do not account for objectives that arise from players with differing beliefs. Concretely, consider two players with different beliefs and utility functions who may cooperate to build a machine that takes actions on their behalf. A representation is needed for how much the machine’s policy will prioritize each player’s interests over time. Assuming the players have reached common knowledge of their situation, this paper derives a recursion that any Pareto optimal policy must satisfy. Two qualitative observations can be made from the recursion: the machine must (1) use each player’s own beliefs in evaluating how well an action will serve that player’s utility function, and (2) shift the relative priority it assigns to each player’s expected utilities over time, by a factor proportional to how well that player’s beliefs predict the machine’s inputs. Observation (2) represents a substantial divergence from näıve linear utility aggregation (as in Harsanyi’s utilitarian theorem, and existing MORL algorithms), which is shown here to be inadequate for Pareto optimal sequential decision-making on behalf of players with different beliefs.

In this paper, we apply reinforcement learning for automatically learning cooperative persuasive dialogue system policies using framing, the use of emotionally charged statements common in persuasive dialogue between humans. In order to apply reinforcement learning, we describe a method to construct user simulators and reward functions specifically tailored to persuasive dialogue based on a corpus of persuasive dialogues between human interlocutors. Then, we evaluate the learned policy and the effect of framing through experiments both with a user simulator and with real users. The experimental evaluation indicates that applying reinforcement learning is effective for construction of cooperative persuasive dialogue systems which use framing.

Bayesian approaches provide a principled solution to the explorationexploitation trade-off in Reinforcement Learning. Typical approaches, however, either assume a fully observable environment or scale poorly. This work introduces the Factored Bayes-Adaptive POMDP model, a framework that is able to exploit the underlying structure while learning the dynamics in partially observable systems. We also present a belief tracking method to approximate the joint posterior over state and model variables, and an adaptation of the Monte-Carlo Tree Search solution method, which together are capable of solving the underlying problem near-optimally. Our method is able to learn efficiently given a known factorization or also learn the factorization and the model parameters at the same time. We demonstrate that this approach is able to outperform current methods and tackle problems that were previously infeasible.

As described in the main paper, we collected a dataset D = (x(n), e(n), a(n), π(n))500 n=1. Environment Complexity. Our environments are 3D scenarios consisting of complex objects such as fridge, microwave and television with many states. These objects can be in different spatial relations with respect to other objects. For example, “bag of chips” can be found behind the television. Figure 1 shows some sample environments from our dataset. For example, an object of category television consists of 6 channels, volume level and power status. An object can have different values of states in different environment and different environment consists of different set of objects and their placement. For example, television might be powered on in one environment and closed in another, microwave might have an object inside it or not in different environment, etc. Moreover, there are often more than one object of the same category. For example, our environment typically have two books, five couches, four pillows etc. Objects of the same category can have different appearance. For example, a book can have the cover of a math book or a Guinness book of world record; resulting in complex object descriptions such as in “throw the sticky stuff in the bowl”. They can also have the same appearance making people use spatial relations or other means while describing them such as in “get me the cup next to the microwave”. This dataset is significantly more challenging compared to the 2D navigation dataset or GUI actions in windows dataset considered earlier.

We propose the neural programmer-interpreter (NPI): a recurrent and compositional neural network that learns to represent and execute programs. NPI has three learnable components: a task-agnostic recurrent core, a persistent key-value program memory, and domain-specific encoders that enable a single NPI to operate in multiple perceptually diverse environments with distinct affordances. By learning to compose lower-level programs to express higher-level programs, NPI reduces sample complexity and increases generalization ability compared to sequence-tosequence LSTMs. The program memory allows efficient learning of additional tasks by building on existing programs. NPI can also harness the environment (e.g. a scratch pad with read-write pointers) to cache intermediate results of computation, lessening the long-term memory burden on recurrent hidden units. In this work we train the NPI with fully-supervised execution traces; each program has example sequences of calls to the immediate subprograms conditioned on the input. Rather than training on a huge number of relatively weak labels, NPI learns from a small number of rich examples. We demonstrate the capability of our model to learn several types of compositional programs: addition, sorting, and canonicalizing 3D models. Furthermore, a single NPI learns to execute these programs and all 21 associated subprograms.

We present a unified framework for learning continuous control policies using backpropagation. It supports stochastic control by treating stochasticity in the Bellman equation as a deterministic function of exogenous noise. The product is a spectrum of general policy gradient algorithms that range from model-free methods with value functions to model-based methods without value functions. We use learned models but only require observations from the environment instead of observations from model-predicted trajectories, minimizing the impact of compounded model errors. We apply these algorithms first to a toy stochastic control problem and then to several physics-based control problems in simulation. One of these variants, SVG(1), shows the effectiveness of learning models, value functions, and policies simultaneously in continuous domains.