Question: Prospective insights into the next-generation e-sports infrastructure and uncovering its advantages.

Exploring the Immersive Realities: The Evolution of Presence in Virtual Environments

This paper delves into the transformative experience of presence in virtual environments, a phenomenon that has become increasingly significant with the advent of advanced gaming and e-sports infrastructures. Presence, the sensation of being physically located within a virtual world, is a crucial aspect that enhances user engagement and immersion. As we look towards the future of e-sports and gaming, the development of next-generation infrastructures will play a pivotal role in enhancing the subjective experience of presence. These advancements will not only push the boundaries of what is possible in virtual worlds but also unlock new dimensions of entertainment, competition, and social interaction. By examining the current state of presence technology and projecting its future trajectories, we can anticipate the benefits that these innovations will bring to e-sports enthusiasts and the broader gaming community. This includes improved realism, heightened emotional responses, and more profound connections between players, all of which contribute to a richer and more fulfilling virtual experience.

Human computation games (HCGs) can be seen as a paradigm that makes use of human brain power to solve computational problems as a byproduct of game play. In recent years, HCGs are increasing in popularity within various application areas which aim to harvest human problem solving skills by seamlessly integrating computational tasks into games. Thus, a proper system of describing such games is necessary in order to obtain a better understanding of the current body of work and identify new opportunities for future research directions. As a starting point, in this paper, we review existing literature on HCGs and present a typology consisting of 12 dimensions based on the key features of HCGs. With this typology, all HCGs could be classified and compared in a systematic manner.

The Internet of Things (IoT) holds big promises for Health Care, especially in Proactive Personal eHealth. To fulfil the promises major challenges must be overcome, particularly regarding privacy and security. This paper explores these issues, discusses use case scenarios, and advances a secure architecture framework. We close the paper with a discussion of the state of various standard organizations, in the conviction of the critical role they will play in making eHealth bloom.

* Archer's capability of attention and relaxation control during shooting process was evaluated using EEG technology. Attention and meditation algorithms were used to represent the levels of mental concentration and relaxation levels. Elite, mid-level, and novice archers were tested for short and long distance shootings in the archery field. Single channel EEG was recorded on the forehead (Fp1) during the shooting process, and attention and meditation levels were computed by real time. Four types of variations were defined based on the increasing and decreasing patterns of attention and meditation levels during shooting process. Elite archers showed increases in both attention and relaxation while mid-level archers showed increased attention but decreased relaxation. Elite archers also showed higher levels of attention at the release than mid-level and novice archers. Levels of attention and relaxation and their variation patterns were useful to categorize archers and to provide feedback in training.

The WTFast's Gamers Private Network (GPN®) is a client/server solution that makes online games faster. GPN® connects online video-game players with a common game service across a wide-area network. Online games are interactive competitions by individual players who compete in a virtual environment. Response time, latency and its predictability are keys to GPN® success and runs against the vast complexity of internet-wide systems. We have built an experimental network of virtualized GPN® components so as to carefully measure the statistics of latency for distributed Minecraft games and to do so in a controlled laboratory environment. This has led to a better understanding of the coupling between parameters such as: the number of players, the subset of players that are idle or active, the volume of packets exchanged, the size of packets, latency to and from the game servers, and time-series for most of those parameters. In this paper we present a mathematical model of those system game network parameters and show how it leads to: (1) realistic simulation of each of those network or game parameters, without relying on the experimental setup; (2) very large-scale numerical simulation of the game setup so as to explore various internet-wide performance scenarios that: (a) are impossible to isolate from internet “noise” in their real environment and; (b) would require vast supercomputing resources if they were to be simulated exhaustively. We motivate all elements of our mathematical model and estimate the savings in computational costs they will bring for very large-scale simulation of the GPN®. Such simulations will improve quality of service for GPN® systems and their reliability.

BACKGROUND
Active videogames (AVGs) may be useful for promoting physical activity for therapeutic uses, including for balance, rehabilitation, and management of illness or disease. The literature from 64 peer-reviewed publications that assessed health outcomes of AVGs for therapeutic purposes was synthesized.

MATERIALS AND METHODS
PubMed, Medline, and PyschInfo were queried for original studies related to the use of AVGs to improve physical outcomes in patients who were ill or undergoing rehabilitation related to balance, burn treatment, cancer, cerebral palsy, Down's syndrome, extremity dysfunction or amputation, hospitalization, lupus, Parkinson's disease, spinal injury, or stroke. The following inclusion criteria were used: (1) human subjects; (2) English language; (3) not duplicates; (4) new empirical data; and (5) tests an AVG, including commercially available or custom-designed. Studies were included regardless of participants' age or the study design.

RESULTS AND LIMITATIONS
Overall, the vast majority of studies demonstrated promising results for improved health outcomes related to therapy, including significantly greater or comparable effects of AVG play versus usual care. However, many studies were pilot trials with small, homogeneous samples, and many studies lacked a control or comparison group. Some trials tested multiweek or multimonth interventions, although many used a single bout of gameplay, and few included follow-up assessments to test sustainability of improved health.

CONCLUSIONS AND IMPLICATIONS
AVGs were acceptable and enjoyable to the populations examined and appear as a promising tool for balance, rehabilitation, and illness management. Future research directions and implications for clinicians are discussed.

Mobile Augmented Reality (MAR) blends the real world with digital objects especially in ubiquitous devices such as smartphones. The MAR applications provide an intelligent interface for users. In this, valuable digital information is advertised in physical spaces. However, the success of these applications is tied directly to the degree of user acceptance. This makes understanding the needs and expectations of the MAR’s potential users of paramount importance for designing and building the proper application. The objective of the paper is to expose an important gap in the development of novel applications in the virtual world. Previous research has shown that it is essential to study and understand the needs and expectations of the potential users of the upcoming application or system. Studying user needs and expectations before offering the developed application ensures a minimum level of acceptance and, of course, success. This paper presents a detailed study comprising of a userexperience (UX) evaluation of different prototypes through the use of three different UX evaluation methods. This kind of evaluation allows new developments to offer systems, which do not fail. The main contributions of this study are that it: 1) solicits expectations when consumers use MAR applications, 2) assesses the UX over different prototypes using three different metrics, 3) provides methodological insights on UX evaluation experiments and, 4) is useful for anyone who wants to develop handheld applications after understanding user expectations and how his experience should progress. The results of the study show that users value concreteness, realizability, personalization, novelty, intuitiveness and the usefulness of presented information. Paying attention to these factors can help develop more acceptable MAR applications and lead to more novel future designs.

The FuturICT project seeks to use the power of big data, analytic models grounded in complexity science, and the collective intelligence they yield for societal benefit. Accordingly, this paper argues that these new tools should not remain the preserve of restricted 110 The European Physical Journal Special Topics government, scientific or corporate élites, but be opened up for societal engagement and critique. To democratise such assets as a public good, requires a sustainable ecosystem enabling different kinds of stakeholder in society, including but not limited to, citizens and advocacy groups, school and university students, policy analysts, scientists, software developers, journalists and politicians. Our working name for envisioning a sociotechnical infrastructure capable of engaging such a wide constituency is the Global Participatory Platform (GPP). We consider what it means to develop a GPP at the different levels of data, models and deliberation, motivating a framework for different stakeholders to find their ecological niches at different levels within the system, serving the functions of (i) sensing the environment in order to pool data, (ii) mining the resulting data for patterns in order to model the past/present/future, and (iii) sharing and contesting possible interpretations of what those models might mean, and in a policy context, possible decisions. A research objective is also to apply the concepts and tools of complexity science and social science to the project’s own work. We therefore conceive the global participatory platform as a resilient, epistemic ecosystem, whose design will make it capable of selforganization and adaptation to a dynamic environment, and whose structure and contributions are themselves networks of stakeholders, challenges, issues, ideas and arguments whose structure and dynamics can be modelled and analysed.

Mobile computer-vision technology will soon become as ubiquitous as touch interfaces.

The Robot World Cup Initiative (RoboCup) is an attempt to foster AI and intelligent robotics research by providing a standard problem where wide range of technologies can be integrated and examined. The rst RoboCup competition will be held at IJCAI-97, Nagoya. In order for a robot team to actually perform a soccer game, various technologies must be incorporated including: design principles of autonomous agents, multi-agent collaboration, strategy acquisition, real-time reasoning, robotics, and sensor-fusion. Unlike AAAI robot competition, which is tuned for a single heavyduty slow-moving robot, RoboCup is a task for a team of multiple fast-moving robots under a dynamic environment. Although RoboCup's nal target is a world cup with real robots, RoboCup o ers a software platform for research on the software aspects of RoboCup. This paper describes technical challenges involved in RoboCup, rules, and simulation environment.

Gaming on demand is an emerging service that has recently started to garner prominence in the gaming industry. Cloud-based video games provide affordable, flexible, and high-performance solutions for end-users with constrained computing resources and enables them to play high-end graphic games on low-end thin clients. Despite its advantages, cloud gaming's Quality of Experience (QoE) suffers from high and varying end-to-end delay. Since the significant part of computational processing, including game rendering and video compression, is performed in data centers, controlling the transfer of information within the cloud has an important impact on the quality of cloud gaming services. In this article, a novel method for minimizing the end-to-end latency within a cloud gaming data center is proposed. We formulate an optimization problem for reducing delay, and propose a Lagrangian Relaxation (LR) time-efficient heuristic algorithm as a practical solution. Simulation results indicate that the heuristic method can provide close-to-optimal solutions. Also, the proposed model reduces end-to-end delay and delay variation by almost 11% and 13.5%, respectively, and outperforms the existing server-centric and network-centric models. As a byproduct, our proposed method also achieves better fairness among multiple competing players by almost 45%, on average, in comparison with existing methods.


Question: A point pair feature method for object detection in random bin picking scenarios.

This paper explores the use of visibility context in 3D object recognition within range images, a critical component for enhancing robotic manipulation and navigation capabilities. Despite ongoing advancements in the field, accurately identifying and localizing objects in range images, particularly in environments characterized by occlusion and clutter, remains a significant challenge. Visibility context offers a novel perspective by considering the visibility patterns of object surfaces in relation to their surroundings, which can significantly improve recognition accuracy in complex scenes. This approach is particularly relevant for robotic applications where the ability to effectively handle random bin picking tasks is crucial. By leveraging visibility context, robots can better discern object boundaries and orientations, even when objects are partially occluded or densely packed, thereby enhancing their ability to perform tasks with precision and efficiency. This research contributes to the broader goal of developing more sophisticated and adaptable robotic systems capable of operating in real-world, unstructured environments.

This paper describes a visual object detection framework that is capable of processing images extremely rapidly while achieving high detection rates. There are three key contributions. The first is the introduction of a new image representation called the “Integral Image” which allows the features used by our detector to be computed very quickly. The second is a learning algorithm, based on AdaBoost, which selects a small number of critical visual features and yields extremely efficient classifiers [6]. The third contribution is a method for combining classifiers in a “cascade” which allows background regions of the image to be quickly discarded while spending more computation on promising object-like regions. A set of experiments in the domain of face detection are presented. The system yields face detection performace comparable to the best previous systems [18, 13, 16, 12, 1]. Implemented on a conventional desktop, face detection proceeds at 15 frames per second.

We propose a novel and general framework to learn compact but highly discriminative floating-point and binary local feature descriptors. By leveraging the boosting-trick we first show how to efficiently train a compact floating-point descriptor that is very robust to illumination and viewpoint changes. We then present the main contribution of this paper—a binary extension of the framework that demonstrates the real advantage of our approach and allows us to compress the descriptor even further. Each bit of the resulting binary descriptor, which we call BinBoost, is computed with a boosted binary hash function, and we show how to efficiently optimize the hash functions so that they are complementary, which is key to compactness and robustness. As we do not put any constraints on the weak learner configuration underlying each hash function, our general framework allows us to optimize the sampling patterns of recently proposed hand-crafted descriptors and significantly improve their performance. Moreover, our boosting scheme can easily adapt to new applications and generalize to other types of image data, such as faces, while providing state-of-the-art results at a fraction of the matching time and memory footprint.

We present a 3-D shape-based object recognition system for simultaneous recognition of multiple objects in scenes containing clutter and occlusion. Recognition is based on matching surfaces by matching points using the spin-image representation. The spin-image is a data level shape descriptor that is used to match surfaces represented as surface meshes. We present a compression scheme for spin-images that results in efficient multiple object recognition which we verify with results showing the simultaneous recognition of multiple objects from a library of 20 models. Furthermore, we demonstrate the robust performance of recognition in the presence of clutter and occlusion through analysis of recognition trials on 100 scenes. This research was performed at Carnegie Mellon University and was supported by the US Department of Energy under contract DE-AC21-92MC29104. 1

In our recent work [1], [2], we proposed Point Feature Histograms (PFH) as robust multi-dimensional features which describe the local geometry around a point p for 3D point cloud datasets. In this paper, we modify their mathematical expressions and perform a rigorous analysis on their robustness and complexity for the problem of 3D registration for overlapping point cloud views. More concretely, we present several optimizations that reduce their computation times drastically by either caching previously computed values or by revising their theoretical formulations. The latter results in a new type of local features, called Fast Point Feature Histograms (FPFH), which retain most of the discriminative power of the PFH. Moreover, we propose an algorithm for the online computation of FPFH features for realtime applications. To validate our results we demonstrate their efficiency for 3D registration and propose a new sample consensus based method for bringing two datasets into the convergence basin of a local non-linear optimizer: SAC-IA (SAmple Consensus Initial Alignment).

In this paper, we present a new approach for establishing correspondences between sparse image features related by an unknown nonrigid mapping and corrupted by clutter and occlusion, such as points extracted from images of different instances of the same object category. We formulate this matching task as an energy minimization problem by defining an elaborate objective function of the appearance and the spatial arrangement of the features. Optimization of this energy is an instance of graph matching, which is in general an NP-hard problem. We describe a novel graph matching optimization technique, which we refer to as dual decomposition (DD), and demonstrate on a variety of examples that this method outperforms existing graph matching algorithms. In the majority of our examples, DD is able to find the global minimum within a minute. The ability to globally optimize the objective allows us to accurately learn the parameters of our matching model from training examples. We show on several matching tasks that our learned model yields results superior to those of state-of-the-art methods.

Localizing functional regions of objects or affordances is an important aspect of scene understanding and relevant for many robotics applications. In this work, we introduce a pixel-wise annotated affordance dataset of 3090 images containing 9916 object instances. Since parts of an object can have multiple affordances, we address this by a convolutional neural network for multilabel affordance segmentation. We also propose an approach to train the network from very few keypoint annotations. Our approach achieves a higher affordance detection accuracy than other weakly supervised methods that also rely on keypoint annotations or image annotations as weak supervision.

We introduce a framework for learning regrasping behaviors based on tactile data. First, we present a grasp stability predictor that uses spatio-temporal tactile features collected from the early-object-lifting phase to predict the grasp outcome with a high accuracy. Next, the trained predictor is used to supervise and provide feedback to a reinforcement learning algorithm that learns the required grasp adjustments based on tactile feedback. Our results gathered over more than 50 hours of real robot experiments indicate that the robot is able to predict the grasp outcome with 93% accuracy. In addition, the robot is able to improve the grasp success rate from 42% when randomly grasping an object to up to 97% when allowed to regrasp the object in case of a predicted failure.

Hand segmentation for hand-object interaction is a necessary preprocessing step in many applications such as augmented reality, medical application, and human-robot interaction. However, typical methods are based on color information which is not robust to objects with skin color, skin pigment difference, and light condition variations. Thus, we propose hand segmentation method for hand-object interaction using only a depth map. It is challenging because of the small depth difference between a hand and objects during an interaction. To overcome this challenge, we propose the two-stage random decision forest (RDF) method consisting of detecting hands and segmenting hands. To validate the proposed method, we demonstrate results on the publicly available dataset of hand segmentation for hand-object interaction. The proposed method achieves high accuracy in short processing time comparing to the other state-of-the-art methods.

We assess the applicability of several popular learning methods for the problem of recognizing generic visual categories with invariance to pose, lighting, and surrounding clutter. A large dataset comprising stereo image pairs of 50 uniform-colored toys under 36 azimuths, 9 elevations, and 6 lighting conditions was collected (for a total of 194,400 individual images). The objects were 10 instances of 5 generic categories: four-legged animals, human figures, airplanes, trucks, and cars. Five instances of each category were used for training, and the other five for testing. Low-resolution grayscale images of the objects with various amounts of variability and surrounding clutter were used for training and testing. Nearest neighbor methods, support vector machines, and convolutional networks, operating on raw pixels or on PCA-derived features were tested. Test error rates for unseen object instances placed on uniform backgrounds were around 13% for SVM and 7% for convolutional nets. On a segmentation/recognition task with highly cluttered images, SVM proved impractical, while convolutional nets yielded 16/7% error. A real-time version of the system was implemented that can detect and classify objects in natural scenes at around 10 frames per second.

Estimating the distance of a target object from a single image is a challenging task since a large variation in the object appearance makes the regression of the distance difficult. In this paper, to tackle such the challenge, we propose 2.5D anchors which provide the candidate of distances based on a perspective camera model. This candidate is expected to relax the difficulty of the regression model since only the residual from the candidate distance needs to be taken into account. We show the effectiveness of the regression with our proposed anchors, by comparing with ordinary regression methods and state-of-the-art 3D object detection methods, through Pascal 3D+ TV monitor and KITTI car experiments. In addition, we also show an example of practical uses of our proposed method in a real-time system, robot navigation, by integrating with ROS-based simultaneous localization and mapping.


Question: Object detection for random bin picking using a point pair feature-based method.

This paper introduces an effective 3D object recognition and pose estimation technique for the grasping of known objects within cluttered and partially obscured settings. Unlike conventional appearance-based methods, our approach exclusively utilizes 3D geometry data. The methodology hinges on a robust geometric descriptor, an efficient hashing mechanism, and an optimized, localized RANSAC-like sampling strategy. Each object is characterized by a model composed of a set of points with associated surface normals. Our method is capable of concurrently identifying multiple instances of these models and estimating their orientations within the scene. Extensive testing confirms the method's efficacy in handling noisy, cluttered, and unsegmented range scans where only portions of the objects are visible. The algorithm's primary process features linear time complexity, ensuring rapid recognition speeds and enabling seamless integration into ongoing manipulation tasks. The method's practicality is demonstrated through experimental validation with a 7-degrees-of-freedom Cartesian impedance controlled robot, showcasing its ability to grasp objects from a complex, randomly stacked pile. This application exemplifies the synergy between computer vision and soft robotics in creating a robotic system adept at operating within unstructured and obscured environments.

We describe a method of mobile robot monocular visual navigation, which uses multiple visual cues to detect and segment the ground plane in the robot’s field of view. Corner points are tracked through an image sequence and grouped into coplanar regions using a method which we call an H-based tracker. The H-based tracker employs planar homographys and is initialised by 5-point planar projective invariants. This allows us to detect ground plane patches and the colour within such patches is subsequently modelled. These patches are grown by colour classification to give a ground plane segmentation, which is then used as an input to a new variant of the artificial potential field algorithm.

We propose a machine learning approach to the perception of a stable robotic grasp based on tactile feedback and hand kinematic data, which we call blind grasping. We first discuss a method for simulating tactile feedback using a soft finger contact model in GraspIt!, which is a robotic grasping simulator [10]. Using this simulation technique, we compute tactile contacts of thousands of grasps with a robotic hand using the Columbia Grasp Database [6]. The tactile contacts along with the hand kinematic data are then input to a Support Vector Machine (SVM) which is trained to estimate the stability of a given grasp based on this tactile feedback and also the robotic hand kinematics. Experimental results indicate that the tactile feedback along with the hand kinematic data carry meaningful information for the prediction of the stability of a blind robotic grasp.

In this paper we propose and evaluate an algorithm that learns a similarity measure for comparing never seen objects. The measure is learned from pairs of training images labeled "same" or "different". This is far less informative than the commonly used individual image labels (e.g., "car model X"), but it is cheaper to obtain. The proposed algorithm learns the characteristic differences between local descriptors sampled from pairs of "same" and "different" images. These differences are vector quantized by an ensemble of extremely randomized binary trees, and the similarity measure is computed from the quantized differences. The extremely randomized trees are fast to learn, robust due to the redundant information they carry and they have been proved to be very good clusterers. Furthermore, the trees efficiently combine different feature types (SIFT and geometry). We evaluate our innovative similarity measure on four very different datasets and consistently outperform the state-of-the-art competitive approaches.

Reinforcement learning provides a powerful and flexible framework for automated acquisition of robotic motion skills. However, applying reinforcement learning requires a sufficiently detailed representation of the state, including the configuration of task-relevant objects. We present an approach that automates state-space construction by learning a state representation directly from camera images. Our method uses a deep spatial autoencoder to acquire a set of feature points that describe the environment for the current task, such as the positions of objects, and then learns a motion skill with these feature points using an efficient reinforcement learning method based on local linear models. The resulting controller reacts continuously to the learned feature points, allowing the robot to dynamically manipulate objects in the world with closed-loop control. We demonstrate our method with a PR2 robot on tasks that include pushing a free-standing toy block, picking up a bag of rice using a spatula, and hanging a loop of rope on a hook at various positions. In each task, our method automatically learns to track task-relevant objects and manipulate their configuration with the robot's arm.

This paper introduces Push-Net, a deep recurrent neural network model, which enables a robot to push objects of unknown physical properties for re-positioning and re-orientation, using only visual camera images as input. The unknown physical properties is a major challenge for pushing. Push-Net overcomes the challenge by tracking a history of push interactions with an LSTM module and training an auxiliary objective function that estimates an object’s center of mass. We trained Push-Net entirely in simulation and tested it extensively on many different objects in both simulation and on two real robots, a Fetch arm and a Kinova MICO arm. Experiments suggest that Push-Net is robust and efficient. It achieved over 97% success rate in simulation on average and succeeded in all real robot experiments with a small number of pushes.

To understand driving environments effectively, it is important to achieve accurate detection and classification of objects detected by sensor-based intelligent vehicle systems, which are significantly important tasks. Object detection is performed for the localization of objects, whereas object classification recognizes object classes from detected object regions. For accurate object detection and classification, fusing multiple sensor information into a key component of the representation and perception processes is necessary. In this paper, we propose a new object-detection and classification method using decision-level fusion. We fuse the classification outputs from independent unary classifiers, such as 3D point clouds and image data using a convolutional neural network (CNN). The unary classifiers for the two sensors are the CNN with five layers, which use more than two pre-trained convolutional layers to consider local to global features as data representation. To represent data using convolutional layers, we apply region of interest (ROI) pooling to the outputs of each layer on the object candidate regions generated using object proposal generation to realize color flattening and semantic grouping for charge-coupled device and Light Detection And Ranging (LiDAR) sensors. We evaluate our proposed method on a KITTI benchmark dataset to detect and classify three object classes: cars, pedestrians and cyclists. The evaluation results show that the proposed method achieves better performance than the previous methods. Our proposed method extracted approximately 500 proposals on a 1226 × 370 image, whereas the original selective search method extracted approximately 10 6 × n proposals. We obtained classification performance with 77.72% mean average precision over the entirety of the classes in the moderate detection level of the KITTI benchmark dataset.

A method is proposed that can generate a ranked list of plausible three-dimensional hand configurations that best match an input image. Hand pose estimation is formulated as an image database indexing problem, where the closest matches for an input hand image are retrieved from a large database of synthetic hand images. In contrast to previous approaches, the system can function in the presence of clutter, thanks to two novel clutter-tolerant indexing methods. First, a computationally efficient approximation of the image-to-model chamfer distance is obtained by embedding binary edge images into a high-dimensional Euclidean space. Second, a general-purpose, probabilistic line matching method identifies those line segment correspondences between model and input images that are the least likely to have occurred by chance. The performance of this cluttertolerant approach is demonstrated in quantitative experiments with hundreds of real hand images.

We describe a model based recognition system, called LEWIS, for the identification of planar objects based on a projectively invariant representation of shape. The advantages of this shape description include simple model acquisition (direct from images), no need for camera calibration or object pose computation, and the use of index functions. We describe the feature construction and recognition algorithms in detail and provide an analysis of the combinatorial advantages of using index functions. Index functions are used to select models from a model base and are constructed from projective invariants based on algebraic curves and a canonical projective coordinate frame. Examples are given of object recognition from images of real scenes, with extensive object libraries. Successful recognition is demonstrated despite partial occlusion by unmodelled objects, and realistic lighting conditions.

We present a new approach for selecting image pairs that are more likely to match in Structure from Motion (SfM). We propose to use Jaccard Similarity (JacS) which shows how many different visual words is shared by an image pair. In our method, the similarity between images is evaluated using JacS of bag-of-visual-words in addition to tf-idf, which is popular for this purpose. To evaluate the efficiency of our method, we carry out experiments on our original datasets as well as on “Pantheon” dataset, which is derived from Flickr. The result of our method using both JacS and tf-idf is better than the results of a standard method using tf-idf only.

We present a new methodology for exact robot motion planning and control that unifies the purely kinematic path planning problem with the lower level feedback controller design. Complete information about the freespace and goal is encoded in the form of a special artificial potential function a navigation function that connects the kinematic planning problem with the dynamic execution problem in a provably correct fashion. The navigation function automatically gives rise to a bounded-torque feedback controller for the robot's actuators that guarantees collision-free motion and convergence to the destination from almost all initial free configurations. Since navigation functions exist for any robot and obstacle course, our methodology is completely general in principle. However, this paper is mainly concerned with certain constructive techniques for a particular class of motion planning problems. Specifically, we present a formula for navigation functions that guide a point-mass robot in a generalized sphere world. The simplest member of this family is a space obtained by puncturing a disc by an arbitrary number of smaller disjoint discs representing obstacles. The other spaces are obtained from this model by a suitable coordinate transformation that we show how to build. Our constructions exploit these coordinate transformations to adapt a navigation function on the model space to its more geometrically complicated (but topologically equivalent) instances. The formula that we present admits sphere-worlds of arbitrary dimension and is directly applicable to configuration spaces whose forbidden regions can be modeled by such generalized discs. We have implemented these navigation functions on planar scenarios, and simulation results are provided throughout the paper. Disciplines Robotics Comments Copyright 1992 IEEE. Reprinted from IEEE Transactions on Robotics and Automation, Volume 8, Issue 5, October 1992, pages 501-518. This material is posted here with permission of the IEEE. Such permission of the IEEE does not in any way imply IEEE endorsement of any of the University of Pennsylvania's products or services. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to pubs-permissions@ieee.org. By choosing to view this document, you agree to all provisions of the copyright laws protecting it. NOTE: At the time of publication, Daniel Koditschek was affiliated with the University of Michigan. Currently, he is a faculty member at the School of Engineering of the University of Pennsylvania. This journal article is available at ScholarlyCommons: http://repository.upenn.edu/ese_papers/323


Question: A point pair feature method for object detection in random bin picking scenarios.

This work introduces an enhanced methodology for the real-time detection of multiple specific 3D objects, building upon the LINE2D/LINEMOD representation framework initially presented by Hinterstoisser et al. Our proposed approach makes two significant improvements to the template-based detection method. Firstly, we advocate for the discriminative training of templates, a process that can be seamlessly integrated into the collection of example images, taking only a few milliseconds. This refinement significantly boosts the detector's accuracy. Secondly, we introduce a cascade-based scheme that accelerates the detection process. Given the rapid detection speed of individual objects, the addition of new objects incurs minimal computational overhead, ensuring scalability. Our experimental results demonstrate the capability to efficiently manage 10-30 3D objects at frame rates exceeding 10fps, utilizing a single CPU core. Comparative analysis across three distinct datasets confirms our method's superiority in terms of both speed and accuracy, outperforming current state-of-the-art techniques. This holds true for both monocular color images (using LINE2D) and RGBD images (employing LINEMOD). Furthermore, we present a novel, challenging dataset comprising 12 objects, designed to serve as a benchmark for future methods focusing on monocular color images.

The detection of small road hazards, such as lost cargo, is a vital capability for self-driving cars. We tackle this challenging and rarely addressed problem with a vision system that leverages appearance, contextual as well as geometric cues. To utilize the appearance and contextual cues, we propose a new deep learning-based obstacle detection framework. Here a variant of a fully convolutional network is proposed to predict a pixel-wise semantic labeling of (i) free-space, (ii) on-road unexpected obstacles, and (iii) background. The geometric cues are exploited using a state-of-the-art detection approach that predicts obstacles from stereo input images via model-based statistical hypothesis tests. We present a principled Bayesian framework to fuse the semantic and stereo-based detection results. The mid-level Stixel representation is used to describe obstacles in a flexible, compact and robust manner. We evaluate our new obstacle detection system on the Lost and Found dataset, which includes very challenging scenes with obstacles of only 5 cm height. Overall, we report a major improvement over the state-of-the-art, with a performance gain of 27.4%. In particular, we achieve a detection rate of over 90% for distances of up to 50 m. Our system operates at 22 Hz on our self-driving platform.

We take a new approach to computing dense scene flow between a pair of consecutive RGB-D frames. We exploit the availability of depth data by seeking correspondences with respect to patches specified not as the pixels inside square windows, but as the 3D points that are the inliers of spheres in world space. Our primary contribution is to show that by reasoning in terms of such patches under 6 DoF rigid body motions in 3D, we succeed in obtaining compelling results at displacements large and small without relying on either of two simplifying assumptions that pervade much of the earlier literature: brightness constancy or local surface planarity. As a consequence of our approach, our output is a dense field of 3D rigid body motions, in contrast to the 3D translations that are the norm in scene flow. Reasoning in our manner additionally allows us to carry out occlusion handling using a 6 DoF consistency check for the flow computed in both directions and a patchwise silhouette check to help reason about alignments in occlusion areas, and to promote smoothness of the flow fields using an intuitive local rigidity prior. We carry out our optimization in two steps, obtaining a first correspondence field using an adaptation of PatchMatch, and subsequently using alpha-expansion to jointly handle occlusions and perform regularization. We show attractive flow results on challenging synthetic and real-world scenes that push the practical limits of the aforementioned assumptions.

We propose an approach to estimate 3D human pose in real world units from a single RGBD image and show that it exceeds performance of monocular 3D pose estimation approaches from color as well as pose estimation exclusively from depth. Our approach builds on robust human keypoint detectors for color images and incorporates depth for lifting into 3D. We combine the system with our learning from demonstration framework to instruct a service robot without the need of markers. Experiments in real world settings demonstrate that our approach enables a PR2 robot to imitate manipulation actions observed from a human teacher.

In our recent work [1], [2], we proposed Point Feature Histograms (PFH) as robust multi-dimensional features which describe the local geometry around a point p for 3D point cloud datasets. In this paper, we modify their mathematical expressions and perform a rigorous analysis on their robustness and complexity for the problem of 3D registration for overlapping point cloud views. More concretely, we present several optimizations that reduce their computation times drastically by either caching previously computed values or by revising their theoretical formulations. The latter results in a new type of local features, called Fast Point Feature Histograms (FPFH), which retain most of the discriminative power of the PFH. Moreover, we propose an algorithm for the online computation of FPFH features for realtime applications. To validate our results we demonstrate their efficiency for 3D registration and propose a new sample consensus based method for bringing two datasets into the convergence basin of a local non-linear optimizer: SAC-IA (SAmple Consensus Initial Alignment).

Feature description for the 3D local shape in the presence of noise, varying mesh resolutions, clutter and occlusion is a quite challenging task in 3D computer vision. This paper tackles the problem by proposing a new local reference frame (LRF) together with a novel triple orthogonal local depth images (TOLDI) representation, forming the TOLDI method for local shape description. Compared with previous methods, TOLDI manages to perform efficient, distinctive and robust description for the 3D local surface simultaneously under various feature matching contexts. The proposed LRF differs from many prior ones in its calculation of the z-axis and x-axis, the z-axis is calculated using the normal of the keypoint and the x-axis is computed by aggregating the weighted projection vectors of the radius neighbors. TOLDI feature descriptors are then obtained by concatenating three local depth images (LDI) captured from three orthogonal view planes in the LRF into feature vectors. The performance of our TOLDI approach is rigorously evaluated on several public datasets, which contain three major surface matching scenarios, namely shape retrieval, object recognition and 3D registration. Experimental results and comparisons with the state-of-the-arts validate the effectiveness, robustness, high efficiency, and overall superiority of our method. Our method is also applied to aligning 3D object ∗Corresponding author Email addresses: jqyang@hust.edu.cn (Jiaqi Yang), hangfanzq@163.com (Qian Zhang), Yang Xiao@hust.edu.cn (Yang Xiao), zgcao@hust.edu.cn (Zhiguo Cao) Preprint submitted to Journal of LTEX Templates November 23, 2016 and indoor scene point clouds obtained by different devices (i.e., LiDAR and Kinect), the accurate outcomes further confirm the effectiveness of our method.

Real-time simultaneous tracking of hands manipulating and interacting with external objects has many potential applications in augmented reality, tangible computing, and wearable computing. However, due to difficult occlusions, fast motions, and uniform hand appearance, jointly tracking hand and object pose is more challenging than tracking either of the two separately. Many previous approaches resort to complex multi-camera setups to remedy the occlusion problem and often employ expensive segmentation and optimization steps which makes real-time tracking impossible. In this paper, we propose a real-time solution that uses a single commodity RGB-D camera. The core of our approach is a 3D articulated Gaussian mixture alignment strategy tailored to handobject tracking that allows fast pose optimization. The alignment energy uses novel regularizers to address occlusions and hand-object contacts. For added robustness, we guide the optimization with discriminative part classification of the hand and segmentation of the object. We conducted extensive experiments on several existing datasets and introduce a new annotated hand-object dataset. Quantitative and qualitative results show the key advantages of our method: speed, accuracy, and robustness.

Hand segmentation for hand-object interaction is a necessary preprocessing step in many applications such as augmented reality, medical application, and human-robot interaction. However, typical methods are based on color information which is not robust to objects with skin color, skin pigment difference, and light condition variations. Thus, we propose hand segmentation method for hand-object interaction using only a depth map. It is challenging because of the small depth difference between a hand and objects during an interaction. To overcome this challenge, we propose the two-stage random decision forest (RDF) method consisting of detecting hands and segmenting hands. To validate the proposed method, we demonstrate results on the publicly available dataset of hand segmentation for hand-object interaction. The proposed method achieves high accuracy in short processing time comparing to the other state-of-the-art methods.

Search-and-rescue (SAR) robots operating in post-disaster urban areas need to accurately identify physical site information to perform navigation, mapping and manipulation tasks. This can be achieved by acquiring a 3D point cloud of the environment and performing object recognition from the point cloud data. However, this task is complicated by the unstructured environments and potentially-deformed objects encountered during disaster relief operations. Current 3D object recognition methods rely on point cloud input acquired under suitable conditions and do not consider deformations such as outlier noise, bending and truncation. This work introduces a deep learning architecture for 3D class recognition from point clouds of deformed building elements. The classification network, consisting of stacked convolution and average pooling layers applied directly to point coordinates, was trained using point clouds sampled from a database of mesh models. The proposed method achieves robustness to input variability using point sorting, resampling, and rotation normalization techniques. Experimental results on synthetically-deformed object datasets show that the proposed method outperforms the conventional deep learning methods in terms of classification accuracy and computational efficiency.

This paper reports on a method to perform robust visual relocalization between temporally separated sets of underwater images gathered by a robot. The place recognition and relocalization problem is more challenging in the underwater environment mainly due to three factors: 1) changes in illumination; 2) long-term changes in the visual appearance of features because of phenomena like biofouling on man-made structures and growth or movement in natural features; and 3) low density of visually salient features for image matching. To address these challenges, a patch-based feature matching approach is proposed, which uses image segmentation and local intensity contrast to locate salient patches and HOG description to make correspondences between patches. Compared to traditional point-based features that are sensitive to dramatic appearance changes underwater, patch-based features are able to encode higher level information such as shape or structure which tends to persist across years in underwater environments. The algorithm is evaluated on real data, from multiple years, collected by a Hovering Autonomous Underwater Vehicle for ship hull inspection. Results in relocalization performance across missions from different years are compared to other traditional methods.

We propose a machine learning approach to the perception of a stable robotic grasp based on tactile feedback and hand kinematic data, which we call blind grasping. We first discuss a method for simulating tactile feedback using a soft finger contact model in GraspIt!, which is a robotic grasping simulator [10]. Using this simulation technique, we compute tactile contacts of thousands of grasps with a robotic hand using the Columbia Grasp Database [6]. The tactile contacts along with the hand kinematic data are then input to a Support Vector Machine (SVM) which is trained to estimate the stability of a given grasp based on this tactile feedback and also the robotic hand kinematics. Experimental results indicate that the tactile feedback along with the hand kinematic data carry meaningful information for the prediction of the stability of a blind robotic grasp.


Question: Object detection using point pair features for random bin picking tasks.

A novel object recognition system has been devised, employing a distinctive set of local image features that demonstrate invariance to transformations such as scaling, translation, rotation, and partial resilience to alterations in illumination and affine or three-dimensional projections. These features mirror the characteristics of neurons found in the inferior temporal cortex, crucial for object recognition in primate vision. The system employs a multi-stage filtering technique to detect these features, ensuring their stability across scale space. Image descriptors, or keys, are created to accommodate local geometric deformations by encoding blurred image gradients across various orientation planes and scales. These keys serve as inputs to a nearest-neighbor indexing mechanism, which facilitates the identification of potential object matches. The final validation of each match is accomplished through the calculation of a low-residual least-squares solution for the unknown model parameters. Empirical evaluations have demonstrated that the system is capable of achieving robust object recognition in complex, partially occluded scenes, all while maintaining a processing time of less than 2 seconds.

This system's robustness and efficiency make it particularly suitable for applications such as random bin picking, where objects are often in cluttered environments and may be partially occluded. The use of scale-invariant features ensures that the system can recognize objects regardless of their size in the image, a critical requirement for bin picking tasks where objects may be at various distances from the camera. Moreover, the system's ability to handle occlusions and cluttered scenes is essential in real-world bin picking scenarios, where objects are typically jumbled together. The low computation time ensures that the system can process images rapidly, making it viable for real-time or near real-time applications. Overall, the system's performance characteristics make it a promising solution for automating the challenging task of object detection in random bin picking scenarios.

As a core robotic and vision problem, camera and range sensor calibration have been researched intensely over the last decades. However, robotic research efforts still often get heavily delayed by the requirement of setting up a calibrated system consisting of multiple cameras and range measurement units. With regard to removing this burden, we present a toolbox with web interface for fully automatic camera-to-camera and camera-to-range calibration. Our system is easy to setup and recovers intrinsic and extrinsic camera parameters as well as the transformation between cameras and range sensors within one minute. In contrast to existing calibration approaches, which often require user intervention, the proposed method is robust to varying imaging conditions, fully automatic, and easy to use since a single image and range scan proves sufficient for most calibration scenarios. Experimentally, we demonstrate that the proposed checkerboard corner detector significantly outperforms current state-of-the-art. Furthermore, the proposed camera-to-range registration method is able to discover multiple solutions in the case of ambiguities. Experiments using a variety of sensors such as grayscale and color cameras, the Kinect 3D sensor and the Velodyne HDL-64 laser scanner show the robustness of our method in different indoor and outdoor settings and under various lighting conditions.

Loop closure detection benefits simultaneous localization and mapping (SLAM) in building a consistent map of the environment by reducing the accumulate error. Handcrafted features have been successfully used in traditional approaches, whereas in this paper, we show that unsupervised features extracted by deep learning models, can improves the accuracy of loop closure detection. In particular, we employ a cascaded deep network, namely the PCANet, to extract features as image descriptors. We tested the performance of our proposed method on open datasets to compare with traditional approaches. We found that the PCANet features outperform state-of-the-art handcrafted competitors, and are computational efficient to be implemented in practical robotics.

This paper introduces Push-Net, a deep recurrent neural network model, which enables a robot to push objects of unknown physical properties for re-positioning and re-orientation, using only visual camera images as input. The unknown physical properties is a major challenge for pushing. Push-Net overcomes the challenge by tracking a history of push interactions with an LSTM module and training an auxiliary objective function that estimates an object’s center of mass. We trained Push-Net entirely in simulation and tested it extensively on many different objects in both simulation and on two real robots, a Fetch arm and a Kinova MICO arm. Experiments suggest that Push-Net is robust and efficient. It achieved over 97% success rate in simulation on average and succeeded in all real robot experiments with a small number of pushes.

In this paper, a new method based on deep learning for robotics autonomous navigation is presented. Different from the most traditional methods based on fixed models, a convolutional neural network (CNN) modelling technique in Deep learning is selected to extract the feature inspired by the working pattern of the biological brain. This neural network model has muti-layer features where the ambient scenes can be recognized and useful information such as the location of door can be identified. The extracted information can be used for robot navigation, so does the robot can approach the target accurately. In the field experiments, detecting doors and predicting the door poses such tasks are designed in the indoor environment to verify the proposed method. The experimental results demonstrate that the doors can be identified with good performance and the deep learning model is suitable for robot navigation.

We present a new methodology for exact robot motion planning and control that unifies the purely kinematic path planning problem with the lower level feedback controller design. Complete information about the freespace and goal is encoded in the form of a special artificial potential function a navigation function that connects the kinematic planning problem with the dynamic execution problem in a provably correct fashion. The navigation function automatically gives rise to a bounded-torque feedback controller for the robot's actuators that guarantees collision-free motion and convergence to the destination from almost all initial free configurations. Since navigation functions exist for any robot and obstacle course, our methodology is completely general in principle. However, this paper is mainly concerned with certain constructive techniques for a particular class of motion planning problems. Specifically, we present a formula for navigation functions that guide a point-mass robot in a generalized sphere world. The simplest member of this family is a space obtained by puncturing a disc by an arbitrary number of smaller disjoint discs representing obstacles. The other spaces are obtained from this model by a suitable coordinate transformation that we show how to build. Our constructions exploit these coordinate transformations to adapt a navigation function on the model space to its more geometrically complicated (but topologically equivalent) instances. The formula that we present admits sphere-worlds of arbitrary dimension and is directly applicable to configuration spaces whose forbidden regions can be modeled by such generalized discs. We have implemented these navigation functions on planar scenarios, and simulation results are provided throughout the paper. Disciplines Robotics Comments Copyright 1992 IEEE. Reprinted from IEEE Transactions on Robotics and Automation, Volume 8, Issue 5, October 1992, pages 501-518. This material is posted here with permission of the IEEE. Such permission of the IEEE does not in any way imply IEEE endorsement of any of the University of Pennsylvania's products or services. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to pubs-permissions@ieee.org. By choosing to view this document, you agree to all provisions of the copyright laws protecting it. NOTE: At the time of publication, Daniel Koditschek was affiliated with the University of Michigan. Currently, he is a faculty member at the School of Engineering of the University of Pennsylvania. This journal article is available at ScholarlyCommons: http://repository.upenn.edu/ese_papers/323

In this paper, we propose a new learning scheme for generating geometric correspondences to be used for visual odometry. A convolutional neural network (CNN) combined with a recurrent neural network (RNN) are trained together to detect the location of keypoints as well as to generate corresponding descriptors in one unified structure. The network is optimized by warping points from source frame to reference frame, with a rigid body transform. Essentially, learning from warping. The overall training is focused on movements of the camera rather than movements within the image, which leads to better consistency in the matching and ultimately better motion estimation. Experimental results show that the proposed method achieves better results than both related deep learning and hand crafted methods. Furthermore, as a demonstration of the promise of our method we use a naive SLAM implementation based on these keypoints and get a performance on par with ORB-SLAM.

|We propose a reliable method to detect occluding contours and occlusion by active binocular stereo with an occluding contour model that describes the correspondence between the contour of curved objects and its projected image. Applying the image ow generated by moving one camera to the model, we can restrict possible matched points seen by the other camera. We detect occluding contours by tting these matched points to the model. This method can nd occluding contours and occlusion more reliably than conventional ones because stereo matching is performed by using the geometric constraint based on the occluding contour model instead of heuristic constraints. Experimental results of the proposed method applied to the actual scene are presented.

Insertion is a challenging haptic and visual control problem with significant practical value for manufacturing. Existing approaches in the model-based robotics community can be highly effective when task geometry is known, but are complex and cumbersome to implement, and must be tailored to each individual problem by a qualified engineer. Within the learning community there is a long history of insertion research, but existing approaches are typically either too sample-inefficient to run on real robots, or assume access to high-level object features, e.g. socket pose. In this paper we show that relatively minor modifications to an off-the-shelf Deep-RL algorithm (DDPG), combined with a small number of human demonstrations, allows the robot to quickly learn to solve these tasks efficiently and robustly. Our approach requires no modeling or simulation, no parameterized search or alignment behaviors, no vision system aside from raw images, and no reward shaping. We evaluate our approach on a narrowclearance peg-insertion task and a deformable clip-insertion task, both of which include variability in the socket position. Our results show that these tasks can be solved reliably on the real robot in less than 10 minutes of interaction time, and that the resulting policies are robust to variance in the socket position and orientation.

Using vision systems with SCARA robotic manipulators is beneficial in many industrial applications specially in automated object picking and placing tasks. In this paper, the equations of forward and inverse kinematics of a SCARA manipulator are derived using D-H parameters convention. The mechanical structure, as well as the electrical, electronic and pneumatic components of the manipulator were designed and developed. The manipulator's control software, animation and simulation were developed using LabVIEW and Arduino. In addition to that, a vision system was developed and integrated with the manipulator system where a mobile phone camera was used to acquire images, then transmit them wirelessly to LabVIEW's Vision Assistant codes for processing and calibration. The developed vision system allowed detection and calculation of object positions within the manipulator's workspace. Finally, the manipulator's positioning errors were determined and discussed based on a number of repeatability experiments.

The objective of this paper is to propose a new homography-based approach to image-based visual tracking and servoing. The visual tracking algorithm proposed in the paper is based on a new efficient second-order minimization method. Theoretical analysis and comparative experiments with other tracking approaches show that the proposed method has a higher convergence rate than standard first-order minimization techniques. Therefore, it is well adapted to real-time robotic applications. The output of the visual tracking is a homography linking the current and the reference image of a planar target. Using the homography, a task function isomorphic to the camera pose has been designed. A new image-based control law is proposed which does not need any measure of the 3D structure of the observed target (e.g. the normal to the plane). The theoretical proof of the existence of the isomorphism between the task function and the camera pose and the theoretical proof of the stability of the control law are provided. The experimental results, obtained with a 6 d.o.f. robot, show the advantages of the proposed method with respect to the existing approaches. KEY WORDS—visual tracking, visual servoing, efficient second-order minimization, homography-based control law


Question: Point pair feature-based object detection for random bin picking tasks.

This research paper introduces a method for object recognition and shape matching that is both computationally efficient and robust to fundamental transformations such as translation, scaling, and rotation. The core innovation is the use of shape context as a distinctive feature for shape detection. Shape context is a descriptor that characterizes the boundary points of an object in relation to every single boundary point, effectively capturing the overall form of the object. By comparing this feature to pre-existing knowledge of the shape context associated with the object's boundary points, object recognition can be accomplished. The approach has been validated through experiments with handwritten digits and trademark images, yielding encouraging outcomes.

**Question**: A Direct Search Method to solve Economic Dispatch Problem with Valve-Point Effect

**Documents**: A novel direct search algorithm for economic dispatch
A new direct search method for solving the economic dispatch problem with valve-point effect is presented. The proposed method is based on the Nelder-Mead simplex algorithm, which is a well-known direct search technique. The valve-point effect is represented by a non-smooth cost function that is approximated by a piecewise linear function. The proposed method is tested on a number of test systems, and the results are compared with those obtained by other methods. The proposed method is shown to be effective and robust for solving the economic dispatch problem with valve-point effect.

**Similar Documents**: This paper introduces a unique direct search algorithm for addressing the economic dispatch problem, which includes the valve-point effect. The method is built upon the Nelder-Mead simplex algorithm, a renowned technique for direct search. To account for the valve-point effect, a non-smooth cost function is utilized and approximated using a piecewise linear function. The proposed method's efficacy and reliability are demonstrated through its application to various test systems, with the outcomes compared against results from alternative methods. The findings reveal that the proposed method is highly effective in solving the economic dispatch problem in the presence of the valve-point effect.

The objective of this paper is to propose a new homography-based approach to image-based visual tracking and servoing. The visual tracking algorithm proposed in the paper is based on a new efficient second-order minimization method. Theoretical analysis and comparative experiments with other tracking approaches show that the proposed method has a higher convergence rate than standard first-order minimization techniques. Therefore, it is well adapted to real-time robotic applications. The output of the visual tracking is a homography linking the current and the reference image of a planar target. Using the homography, a task function isomorphic to the camera pose has been designed. A new image-based control law is proposed which does not need any measure of the 3D structure of the observed target (e.g. the normal to the plane). The theoretical proof of the existence of the isomorphism between the task function and the camera pose and the theoretical proof of the stability of the control law are provided. The experimental results, obtained with a 6 d.o.f. robot, show the advantages of the proposed method with respect to the existing approaches. KEY WORDS—visual tracking, visual servoing, efficient second-order minimization, homography-based control law

Several randomized path planners have been proposed during the last few years. Their attractiveness stems from their applicability to virtually any type of robots, and their empirically observed success. In this paper we attempt to present a unifying view of these planners and to theoretically explain their success. First, we introduce a general planning scheme that consists of randomly sampling the robot’s configuration space. We then describe two previously developed planners as instances of planners based on this scheme, but applying very different sampling strategies. These planners are probabilistically complete: if a path exists, they will find one with high probability, if we let them run long enough. Next, for one of the planners, we analyze the relation between the probability of failure and the running time. Under assumptions characterizing the “goodness” of the robot’s free space, we show that the running time only grows as the absolute value of the logarithm of the probability of failure that we are willing to tolerate. We also show that it increases at a reasonable rate as the space goodness degrades. In the last section we suggest directions for future research.

The detection of small road hazards, such as lost cargo, is a vital capability for self-driving cars. We tackle this challenging and rarely addressed problem with a vision system that leverages appearance, contextual as well as geometric cues. To utilize the appearance and contextual cues, we propose a new deep learning-based obstacle detection framework. Here a variant of a fully convolutional network is proposed to predict a pixel-wise semantic labeling of (i) free-space, (ii) on-road unexpected obstacles, and (iii) background. The geometric cues are exploited using a state-of-the-art detection approach that predicts obstacles from stereo input images via model-based statistical hypothesis tests. We present a principled Bayesian framework to fuse the semantic and stereo-based detection results. The mid-level Stixel representation is used to describe obstacles in a flexible, compact and robust manner. We evaluate our new obstacle detection system on the Lost and Found dataset, which includes very challenging scenes with obstacles of only 5 cm height. Overall, we report a major improvement over the state-of-the-art, with a performance gain of 27.4%. In particular, we achieve a detection rate of over 90% for distances of up to 50 m. Our system operates at 22 Hz on our self-driving platform.

Recognizing object classes and their 3D viewpoints is an important problem in computer vision. Based on a part-based probabilistic representation [31], we propose a new 3D object class model that is capable of recognizing unseen views by pose estimation and synthesis. We achieve this by using a dense, multiview representation of the viewing sphere parameterized by a triangular mesh of viewpoints. Each triangle of viewpoints can be morphed to synthesize new viewpoints. By incorporating 3D geometrical constraints, our model establishes explicit correspondences among object parts across viewpoints. We propose an incremental learning algorithm to train the generative model. A cellphone video clip of an object is first used to initialize model learning. Then the model is updated by a set of unsorted training images without viewpoint labels. We demonstrate the robustness of our model on object detection, viewpoint classification and synthesis tasks. Our model performs superiorly to and on par with state-of-the-art algorithms on the Savarese et al. 2007 and PASCAL datasets in object detection. It outperforms all previous work in viewpoint classification and offers promising results in viewpoint synthesis.

In this paper we consider the problem of recovering the free space of an indoor scene from its single image. We show that exploiting the box like geometric structure of furniture and constraints provided by the scene, allows us to recover the extent of major furniture objects in 3D. Our “boxy” detector localizes box shaped objects oriented parallel to the scene across different scales and object types, and thus blocks out the occupied space in the scene. To localize the objects more accurately in 3D we introduce a set of specially designed features that capture the floor contact points of the objects. Image based metrics are not very indicative of performance in 3D. We make the first attempt to evaluate single view based occupancy estimates for 3D errors and propose several task driven performance measures towards it. On our dataset of 592 indoor images marked with full 3D geometry of the scene, we show that: (a) our detector works well using image based metrics; (b) our refinement method produces significant improvements in localization in 3D; and (c) if one evaluates using 3D metrics, our method offers major improvements over other single view based scene geometry estimation methods.

Detecting objects and estimating their pose remains as one of the major challenges of the computer vision research community. There exists a compromise between localizing the objects and estimating their viewpoints. The detector ideally needs to be viewinvariant, while the pose estimation process should be able to generalize towards the category-level. This work is an exploration of using deep learning models for solving both problems simultaneously. For doing so, we propose three novel deep learning architectures, which are able to perform a joint detection and pose estimation, where we gradually decouple the two tasks. We also investigate whether the pose estimation problem should be solved as a classification or regression problem, being this still an open question in the computer vision community. We detail a comparative analysis of all our solutions and the methods that currently define the state of the art for this problem. We use PASCAL3D+ and ObjectNet3D datasets to present the thorough experimental evaluation and main results. With the proposed models we achieve the state-of-the-art performance in both datasets.

This paper describes the simultaneous localization and mapping (SLAM) problem and the essential methods for solving the SLAM problem and summarizes key implementations and demonstrations of the method. While there are still many practical issues to overcome, especially in more complex outdoor environments, the general SLAM method is now a well understood and established part of robotics. Another part of the tutorial summarized more recent works in addressing some of the remaining issues in SLAM, including computation, feature representation, and data association

Deep reinforcement learning is becoming increasingly popular for robot control algorithms, with the aim for a robot to self-learn useful feature representations from unstructured sensory input leading to the optimal actuation policy. In addition to sensors mounted on the robot, sensors might also be deployed in the environment, although these might need to be accessed via an unreliable wireless connection. In this paper, we demonstrate deep neural network architectures that are able to fuse information generated by multiple sensors and are robust to sensor failures at runtime. We evaluate our method on a search and pick task for a robot both in simulation and the real world.

In this article a new method for the calibration of a vision system which consists of two (or more) cameras is presented. The proposed method, which uses simple properties of vanishing points, is divided into two steps. In the first step, the intrinsic parameters of each camera, that is, the focal length and the location of the intersection between the optical axis and the image plane, are recovered from a single image of a cube. In the second step, the extrinsic parameters of a pair of cameras, that is, the rotation matrix and the translation vector which describe the rigid motion between the coordinate systems fixed in the two cameras are estimated from an image stereo pair of a suitable planar pattern. Firstly, by matching the corresponding vanishing points in the two images the rotation matrix can be computed, then the translation vector is estimated by means of a simple triangulation. The robustness of the method against noise is discussed, and the conditions for optimal estimation of the rotation matrix are derived. Extensive experimentation shows that the precision that can be achieved with the proposed method is sufficient to efficiently perform machine vision tasks that require camera calibration, like depth from stereo and motion from image sequence.

The impressive results of applying deep neural networks in tasks such as object recognition, language translation, and solving digital games are largely attributed to the availability of massive amounts of high quality labelled data. However, despite numerous promising steps in incorporating these techniques in robotic tasks, the cost of gathering data with a real robot has halted the proliferation of deep learning in robotics. In this work, a plugin for the Gazebo simulator is presented, which allows rapid generation of synthetic data. By introducing variations in simulator-specific or irrelevant aspects of a task, one can train a model which exhibits some degree of robustness against those aspects, and ultimately narrow the reality gap between simulated and real-world data. To show a use-case of the developed software, we build a new dataset for detection and localisation of three object classes: box, cylinder and sphere. Our results in the object detection and localisation task demonstrate that with small datasets generated only in simulation, one can achieve comparable performance to that achieved when training on real-world images.


Question: Variance minimization techniques to accelerate non-convex optimization processes.

This work introduces an enhanced analysis of Stochastic Dual Coordinate Ascent (SDCA) methods, which have gained traction for their application in large-scale supervised machine learning optimization problems, particularly in scenarios involving SVM. Despite the widespread implementation of SDCA in various software packages, its convergence properties have remained understudied until now. The paper offers a comprehensive theoretical examination that positions SDCA as a highly competitive alternative to Stochastic Gradient Descent (SGD), with guarantees that are at par or even superior in certain aspects. This analysis reinforces the practical viability of SDCA for real-world applications, highlighting its potential as a leading method in the field of regularized loss minimization.

This paper examines the generalization properties of online convex programming algorithms when the loss function is Lipschitz and strongly convex. Our main result is a sharp bound, that holds with high probability, on the excess risk of the output of an online algorithm in terms of the average regret. This allows one to use recent algorithms with logarithmic cumulative regret guarantees to achieve fast convergence rates for the excess risk with high probability. As a corollary, we characterize the convergence rate of P EGASOS(with high probability), a recently proposed method for solving the SVM optimization problem.

One of the mysteries in the success of neural networks is randomly initialized first order methods like gradient descent can achieve zero training loss even though the objective function is non-convex and non-smooth. This paper demystifies this surprising phenomenon for two-layer fully connected ReLU activated neural networks. For anm hidden node shallow neural network with ReLU activation and n training data, we show as long asm is large enough and no two inputs are parallel, randomly initialized gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function. Our analysis relies on the following observation: over-parameterization and random initialization jointly restrict every weight vector to be close to its initialization for all iterations, which allows us to exploit a strong convexity-like property to show that gradient descent converges at a global linear rate to the global optimum. We believe these insights are also useful in analyzing deep models and other first order methods.

Alternating minimization represents a widely applicable and empirically successful approach for finding low-rank matrices that best fit the given data. For example, for the problem of low-rank matrix completion, this method is believed to be one of the most accurate and efficient, and formed a major component of the winning entry in the Netflix Challenge [17].
 In the alternating minimization approach, the low-rank target matrix is written in a bi-linear form, i.e. X = UV†; the algorithm then alternates between finding the best U and the best V. Typically, each alternating step in isolation is convex and tractable. However the overall problem becomes non-convex and is prone to local minima. In fact, there has been almost no theoretical understanding of when this approach yields a good result.
 In this paper we present one of the first theoretical analyses of the performance of alternating minimization for matrix completion, and the related problem of matrix sensing. For both these problems, celebrated recent results have shown that they become well-posed and tractable once certain (now standard) conditions are imposed on the problem. We show that alternating minimization also succeeds under similar conditions. Moreover, compared to existing results, our paper shows that alternating minimization guarantees faster (in particular, geometric) convergence to the true matrix, while allowing a significantly simpler analysis.

Total variation (TV) is a one of the most popular regularizers for stabilizing the solution of ill-posed inverse problems. This paper proposes a novel proximal-gradient algorithm for minimizing TV regularized least-squares cost functionals. Unlike traditional methods that require nested iterations for computing the proximal step of TV, our algorithm approximates the latter with several simple proximals that have closed form solutions. We theoretically prove that the proposed parallel proximal method achieves the TV solution with arbitrarily high precision at a global rate of converge that is equivalent to the fast proximal-gradient methods. The results in this paper have the potential to enhance the applicability of TV for solving very large-scale imaging inverse problems.

In many applications of black-box optimization, one can evaluate multiple points simultaneously, e.g. when evaluating the performances of several different neural networks in a parallel computing environment. In this paper, we develop a novel batch Bayesian optimization algorithm — the parallel knowledge gradient method. By construction, this method provides the one-step Bayes optimal batch of points to sample. We provide an efficient strategy for computing this Bayes-optimal batch of points, and we demonstrate that the parallel knowledge gradient method finds global optima significantly faster than previous batch Bayesian optimization algorithms on both synthetic test functions and when tuning hyperparameters of practical machine learning algorithms, especially when function evaluations are noisy.

This paper presents an algorithm to estimate simultaneously both mean and variance of a non parametric regression problem. The key point is that we are able to estimate variance locally unlike standard Gaussian Process regression or SVMs. This means that our estimator adapts to the local noise. The problem is cast in the setting of maximum a posteriori estimation in exponential families. Unlike previous work, we obtain a convex optimization problem which can be solved via Newton's method.

We propose a new fast algorithm for solving one of the standard formulations of image restoration and reconstruction which consists of an unconstrained optimization problem where the objective includes an l2 data-fidelity term and a nonsmooth regularizer. This formulation allows both wavelet-based (with orthogonal or frame-based representations) regularization or total-variation regularization. Our approach is based on a variable splitting to obtain an equivalent constrained optimization formulation, which is then addressed with an augmented Lagrangian method. The proposed algorithm is an instance of the so-called alternating direction method of multipliers, for which convergence has been proved. Experiments on a set of image restoration and reconstruction benchmark problems show that the proposed algorithm is faster than the current state of the art methods.

Many modern unsupervised or semi-supervised machine learning algorithms rely on Bayesian probabilistic models. These models are usually intractable and thus require approximate inference. Variational inference (VI) lets us approximate a high-dimensional Bayesian posterior with a simpler variational distribution by solving an optimization problem. This approach has been successfully used in various models and large-scale applications. In this review, we give an overview of recent trends in variational inference. We first introduce standard mean field variational inference, then review recent advances focusing on the following aspects: (a) scalable VI, which includes stochastic approximations, (b) generic VI, which extends the applicability of VI to a large class of otherwise intractable models, such as non-conjugate models, (c) accurate VI, which includes variational models beyond the mean field approximation or with atypical divergences, and (d) amortized VI, which implements the inference over local latent variables with inference networks. Finally, we provide a summary of promising future research directions.

Convex empirical risk minimization is a basic tool in machine learning and statistics. We provide new algorithms and matching lower bounds for differentially private convex empirical risk minimization assuming only that each data point's contribution to the loss function is Lipschitz and that the domain of optimization is bounded. We provide a separate set of algorithms and matching lower bounds for the setting in which the loss functions are known to also be strongly convex. Our algorithms run in polynomial time, and in some cases even match the optimal nonprivate running time (as measured by oracle complexity). We give separate algorithms (and lower bounds) for (ε, 0)and (ε, δ)-differential privacy; perhaps surprisingly, the techniques used for designing optimal algorithms in the two cases are completely different. Our lower bounds apply even to very simple, smooth function families, such as linear and quadratic functions. This implies that algorithms from previous work can be used to obtain optimal error rates, under the additional assumption that the contributions of each data point to the loss function is smooth. We show that simple approaches to smoothing arbitrary loss functions (in order to apply previous techniques) do not yield optimal error rates. In particular, optimal algorithms were not previously known for problems such as training support vector machines and the high-dimensional median.

This paper introduces a novel algorithm to approximate the matrix with minimum nuclear norm among all matrices obeying a set of convex constraints. This problem may be understood as the convex relaxation of a rank minimization problem and arises in many important applications as in the task of recovering a large matrix from a small subset of its entries (the famous Netflix problem). Off-the-shelf algorithms such as interior point methods are not directly amenable to large problems of this kind with over a million unknown entries. This paper develops a simple first-order and easy-to-implement algorithm that is extremely efficient at addressing problems in which the optimal solution has low rank. The algorithm is iterative, produces a sequence of matrices {Xk ,Y k}, and at each step mainly performs a soft-thresholding operation on the singular values of the matrix Y k. There are two remarkable features making this attractive for low-rank matrix completion problems. The first is that the soft-thresholding operation is applied to a sparse matrix; the second is that the rank of the iterates {Xk} is empirically nondecreasing. Both these facts allow the algorithm to make use of very minimal storage space and keep the computational cost of each iteration low. On the theoretical side, we provide a convergence analysis showing that the sequence of iterates converges. On the practical side, we provide numerical examples in which 1, 000 × 1, 000 matrices are recovered in less than a minute on a modest desktop computer. We also demonstrate that our approach is amenable to very large scale problems by recovering matrices of rank about 10 with nearly a billion unknowns from just about 0.4% of their sampled entries. Our methods are connected with the recent literature on linearized Bregman iterations for 1 minimization, and we develop a framework in which one can understand these algorithms in terms of well-known Lagrange multiplier algorithms.