Question: Neural variational inference approach for embedding knowledge graphs.

This paper introduces a novel class of subgradient methods designed to dynamically integrate insights from the geometric structure of previously observed data, thereby enhancing gradient-based learning by identifying highly predictive yet infrequent features. Our approach is rooted in recent advancements in stochastic optimization and online learning, which utilize proximal functions to regulate gradient steps. We propose a mechanism for adaptively adjusting the proximal function, which considerably streamlines the learning rate setting process and ensures regret bounds that match those of the best hindsight-selected proximal function. We provide efficient algorithms tailored for empirical risk minimization tasks involving prevalent regularization functions and domain constraints. Through experimental validation of our theoretical findings, we demonstrate that adaptive subgradient methods outperform state-of-the-art non-adaptive subgradient algorithms in practical applications.

The mathematical representation of semantics is a key issue for Natural Language Processing (NLP). A lot of research has been devoted to finding ways of representing the semantics of individual words in vector spaces. Distributional approaches—meaning distributed representations that exploit co-occurrence statistics of large corpora—have proved popular and successful across a number of tasks. However, natural language usually comes in structures beyond the word level, with meaning arising not only from the individual words but also the structure they are contained in at the phrasal or sentential level. Modelling the compositional process by which the meaning of an utterance arises from the meaning of its parts is an equally fundamental task of NLP. This dissertation explores methods for learning distributed semantic representations and models for composing these into representations for larger linguistic units. Our underlying hypothesis is that neural models are a suitable vehicle for learning semantically rich representations and that such representations in turn are suitable vehicles for solving important tasks in natural language processing. The contribution of this thesis is a thorough evaluation of our hypothesis, as part of which we introduce several new approaches to representation learning and compositional semantics, as well as multiple state-of-the-art models which apply distributed semantic representations to various tasks in NLP. Part I focuses on distributed representations and their application. In particular, in Chapter 3 we explore the semantic usefulness of distributed representations by evaluating their use in the task of semantic frame identification. Part II describes the transition from semantic representations for words to compositional semantics. Chapter 4 covers the relevant literature in this field. Following this, Chapter 5 investigates the role of syntax in semantic composition. For this, we discuss a series of neural network-based models and learning mechanisms, and demonstrate how syntactic information can be incorporated into semantic composition. This study allows us to establish the effectiveness of syntactic information as a guiding parameter for semantic composition, and answer questions about the link between syntax and semantics. Following these discoveries regarding the role of syntax, Chapter 6 investigates whether it is possible to further reduce the impact of monolingual surface forms and syntax when attempting to capture semantics. Asking how machines can best approximate human signals of semantics, we propose multilingual information as one method for grounding semantics, and develop an extension to the distributional hypothesis for multilingual representations. Finally, Part III summarizes our findings and discusses future work.

This tutorial gives an overview on state-of-the-art methods for the automatic construction of large knowledge bases and harnessing them for data and text analytics. It covers both big-data methods for building knowledge bases and knowledge bases being assets for big-data applications. The tutorial also points out challenges and research opportunities. 1. MOTIVATION AND SCOPE Comphrehensive machine-readable knowledge bases (KB’s) have been pursued since the seminal projects Cyc [19, 20] and WordNet [12]. In contrast to these manually created KB’s, great advances have recently been made on automating the building and curation of large KB’s [1, 16], using information extraction (IE) techniques and harnessing highquality Web sources like Wikipedia. Prominent endeavors of this kind include academic research projects such as DBpedia [3], KnowItAll [10], NELL [5] and YAGO [29], as well as industrial ones such as Freebase. These projects provide automatically constructed KB’s of facts about named entities, their semantic classes, and their mutual relationships. They contain millions of entities and billions of facts about them. Moreover, several KB’s are interlinked at the entity level, forming the backbone of the Web of Linked Data [14]. Such world knowledge in turn enables cognitive applications and knowledge-centric services like disambiguating natural-language text, entity linking, text summarization, deep question answering, and semantic search and analytics over entities and relations in Web and enterprise data (e.g., [2, 6, 8, 13]). Prominent examples of how KB’s can be harnessed include the Google Knowledge Graph [27] and the IBM Watson question answering system [17]. This tutorial presents state-of-the-art methods, recent advances, research opportunities, and open challenges along this avenue of knowledge harvesting and its applications. Particular emphasis is on the twofold role of KB’s for bigdata analytics: using scalable distributed algorithms for harvesting knowledge from Web and text sources, and leveraging entity-centric knowledge for deeper interpretation of and This work is licensed under the Creative Commons AttributionNonCommercial-NoDerivs 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/. Obtain permission prior to any use beyond those covered by the license. Contact copyright holder by emailing info@vldb.org. Articles from this volume were invited to present their results at the 40th International Conference on Very Large Data Bases, September 1st 5th 2014, Hangzhou, China. Proceedings of the VLDB Endowment, Vol. 7, No. 13 Copyright 2014 VLDB Endowment 2150-8097/14/08. better intelligence with big data. The following sections outline the structure of the tutorial. An extensive bibliography on this theme is given in [30]. 2. BUILDING KNOWLEDGE BASES Digital Knowledge: Today’s KB’s represent their data mostly in RDF-style SPO (subject-predicate-object) triples. We introduce this data model and the most salient KB projects, which include KnowItAll [10, 11], BabelNet [22], ConceptNet [28], DBpedia [3, 18], DeepDive [24], Freebase [4], ImageNet [7], NELL [5], Wikidata [31], WikiNet [21], WikiTaxonomy [26], and YAGO [29, 15]. We briefly discuss industrial projects like the Google Knowledge Graph and related work at Google [9, 13, 25], the EntityCube and Probase projects at Microsoft Research [23, 32], and IBM’s Watson project [17]. Harvesting Knowledge on Entities and Classes: Every entity in a KB (e.g., Steve Jobs) belongs to one or multiple classes (e.g., computer pioneer, entrepreneur). These classes are organized into a taxonomy, where more special classes are subsumed by more general classes (e.g., person). We discuss two families of methods to harvest such information: Wikipedia-based approaches that analyze the category system, and Web-based approaches that use techniques like set expansion. 3. HARVESTING FACTS AT WEB SCALE Harvesting Relational Facts: Relational facts express properties of and relationships between entities. There is a large spectrum of methods to extract such facts from Web documents. We give an overview on methods from pattern matching (e.g., regular expressions), computational linguistics (e.g., dependency parsing), statistical learning (e.g., factor graphs and MLN’s), and logical consistency reasoning (e.g., weighted MaxSat or ILP solvers). We also discuss to what extent these approaches scale to handle big data. Open Information Extraction: Alternatively to methods that operate on a pre-specified set of relations and entities, open information extraction harvests arbitrary SPO triples from natural language documents. It aggressively taps into noun phrases as entity candidates and verbal phrases as prototypic patterns for relations. We discuss recent methods that follow this direction. Some methods along these lines make clever use of big-data techniques like frequent sequence mining and map-reduce computation. Temporal and Multilingual Knowledge: Properly interpreting entities and facts in a KB often requires additional meta-information like entity names in different languages and the temporal scope of facts. We discuss tech-

Words are polysemous. However, most approaches to representation learning for lexical semantics assign a single vector to every surface word type. Meanwhile, lexical ontologies such as WordNet provide a source of complementary knowledge to distributional information, including a word sense inventory. In this paper we propose two novel and general approaches for generating sense-specific word embeddings that are grounded in an ontology. The first applies graph smoothing as a postprocessing step to tease the vectors of different senses apart, and is applicable to any vector space model. The second adapts predictive maximum likelihood models that learn word embeddings with latent variables representing senses grounded in an specified ontology. Empirical results on lexical semantic tasks show that our approaches effectively captures information from both the ontology and distributional statistics. Moreover, in most cases our sense-specific models outperform other models we compare against.

Learning graph representations is a fundamental task aimed at capturing various properties of graphs in vector space. The most recent methods learn such representations for static networks. However, real world networks evolve over time and have varying dynamics. Capturing such evolution is key to predicting the properties of unseen networks. To understand how the network dynamics affect the prediction performance, we propose an embedding approach which learns the structure of evolution in dynamic graphs and can predict unseen links with higher precision. Our model, dyngraph2vec, learns the temporal transitions in the network using a deep architecture composed of dense and recurrent layers. We motivate the need of capturing dynamics for prediction on a toy data set created using stochastic block models. We then demonstrate the efficacy of dyngraph2vec over existing state-of-the-art methods on two real world data sets. We observe that learning dynamics can improve the quality of embedding and yield better performance in link prediction.

Word embeddings provide point representations of words containing useful semantic information. We introduce multimodal word distributions formed from Gaussian mixtures, for multiple word meanings, entailment, and rich uncertainty information. To learn these distributions, we propose an energy-based max-margin objective. We show that the resulting approach captures uniquely expressive semantic information, and outperforms alternatives, such as word2vec skip-grams, and Gaussian embeddings, on benchmark datasets such as word similarity and entailment.

We present the Neural Physics Engine (NPE), a framework for learning simulators of intuitive physics that naturally generalize across variable object count and different scene configurations. We propose a factorization of a physical scene into composable object-based representations and a neural network architecture whose compositional structure factorizes object dynamics into pairwise interactions. Like a symbolic physics engine, the NPE is endowed with generic notions of objects and their interactions; realized as a neural network, it can be trained via stochastic gradient descent to adapt to specific object properties and dynamics of different worlds. We evaluate the efficacy of our approach on simple rigid body dynamics in two-dimensional worlds. By comparing to less structured architectures, we show that the NPE’s compositional representation of the structure in physical interactions improves its ability to predict movement, generalize across variable object count and different scene configurations, and infer latent properties of objects such as mass.

• Neural network language model and distributed representation for words (Vector representation) • Capture syntactic and remantic regularities in language • Outperform state-of-the-art

Network embeddings have become very popular in learning effective feature representations of networks. Motivated by the recent successes of embeddings in natural language processing, researchers have tried to nd network embeddings in order to exploit machine learning algorithms for mining tasks like node classication and edge prediction. However, most of the work focuses on nding distributed representations of nodes, which are inherently ill-suited to tasks such as community detection which are intuitively dependent on subgraphs. Here, we propose Sub2Vec, an unsupervised scalable algorithm to learn feature representations of arbitrary subgraphs. We provide means to characterize similarties between subgraphs and provide theoretical analysis of Sub2Vec and demonstrate that it preserves the so-called local proximity. We also highlight the usability of Sub2Vec by leveraging it for network mining tasks, like community detection. We show that Sub2Vec gets signicant gains over stateof-the-art methods and node-embedding methods. In particular, Sub2Vec oers an approach to generate a richer vocabulary of features of subgraphs to support representation and reasoning. ACM Reference format: Bijaya Adhikari, Yao Zhang, Naren Ramakrishnan and B. Aditya Prakash. 2016. Distributed Representation of Subgraphs. In Proceedings of ACM Conference, Washington, DC, USA, July 2017 (Conference’17), 9 pages. DOI: 10.1145/nnnnnnn.nnnnnnn

Recent advances of kernel methods have yielded a framework for representing probabilities using a reproducing kernel Hilbert space, called kernel embedding of distributions. In this paper, we propose a Monte Carlo filtering algorithm based on kernel embeddings. The proposed method is applied to state-space models where sampling from the transition model is possible, while the observation model is to be learned from training samples without assuming a parametric model. As a theoretical basis of the proposed method, we prove consistency of the Monte Carlo method combined with kernel embeddings. Experimental results on synthetic models and real vision-based robot localization confirm the effectiveness of the proposed approach.

Most of the existing graph embedding methods focus on nodes, which aim to output a vector representation for each node in the graph such that two nodes being “close” on the graph are close too in the low-dimensional space. Despite the success of embedding individual nodes for graph analytics, we notice that an important concept of embedding communities (i.e., groups of nodes) is missing. Embedding communities is useful, not only for supporting various community-level applications, but also to help preserve community structure in graph embedding. In fact, we see community embedding as providing a higher-order proximity to define the node closeness, whereas most of the popular graph embedding methods focus on first-order and/or second-order proximities. To learn the community embedding, we hinge upon the insight that community embedding and node embedding reinforce with each other. As a result, we propose ComEmbed, the first community embedding method, which jointly optimizes the community embedding and node embedding together. We evaluate ComEmbed on real-world data sets. We show it outperforms the state-of-the-art baselines in both tasks of node classification and community prediction.


Question: Multi-view laser reconstruction for autonomous underwater object grasping.

This document outlines an innovative technique for semi-autonomous underwater grasping and recovery of unfamiliar objects by a robot operating in a floating position. The complexity of autonomous grasping is significantly heightened in underwater settings due to the chaotic environments, restricted sensor capabilities, and challenging conditions that can adversely impact the robot's sensory and control systems. The proposed method involves the integration of template tracking and laser peak detection to achieve 3D reconstruction and facilitate grasping in these demanding underwater scenarios. A laser stripe projector is affixed to the robot's forearm, which performs a scan of the object of interest. Simultaneously, a camera captures the scan and gauges the motion of the floating vehicle during the scanning process. The points acquired from the scan are triangulated and adjusted according to the motion estimation, allowing for the partial reconstruction of the 3D scene in relation to a static frame. Following this, a user selects the specific area on the object to be grasped, and the system automatically devises a grasp plan for that designated region. The effectiveness of this approach has been evaluated and substantiated through experiments conducted in controlled water tank settings.

We present a Bayesian object observation model for complete probabilistic semantic SLAM. Recent studies on object detection and feature extraction have become important for scene understanding and 3D mapping. However, 3D shape of the object is too complex to formulate the probabilistic observation model; therefore, performing the Bayesian inference of the object-oriented features as well as their pose is less considered. Besides, when the robot equipped with an RGB mono camera only observes the projected single view of an object, a significant amount of the 3D shape information is abandoned. Due to these limitations, semantic SLAM and viewpoint-independent loop closure using volumetric 3D object shape is challenging. In order to enable the complete formulation of probabilistic semantic SLAM, we approximate the observation model of a 3D object with a tractable distribution. We also estimate the variational likelihood from the 2D image of the object to exploit its observed single view. In order to evaluate the proposed method, we perform pose and feature estimation, and demonstrate that the automatic loop closure works seamlessly without additional loop detector in various environments.

This paper presents planning and control synthesis for multiple aerial manipulators to transport a common object. Each aerial manipulator that consists of a hexacopter and a two-degree-of-freedom robotic arm is controlled by an augmented adaptive sliding mode controller based on a closed-chain robot dynamics. We propose a motion planning algorithm by exploiting rapidly exploring random tree star (RRT*) and dynamic movement primitives (DMPs). The desired path for each aerial manipulator is obtained by using RRT* with Bezier curve, which is designed to handle environmental obstacles, such as buildings or equipments. During aerial transportation, to avoid unknown obstacle, DMPs modify the trajectory based on the virtual leader–follower structure. By the combination of RRT* and DMPs, the cooperative aerial manipulators can carry a common object to keep reducing the interaction force between multiple robots while avoiding an obstacle in the unstructured environment. To validate the proposed planning and control synthesis, two experiments with multiple custom-made aerial manipulators are presented, which involve user-guided trajectory and RRT*-planned trajectory tracking in unstructured environments.Note to Practitioners—This paper presents a viable approach to autonomous aerial transportation using multiple aerial manipulators equipped with a multidegree-of-freedom robotic arm. Existing approaches for cooperative manipulation based on force decomposition or impedance-based control often require a heavy or expensive force/torque sensor. However, this paper suggests a method without using a heavy or expensive force/torque sensor based on closed-chain dynamics in joint space and rapidly exploring random tree star (RRT*) that generates the desired trajectory of aerial manipulators. Unlike conventional RRT*, in this paper, our method can also avoid an unknown moving obstacle during aerial transportation by exploiting RRT* and dynamic movement primitives. The proposed planning and control synthesis is tested to demonstrate performance in a lab environment with two custom-made aerial manipulators and a common object.

We describe a novel system for rapid prototyping of laserbased optical and holographic layouts. Users of this optical prototyping tool – called the Illuminating Light system – move physical representations of various optical elements about a workspace, while the system tracks these components and projects back onto the workspace surface the simulated propagation of laser light through the evolving layout. This application is built atop the Luminous Room infrastructure, an aggregate of interlinked, computer-controlled projector-camera units called I/O Bulbs. Philosophically, the work embodies the emerging ideas of the Luminous Room and builds on the notions of ‘graspable media’. We briefly introduce the I/O Bulb and Luminous Room concepts and discuss their current implementations. After an overview of the optical domain that the Illuminating Light system is designed to address, we present the overall system design and implementation, including that of an intermediary toolkit called voodoo which provides a general facility for object identification and tracking.

In this paper we consider the problem of computing the 3D shape of an unknown, arbitrarily-shaped scene from multiple photographs taken at known but arbitrarily-distributed viewpoints. By studying the equivalence class of all 3D shapes that reproduce the input photographs, we prove the existence of a special member of this class, the photo hull, that (1) can be computed directly from photographs of the scene, and (2) subsumes all other members of this class. We then give a provably-correct algorithm, called Space Carving, for computing this shape and present experimental results on complex real-world scenes. The approach is designed to (1) capture photorealistic shapes that accurately model scene appearance from a wide range of viewpoints, and (2) account for the complex interactions between occlusion, parallax, shading, and their view-dependent effects on scene-appearance.

This paper introduces Push-Net, a deep recurrent neural network model, which enables a robot to push objects of unknown physical properties for re-positioning and re-orientation, using only visual camera images as input. The unknown physical properties is a major challenge for pushing. Push-Net overcomes the challenge by tracking a history of push interactions with an LSTM module and training an auxiliary objective function that estimates an object’s center of mass. We trained Push-Net entirely in simulation and tested it extensively on many different objects in both simulation and on two real robots, a Fetch arm and a Kinova MICO arm. Experiments suggest that Push-Net is robust and efficient. It achieved over 97% success rate in simulation on average and succeeded in all real robot experiments with a small number of pushes.

The ability to segment unknown objects in depth images has potential to enhance robot skills in grasping and object tracking. Recent computer vision research has demonstrated that Mask R-CNN can be trained to segment specific categories of objects in RGB images when massive handlabeled datasets are available. As generating these datasets is time-consuming, we instead train with synthetic depth images. Many robots now use depth sensors, and recent results suggest training on synthetic depth data can generalize well to the real world. We present a method for automated dataset generation and rapidly generate a training dataset of 50k depth images and 320k object masks synthetically using simulated scenes of 3D CAD models. We train a variant of Mask R-CNN on the generated dataset to perform category-agnostic instance segmentation without hand-labeled data. We evaluate the trained network, which we refer to as Synthetic Depth (SD) Mask R-CNN, on a set of real, high-resolution images of challenging, denselycluttered bins containing objects with highly-varied geometry. SD Mask R-CNN outperforms point cloud clustering baselines by an absolute 15% in Average Precision and 20% in Average Recall, and achieves performance levels similar to a Mask RCNN trained on a massive, hand-labeled RGB dataset and fine-tuned on real images from the experimental setup. The network also generalizes well to a lower-resolution depth sensor. We deploy the model in an instance-specific grasping pipeline to demonstrate its usefulness in a robotics application. Code, the synthetic training dataset, and supplementary material are available at https://bit.ly/2letCuE.

2D and 3D computer vision systems are frequently being used in automated production to detect and determine the position of objects. Accuracy is important in the production industry, and computer vision systems require structured environments to function optimally. For 2D vision systems, a change in surfaces, lighting and viewpoint angles can reduce the accuracy of a method, maybe even to a degree that it will be erroneous, while for 3D vision systems, the accuracy mainly depends on the 3D laser sensors. Commercially available 3D cameras lack the precision found in high-grade 3D laser scanners, and are therefore not suited for accurate measurements in industrial use. In this paper, we show that it is possible to identify and locate objects using a combination of 2D and 3D cameras. A rough estimate of the object pose is first found using a commercially available 3D camera. Then, a robotic arm with an eye-in-hand 2D camera is used to determine the pose accurately. We show that this increases the accuracy to < 1 mm and < 1◦. This was demonstrated in a real industrial assembly task where high accuracy is required.

This paper relates first experiences using a state-of-the-art, time-of-flight sensor that is able to deliver 3D images. The properties and capabilities of the sensor make it a potential powerful tool for applications within mobile robotics especially for real-time tasks, as the sensor features a frame rate of up to 30 frames per second. Its capabilities in terms of basic obstacle avoidance and local path-planning are evaluated and compared to the performance of a standard laser scanner.

Simulators have played a critical role in robotics research as tools for quick and efficient testing of new concepts, strategies, and algorithms. To date, most simulators have been restricted to 2D worlds, and few have matured to the point where they are both highly capable and easily adaptable. Gazebo is designed to fill this niche by creating a 3D dynamic multi-robot environment capable of recreating the complex worlds that would be encountered by the next generation of mobile robots. Its open source status, fine grained control, and high fidelity place Gazebo in a unique position to become more than just a stepping stone between the drawing board and real hardware: data visualization, simulation of remote environments, and even reverse engineering of blackbox systems are all possible applications. Gazebo is developed in cooperation with the Player and Stage projects (Gerkey, B. P., et al., July 2003), (Gerkey, B. P., et al., May 2001), (Vaughan, R. T., et al., Oct. 2003), and is available from http://playerstage.sourceforge.net/gazebo/ gazebo.html.

There have been increasing demands for developing robotic system combining camera and inertial measurement unit in navigation task, due to their low-cost, lightweight and complementary properties. In this paper, we present a Visual Inertial Odometry (VIO) system which can utilize sparse depth to estimate 6D pose in GPS-denied and unstructured environments. The system is based on Multi-State Constraint Kalman Filter (MSCKF), which benefits from low computation load when compared to optimization-based method, especially on resource-constrained platform. Features are enhanced with depth information forming 3D landmark position measurements in space, which reduces uncertainty of position estimate. And we derivate measurement model to access compatibility with both 2D and 3D measurements. In experiments, we evaluate the performance of the system in different in-flight scenarios, both cluttered room and industry environment. The results suggest that the estimator is consistent, substantially improves the accuracy compared with original monocular-based MSKCF and achieves competitive accuracy with other research.


Question: Multi-view laser reconstruction for autonomous underwater object grasping.

This paper introduces an innovative computer vision technique aimed at significantly enhancing underwater imaging quality, crucial for scientific, technological, and recreational purposes. The method focuses on counteracting the detrimental effects that impair underwater visibility. By examining the physics behind visibility degradation, we identify partial polarization of light as a key factor. To address this, we propose an algorithm that reverses the image formation process, effectively restoring clear visibility of underwater objects. This algorithm leverages a series of images captured through a polarizer at varying angles. An added benefit is the generation of a distance map of the underwater scene. Our approach was thoroughly tested in real-world oceanic environments using a custom-built system, yielding remarkable results. Scene contrast was markedly improved, colors were accurately corrected, and the underwater visibility range was nearly doubled, showcasing the effectiveness of our technique in enhancing underwater vision.

The limited nature of robot sensors make many important robo tics problems partially observable. These problems may require the s yst m to perform complex information-gathering operations. One approach to so lving these problems is to create plans inbelief-space , the space of probability distributions over the underlying state of the system. The belief-space plan encodes a st rategy for performing a task while gaining information as necessary. Most approach es to belief-space planning rely upon representing belief state in a particular way (typically as a Gaussian). Unfortunately, this can lead to large errors between the assumed density representation of belief state and the true belief state. Th is paper proposes a new sample-based approach to belief-space planning that has fix ed computational complexity while allowing arbitrary implementations of Bayes filtering to be used to track belief state. The approach is illustrated in the conte xt of a simple example and compared to a prior approach. Then, we propose an applicatio n of the technique to an instance of the grasp synthesis problem where a robot mu st simultaneously localize and grasp an object given initially uncertain obje ct parameters by planning information-gathering behavior. Experimental results ar e presented that demonstrate the approach to be capable of actively localizing and graspi ng boxes that are presented to the robot in uncertain and hard-to-localize config urations.

This paper presents the design and analysis of a computer vision-aided motion sensing algorithm for wheeled mobile robot's indoor navigation. The algorithm is realized using two vision cameras attached on a wheeled mobile robot. The first camera is positioned at front-looking direction while the second camera is positioned at downward-looking direction. An algorithm is developed to process the images acquired from the cameras to yield the mobile robot's positions and orientations. The proposed algorithm is implemented on a wheeled mobile robot for real-world effectiveness testing. Results are compared and shown the accuracy of the proposed algorithm. At the end of the paper, an artificial landmark approach is introduced to improve the navigation efficiency. Future work involved implementing the proposed artificial landmark for indoor navigation applications with minimized accumulated errors.

The ability to segment unknown objects in depth images has potential to enhance robot skills in grasping and object tracking. Recent computer vision research has demonstrated that Mask R-CNN can be trained to segment specific categories of objects in RGB images when massive handlabeled datasets are available. As generating these datasets is time-consuming, we instead train with synthetic depth images. Many robots now use depth sensors, and recent results suggest training on synthetic depth data can generalize well to the real world. We present a method for automated dataset generation and rapidly generate a training dataset of 50k depth images and 320k object masks synthetically using simulated scenes of 3D CAD models. We train a variant of Mask R-CNN on the generated dataset to perform category-agnostic instance segmentation without hand-labeled data. We evaluate the trained network, which we refer to as Synthetic Depth (SD) Mask R-CNN, on a set of real, high-resolution images of challenging, denselycluttered bins containing objects with highly-varied geometry. SD Mask R-CNN outperforms point cloud clustering baselines by an absolute 15% in Average Precision and 20% in Average Recall, and achieves performance levels similar to a Mask RCNN trained on a massive, hand-labeled RGB dataset and fine-tuned on real images from the experimental setup. The network also generalizes well to a lower-resolution depth sensor. We deploy the model in an instance-specific grasping pipeline to demonstrate its usefulness in a robotics application. Code, the synthetic training dataset, and supplementary material are available at https://bit.ly/2letCuE.

We propose a new method for the localization of a Micro Aerial Vehicle (MAV) with respect to a ground robot. We solve the problem of registering the 3D maps computed by the robots using different sensors: a dense 3D reconstruction from the MAV monocular camera is aligned with the map computed from the depth sensor on the ground robot. Once aligned, the dense reconstruction from the MAV is used to augment the map computed by the ground robot, by extending it with the information conveyed by the aerial views. The overall approach is novel, as it builds on recent developments in live dense reconstruction from moving cameras to address the problem of air-ground localization. The core of our contribution is constituted by a novel algorithm integrating dense reconstructions from monocular views, Monte Carlo localization, and an iterative pose refinement. In spite of the radically different vantage points from which the maps are acquired, the proposed method achieves high accuracy whereas appearance-based, state-of-the-art approaches fail. Experimental validation in indoor and outdoor scenarios reported an accuracy in position estimation of 0.08 meters and real time performance. This demonstrates that our new approach effectively overcomes the limitations imposed by the difference in sensors and vantage points that negatively affect previous techniques relying on matching visual features.

Robot navigation has multi-spectral applications across a plethora of industries. Being one of the fastest evolving field of Artificial intelligence, it has garnered a lot of speculation and the problem of efficient Robot navigation has become a talking point in the “Artificial Intelligence research community”. Robot Navigation can be thought of as a collection of sub-problems that when solved efficiently will (most likely) give the complete and the most efficient solution. While the main Navigation problem can be divided into numerous smaller ones some of the prevalent (sub)-tasks associated with Robot Navigation are Path Planning, Collision Prevention, Search Algorithm and an appropriate pictorial (graphical) representation of the given environment. Path Planning gives us the most optimal path finding approach while keeping the dynamism of the environment and the objects in mind. Collision Prevention ensures the Robot reaches its goal without colliding with any of the objects present in the environment. Collision Prevention techniques' vitality increases considerably when the number of objects in the input space is large (>10, approximately). Search Algorithms are the functional units of this problem as they determine the path the Robot follows while traveling to its destination. Since the environment greatly affects the choice of technique for all the other tasks it is vital to have a clear, concise and a non-ambiguous representation of the same. The main aim of the paper is to summarise the development of various techniques in the multiple domains of the Robot Navigation problem in the last fifteen years (2000-2015) while stating the scope and the restriction of each of them.

Micro aerial vehicles, such as multirotors, are particular well suited for the autonomous monitoring, inspection, and surveillance of buildings, e.g., for maintenance in industrial plants. Key prerequisites for the fully autonomous operation of micro aerial vehicles in restricted environments are 3D mapping, real-time pose tracking, obstacle detection, and planning of collision-free trajectories. In this article, we propose a complete navigation system with a multimodal sensor setup for omnidirectional environment perception. Measurements of a 3D laser scanner are aggregated in egocentric local multiresolution grid maps. Local maps are registered and merged to allocentric maps in which the MAV localizes. For autonomous navigation, we generate trajectories in a multi-layered approach: from mission planning over global and local trajectory planning to reactive obstacle avoidance. We evaluate our approach in a GNSS-denied indoor environment where multiple collision hazards require reliable omnidirectional perception and quick navigation reactions.

There is a large variety of objects and appliances in human environments, such as stoves, coffee dispensers, juice extractors, and so on. It is challenging for a roboticist to program a robot for each of these object types and for each of their instantiations. In this work, we present a novel approach to manipulation planning based on the idea that many household objects share similarly-operated object parts. We formulate the manipulation planning as a structured prediction problem and learn to transfer manipulation strategy across different objects by embedding point-cloud, natural language, and manipulation trajectory data into a shared embedding space using a deep neural network. In order to learn semantically meaningful spaces throughout our network, we introduce a method for pre-training its lower layers for multimodal feature embedding and a method for fine-tuning this embedding space using a loss-based margin. In order to collect a large number of manipulation demonstrations for different objects, we develop a new crowd-sourcing platform called Robobarista. We test our model on our dataset consisting of 116 objects and appliances with 249 parts along with 250 language instructions, for which there are 1225 crowd-sourced manipulation demonstrations. We further show that our robot with our model can even prepare a cup of a latte with appliances it has never seen before.

In order to perform useful tasks, a service robot needs to manipulate objects in its environment. In this paper, we propose a method for experimental evaluation of the suitability of a robotic hand for grasping tasks in service robotics. The method is applied to the Schunk 5-Finger Gripping Hand, which is a mechatronic gripper designed for service robots. During evaluation, it is shown, that it is able to grasp various common household objects and execute the grasps from the well known Cutkosky grasp taxonomy [1]. The result is, that it is a suitable hand for service robot tasks.

This study presents the development of vision-based techniques for autonomous collision avoidance by an unmanned surface ship using an onboard monocular camera. In order to determine the initiation of an evasive maneuver, the range and bearing measurements of each target traffic ship with respect to the observer (e.g., own ship) need to be provided for trajectory estimation. A tracking estimator is used to estimate the target ship trajectory in the framework of bearings-only tracking based on the extended Kalman filter (EKF) algorithm with a Continuous White Noise Acceleration (CWNA) model. To enhance the observability of the tracking filter, the vertical pixel distance from the water horizon to each target ship is used as a range measurement. When the estimated separation distance between the target and own ships is less than a predefined minimum separation, the own ship alters its heading angle to avoid an imminent collision following the standard marine traffic rules. Field experiment results are presented and discussed to demonstrate the feasibility and validity of the proposed approach.

A successful grasp requires careful balancing of the contact forces. Deducing whether a particular grasp will be successful from indirect measurements, such as vision, is therefore quite challenging, and direct sensing of contacts through touch sensing provides an appealing avenue toward more successful and consistent robotic grasping. However, in order to fully evaluate the value of touch sensing for grasp outcome prediction, we must understand how touch sensing can influence outcome prediction accuracy when combined with other modalities. Doing so using conventional model-based techniques is exceptionally difficult. In this work, we investigate the question of whether touch sensing aids in predicting grasp outcomes within a multimodal sensing framework that combines vision and touch. To that end, we collected more than 9,000 grasping trials using a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger, and evaluated visuo-tactile deep neural network models to directly predict grasp outcomes from either modality individually, and from both modalities together. Our experimental results indicate that incorporating tactile readings substantially improve grasping performance.


Question: Multi-view laser reconstruction for autonomous underwater object grasping.

The Evolution of the Girona 500 AUV: A Versatile Platform for Underwater Operations

This document details the design and capabilities of the Girona 500, an advanced autonomous underwater vehicle (AUV) distinguished by its adaptability for various underwater missions. The Girona 500 is uniquely engineered to undergo reconfiguration, enabling it to excel in a spectrum of tasks, from comprehensive seafloor mapping to intricate inspection and intervention operations. Its design philosophy underscores the importance of flexibility, allowing the vehicle to pivot seamlessly between surveying environments and executing targeted missions with precision.

**Question**: Autonomous underwater grasping using multi-view laser reconstruction

**Documents**: Robotic Grasping Underwater: A Review of Techniques and Challenges
This paper provides an overview of the current state of robotic grasping in underwater environments, highlighting the methods, challenges, and advancements in the field. It discusses the use of multi-view laser reconstruction for improving the accuracy and reliability of grasping tasks, emphasizing the importance of precise object detection and manipulation in aquatic settings.

**Similar Documents**:

Advancements in Underwater Robotic Grasping: A Comprehensive Review

This review paper delves into the contemporary landscape of robotic grasping in underwater contexts, exploring the strategies, obstacles, and innovations that define this field. It particularly focuses on the application of multi-view laser reconstruction techniques, which significantly enhance the precision and dependability of underwater grasping operations. The paper underscores the critical role of accurate object localization and manipulation, particularly in the complex and often unpredictable aquatic environment. By synthesizing the latest research and practical insights, this review offers a comprehensive perspective on the evolving capabilities of underwater robotic grasping.

This paper describes a data set collected by an autonomous ground vehicle testbed, based upon a modified Ford F-250 pickup truck. The vehicle is outfitted with a professional (Applanix POS-LV) and consumer (Xsens MTi-G) inertial measurement unit (IMU), a Velodyne 3D-lidar scanner, two push-broom forward looking Riegl lidars, and a Point Grey Ladybug3 omnidirectional camera system. Here we present the time-registered data from these sensors mounted on the vehicle, collected while driving the vehicle around the Ford Research campus and downtown Dearborn, Michigan during NovemberDecember 2009. The vehicle path trajectory in these data sets contain several large and small-scale loop closures, which should be useful for testing various state of the art computer vision and simultaneous localization and mapping (SLAM) algorithms. Fig. 1. The modified Ford F-250 pickup truck.

This paper introduces a stereo vision system to automatically generate 3D models of real objects. 3D model generation is based on the merging of multiview range images obtained from a digital stereo camera. Stereo images obtained from the camera are rectified, and a correlation-based stereo matching technique reconstructs range images from them. A turntable stage is also employed to obtain multiple range images of the objects. To register range images into a common coordinate system automatically, we introduce and calibrate a turntable coordinate system with respect to the camera coordinate system. After the registration of multiview range images, a 3D model is reconstructed using a volumetric integration technique. Error analysis on turntable calibration and 3D model reconstruction shows the accuracy of our 3D modeling system.

In this paper, we concentrate on the design of a new whole-sensitive robot arm enabling torque measurement in each joint by means of developed optical torque sensors. When the contact of arm with an object occurs, local impedance algorithm provides active compliance of corresponding robot arm joint. Thus, the whole structure of the manipulator can safely interact with unstructured environment. The detailed design procedure of the 4-DOF robot arm and optical torque sensors is described in the paper. The gravity compensation algorithm was elaborated and verified by implementation of the local impedance control of a simple two-link arm manipulator

In this brief, a convolutional learning system for classification of segmented objects represented in 3-D as point clouds of laser reflections is proposed. Several novelties are discussed: (1) extension of the existing convolutional neural network (CNN) framework to direct processing of 3-D data in a multiview setting which may be helpful for rotation-invariant consideration, (2) improvement of CNN training effectiveness by employing a stochastic meta-descent (SMD) method, and (3) combination of unsupervised and supervised training for enhanced performance of CNN. CNN performance is illustrated on a two-class data set of objects in a segmented outdoor environment.

During the last few decades, robotic grippers are developed by research community to solve grasping complexities of several objects as their primary objective. Due to the increasing demands of industries, many issues are rising and remain unsolved such as in-hand manipulation, placing object with appropriate posture. Operations like twisting, altering orientation of object, in a hand, requires significant dexterity of the gripper that must be achieved from a compact mechanical design at the first place. In this paper, a newly designed gripper is proposed whose primary goal is to solve in-hand manipulation. The gripper is derived from four identical finger modules; each module has four DOFs, consists of a five bar linkage to grasp and additional two DOFs mechanism similar to ball spline is conceived to manipulate object. Hence the easily constructible gripper is capable to grasp and manipulate plurality of object. As a preliminary inspection, an optimized platform that represents the central concept of the proposed gripper is developed, to the aim of evaluating kinematic feasibilities of manipulation of different objects.

We present an extremely simple yet robust multi-view stereo algorithm and analyze its properties. The algorithm first computes individual depth maps using a window-based voting approach that returns only good matches. The depth maps are then merged into a single mesh using a straightforward volumetric approach. We show results for several datasets, showing accuracy comparable to the best of the current state of the art techniques and rivaling more complex algorithms.

We propose a technique for multi-task learning from demonstration that trains the controller of a low-cost robotic arm to accomplish several complex picking and placing tasks, as well as non-prehensile manipulation. The controller is a recurrent neural network using raw images as input and generating robot arm trajectories, with the parameters shared across the tasks. The controller also combines VAE-GAN-based reconstruction with autoregressive multimodal action prediction. Our results demonstrate that it is possible to learn complex manipulation tasks, such as picking up a towel, wiping an object, and depositing the towel to its previous position, entirely from raw images with direct behavior cloning. We show that weight sharing and reconstruction-based regularization substantially improve generalization and robustness, and training on multiple tasks simultaneously increases the success rate on all tasks.

We present a Bag of Words-based active object categorization technique implemented and tested on a humanoid robot. The robot is trained to categorize objects that are handed to it by a human operator. The robot uses hand and head motions to actively acquire a number of different views. A view planning scheme using entropy minimization reduces the number of views needed to achieve a valid decision. Categorization results are significantly improved by active elimination of background features using robot arm motion. Our experiments cover both, categorization when the object is handed to the robot in a fixed pose at training and testing, and object pose independent categorization. Results on a 4-class object database demonstrate the classification efficiency, a significant gain from multi-view compared to single-view classification, and the advantage of view planning. We conclude that humanoid robotic systems can be successfully applied to actively categorize objects a task with many potential applications ranging from edutainment to active surveillance.

Robot vision refers to the capability of a robot to visually perceive the environment and use this information for execution of different tasks. Visual feedback has been used extensively for robot navigation and obstacle avoidance. In the recent years, there are also examples that include interaction with people and manipulation of objects. In this paper, we review some of the work that goes beyond of using artificial landmarks and fiducial markers for the purpose of implementing vision based control in robots. We discuss different application areas, both from the systems perspective and individual problems such as object tracking and recognition.

Mobile robots with complex morphology are essential for traversing rough terrains in Urban Search & Rescue missions. Since teleoperation of the complex morphology causes high cognitive load of the operator, the morphology is controlled autonomously. The autonomous control measures the robot state and surrounding terrain which is usually only partially observable, and thus the data are often incomplete. We marginalize the control over the missing measurements and evaluate an explicit safety condition. If the safety condition is violated, tactile terrain exploration by the body-mounted robotic arm gathers the missing data.


Question: Multi-view laser reconstruction for autonomous underwater grasping tasks.

This article provides an in-depth look at ROS (Robot Operating System), an open-source framework designed for robotics software development. Unlike traditional operating systems that manage processes and scheduling, ROS functions as a communication layer above the host operating systems of a diverse computing cluster. The paper explores the relationship between ROS and other established robot software frameworks, highlighting its unique contributions to the field. Additionally, it offers a concise summary of the application software that leverages ROS, showcasing the system's versatility and the breadth of its applications in robotics.

**Question**: A Direct Search Method to solve Economic Dispatch Problem with Valve-Point Effect

**Documents**: A hybrid of genetic algorithm and particle swarm optimization for recurrent network design
An evolutionary recurrent network which automates the design of recurrent neural/fuzzy networks using a new evolutionary learning algorithm is proposed in this paper. This new evolutionary learning algorithm is based on a hybrid of genetic algorithm (GA) and particle swarm optimization (PSO), and is thus called HGAPSO. In HGAPSO, individuals in a new generation are created, not only by crossover and mutation operation as in GA, but also by PSO. The concept of elite strategy is adopted in HGAPSO, where the upper-half of the best-performing individuals in a population are regarded as elites. However, instead of being reproduced directly to the next generation, these elites are first enhanced. The group constituted by the elites is regarded as a swarm, and each elite corresponds to a particle within it. In this regard, the elites are enhanced by PSO, an operation which mimics the maturing phenomenon in nature. These enhanced elites constitute half of the population in the new generation, whereas the other half is generated by performing crossover and mutation operation on these enhanced elites. HGAPSO is applied to recurrent neural/fuzzy network design as follows. For recurrent neural network, a fully connected recurrent neural network is designed and applied to a temporal sequence production problem. For recurrent fuzzy network design, a Takagi-Sugeno-Kang-type recurrent fuzzy network is designed and applied to dynamic plant control. The performance of HGAPSO is compared to both GA and PSO in these recurrent networks design problems, demonstrating its superiority.

**Similar Documents**: This article introduces a novel evolutionary learning algorithm, coined HGAPSO, which combines the strengths of genetic algorithms (GA) and particle swarm optimization (PSO) to automate the design of recurrent neural/fuzzy networks. The HGAPSO approach deviates from conventional GA methods by incorporating PSO techniques in the creation of new generations. Specifically,

An explicit three-dimensional representation is constructed from feature-points extracted from a sequence of images taken by a moving camera. The points are tracked through the sequence, and their 3D locations accurately determined by use of Kalman Filters. The ego-motion of the camera is also solved for.

This paper summarizes recent activities carried out for the development of an innovative anthropomorphic robotic hand called the DEXMART Hand. The main goal of this research is to face the problems that affect current robotic hands by introducing suitable design solutions aimed at achieving simplification and cost reduction while possibly enhancing robustness and performance. While certain aspects of the DEXMART Hand development have been presented in previous papers, this paper is the first to give a comprehensive description of the final hand version and its use to replicate humanlike grasping. In this paper, particular emphasis is placed on the kinematics of the fingers and of the thumb, the wrist architecture, the dimensioning of the actuation system, and the final implementation of the position, force and tactile sensors. The paper focuses also on how these solutions have been integrated into the mechanical structure of this innovative robotic hand to enable precise force and displacement control of the whole system. Another important aspect is the lack of suitable control tools that severely limits the development of robotic hand applications. To address this issue, a new method for the observation of human hand behavior during interaction with common day-to-day objects by means of a 3D computer vision system is presented in this work together with a strategy for mapping human hand postures to the robotic hand. A simple control strategy based on postural synergies has been used to reduce the complexity of the grasp planning problem. As a preliminary evaluation of the DEXMART Hand’s capabilities, this approach has been adopted in this paper to simplify and speed up the transfer of human actions to the robotic hand, showing its effectiveness in reproducing human-like grasping.

During last decade the scientific research on Unmanned Aerial Vehicless (UAVs) increased spectacularly and led to the design of multiple types of aerial platforms. The major challenge today is the development of autonomously operating aerial agents capable of completing missions independently of human interaction. To this extent, visual sensing techniques have been integrated in the control pipeline of the UAVs in order to enhance their navigation and guidance skills. The aim of this article is to present a comprehensive literature review on vision based applications for UAVs focusing mainly on current developments and trends. These applications are sorted in different categories according to the research topics among various research groups. More specifically vision based position-attitude control, pose estimation and mapping, obstacle detection as well as target tracking are the identified components towards autonomous agents. Aerial platforms could reach greater level of autonomy by integrating all these technologies onboard. Additionally, throughout this article the concept of fusion multiple sensors is highlighted, while an overview on the challenges C. Kanellakis ( ) · G. Nikolakopoulos Luleå University of Technology, Luleå, Sweden e-mail: christoforos.kanellakis@ltu.se G. Nikolakopoulos e-mail: geonik@ltu.se addressed and future trends in autonomous agent development will be also provided.

This paper introduces a stereo vision system to automatically generate 3D models of real objects. 3D model generation is based on the merging of multiview range images obtained from a digital stereo camera. Stereo images obtained from the camera are rectified, and a correlation-based stereo matching technique reconstructs range images from them. A turntable stage is also employed to obtain multiple range images of the objects. To register range images into a common coordinate system automatically, we introduce and calibrate a turntable coordinate system with respect to the camera coordinate system. After the registration of multiview range images, a 3D model is reconstructed using a volumetric integration technique. Error analysis on turntable calibration and 3D model reconstruction shows the accuracy of our 3D modeling system.

This paper describes the environment perception system designed for intelligent vehicle SmartV-II, which won the 2010 Future Challenge. This system utilizes the cooperation of multiple lasers and cameras to realize several necessary functions of autonomous navigation: road curb detection, lane detection and traffic sign recognition. Multiple single scan lasers are integrated to detect the road curb based on Z-variance method. Vision based lane detection is realized by two scans method combining with image model. Haar-like feature based method is applied for traffic sign detection and SURF matching method is used for sign classification. The results of experiments validate the effectiveness of the proposed algorithms and the whole system.

This paper addresses the problem of obtaining complete, detailed reconstructions of textureless shiny objects. We present an algorithm which uses silhouettes of the object, as well as images obtained under changing illumination conditions. In contrast with previous photometric stereo techniques, ours is not limited to a single viewpoint but produces accurate reconstructions in full 3D. A number of images of the object are obtained from multiple viewpoints, under varying lighting conditions. Starting from the silhouettes, the algorithm recovers camera motion and constructs the object's visual hull. This is then used to recover the illumination and initialize a multiview photometric stereo scheme to obtain a closed surface reconstruction. There are two main contributions in this paper: First, we describe a robust technique to estimate light directions and intensities and, second, we introduce a novel formulation of photometric stereo which combines multiple viewpoints and, hence, allows closed surface reconstructions. The algorithm has been implemented as a practical model acquisition system. Here, a quantitative evaluation of the algorithm on synthetic data is presented together with complete reconstructions of challenging real objects. Finally, we show experimentally how, even in the case of highly textured objects, this technique can greatly improve on correspondence-based multiview stereo results.

We present a Bayesian object observation model for complete probabilistic semantic SLAM. Recent studies on object detection and feature extraction have become important for scene understanding and 3D mapping. However, 3D shape of the object is too complex to formulate the probabilistic observation model; therefore, performing the Bayesian inference of the object-oriented features as well as their pose is less considered. Besides, when the robot equipped with an RGB mono camera only observes the projected single view of an object, a significant amount of the 3D shape information is abandoned. Due to these limitations, semantic SLAM and viewpoint-independent loop closure using volumetric 3D object shape is challenging. In order to enable the complete formulation of probabilistic semantic SLAM, we approximate the observation model of a 3D object with a tractable distribution. We also estimate the variational likelihood from the 2D image of the object to exploit its observed single view. In order to evaluate the proposed method, we perform pose and feature estimation, and demonstrate that the automatic loop closure works seamlessly without additional loop detector in various environments.

The majority of methods for the automatic surface reconstruction of an environment from an image sequence have two steps: Structure-from-Motion and dense stereo. From the computational standpoint, it would be interesting to avoid dense stereo and to generate a surface directly from the sparse cloud of 3D points and their visibility information provided by Structure-fromMotion. The previous attempts to solve this problem are currently very limited: the surface is non-manifold or has zero genus, the experiments are done on small scenes or objects using a few dozens of images. Our solution does not have these limitations. Furthermore, we experiment with hand-held or helmet-held catadioptric cameras moving in a city and generate 3D models such that the camera trajectory can be longer than one kilometer.

Wireless optical communications are emerging as a viable solution for high-speed data transmission over short ranges in the ocean, complementing mainstream acoustic communication systems that operate over much longer ranges, but at lower data rates. The current drive to develop cooperative autonomous vehicular systems to carry out surveying and other complex missions in the ocean critically depends on the existence of such communication links to share sensory and coordination information. This paper presents a compact, low power consumption, cost-efficient, and lightweight optical modem designed for fast data transmission between MEDUSA underwater vehicles at ranges on the order of 10 m. The transmitter uses LEDs for ON-OFF keying, while the receiver front-end adopts a transimpedance architecture with a photodiode detector. This simplistic transmission and reception technique reduces the overall design and hardware complexities compared to, e.g., laser diodes and photomultiplier tubes. The LEDs are arranged in a circular array, with a photodiode at the centre to enable half-duplex operation. These modems are designed to achieve data rates of 20-200 kb/s over short-range, line-of-sight, configurations. A transparent casing is customised to fit the Medusa vehicles, with proper alignment for inter-vehicle communication when they move in formation.

This paper introduces Push-Net, a deep recurrent neural network model, which enables a robot to push objects of unknown physical properties for re-positioning and re-orientation, using only visual camera images as input. The unknown physical properties is a major challenge for pushing. Push-Net overcomes the challenge by tracking a history of push interactions with an LSTM module and training an auxiliary objective function that estimates an object’s center of mass. We trained Push-Net entirely in simulation and tested it extensively on many different objects in both simulation and on two real robots, a Fetch arm and a Kinova MICO arm. Experiments suggest that Push-Net is robust and efficient. It achieved over 97% success rate in simulation on average and succeeded in all real robot experiments with a small number of pushes.


Question: Multi-view laser reconstruction for autonomous underwater object grasping.

Advancements in Autonomous Underwater Vehicle Design and Control: An Overview

Over the past few decades, there has been a surge in global research and development in underwater robotics, particularly in the realm of autonomous underwater vehicles (AUVs). As environmental concerns, resource exploration, and scientific and military objectives increasingly focus on the ocean, the demand for underwater robotic systems has grown. To address the complexities and risks posed by the unpredictable and challenging marine environment, significant strides have been made in AUV development. Throughout the 1990s, approximately 30 new AUV models were constructed globally, driven by innovations in materials science, computing, sensor technology, and theoretical foundations. These advancements have catalyzed a proliferation of research and development activities within the AUV community. Yet, this is merely the start of a journey towards the creation of more sophisticated, practical, and dependable AUVs. This paper offers a comprehensive review of select critical areas in the cutting-edge technologies of underwater robotics, providing a curated bibliography for future advancements. The 21st century promises to usher in technological breakthroughs that will facilitate the creation of AUVs that are more effective, resilient, and serviceable in real-world applications.

In urban environments, moving obstacles detection and free space determination are key issues for driving assistance systems and autonomous vehicles. When using lidar sensors scanning in front of the vehicle, uncertainty arises from ignorance and errors. Ignorance is due to the perception of new areas and errors come from imprecise pose estimation and noisy measurements. Complexity is also increased when the lidar provides multi-echo and multi-layer information. This paper presents an occupancy grid framework that has been designed to manage these different sources of uncertainty. A way to address this problem is to use grids projected onto the road surface in global and local frames. The global one generates the mapping and the local one is used to deal with moving objects. A credibilist approach is used to model the sensor information and to do a global fusion with the world-fixed map. Outdoor experimental results carried out with a precise positioning system show that such a perception strategy increases significantly the performance compared to a standard approach.

In this paper we propose a new method for detecting multiple specific 3D objects in real time. We start from the template-based approach based on the LINE2D/LINEMOD representation introduced recently by Hinterstoisser et al., yet extend it in two ways. First, we propose to learn the templates in a discriminative fashion. We show that this can be done online during the collection of the example images, in just a few milliseconds, and has a big impact on the accuracy of the detector. Second, we propose a scheme based on cascades that speeds up detection. Since detection of an object is fast, new objects can be added with very low cost, making our approach scale well. In our experiments, we easily handle 10-30 3D objects at frame rates above 10fps using a single CPU core. We outperform the state-of-the-art both in terms of speed as well as in terms of accuracy, as validated on 3 different datasets. This holds both when using monocular color images (with LINE2D) and when using RGBD images (with LINEMOD). Moreover, we propose a challenging new dataset made of 12 objects, for future competing methods on monocular color images.

Using vision systems with SCARA robotic manipulators is beneficial in many industrial applications specially in automated object picking and placing tasks. In this paper, the equations of forward and inverse kinematics of a SCARA manipulator are derived using D-H parameters convention. The mechanical structure, as well as the electrical, electronic and pneumatic components of the manipulator were designed and developed. The manipulator's control software, animation and simulation were developed using LabVIEW and Arduino. In addition to that, a vision system was developed and integrated with the manipulator system where a mobile phone camera was used to acquire images, then transmit them wirelessly to LabVIEW's Vision Assistant codes for processing and calibration. The developed vision system allowed detection and calculation of object positions within the manipulator's workspace. Finally, the manipulator's positioning errors were determined and discussed based on a number of repeatability experiments.

This paper studies the problem of detecting unknown objects within indoor environments in an active and natural manner. The visual saliency scheme utilizing both color and depth cues is proposed to arouse the interests of the machine system for detecting unknown objects at salient positions in a 3D scene. The 3D points at the salient positions are selected as seed points for generating object hypotheses using the 3D shape. We perform multi-class labeling on a Markov random field (MRF) over the voxels of the 3D scene, combining cues from object hypotheses and 3D shape. The results from MRF are further refined by merging the labeled objects, which are spatially connected and have high correlation between color histograms. Quantitative and qualitative evaluations on two benchmark RGB-D datasets illustrate the advantages of the proposed method. The experiments of object detection and manipulation performed on a mobile manipulator validate its effectiveness and practicability in robotic applications.

Grabbing and manipulating virtual objects is an important user interaction for immersive virtual environments. We present implementations and discussion of six techniques which allow manipulation of remote objects. A user study of these techniques was performed which revealed their characteristics and deficiencies, and led to the development of a new class of techniques. These hybrid techniques provide istinct advantages in terms of ease of use and efficiency because they consider the tasks of grabbing and manipulation separately. CR Categories and Subject Descr iptors: I.3.7 [Computer Graphics]:Three-Dimensional Graphics and Realism Virtual Reality; I.3.6 [Computer Graphics]:Methodology and Techniques Interaction Techniques.

In this paper, we present an infrastructure-free mapping and localization framework for rail vehicles using only a lidar sensor. Our method is designed to handle the pathological environment found in modern underground tunnels: narrow, parallel, and relatively smooth concrete walls with very little infrastructure to break up the empty spaces in the tunnel. By using an RQE-based, point-cloud alignment approach, we are able to implement a sliding-window algorithm, used for both mapping and localization. We demonstrate the proposed method with datasets gathered on a subway train travelling at high speeds (up to 70 km/h) in an underground tunnel for a total of 20 km across 6 runs. Our method is capable of mapping the tunnel with less than 0.6% error over the total length of the generated map. It is capable of continuously localizing, relative to the generated map, to within 10 cm in stations and at crossovers, and 1.8 m in pathological sections of tunnel. This method improves rail-based localization in a tunnel, which can be used to increase capacity on existing railways and for automated trains.

In this paper we introduce Co-Fusion, a dense SLAM system that takes a live stream of RGB-D images as input and segments the scene into different objects (using either motion or semantic cues) while simultaneously tracking and reconstructing their 3D shape in real time. We use a multiple model fitting approach where each object can move independently from the background and still be effectively tracked and its shape fused over time using only the information from pixels associated with that object label. Previous attempts to deal with dynamic scenes have typically considered moving regions as outliers, and consequently do not model their shape or track their motion over time. In contrast, we enable the robot to maintain 3D models for each of the segmented objects and to improve them over time through fusion. As a result, our system can enable a robot to maintain a scene description at the object level which has the potential to allow interactions with its working environment; even in the case of dynamic scenes.

In this paper we will address the problem of planning optimal grasps. Two general optimality criteria, that consider the total finger force and the maximum finger force will be introduced and discussed. Moreover their formalization, using various metrics on a space of generalized forces, will be detailed. The geometric interpretation of the two criteria will lead to an efficient planning algorithm. An example of its use in a robotic environment equipped with two-jaw and three-jaw grippers will also be shown.

We propose an intelligent visuomotor system that interacts with the environment and memorizes the consequences of actions. As more memories are recorded and more interactions are observed, the agent becomes more capable of predicting the consequences of actions and is, thus, better at planning sequences of actions to solve tasks. In previous work, we introduced the aspect transition graph (ATG) which represents how actions lead from one observation to another using a directed multi-graph. In this work, we propose a novel aspect representation based on hierarchical CNN features, learned with convolutional neural networks, that supports manipulation and captures the essential affordances of an object based on RGB-D images. In a traditional planning system, robots are given a pre-defined set of actions that take the robot from one symbolic state to another. However symbolic states often lack the flexibility to generalize across similar situations. Our proposed representation is grounded in the robot's observations and lies in a continuous space that allows the robot to handle similar unseen situations. The hierarchical CNN features within a representation also allow the robot to act precisely with respect to the spatial location of individual features. We evaluate the robustness of this representation using the Washington RGB-D Objects Dataset and show that it achieves state of the art results for instance pose estimation. We then test this representation in conjunction with an ATG on a drill grasping task on Robonaut-2. We show that given grasp, drag, and turn demonstrations on the drill, the robot is capable of planning sequences of learned actions to compensate for reachability constraints.

In this paper we show and evaluate the design of a novel telemanipulation system that maps proximity values, acquired inside of a gripper, to forces a user can feel through a haptic input device. The command console is complemented by input-devices that give the user an intuitive control over parameters relevant to the system. Furthermore, proximity sensors enable the autonomous alignment/centering of the gripper to objects in user-selected DoFs with the potential of aiding the user and lowering the workload. We evaluate our approach in a user study that shows that the telemanipulation system benefits from the supplementary proximity information and that the workload can indeed be reduced when the system operates with partial autonomy.


Question: A hierarchical multi-label categorization method for ticket data utilizing contextual loss.

This paper delves into the realm of hierarchical classification, focusing on scenarios where the taxonomy allows for labels corresponding to partial or multiple paths. We propose a novel hierarchical loss function, termed the H-loss, which embodies the principle that errors within the subtree of an already mistaken class should not incur additional penalties. Drawing from a probabilistic data model established in prior research, we ascertain the Bayes-optimal classifier for the H-loss. Subsequently, we conduct empirical comparisons between two incremental approximations of this Bayes-optimal classifier and a flat SVM classifier, alongside classifiers generated by hierarchical variants of the Perceptron and SVM algorithms. Our findings reveal that the simplest incremental approximation of the Bayes-optimal classifier achieves, after a single training iteration, performance levels nearly equivalent to those of the hierarchical SVM classifier, which stands out as the most effective. Moreover, for this incremental algorithm, we derive an H-loss bound, which reveals an exponential convergence to the H-loss of the hierarchical classifier based on the true model parameters when data adhere to our probabilistic data model.

Multilabel image annotation is one of the most important open problems in computer vision field. Unlike existing works that usually use conventional visual features to annotate images, features based on deep learning have shown potential to achieve outstanding performance. In this work, we propose a multimodal deep learning framework, which aims to optimally integrate multiple deep neural networks pretrained with convolutional neural networks. In particular, the proposed framework explores a unified two-stage learning scheme that consists of (i) learning to fune-tune the parameters of deep neural network with respect to each individual modality, and (ii) learning to find the optimal combination of diverse modalities simultaneously in a coherent process. Experiments conducted on the NUS-WIDE dataset evaluate the performance of the proposed framework for multilabel image annotation, in which the encouraging results validate the effectiveness of the proposed algorithms.

Automatic Text Classification is a semi-supervised machine learning task that automatically assigns a given document to a set of pre-defined categories based on its textual content and extracted features. Automatic Text Classification has important applications in content management, contextual search, opinion mining, product review analysis, spam filtering and text sentiment mining. This paper explains the generic strategy for automatic text classification and surveys existing solutions to major issues such as dealing with unstructured text, handling large number of attributes and selecting a machine learning technique appropriate to the text-classification application.

Search task identification aims to understand a user's information needs to improve search quality for applications such as query suggestion, personalized search, and advertisement retrieval. To properly identify the search task within long query sessions, it is important to partition these sessions into segments before further processing. In this paper, we present the first search session segmentation model that uses a long short-term memory (LSTM) network with an attention mechanism. This model considers sequential context knowledge, including temporal information, word and character, and essentially learns which parts of the input query sequence are relevant to the current segmentation task and determines the influence of these parts. This segmentation technique is also combined with an efficient clustering method using a novel query relevance metric for end-to-end search task identification. Using real-world datasets, we demonstrate that our segmentation technique improves task identification accuracy of existing clustering-based algorithms.

In multi-label classification, the main focus has been to develop ways of learning the underlying dependencies between labels, and to take advantage of this at classification time. Developing better feature-space representations has been predominantly employed to reduce complexity, e.g., by eliminating non-helpful feature attributes from the input space prior to (or during) training. This is an important task, since many multilabel methods typically create many different copies or views of the same input data as they transform it, and considerable memory can be saved by taking advantage of redundancy. In this paper, we show that a proper development of the feature space can make labels less interdependent and easier to model and predict at inference time. For this task we use a deep learning approach with restricted Boltzmann machines. We present a deep network that, in an empirical evaluation, outperforms a number of competitive methods from the literature.

A challenging problem in hierarchical classification is to leverage the hierarchical relations among classes for improving classification performance. An even greater challenge is to do so in a manner that is computationally feasible for large scale problems. This paper proposes a set of Bayesian methods to model hierarchical dependencies among class labels using multivariate logistic regression. Specifically, the parent-child relationships are modeled by placing a hierarchical prior over the children nodes centered around the parameters of their parents; thereby encouraging classes nearby in the hierarchy to share similar model parameters. We present variational algorithms for tractable posterior inference in these models, and provide a parallel implementation that can comfortably handle largescale problems with hundreds of thousands of dimensions and tens of thousands of classes. We run a comparative evaluation on multiple large-scale benchmark datasets that highlights the scalability of our approach and shows improved performance over the other state-of-the-art hierarchical methods.

Asynchronous conversations are conversations where participants collaborate with each other at different times (e.g., email, blog, forum). The huge amount of textual data generated everyday in these conversations calls for automated methods of conversational text analysis. Topic segmentation and labeling is often considered a prerequisite for higher-level conversation analysis and has been shown to be useful in many NLP applications including summarization, information extraction and conversation visualization. Topic segmentation in asynchronous conversation refers to the task of clustering the sentences into a set of coherent topical clusters. In [6, 7], we presented unsupervised and supervised topic segmentation models for asynchronous conversations, which to our knowledge are the state-of-the-art. This study concerns topic labeling, that is, given a set of topical clusters in a conversation, the task is to assign appropriate topic labels to the clusters. For example, five topic labels in a Slashdot [2] blog conversation about releasing a new game called Daggerfall are game contents and size, game design, bugs/faults, other gaming options and performance/speed issues. Such topic labels can serve as a concise summary of the conversation and can also be used for indexing. Ideally, topic labels should be meaningful, semantically similar to the underlying topic, general (i.e., broad coverage of the topic) and discriminative (or exclusive) when there are multiple topics [10]. Traditionally, the top K terms in a multinomial topic model (e.g., LDA [3]) are used to represent a topic. However, as pointed out by [10], at the word-level, topic labels may become too general that it can impose cognitive difficulties on a user to interpret the meaning of the topic by associating the words together. On the other hand, if the labels are expressed at the sentence-level, they may become too specific to cover the whole theme of the topic. Based on these observations, [10] and other recent studies (e.g., [9]) advocate for phrase-level topic labels. This is also consistent with the monologue corpora built as part of the Topic Detection and Tracking (TDT) project [1] as well as with our own email and blog conversational corpora in which human annotators without specific instructions spontaneously generated labels at the phrase-level. Considering all these factors, we also treat phrase-level as the right level of granularity for a label in this work. Few prior studies have addressed the topic labeling problem in different settings [10, 9]. Common to their approaches is that they first mine topics in the form of topic-word distributions from the whole corpus using topic models like LDA. Then, they try to label the topics (i.e., topic-word distributions) with an appropriate label using the statistical association metrics (e.g., point-wise mutual information, t-test) computed from either the source corpus or an external knowledge base (e.g., Wikipedia). In contrast, our task is to label the topical clusters in a given conversation, where topics are closely related and distributional variations are subtle (e.g., game contents and size, game design). Therefore, corpus-based statistical association metrics are not reliable in our case. Also at the conversation-level, the topics are too specific to find their labels in an external source. To our knowledge, none has studied this problem before. Therefore, there is no standard corpus and no agreed-upon evaluation metrics available. Our contributions aim to remedy these problems. First, we present a blog and an email corpora annotated with topics. Second, we propose to generate topic labels using an extractive approach, that finds the most representative phrases from the text without relying on an external source. Since, graph-based key phrase ranking has proved to be the state-of-the-art (unsupervised) method [11], we adopt the same framework. We propose a novel biased random walk model that exploits the fact that the leading sentences in a topic often carry the most informative clues for its label. However, the phrases extracted only by considering the sentences in a topic may ignore the global aspect of the conversation. As another contribution, we propose to re-rank the phrases extracted from the whole conversation with respect to the individual topics and include the relevant ones. Experimental results show that our approach outperforms other ranking models including a general random walk model (i.e., TextRank) proposed by [11], a leadoriented model and a frequency-based model, and including the relevant conversation-level phrases improves the performance.

Answer selection in community question answering (cQA) is a challenging task in natural language processing. The difficulty lies in that it not only needs the consideration of semantic matching between question answer pairs but also requires a serious modeling of contextual factors. In this letter, we propose an attentive deep neural network architecture so as to learn the deterministic information for answer selection. The architecture can support various input formats through the organization of convolutional neural networks, attention-based long short-term memory, and conditional random fields. Experiments are carried out on the SemEval-2015 cQA dataset. We attain 58.35% on macroaveraged $\mathrm{F_1}$, which outperforms the Top-1 system in the shared task by 1.16% and improves the state-of-the-art deep-neural-network-based method by 2.21%.

The four-level pachinko allocation model (PAM) (Li & McCallum, 2006) represents correlations among topics using a DAG structure. It does not, however, represent a nested hierarchy of topics, with some topical word distributions representing the vocabulary that is shared among several more specific topics. This paper presents hierarchical PAM---an enhancement that explicitly represents a topic hierarchy. This model can be seen as combining the advantages of hLDA's topical hierarchy representation with PAM's ability to mix multiple leaves of the topic hierarchy. Experimental results show improvements in likelihood of held-out documents, as well as mutual information between automatically-discovered topics and humangenerated categories such as journals.

Microsoft Office users submit hundreds of thousands of pieces of verbatim feedback per month. How can an engineer or manager in Office find the signal in this data to make business decisions? This paper presents an overview of the Office Customer Voice (OCV) system. OCV combines classification, on-demand clustering and other machine learning techniques with a rich web UI to solve this problem. In this paper, we describe the different types of feedback received. Next, we outline the architecture used to build OCV. We then detail the text processing, classification and clustering done to reason on the data. Finally, we present challenges, future plans, and best practices that may be relevant to other teams analyzing customer feedback. We argue that this multi-pronged approach to handling customer feedback presents a pattern that other organizations can use to mature their handling of customer feedback.

AdaBoost.MH is a boosting algorithm that is considered to be one of the most accurate algorithms for multilabel classification. It works by iteratively building a committee of weak hypotheses of decision stumps. To build the weak hypotheses, in each iteration, AdaBoost.MH obtains the whole extracted features and examines them one by one to check their ability to characterize the appropriate category. Using Bag-Of-Words for text representation dramatically increases the computational time of AdaBoost.MH learning, especially for large-scale datasets. In this paper we demonstrate how to improve the efficiency and effectiveness of AdaBoost.MH using latent topics, rather than words. A well-known probabilistic topic modelling method, Latent Dirichlet Allocation, is used to estimate the latent topics in the corpus as features for AdaBoost.MH. To evaluate LDA-AdaBoost.MH, the following four datasets have been used: Reuters-21578-ModApte, WebKB, 20-Newsgroups and a collection of Arabic news. The experimental results confirmed that representing the texts as a small number of latent topics, rather than a large number of words, significantly decreased the computational time of AdaBoost.MH learning and improved its performance for text categorization.