Question: Utilizing a model-assisted bundle adjustment framework for mapping underwater ship hull surfaces.

This paper introduces a comprehensive framework for evaluating and categorizing dense two-frame stereo correspondence algorithms, a critical area within computer vision research. Despite the proliferation of stereo matching techniques, there has been a notable gap in systematically analyzing their efficacy. To address this, we have developed a detailed taxonomy that focuses on the diverse components and design choices integral to stereo algorithms. This taxonomy allows for a comparative analysis of existing methods and facilitates the assessment of various algorithmic permutations. In pursuit of standardizing the evaluation process, we have crafted a versatile C++ software platform that not only evaluates individual components but also offers the flexibility to incorporate new algorithms. Additionally, we have curated a series of multi-frame stereo data sets, complete with ground truth data, which are now accessible online. Our contribution extends to the provision of these resources, alongside a thorough comparative analysis of leading stereo algorithms currently in use. This endeavor not only enriches the field of stereo correspondence but also provides a benchmark for future research and algorithm development.

**Question**: A New Method for the Automatic Detection of Solar Filaments Using Image Processing Techniques

**Documents**: Automatic Detection of Solar Filaments from Full Disk Solar Images Using a Neural Network
In this paper, we present a method for the automatic detection of solar filaments from full disk solar images using a neural network. The method is based on a Convolutional Neural Network (CNN) that has been trained to identify filaments in solar images. The CNN is trained using a dataset of manually labeled filament locations. The method is tested on a set of images that were not used in the training set. The results show that the method is able to detect filaments with a high degree of accuracy. The method is also compared to a traditional image processing method, and it is shown that the neural network method outperforms the traditional method in terms of filament detection accuracy. The method is fast and can be used to process large datasets of solar images in real-time. The method can also be extended to other types of solar features, such as sunspots and prominences, by retraining the neural network with a new dataset of labeled images.

**Similar Documents**: This paper introduces an innovative technique for automatically identifying solar filaments in full disk solar images, employing a Convolutional Neural Network (CNN) for the task. The proposed method leverages a CNN that has been meticulously trained on a dataset of manually annotated filament locations to recognize filament patterns in solar images. The CNN's effectiveness is validated through testing on a separate set of images that were excluded from the training phase. The outcomes

The goal of our work is to provide a fast and accurate method to estimate the camera motion from RGB-D images. Our approach registers two consecutive RGB-D frames directly upon each other by minimizing the photometric error. We estimate the camera motion using non-linear minimization in combination with a coarse-to-fine scheme. To allow for noise and outliers in the image data, we propose to use a robust error function that reduces the influence of large residuals. Furthermore, our formulation allows for the inclusion of a motion model which can be based on prior knowledge, temporal filtering, or additional sensors like an IMU. Our method is attractive for robots with limited computational resources as it runs in real-time on a single CPU core and has a small, constant memory footprint. In an extensive set of experiments carried out both on a benchmark dataset and synthetic data, we demonstrate that our approach is more accurate and robust than previous methods. We provide our software under an open source license.

This paper describes the implementation of a LCD stripe projection system, based on a new processing scheme called line shift processing. One advantage of the method is, that the projection device can, but is not required to be stable over time nor does it have to be calibrated. The new method therefore allows us to use an off-the-shelf multimedia projector to acquire dense 3-D data (several 100K Triangles) of large objects in a matter of seconds. Since we are able to control our source of illumination, we can also acquire registered color information using standard monochrome cameras. Besides the description of the system some practical examples are given. The calibration of the system is also addressed.

RGB-D cameras are novel sensing systems that capture RGB images along with per-pixel depth information. RGB-D cameras rely on either structured light patterns combined with stereo sensing [6,10] or time-of-flight laser sensing [1] to generate depth estimates that can be associated with RGB pixels. Very soon, small, high-quality RGB-D cameras developed for computer gaming and home entertainment applications will become available at cost below $100. In this paper we investigate how such cameras can be used in the context of robotics, specifically for building dense 3D maps of indoor environments. Such maps have applications in robot navigation, manipulation, semantic mapping, and telepresence. The robotics and computer vision communities have developed a variety of techniques for 3D mapping based on laser range scans [8, 11], stereo cameras [7], monocular cameras [3], and unsorted collections of photos [4]. While RGB-D cameras provide the opportunity to build 3D maps of unprecedented richness, they have drawbacks that make their application to 3D mapping difficult: They provide depth only up to a limited distance (typically less than 5m), depth values are much noisier than those provided by laser scanners, and their field of view (∼ 60◦) is far more constrained than that of specialized cameras or laser scanners typically used for 3D mapping (∼ 180◦). In our work, we use a camera developed by PrimeSense [10]. The key insights of this investigation are: first, that existing frame matching techniques are not sufficient to provide robust visual odometry with these cameras; second, that a tight integration of depth and color information can yield robust frame matching and loop closure detection; third, that building on best practice techniques in SLAM and computer graphics makes it possible to build and visualize accurate and extremely rich 3D maps with such cameras; and, fourth, that it will be feasible to build complete robot navigation and interaction systems solely based on cheap depth cameras.

This paper focuses on motion estimation using inertial measurements and observations of naturally occurring point features. To date, this task has primarily been addressed using filtering methods, which track the system state starting from known initial conditions. However, when no prior knowledge of the initial system state is available, (e.g., at the onset of the system's operation), the existing approaches are not applicable. To address this problem, in this work we present algorithms for computing the system's observable quantities (platform attitude and velocity, feature positions, and IMU-camera calibration) directly from the sensor measurements, without any prior knowledge. A key contribution of this work is a convex-optimization based algorithm for computing the rotation matrix between the camera and IMU. We show that once this rotation matrix has been computed, all remaining quantities can be determined by solving a quadratically constrained least-squares problem. To increase their accuracy, the initial estimates are refined by an iterative maximum-likelihood estimator.

A new technique to recognise 3D free-form objects via registration is proposed. This technique attempts to register a free-form surface, represented by a set of % MathType!MTEF!2!1!+-% feaafeart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn% hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr% 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq-Jc9% vqaqpepm0xbba9pwe9Q8fs0-yqaqpepae9pg0FirpepeKkFr0xfr-x% fr-xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaGaaGOmamaala% aabaGaaGymaaqaaiaaikdaaaGaamiraaaa!38F8!\[2\frac{1}{2}D\] sensed data points, to the model surface, represented by another set of % MathType!MTEF!2!1!+-% feaafeart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn% hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr% 4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq-Jc9% vqaqpepm0xbba9pwe9Q8fs0-yqaqpepae9pg0FirpepeKkFr0xfr-x% fr-xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaGaaGOmamaala% aabaGaaGymaaqaaiaaikdaaaGaamiraaaa!38F8!\[2\frac{1}{2}D\] model data points, without prior knowledge of correspondence or view points between the two point sets. With an initial assumption that the sensed surface be part of a more complete model surface, the algorithm begins by selecting three dispersed, reliable points on the sensed surface. To find the three corresponding model points, the method uses the principal curvatures and the Darboux frames to restrict the search over the model space. Invariably, many possible model 3-typles will be found. For each hypothesized model 3-tuple, the transformation to match the sensed 3-tuple to the model 3-tuple can be determined. A heuristic search is proposed to single out the optimal transformation in low order time. For realistic object recognition or registration, where the two range images are often extracted from different view points of the model, the earlier assumption that the sensed surface be part of a more complete model surface cannot be relied on. With this, the sensed 3-tuple must be chosen such that the three sensed points lie on the common region visible to both the sensed and model views. We propose an algorithm to select a minimal non-redundant set of 3-tuples such that at least one of the 3-tuples will lie on the overlap. Applying the previous algorithm to each 3-tuple within this set, the optimal transformation can be determined. Experiments using data obtained from a range finder have indicated fast registration for relatively complex test cases. If the optimal registrations between the sensed data (candidate) and each of a set of model data are found, then, for 3D object recognition purposes, the minimal best fit error can be used as the decision rule.

Ansar, the method by Ansar and Daniilidis (2003), implementation from Xu et al. (2016). Mirzaei, the method by Mirzaei and Roumeliotis (2011). RPnL, the method by Zhang et al. (2013). ASPnL, the method by Xu et al. (2016). LPnL Bar LS, the method by Xu et al. (2016). LPnL Bar ENull, the method by Xu et al. (2016). DLT-Lines, the method by Hartley and Zisserman (2004, p. 180), our implementation. DLT-Plücker-Lines, the method by Přibyl et al. (2015), our implementation. DLT-Combined-Lines, the proposed method.

There have been increasing demands for developing robotic system combining camera and inertial measurement unit in navigation task, due to their low-cost, lightweight and complementary properties. In this paper, we present a Visual Inertial Odometry (VIO) system which can utilize sparse depth to estimate 6D pose in GPS-denied and unstructured environments. The system is based on Multi-State Constraint Kalman Filter (MSCKF), which benefits from low computation load when compared to optimization-based method, especially on resource-constrained platform. Features are enhanced with depth information forming 3D landmark position measurements in space, which reduces uncertainty of position estimate. And we derivate measurement model to access compatibility with both 2D and 3D measurements. In experiments, we evaluate the performance of the system in different in-flight scenarios, both cluttered room and industry environment. The results suggest that the estimator is consistent, substantially improves the accuracy compared with original monocular-based MSKCF and achieves competitive accuracy with other research.

A model based approach for rectifying the camera image of the bound document has been developed, i.e., the surface of the document is represented by a general cylindrical surface. The principle of using the model to unwrap the image is discussed. Practically, the skeleton of each horizontal text line is extracted to help estimate the parameter of the model, and rectify the images. To use the model, only a few priori is required, and no more auxiliary device is necessary. Experiment results are given to demonstrate the feasibility and the stability of the method.

There have been increasing demands for developing microaerial vehicles with vision-based autonomy for search and rescue missions in complex environments. In particular, the monocular visual–inertial system (VINS), which consists of only an inertial measurement unit (IMU) and a camera, forms a great lightweight sensor suite due to its low weight and small footprint. In this paper, we address two challenges for rapid deployment of monocular VINS: 1) the initialization problem and 2) the calibration problem. We propose a methodology that is able to initialize velocity, gravity, visual scale, and camera–IMU extrinsic calibration on the fly. Our approach operates in natural environments and does not use any artificial markers. It also does not require any prior knowledge about the mechanical configuration of the system. It is a significant step toward plug-and-play and highly customizable visual navigation for mobile robots. We show through online experiments that our method leads to accurate calibration of camera–IMU transformation, with errors less than 0.02 m in translation and 1° in rotation. We compare out method with a state-of-the-art marker-based offline calibration method and show superior results. We also demonstrate the performance of the proposed approach in large-scale indoor and outdoor experiments.

This paper presents a visual-inertial odometry framework which tightly fuses inertial measurements with visual data from one or more cameras, by means of an iterated extended Kalman filter (IEKF). By employing image patches as landmark descriptors, a photometric error is derived, which is directly integrated as an innovation term in the filter update step. Consequently, the data association is an inherent part of the estimation process and no additional feature extraction or matching processes are required. Furthermore, it enables the tracking of non-corner shaped features, such as lines, and thereby increases the set of possible landmarks. The filter state is formulated in a fully robocentric fashion, which reduces errors related to nonlinearities. This also includes partitioning of a landmark’s location estimate into a bearing vector and distance and thereby allows an undelayed initialization of landmarks. Overall, this results in a compact approach which exhibits a high level of robustness with respect to low scene texture and motion blur. Furthermore, there is no time-consuming initialization procedure and pose estimates are available starting at the second image frame. We test the filter on different real datasets and compare it to other state-of-the-art visual-inertial frameworks. The experimental results show that robust localization with high accuracy can be achieved with this filterbased framework.


Question: High-precision simulation to assess the performance of robotic vision systems.

This paper explores the design and application paradigms for Gazebo, an open-source, multi-robot simulator that aims to revolutionize the field of robotics research. Historically, simulators have been instrumental in expediting the testing and validation of novel ideas, strategies, and algorithms in robotics. However, the majority of these simulators have been confined to two-dimensional (2D) environments, with only a few reaching a level of maturity that combines capability with adaptability. Gazebo addresses this gap by offering a three-dimensional (3D) dynamic environment that can accurately replicate the intricate landscapes and conditions faced by advanced mobile robots. Its open-source nature, coupled with precise control and high-fidelity simulation capabilities, positions Gazebo as more than just an intermediate step between theoretical development and real-world deployment. It holds potential for a wide array of applications, including data visualization, simulation of inaccessible or remote environments, and the reverse engineering of complex systems with opaque internal workings. Gazebo's development is a collaborative effort with the Player and Stage projects, enhancing its functionality and accessibility. For more information and to access Gazebo, visit http://playerstage.sourceforge.net/gazebo/.

In the context of evaluating robotic vision performance, Gazebo serves as a high-fidelity simulation platform, offering researchers and developers a comprehensive toolset to test and refine vision algorithms. The 3D environments created by Gazebo can be tailored to simulate various lighting conditions, object textures, and environmental factors, providing a realistic testing ground for vision systems. This capability is crucial for ensuring that robots can accurately perceive and navigate complex real-world scenarios, thereby enhancing their overall performance and reliability. Gazebo's adaptability and detailed simulation environment make it an invaluable resource for the continuous improvement of robotic vision technologies.

This paper is about the efficient generation of dense, colored models of city-scale environments from range data and in particular, stereo cameras. Better maps make for better understanding; better understanding leads to better robots, but this comes at a cost. The computational and memory requirements of large dense models can be prohibitive. We provide the theory and the system needed to create cityscale dense reconstructions. To do so, we apply a regularizer over a compressed 3D data structure while dealing with the complex boundary conditions this induces during the data-fusion stage. We show that only with these considerations can we swiftly create neat, large, “well behaved” reconstructions. We evaluate our system using the KITTI dataset and provide statistics for the metric errors in all surfaces created compared to those measured with 3D laser. Our regularizer reduces the median error by 40% in 3.4 km of dense reconstructions with a median accuracy of 6 cm. For subjective analysis, we provide a qualitative review of 6.1 km of our dense reconstructions in an attached video. These are the largest dense reconstructions from a single passive camera we are aware of in the literature. Video: https://youtu.be/FRmF7mH86EQ

We consider the task of driving a remote control car at high speeds through unstructured outdoor environments. We present an approach in which supervised learning is first used to estimate depths from single monocular images. The learning algorithm can be trained either on real camera images labeled with ground-truth distances to the closest obstacles, or on a training set consisting of synthetic graphics images. The resulting algorithm is able to learn monocular vision cues that accurately estimate the relative depths of obstacles in a scene. Reinforcement learning/policy search is then applied within a simulator that renders synthetic scenes. This learns a control policy that selects a steering direction as a function of the vision system's output. We present results evaluating the predictive ability of the algorithm both on held out test data, and in actual autonomous driving experiments.

We present a mobile vision system for multi-person tracking in busy environments. Specifically, the system integrates continuous visual odometry computation with tracking-by-detection in order to track pedestrians in spite of frequent occlusions and egomotion of the camera rig. To achieve reliable performance under real-world conditions, it has long been advocated to extract and combine as much visual information as possible. We propose a way to closely integrate the vision modules for visual odometry, pedestrian detection, depth estimation, and tracking. The integration naturally leads to several cognitive feedback loops between the modules. Among others, we propose a novel feedback connection from the object detector to visual odometry which utilizes the semantic knowledge of detection to stabilize localization. Feedback loops always carry the danger that erroneous feedback from one module is amplified and causes the entire system to become instable. We therefore incorporate automatic failure detection and recovery, allowing the system to continue when a module becomes unreliable. The approach is experimentally evaluated on several long and difficult video sequences from busy inner-city locations. Our results show that the proposed integration makes it possible to deliver stable tracking performance in scenes of previously infeasible complexity.

Dexterous surgical activity is of interest to many researchers in human motion modeling. In this paper, we describe a dataset of surgical activities and release it for public use. The dataset was captured using the da Vinci Surgical System and consists of kinematic and video from eight surgeons with different levels of skill performing five repetitions of three elementary surgical tasks on a bench-top model. The tasks, which include suturing, knot-tying and needle-passing, are standard components of most surgical skills training curricula. In addition to kinematic and video data captured from the da Vinci Surgical System, we are also releasing manual annotations of surgical gestures (atomic activity segments), surgical skill using global rating scores, a standardized cross-validation experimental setup, and a C++/Matlab toolkits for analyzing surgical gestures using hidden Markov models and using linear dynamical systems. We refer to the dataset as the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) to indicate the collaboration between Johns Hopkins University (JHU) and Intuitive Surgical Inc. (ISI), Sunnyvale, CA, on collecting these data.

Robot navigation has multi-spectral applications across a plethora of industries. Being one of the fastest evolving field of Artificial intelligence, it has garnered a lot of speculation and the problem of efficient Robot navigation has become a talking point in the “Artificial Intelligence research community”. Robot Navigation can be thought of as a collection of sub-problems that when solved efficiently will (most likely) give the complete and the most efficient solution. While the main Navigation problem can be divided into numerous smaller ones some of the prevalent (sub)-tasks associated with Robot Navigation are Path Planning, Collision Prevention, Search Algorithm and an appropriate pictorial (graphical) representation of the given environment. Path Planning gives us the most optimal path finding approach while keeping the dynamism of the environment and the objects in mind. Collision Prevention ensures the Robot reaches its goal without colliding with any of the objects present in the environment. Collision Prevention techniques' vitality increases considerably when the number of objects in the input space is large (>10, approximately). Search Algorithms are the functional units of this problem as they determine the path the Robot follows while traveling to its destination. Since the environment greatly affects the choice of technique for all the other tasks it is vital to have a clear, concise and a non-ambiguous representation of the same. The main aim of the paper is to summarise the development of various techniques in the multiple domains of the Robot Navigation problem in the last fifteen years (2000-2015) while stating the scope and the restriction of each of them.

This paper describes an approach for recognizing instances of a 3D object in a single camera image and for determining their 3D poses. A hierarchical model is generated solely based on the geometry information of a 3D CAD model of the object. The approach does not rely on texture or reflectance information of the object's surface, making it useful for a wide range of industrial and robotic applications, e.g., bin-picking. A hierarchical view-based approach that addresses typical problems of previous methods is applied: It handles true perspective, is robust to noise, occlusions, and clutter to an extent that is sufficient for many practical applications, and is invariant to contrast changes. For the generation of this hierarchical model, a new model image generation technique by which scale-space effects can be taken into account is presented. The necessary object views are derived using a similarity-based aspect graph. The high robustness of an exhaustive search is combined with an efficient hierarchical search. The 3D pose is refined by using a least-squares adjustment that minimizes geometric distances in the image, yielding a position accuracy of up to 0.12 percent with respect to the object distance, and an orientation accuracy of up to 0.35 degree in our tests. The recognition time is largely independent of the complexity of the object, but depends mainly on the range of poses within which the object may appear in front of the camera. For efficiency reasons, the approach allows the restriction of the pose range depending on the application. Typical runtimes are in the range of a few hundred ms.

The field of robotics began in the 1960s and 1970s with the design and control of general purpose robotic manipulators. During the 1980s the expense of integrating robotics into manufacturing lines and the difficulty of performing seemingly easy tasks, such as manipulating flexible objects or washing a window, led to disenchantment among many researchers and funding agencies. Since the 1990s the popularity of robotics has resurged with many new exciting areas of application particularly in the medical, defense, and service sectors. Robotics has grown from a field that is primarily concerned with problems such as the manipulation of rigid bodies in well-structured environments to a field that has tackled challenging problems such as extraterrestrial exploration and remote minimally invasive surgery. I began my involvement in the field of robotics as a graduate student studying control theory. My project was to design a control algorithm to induce stable walking in a biped robot. As with many control problems, before formulating a solution I had to determine a suitable model for the plant. Since I was an electrical engineering student with little background in the kinematics and dynamics of rigid-body mechanical systems, this task was daunting. With some guidance from a fellow student, I fumbled my way through Lagrange’s method to derive the robot’s equations of motion. I had little idea at the time why forming the Lagrangian and taking certain derivatives resulted in the right equations. For insight I was pointed to various texts on classical mechanics. While these texts provided the desired insight, I was still left with questions: Given an arbitrary system, how do I systematically construct the Lagrangian so that I can derive the system’s equation of motion? What are the most effective techniques for robot control? What special aspects of the robot’s dynamics should be considered in developing controllers? Five years after my introduction to the field of robotics, I now know that research in the field goes well beyond the dynamics and control of traditional manipulators. The concepts required to analyze and control manipulators, however, are still needed in modern robotics research. These fundamental concepts, which are addressed by many introductory texts on robotics, are the following: 1) Forward kinematics: Given joint displacements, determine the position and orientation of the endeffector with respect to a specified reference frame. 2) Inverse kinematics: Given the position and orientation of the end-effector frame relative to a specified reference frame, determine the joint variables that give rise to this configuration. 3) Differential kinematics: Given the manipulator’s joint rates, determine the linear and angular velocities of the end-effector frame relative to a fixed reference frame. 4) Statics: Given the forces and moments applied to the manipulator, determine the corresponding joint forces and torques required to maintain equilibrium in the presence of applied forces and moments. 5) Dynamics: Determine the equations of motion of the system. Specifically, determine the differential equations relating the applied forces and torques to the joint variables, joint rates, and joint accelerations. 6) Trajectory generation: Given initial and final configurations of the manipulator, find a trajectory of the joint variables connecting these two configurations. 7) Control: Given desired joint trajectories or forces between the end-effector and the environment determine joint torques and forces that effect these joint trajectories and forces.

Vision-aided inertial navigation systems (V-INSs) can provide precise state estimates for the 3-D motion of a vehicle when no external references (e.g., GPS) are available. This is achieved by combining inertial measurements from an inertial measurement unit (IMU) with visual observations from a camera under the assumption that the rigid transformation between the two sensors is known. Errors in the IMU-camera extrinsic calibration process cause biases that reduce the estimation accuracy and can even lead to divergence of any estimator processing the measurements from both sensors. In this paper, we present an extended Kalman filter for precisely determining the unknown transformation between a camera and an IMU. Contrary to previous approaches, we explicitly account for the time correlation of the IMU measurements and provide a figure of merit (covariance) for the estimated transformation. The proposed method does not require any special hardware (such as spin table or 3-D laser scanner) except a calibration target. Furthermore, we employ the observability rank criterion based on Lie derivatives and prove that the nonlinear system describing the IMU-camera calibration process is observable. Simulation and experimental results are presented that validate the proposed method and quantify its accuracy.

In this paper we present a new solution for real-time 3D camera pose estimation for Augmented Reality (AR) applications. The tracking system does not require special engineering of the environment, such as placing markers or beacons. The required input data are a CAD model of the target object to be tracked, and a calibrated reference image of it. We consider the whole process of camera tracking, and developed both an autonomous initialization and a real-time tracking procedure. The system is robust to abrupt camera motions, strong changes of the lighting conditions and partial occlusions. To avoid typical jitter and drift problems the tracker performs feature matching not only in an iterative manner, but also against stable reference features, which are dynamically cached in case of high confidence. We present experimental results generated with help of synthetic ground truth, real off-line and on-line image sequences using different types of target objects.

Real-time robot motion planning using rasterizing computer graphics hardware. In Proc. OY82] C. O'D unlaing and C.K. Yap. A retraction method for planning the motion of a disc. A local approach for path planning of manipulators with a high number of degrees of freedom. a path generation algorithm into oo-line programming of airbus panels. Motion planning for many degrees of freedom-random reeections at c-space obstacles. REFERENCES 31 iterative collision checker. We observed no dramatic slowdown of the planner. A challenging research goal would now be to extend the method to dynamic scenes. One rst question is: How should a roadmap computed for a given workspace be updated if a few obstacles are removed or added? Answering this question would be useful to apply our method to scenes subject to small incremental changes. Such changes occur in many manufacturing (e.g., assembly) cells; while most of the geometry of such a cell is permanent and stationary, a few objects (e.g., xtures) are added or removed between any two consecutive manufacturing operations. Similar incremental changes also occur in automatic graphic animation. A second question is: How should the learning and query phase be modiied if some obstacles are moving along known trajectories? An answer to this question might consist of applying our roadmap method in the conngurationtime space of the robot Lat91]. The roadmap would then have to be built as a directed graph, since local paths between any two nodes must monotonically progress along the time axis, with possibly additional constraints on their slope and curvature to reeect bounds on the robot's velocity and acceleration. A motion planner with performance proportional to task diiculty. 30 In KL94a, KL94b, O S94] prior versions of the method have been applied to a great variety of holonomic robots including planar and spatial articulated robots with revolute, prismatic, and/or spherical joints, xed or free base, and single or multiple kinematic chains. In Sve93, SO94] a variation of the method (essentially one with a diierent general local planner) was also run successfully on examples involving nonholonomic car-like robots. Experimental results show that our method can eeciently solve problems which are beyond the capabilities of other existing methods. For example, for planar articulated robots with many dofs, the customized implementation of Section 5 is much more consistent than the Randomized Path Planner (RPP) of BL91]. Indeed, the latter can be very fast on some diicult problems, but it …


Question: High-precision simulation to assess the performance of robotic vision systems.

This article introduces a cutting-edge feature-based monocular Simultaneous Localization and Mapping (SLAM) system, designed to deliver superior robustness, enhanced camera pose accuracy, and semi-dense reconstructions that match or surpass the current benchmarks. In recent times, direct SLAM methods, which do not rely on features, have gained prominence for their ability to generate semi-dense or dense scene reconstructions, thereby challenging the necessity of feature usage—a cornerstone of established SLAM techniques from the past decade. The proposed feature-based SLAM system excels by processing keyframes optimized through local bundle adjustment, enabling accurate triangulations from wide baselines. A distinguishing feature of our approach is a novel correspondence search algorithm, complemented by measurement fusion and inter-keyframe depth consistency checks, which collectively result in clean reconstructions with minimal outliers. Contrary to the prevailing trend in direct SLAM, we demonstrate that segregating semi-dense reconstruction from trajectory computation yields superior outcomes. This finding reinvigorates the discourse on the merits of features, even in scenarios where semi-dense reconstruction is sought.

A key problem of a mobile robot system is how to localize itself and detect objects in the workspace. In this paper, a multiple sensor based robot localization and object pose estimation method is presented. First, optical encoders and odometry model are utilized to determine the pose of the mobile robot in the workspace, with respect to the global coordinate system. Next, a CCD camera is used as a passive sensor to find an object (a box) in the environment, including the specific vertical surfaces of the box. By identifying and tracking color blobs which are attached to the center of each vertical surface of the box, the robot rotates and adjusts its base pose to move the color blob into the center of the camera view in order to make sure that the box is in the range of the laser scanner. Finally, a laser range finder, which is mounted on the top of the mobile robot, is activated to detect and compute the distances and angles between the laser source and laser contact surface on the box. Based on the information acquired in this manner, the global pose of the robot and the box can be represented using the homogeneous transformation matrix. This approach is validated using the Microsoft Robotics Studio simulation environment.

Accurate metrical localization is one of the central challenges in mobile robotics. Many existing methods aim at localizing after building a map with the robot. In this paper, we present a novel approach that instead uses geo-tagged panoramas from the Google Street View as a source of global positioning. We model the problem of localization as a non-linear least squares estimation in two phases. The first estimates the 3D position of tracked feature points from short monocular camera sequences. The second computes the rigid body transformation between the Street View panoramas and the estimated points. The only input of this approach is a stream of monocular camera images and odometry estimates. We quantified the accuracy of the method by running the approach on a robotic platform in a parking lot by using visual fiducials as ground truth. Additionally, we applied the approach in the context of personal localization in a real urban scenario by using data from a Google Tango tablet.

Autonomous underwater vehicles (AUVs) are usually equipped with one or more optical cameras to obtain visual data of underwater environments. The camera can also be used to estimate the AUV's pose information, along with various navigation sensors such as inertial measurement unit (IMU), Doppler velocity log (DVL), depth sensor, and so on. In this paper, we propose a vision-based simultaneous localization and mapping (SLAM) of AUVs, where underwater artificial landmarks are used to help visual sensing of forward and downward looking cameras. Three types of landmarks are introduced and their detection algorithms are organized in a framework of conventional extended Kalman filter (EKF) SLAM to estimate both robot and landmark states. The proposed method is validated by an experiment performed in a engineering basin. Since DVL suffers from noises in a real ocean environment, we generated synthetic noisy data based on the real sensor data. With this data we verify that the proposed SLAM approach can recover from the erroneous dead reckoning position.

In this study, we propose integrated control of a robotic hand and arm using only proximity sensing from the fingertips. An integrated control scheme for the fingers and for the arm enables quick control of the position and posture of the arm by placing the fingertips adjacent to the surface of an object to be grasped. The arm control scheme enables adjustments based on errors in hand position and posture that would be impossible to achieve by finger motions alone, thus allowing the fingers to grasp an object in a laterally symmetric grasp. This can prevent grasp failures such as a finger pushing the object out of the hand or knocking the object over. Proposed control of the arm and hand allowed correction of position errors on the order of several centimeters. For example, an object on a workbench that is in an uncertain positional relation with the robot, with an inexpensive optical sensor such as a Kinect, which only provides coarse image data, would be sufficient for grasping an object.

Real-time robot motion planning using rasterizing computer graphics hardware. In Proc. OY82] C. O'D unlaing and C.K. Yap. A retraction method for planning the motion of a disc. A local approach for path planning of manipulators with a high number of degrees of freedom. a path generation algorithm into oo-line programming of airbus panels. Motion planning for many degrees of freedom-random reeections at c-space obstacles. REFERENCES 31 iterative collision checker. We observed no dramatic slowdown of the planner. A challenging research goal would now be to extend the method to dynamic scenes. One rst question is: How should a roadmap computed for a given workspace be updated if a few obstacles are removed or added? Answering this question would be useful to apply our method to scenes subject to small incremental changes. Such changes occur in many manufacturing (e.g., assembly) cells; while most of the geometry of such a cell is permanent and stationary, a few objects (e.g., xtures) are added or removed between any two consecutive manufacturing operations. Similar incremental changes also occur in automatic graphic animation. A second question is: How should the learning and query phase be modiied if some obstacles are moving along known trajectories? An answer to this question might consist of applying our roadmap method in the conngurationtime space of the robot Lat91]. The roadmap would then have to be built as a directed graph, since local paths between any two nodes must monotonically progress along the time axis, with possibly additional constraints on their slope and curvature to reeect bounds on the robot's velocity and acceleration. A motion planner with performance proportional to task diiculty. 30 In KL94a, KL94b, O S94] prior versions of the method have been applied to a great variety of holonomic robots including planar and spatial articulated robots with revolute, prismatic, and/or spherical joints, xed or free base, and single or multiple kinematic chains. In Sve93, SO94] a variation of the method (essentially one with a diierent general local planner) was also run successfully on examples involving nonholonomic car-like robots. Experimental results show that our method can eeciently solve problems which are beyond the capabilities of other existing methods. For example, for planar articulated robots with many dofs, the customized implementation of Section 5 is much more consistent than the Randomized Path Planner (RPP) of BL91]. Indeed, the latter can be very fast on some diicult problems, but it …

This paper describes the simultaneous localization and mapping (SLAM) problem and the essential methods for solving the SLAM problem and summarizes key implementations and demonstrations of the method. While there are still many practical issues to overcome, especially in more complex outdoor environments, the general SLAM method is now a well understood and established part of robotics. Another part of the tutorial summarized more recent works in addressing some of the remaining issues in SLAM, including computation, feature representation, and data association

As a core robotic and vision problem, camera and range sensor calibration have been researched intensely over the last decades. However, robotic research efforts still often get heavily delayed by the requirement of setting up a calibrated system consisting of multiple cameras and range measurement units. With regard to removing this burden, we present a toolbox with web interface for fully automatic camera-to-camera and camera-to-range calibration. Our system is easy to setup and recovers intrinsic and extrinsic camera parameters as well as the transformation between cameras and range sensors within one minute. In contrast to existing calibration approaches, which often require user intervention, the proposed method is robust to varying imaging conditions, fully automatic, and easy to use since a single image and range scan proves sufficient for most calibration scenarios. Experimentally, we demonstrate that the proposed checkerboard corner detector significantly outperforms current state-of-the-art. Furthermore, the proposed camera-to-range registration method is able to discover multiple solutions in the case of ambiguities. Experiments using a variety of sensors such as grayscale and color cameras, the Kinect 3D sensor and the Velodyne HDL-64 laser scanner show the robustness of our method in different indoor and outdoor settings and under various lighting conditions.

A robot perceives space in order to navigate, map, search and investigate its surroundings. One application area where humans interact with robots is search and rescue. The robot may have unique capabilities such as seeing outside the visible spectrum and being able to navigate in tight and hazardous spaces. In such an operation, the human may also be an effective search agent. Thus, it would be beneficial if the robot’s spatial perception was conveyed to the human via a secondary peripheral modality. We have worked on developing a visual to tactile substitution.device for people who are visually impaired or blind. We propose to use this similar technique to the remote robot in order that it convey its spatial perception to the human operator or co-worker.

Dexterous surgical activity is of interest to many researchers in human motion modeling. In this paper, we describe a dataset of surgical activities and release it for public use. The dataset was captured using the da Vinci Surgical System and consists of kinematic and video from eight surgeons with different levels of skill performing five repetitions of three elementary surgical tasks on a bench-top model. The tasks, which include suturing, knot-tying and needle-passing, are standard components of most surgical skills training curricula. In addition to kinematic and video data captured from the da Vinci Surgical System, we are also releasing manual annotations of surgical gestures (atomic activity segments), surgical skill using global rating scores, a standardized cross-validation experimental setup, and a C++/Matlab toolkits for analyzing surgical gestures using hidden Markov models and using linear dynamical systems. We refer to the dataset as the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) to indicate the collaboration between Johns Hopkins University (JHU) and Intuitive Surgical Inc. (ISI), Sunnyvale, CA, on collecting these data.

In this paper, a graphical, flexible, interactive, and systematic 3D simulation helps facilitate analyzing and previewing kinematics of PA10-7C robot arm in terms of forward kinematics, inverse kinematics, and the Denavit-Hartenberg convention. Modeling and control are of critical importance when the robot arm is used for practical applications. In the paper, the D-H model of PA10-7C robot arm is given first to describe the relationship between two consecutive frames of joints. Based on this D-H model, forward kinematics is calculated efficiently. To describe the relationship between joint angular velocities in the joint space and the end effector's velocities in Cartesian space, Jacobian matrix for PA10-7C robot arm is derived. By Jacobian matrix, inverse kinematics approach is obtained in the paper. VRML-based model of PA10-7C is built in the paper for the simulation. Finally, simulation results are discussed and the paper is concluded.


Question: High-fidelity simulation method to assess the performance of robotic vision systems.

SeqSLAM: Robust Visual Navigation Across Diverse Environmental Conditions
Navigating a route under varying environmental conditions, such as daytime versus nighttime, clear versus inclement weather, and different seasons, poses significant challenges for contemporary computer vision and robotics algorithms. In this paper, we introduce SeqSLAM, a novel approach to visual navigation that excels under diverse conditions. Unlike methods that focus on identifying the single most likely location based on a current image, SeqSLAM calculates the optimal match for every local sequence of images within a navigation route. Localization is achieved by recognizing coherent sequences of these "local best matches", eliminating the requirement for the vision front-end to perform global matching. Instead, it must only select the best match within any short image sequence. This method is particularly effective in environments where traditional feature-based techniques are rendered ineffective. Through the analysis of two datasets captured using car-mounted cameras, we showcase the efficacy of SeqSLAM and compare it to FAB-MAP, a leading feature-based SLAM algorithm. The datasets encompass extreme perceptual changes, including traversals through environments during the day and night, separated by months or years and across different seasons, as well as in clear weather and heavy rain. Despite the feature-based method's failure, SeqSLAM successfully matches trajectory segments with 100% precision and recall rates of up to 60%, demonstrating its robustness and versatility in challenging conditions.

This paper introduces a four-degree-of-freedom parallel manipulator dedicated to pick-and-place. It has been developed with the goal of reaching very high speed. This paper shows that its architecture is particularly well adapted to high dynamics. Indeed, it is an evolution of Delta, H4 and 14 robots architectures: it keeps the advantages of these existing robots, while overcoming their drawbacks. In addition, an optimization method based on velocity using adept motion has been developed and applied to this high speed parallel robot. All these considerations led to experimentations that proved we can reach high accelerations (13 G) and obtain a cycle time of 0.28 s.

The research and use of UAVs (Unmanned Aerial Vehicles) have grown in attention and new ideas in the last couple of decades, mainly due to technology improvement of its construction parts, sensors and controllers. When it comes to its autonomous behavior, robust techniques make themselves necessary to assure the security of the aircraft and people around it. This work presents the use of computer vision techniques to improve the positioning of a quadrotor UAV in the landing moment, adding security and precision, overcoming intrinsic errors of other positioning sensors, such as GPS. A landmark was built to be detected by the camera with contour detection, image thresholding and other mathematical techniques in order to obtain fast, simple and robust processing and results. A GoPro camera model was used with a developed algorithm to remove the image distortion. With this set up, the vehicle can be programmed to develop any mission, with this works control algorithm taking part with the vision aided positioning in the landing moment. The quadrotor was conceived and built at the GRIn (Grupo de Robtica Inteligente) laboratory at UFJF (Juiz de Fora Federal University), and uses open source code for its control, with Mavlink communication protocol to exchange messages with the vision-based control algorithm for mission and landing. This algorithm was developed in Python language, and is in charge of image characteristics segmentation, extracting distances and altitude from the data. The control commands are sent to the quadrotor using the mentioned protocol. The system is structured with a remote laptop connected with a serial port to the quadrotor, and a router transmits messages back and forth among the GCS (Ground Control Station, on the laptop) and the high-level control algorithm (mission planning and vision based control) through TCP ports to the quadrotor itself. The camera is connected via Wi-Fi built-in network to the laptop, so all the image processing is done inside the last one. The results obtained in outdoor tests show the efficacy and fast processing of the vision methods and ensure safety in the landing moment. Furthermore, the positioning techniques can be extended to other UAV applications, such as transmission lines and utility equipment inspection and monitoring. Future works look forward to embedding the process in a companion board attached to the UAV, so the process would become automatic and the risk of communication interference and lost heavily reduced.

We use reinforcement learning (RL) to learn dexterous in-hand manipulation policies which can perform vision-based object reorientation on a physical Shadow Dexterous Hand. The training is performed in a simulated environment in which we randomize many of the physical properties of the system like friction coefficients and an object’s appearance. Our policies transfer to the physical robot despite being trained entirely in simulation. Our method does not rely on any human demonstrations, but many behaviors found in human manipulation emerge naturally, including finger gaiting, multi-finger coordination, and the controlled use of gravity. Our results were obtained using the same distributed RL system that was used to train OpenAI Five [43]. We also include a video of our results: https://youtu. be/jwSbzNHGflM.

The concept of active tracking is presented to simulate the characteristics of human vision in intelligent visual surveillance. The Pan/Tilt/Zoom (PTZ) camera is generally used for active tracking. In this paper, we present a novel and effective approach for active object tracking with a PTZ camera, and construct a near real-time system for indoor and outdoor scenes. The tracking algorithm of our system is based on the feature matching, with the PID control to drive the camera. The feature extracted from moving people is described as a region covariance matrix which combines the spatial and statistical properties of the targets (e.g. coordinates, color, and gradient). Results from indoor and outdoor experiments demonstrate the effectiveness and accuracy of our approach.

We present a new methodology for exact robot motion planning and control that unifies the purely kinematic path planning problem with the lower level feedback controller design. Complete information about the freespace and goal is encoded in the form of a special artificial potential function a navigation function that connects the kinematic planning problem with the dynamic execution problem in a provably correct fashion. The navigation function automatically gives rise to a bounded-torque feedback controller for the robot's actuators that guarantees collision-free motion and convergence to the destination from almost all initial free configurations. Since navigation functions exist for any robot and obstacle course, our methodology is completely general in principle. However, this paper is mainly concerned with certain constructive techniques for a particular class of motion planning problems. Specifically, we present a formula for navigation functions that guide a point-mass robot in a generalized sphere world. The simplest member of this family is a space obtained by puncturing a disc by an arbitrary number of smaller disjoint discs representing obstacles. The other spaces are obtained from this model by a suitable coordinate transformation that we show how to build. Our constructions exploit these coordinate transformations to adapt a navigation function on the model space to its more geometrically complicated (but topologically equivalent) instances. The formula that we present admits sphere-worlds of arbitrary dimension and is directly applicable to configuration spaces whose forbidden regions can be modeled by such generalized discs. We have implemented these navigation functions on planar scenarios, and simulation results are provided throughout the paper. Disciplines Robotics Comments Copyright 1992 IEEE. Reprinted from IEEE Transactions on Robotics and Automation, Volume 8, Issue 5, October 1992, pages 501-518. This material is posted here with permission of the IEEE. Such permission of the IEEE does not in any way imply IEEE endorsement of any of the University of Pennsylvania's products or services. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to pubs-permissions@ieee.org. By choosing to view this document, you agree to all provisions of the copyright laws protecting it. NOTE: At the time of publication, Daniel Koditschek was affiliated with the University of Michigan. Currently, he is a faculty member at the School of Engineering of the University of Pennsylvania. This journal article is available at ScholarlyCommons: http://repository.upenn.edu/ese_papers/323

The synthesis of human hand motion and grasping of arbitrary shaped objects is a very complex problem. Therefore high-level control is needed to perform these actions. In order to satisfy the kinematic and physical constraints associated with the human hand and to reduce the enormous search space associated with the problem of grasping objects, a knowledge based approach is used. A three-phased scheme is presented which incorporates the role of the hand, the object, the environment and the animator. The implementation of a hand simulation system HANDS is discussed.

This paper presents a tracking system for ego-motion estimation which fuses vision and inertial measurements using EKF and UKF (Extended and Unscented Kalman Filters), where a comparison of their performance has been done. It also considers the multi-rate nature of the sensors: inertial sensing is sampled at a fast sampling frequency while the sampling frequency of vision is lower. the proposed approach uses a constant linear acceleration model and constant angular velocity model based on quaternions, which yields a non-linear model for states and a linear model in measurement equations. Results show that a significant improvement is obtained on the estimation when fusing both measurements with respect to just vision or just inertial measurements. It is also shown that the proposed system can estimate fast-motions even when vision system fails. Moreover, a study of the influence of the noise covariance is also performed, which aims to select their appropriate values at the tuning process. The setup is an end-effector mounted camera, which allow us to pre-define basic rotational and translational motions for validating results. KEY WORDS—vision and inertial, multi-rate systems, sensor fusion, tracking

In this paper the possibilities that Matlab provides to design, implementation and monitoring programs of autonomous navigation for mobile robots, on both simulated and real platforms, through its new toolbox for robotics will be explored. Robotics System Toolbox has established itself as a solid tool of integration between Matlab and robot operating under ROS environment. It will be studied the tools available in this toolbox that allow achieving the connection between the two platforms, in addition to the generation of algorithms of location, path planning, mapping and autonomous navigation.

We present an automated framework for visual assessment of the expertise level of surgeons using the OSATS (Objective Structured Assessment of Technical Skills) criteria. Video analysis techniques for extracting motion quality via frequency coefficients are introduced. The framework is tested on videos of medical students with different expertise levels performing basic surgical tasks in a surgical training lab setting. We demonstrate that transforming the sequential time data into frequency components effectively extracts the useful information differentiating between different skill levels of the surgeons. The results show significant performance improvements using DFT and DCT coefficients over known state-of-the-art techniques.

Learning by imitation is becoming increasingly important for teaching humanoid robots new skills. The simplest form of imitation is behavior copying in which the robot is minimizing the difference between its perceived motion and that of the imitated agent. One problem that must be solved even in this simplest of all imitation tasks is calculating the learner's pose corresponding to the perceived pose of the agent it is imitating. This paper presents a general framework for solving this problem in closed form for the arms of a generalized humanoid robot of which most available humanoids are special cases. The paper also reports the evaluation of the proposed system for real and simulated robots.


Question: High-fidelity simulation method to assess the performance of robotic vision systems.

Introducing a Comprehensive Synthetic Dataset for Optical Flow Analysis

Creating accurate ground truth optical flow data for real-world scenes with natural motion is a challenging task, leading to limitations in the size, complexity, and diversity of existing optical flow datasets. This restricts the training and testing of optical flow algorithms on realistic and varied data. To address this gap, we present a novel optical flow dataset derived from the open-source 3D animated short film Sintel. This dataset offers several key advantages over popular benchmarks like the Middlebury flow evaluation, including longer sequences, greater motions, the presence of specular reflections, motion blur, defocus blur, and atmospheric effects.

One of the unique aspects of our dataset is the availability of the graphics data that generated the movie, enabling us to render scenes under various conditions of complexity. This feature allows us to systematically evaluate the performance of existing optical flow algorithms and pinpoint their limitations. Upon testing several recent optical flow algorithms, we observed that those currently ranking high on the Middlebury evaluation struggle with the increased complexity of our dataset. This finding underscores the need for further research and development in optical flow estimation.

To ensure the validity of using synthetic data, we conducted a comparative analysis of the image and flow statistics between Sintel and real films and videos. Our results confirm that the statistics are comparable, supporting the use of synthetic data for optical flow evaluation. The dataset, along with the evaluation metrics and website, is made publicly available to facilitate research and algorithm development in the field of optical flow analysis.

This comprehensive synthetic dataset not only provides a more realistic and challenging environment for testing optical flow algorithms but also opens up new avenues for research and algorithm optimization in the realm of robotic vision performance.

No feature-based vision system can work unless good features can be identi ed and tracked from frame to frame. Although tracking itself is by and large a solved problem, selecting features that can be tracked well and correspond to physical points in the world is still hard. We propose a feature selection criterion that is optimal by construction because it is based on how the tracker works, and a feature monitoring method that can detect occlusions, disocclusions, and features that do not correspond to points in the world. These methods are based on a new tracking algorithm that extends previous Newton-Raphson style search methods to work under a ne image transformations. We test performance with several simulations and experiments.

In this paper we show and evaluate the design of a novel telemanipulation system that maps proximity values, acquired inside of a gripper, to forces a user can feel through a haptic input device. The command console is complemented by input-devices that give the user an intuitive control over parameters relevant to the system. Furthermore, proximity sensors enable the autonomous alignment/centering of the gripper to objects in user-selected DoFs with the potential of aiding the user and lowering the workload. We evaluate our approach in a user study that shows that the telemanipulation system benefits from the supplementary proximity information and that the workload can indeed be reduced when the system operates with partial autonomy.

This paper introduces a four-degree-of-freedom parallel manipulator dedicated to pick-and-place. It has been developed with the goal of reaching very high speed. This paper shows that its architecture is particularly well adapted to high dynamics. Indeed, it is an evolution of Delta, H4 and 14 robots architectures: it keeps the advantages of these existing robots, while overcoming their drawbacks. In addition, an optimization method based on velocity using adept motion has been developed and applied to this high speed parallel robot. All these considerations led to experimentations that proved we can reach high accelerations (13 G) and obtain a cycle time of 0.28 s.

Pose estimation of object is one of the key problems for the automatic-grasping task of robotics. In this paper, we present a new vision-based robotic grasping system, which can not only recognize different objects but also estimate their poses by using a deep learning model, finally grasp them and move to a predefined destination. The deep learning model demonstrates strong power in learning hierarchical features which greatly facilitates the recognition mission. We apply the Max-pooling Convolutional Neural Network (MPCNN), one of the most popular deep learning models, in this system, and assign different poses of objects as different classes in MPCNN. Besides, a new object detection method is also presented to overcome the disadvantage of the deep learning model. We have built a database comprised of 5 objects with different poses and illuminations for experimental performance evaluation. The experimental results demonstrate that our system can achieve high accuracy on object recognition as well as pose estimation. And the vision-based robotic system can grasp objects successfully regardless of different poses and illuminations.

In this paper, we propose a CAD-based 6-DOF pose estimation design for random bin-picking of multiple different objects using a Kinect RGB-D sensor. 3D CAD models of objects are constructed via a virtual camera, which generates a point cloud database for object recognition and pose estimation. A voxel grid filter is suggested to reduce the number of 3D point cloud of objects for reducing computing time of pose estimation. A voting-scheme method was adopted for the 6-DOF pose estimation a swell as object recognition of different type objects in the bin. Furthermore, an outlier filter is designed to filter out bad matching poses and occluded ones, so that the robot arm always picks up the upper object in the bin to increase pick up success rate. A series of experiments on a Kuka 6-axis robot revels that the proposed system works satisfactorily to pick up all random objects in the bin. The average recognition rate of three different type objects is 93.9% and the pickup success rate is 89.7%.

Human visual object recognition is typically rapid and seemingly effortless, as well as largely independent of viewpoint and object orientation. Until very recently, animate visual systems were the only ones capable of this remarkable computational feat. This has changed with the rise of a class of computer vision algorithms called deep neural networks (DNNs) that achieve human-level classification performance on object recognition tasks. Furthermore, a growing number of studies report similarities in the way DNNs and the human visual system process objects, suggesting that current DNNs may be good models of human visual object recognition. Yet there clearly exist important architectural and processing differences between stateof-the-art DNNs and the primate visual system. The potential behavioural consequences of these differences are not well understood. We aim to address this issue by comparing human and DNN generalisation abilities towards image degradations. We find the human visual system to be more robust to image manipulations like contrast reduction, additive noise or novel eidolon-distortions. In addition, we find progressively diverging classification error-patterns between man and DNNs when the signal gets weaker, indicating that there may still be marked differences in the way humans and current DNNs perform visual object recognition. We envision that our findings as well as our carefully measured and freely available behavioural datasets1 provide a new useful benchmark for the computer vision community to improve the robustness of DNNs and a motivation for neuroscientists to search for mechanisms in the brain that could facilitate this robustness. Data and materials available at https://github.com/rgeirhos/object-recognition 1 ar X iv :1 70 6. 06 96 9v 1 [ cs .C V ] 2 1 Ju n 20 17

In this letter, we address the autonomous flight of a small quadrotor, enabling tracking of a moving object. The 15-cm diameter, 250-g robot relies only on onboard sensors (a single camera and an inertial measurement unit) and computers, and can detect, localize, and track moving objects. Our key contributions include the relative pose estimate of a spherical target as well as the planning algorithm, which considers the dynamics of the underactuated robot, the actuator limitations, and the field of view constraints. We show simulation and experimental results to demonstrate feasibility and performance, as well as robustness to abrupt variations in target motion.

This paper presents a robust and flexible framework for augmented reality which does not require instrumenting either the environment or the workpiece. A model-based visual tracking system is combined with with rate gyroscopes to produce a system which can track the rapid camera rotations generated by a head-mounted camera, even if images are substantially degraded by motion blur. This tracking yields estimates of head position at video field rate (50Hz) which are used to align computer-generated graphics on an optical see-through display. Nonlinear optimisation is used for the calibration of display parameters which include a model of optical distortion. Rendered visuals are pre-distorted to correct the optical distortion of the display.

In this paper we propose and evaluate a control system to (1) learn and (2) adapt robot motion for continuous non-rigid contact with the environment. We present the approach in the context of wiping surfaces with robots. Our approach is based on learning by demonstration. First an initial periodicmotion, covering the essence of the wiping task, is transferred from a human to a robot. The system extracts and learns one period of motion. Once the user/demonstrator is content with the motion, the robot seeks and establishes contactwith a given surface,maintaining a predefined force of contact through force feedback. The shape of the surface is encoded for the complete period ofmotion, but the robot can adapt to a different surface, perturbations or obstacles. The novelty stems from the fact that the feedforward component is learned and encoded in a dynamic movement primitive. By using the feedforward component, the feedback component is greatly reduced if not completely canceled. Finally, if the user is not satisfied with the periodic pattern, he/she can change parts of motion through predefined gestures or through physical contact in a manner of a tutor or a coach. The complete system thus allows not only a transfer ofmotion, but a transfer ofmotionwithmatching correspondences, i.e. wiping motion is constrained to maintain physical contact with the surface to be wiped. The interface for both learning and adaptation is simple and intuitive and allows for fast and reliable knowledge transfer to the robot. Simulated and real world results in the application domain of wiping a surface are presented on three different robotic platforms. Results of the three robotic platforms, namely a 7 degree-of-freedom Kuka LWR-4 robot, the ARMAR-IIIa humanoid platform and the Sarcos CB-i humanoid robot, depict different methods of adaptation to the environment and coaching. © 2015 Elsevier B.V. All rights reserved. ⇤ Corresponding author. E-mail addresses: andrej.gams@ijs.si (A. Gams), tadej.petric@ijs.si (T. Petri£), martin.do@kit.edu (M. Do), bojan.nemec@ijs.si (B. Nemec), xmorimo@atr.jp (J. Morimoto), asfour@kit.edu (T. Asfour), ales.ude@ijs.si (A. Ude).

A co-simulation on virtual prototype of a type of spot welding robot has been accomplished by combining MATLAB/Simulink with RecurDyn. The control algorithm is executed under MATLAB to generate command forces, which are then exported to RecurDyn environment and applied to actuators of the virtual prototype at each cycle of time. Dynamic control on joint angles was accomplished through the co-simulation. Torques of each joint and the velocity of the clamp end were also acquired by co-simulation, which can give useful information for optimizing robot design.


Question: High-precision simulation to assess the performance of robotic vision systems.

The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) serves as a benchmark for evaluating the performance of robotic vision systems. This annual competition provides a high-fidelity simulation environment designed to test the accuracy and efficiency of computer vision algorithms in recognizing and categorizing a vast array of objects. By participating in ILSVRC, researchers can assess their robotic vision systems against a standardized dataset, ensuring that performance metrics are comparable across different models. The challenge includes various tasks such as object detection, image classification, and semantic segmentation, all of which are critical for robotic systems aiming to navigate and interact with complex real-world environments. Through the rigorous testing provided by ILSVRC, robotic vision researchers can identify areas for improvement and refine their algorithms to enhance the overall functionality and reliability of robotic systems in diverse applications.

Forward and inverse kinematic studies of industrial robots (IR) have been developed and presented in a large number of papers. However, eve n g neral mathematic formalization is usually almost correct, (basically following up general Har tenberg Denavit (H-D) conventions and associated homogenous transformation matrix), only few papers presents kinematic models ready to be directly implemented on a real scale industrial robot or as well able to evaluate kinematics behavior of a real scale IR specific model. That is usually due to som e inconsistencies in modeling, the most frequently of these referring on: the incomplete formalization of the full set of constructive and functional parame ters (that mandatory need to be considered in case of a specific real IR's model), avoidance of considering IR's specific design features, (as joint dimensions a d links dimensions are) leading to wrongly locat ing the reference frames used for expressing homogenous coordinate transformations, as well as missing of the validation procedures able to check the correct itude of the mathematical models, previously to its implementing in a real scale IR's controller. That is why present paper shows first a completely new approach for IR's forward an inverse kinematics, in terms of IR's analytical modeling by taking into account the full set of IR's constructive and funct ional parameters of two different IR's models. Then , for both direct and inverse mathematical models complet e symbolic formalization and full set of solutions for forward and inverse kinematics are presented for bo th IR types. In order to study mathematical models applicability on the real scale IR, two specific IR models were studied: an ABB serial-link open chain kinematics IR and a Fanuc serial-link closed chain ki ematics IR. Numerical results were verified by cross validation using both analytically calculatio ns results and by mean of a constrained 3D CAD mode l used to geometrically verify the results. The param etric form of the model elaborated in PTC Mathcad 1 4 allows a quick reconfiguration for other robot's mo dels having similar configurations. Results can be also used for solving dynamics, path planning and contro l p blems in case of real scale IR.

As part of a continuous research and development of underwater robotics technology at ITB, a visionbased distance measurement system for an Unmanned Underwater vehicle (UUV) has been designed. The proposed system can be used to predict horizontal distance between underwater vehicle and wall in front of vehicle. At the same time, it can be used to predict vertical distance between vehicle and the surface below it as well. A camera and a single laser pointer are used to obtain data needed by our algorithm. The vision-based navigation consists of two main processes which are the detection of a laser spot using image processing and the calculation of the distance based on laser spot position on the image.

RGB-D cameras are novel sensing systems that capture RGB images along with per-pixel depth information. RGB-D cameras rely on either structured light patterns combined with stereo sensing [6,10] or time-of-flight laser sensing [1] to generate depth estimates that can be associated with RGB pixels. Very soon, small, high-quality RGB-D cameras developed for computer gaming and home entertainment applications will become available at cost below $100. In this paper we investigate how such cameras can be used in the context of robotics, specifically for building dense 3D maps of indoor environments. Such maps have applications in robot navigation, manipulation, semantic mapping, and telepresence. The robotics and computer vision communities have developed a variety of techniques for 3D mapping based on laser range scans [8, 11], stereo cameras [7], monocular cameras [3], and unsorted collections of photos [4]. While RGB-D cameras provide the opportunity to build 3D maps of unprecedented richness, they have drawbacks that make their application to 3D mapping difficult: They provide depth only up to a limited distance (typically less than 5m), depth values are much noisier than those provided by laser scanners, and their field of view (∼ 60◦) is far more constrained than that of specialized cameras or laser scanners typically used for 3D mapping (∼ 180◦). In our work, we use a camera developed by PrimeSense [10]. The key insights of this investigation are: first, that existing frame matching techniques are not sufficient to provide robust visual odometry with these cameras; second, that a tight integration of depth and color information can yield robust frame matching and loop closure detection; third, that building on best practice techniques in SLAM and computer graphics makes it possible to build and visualize accurate and extremely rich 3D maps with such cameras; and, fourth, that it will be feasible to build complete robot navigation and interaction systems solely based on cheap depth cameras.

The limited nature of robot sensors make many important robo tics problems partially observable. These problems may require the s yst m to perform complex information-gathering operations. One approach to so lving these problems is to create plans inbelief-space , the space of probability distributions over the underlying state of the system. The belief-space plan encodes a st rategy for performing a task while gaining information as necessary. Most approach es to belief-space planning rely upon representing belief state in a particular way (typically as a Gaussian). Unfortunately, this can lead to large errors between the assumed density representation of belief state and the true belief state. Th is paper proposes a new sample-based approach to belief-space planning that has fix ed computational complexity while allowing arbitrary implementations of Bayes filtering to be used to track belief state. The approach is illustrated in the conte xt of a simple example and compared to a prior approach. Then, we propose an applicatio n of the technique to an instance of the grasp synthesis problem where a robot mu st simultaneously localize and grasp an object given initially uncertain obje ct parameters by planning information-gathering behavior. Experimental results ar e presented that demonstrate the approach to be capable of actively localizing and graspi ng boxes that are presented to the robot in uncertain and hard-to-localize config urations.

During last decade the scientific research on Unmanned Aerial Vehicless (UAVs) increased spectacularly and led to the design of multiple types of aerial platforms. The major challenge today is the development of autonomously operating aerial agents capable of completing missions independently of human interaction. To this extent, visual sensing techniques have been integrated in the control pipeline of the UAVs in order to enhance their navigation and guidance skills. The aim of this article is to present a comprehensive literature review on vision based applications for UAVs focusing mainly on current developments and trends. These applications are sorted in different categories according to the research topics among various research groups. More specifically vision based position-attitude control, pose estimation and mapping, obstacle detection as well as target tracking are the identified components towards autonomous agents. Aerial platforms could reach greater level of autonomy by integrating all these technologies onboard. Additionally, throughout this article the concept of fusion multiple sensors is highlighted, while an overview on the challenges C. Kanellakis ( ) · G. Nikolakopoulos Luleå University of Technology, Luleå, Sweden e-mail: christoforos.kanellakis@ltu.se G. Nikolakopoulos e-mail: geonik@ltu.se addressed and future trends in autonomous agent development will be also provided.

Multiple cameras are increasingly prevalent on robotic and human-driven vehicles. These cameras come in a variety of wide-angle, fish-eye, and catadioptric models. Furthermore, wheel odometry is generally available on the vehicles on which the cameras are mounted. For robustness, vision applications tend to use wheel odometry as a strong prior for camera pose estimation, and in these cases, an accurate extrinsic calibration is required in addition to an accurate intrinsic calibration. To date, there is no known work on automatic intrinsic calibration of generic cameras, and more importantly, automatic extrinsic calibration of a rig with multiple generic cameras and odometry. We propose an easy-to-use automated pipeline that handles both intrinsic and extrinsic calibration; we do not assume that there are overlapping fields of view. At the begining, we run an intrinsic calibration for each generic camera. The intrinsic calibration is automatic and requires a chessboard. Subsequently, we run an extrinsic calibration which finds all camera-odometry transforms. The extrinsic calibration is unsupervised, uses natural features, and only requires the vehicle to be driven around for a short time. The intrinsic parameters are optimized in a final bundle adjustment step in the extrinsic calibration. In addition, the pipeline produces a globally-consistent sparse map of landmarks which can be used for visual localization. The pipeline is publicly available as a standalone C++ package.

This paper describes an improvement of Additive Manufacturing (AM) to process products with complex inner and outer geometries such as turbine blades by using a combination of AM, optical measuring techniques and robotics. One problem of AM is the rough surface caused by the layer-by-layer 3D printing with granular material, so the geometries need to be post-processed with milling techniques. To overcome this problem, we implement an inline quality control management to post-process several manufactured layers by a integrated imaging and robotic techniques, especially while the geometries are still accessible. The produced surfaces are measured by an inline stereo vision system and are compared with the real 3D CAD model data. Detected differences are translated to movements of a robot arm which then follows the contours to smoothen the surfaces with a milling tool. We show our current state on the 3D imaging, on developing the driver for the deployed robot arm and on a 3D simulation using the Robot Operating System (ROS).

A real-time solution for estimating and tracking the 3D pose of a rigid object is presented for image-based visual servo with natural landmarks. The many state-of-the-art technologies that are available for recognizing the 3D pose of an object in a natural setting are not suitable for real-time servo due to their time lags. This paper demonstrates that a real-time solution of 3D pose estimation become feasible by combining a fast tracker such as KLT [7] [8] with a method of determining the 3D coordinates of tracking points on an object at the time of SIFT based tracking point initiation, assuming that a 3D geometric model with SIFT description of an object is known a-priori. Keeping track of tracking points with KLT, removing the tracking point outliers automatically, and reinitiating the tracking points using SIFT once deteriorated, the 3D pose of an object can be estimated and tracked in real-time. This method can be applied to both mono and stereo camera based 3D pose estimation and tracking. The former guarantees higher frame rates with about 1 ms of local pose estimation, while the latter assures of more precise pose results but with about 16 ms of local pose estimation. The experimental investigations have shown the effectiveness of the proposed approach with real-time performance.

This paper describes the simultaneous localization and mapping (SLAM) problem and the essential methods for solving the SLAM problem and summarizes key implementations and demonstrations of the method. While there are still many practical issues to overcome, especially in more complex outdoor environments, the general SLAM method is now a well understood and established part of robotics. Another part of the tutorial summarized more recent works in addressing some of the remaining issues in SLAM, including computation, feature representation, and data association

We are interested in how the concept of affordances can affect our view to autonomous robot control, and how the results obtained from autonomous robotics can be reflected back upon the discussion and studies on the concept of affordances. In this paper, we studied how a mobile robot, equipped with a 3D laser scanner, can learn to perceive the traversability affordance and use it to wander in a room tilled with spheres, cylinders and boxes. The results showed that after learning, the robot can wander around avoiding contact with non-traversable objects (i.e. boxes, upright cylinders, or lying cylinders in certain orientation), but moving over traversable objects (such as spheres, and lying cylinders in a rollable orientation with respect to the robot) rolling them out of its way. We have shown that for each action approximately 1% of the perceptual features were relevant to determine whether it is afforded or not and that these relevant features are positioned in certain regions of the range image. The experiments are conducted both using a physics-based simulator and on a real robot.


Question: Challenges in managing data within cloud computing infrastructures.

This paper explores a series of caching protocols for distributed networks, specifically designed to mitigate or prevent the formation of hot spots, which can significantly degrade performance. These protocols are particularly relevant for vast networks like the Internet, where the impact of hot spots can be particularly severe due to the sheer scale and complexity of the network. Unlike traditional approaches that often rely on comprehensive network state information, our protocols are designed to operate effectively in environments where such information is not readily available or too costly to maintain. The protocols are straightforward to implement, leveraging established network protocols such as TCP/IP, and are characterized by minimal overhead. They operate based on local control, optimizing resource utilization and scaling gracefully as the network expands. Central to these protocols is the concept of consistent hashing, a hashing technique that ensures minimal disruption when the function's range undergoes changes. By developing robust consistent hash functions, we've created caching protocols that do not necessitate users to possess an up-to-date or coherent view of the entire network. We anticipate that consistent hash functions will find applications beyond caching, potentially benefiting areas like distributed name servers and quorum systems, enhancing their efficiency and reliability.

While cloud computing has brought paradigm shifts to computing services, researchers and developers have also found some problems inherent to its nature such as bandwidth bottleneck, communication overhead, and location blindness. The concept of fog/edge computing is therefore coined to extend the services from the core in cloud data centers to the edge of the network. In recent years, many systems are proposed to better serve ubiquitous smart devices closer to the user. This article provides a complete and up-to-date review of edge-oriented computing systems by encapsulating relevant proposals on their architecture features, management approaches, and design objectives.

Cloud computing is expected to be a promising solution for scientific computing. In this paper, we propose a flexible container-based computing platform to run scientific workflows on cloud. We integrate Galaxy, a popular biology workflow system, with four famous container cluster systems. Preliminary evaluation shows that container cluster systems introduce negligible performance overhead for data intensive scientific workflows, meanwhile, they are able to solve tool installation problem, guarantee reproducibility and improve resource utilization. Moreover, we implement four ways of using Docker, the most popular container tool, for our platform. Docker in Docker and Sibling Docker, which run everything within containers, both help scientists easily deploy our platform on any clouds in a few minutes.

Erasure coded storage schemes offer a promising future for cloud storage. Highlights of erasure coded storage systems are that these offer the same level of fault tolerance as that of replication, at lower storage footprints. In the big data era, cloud storage systems based on data replication are of dubious usability due to 200% storage overhead in data replication systems. This has prompted storage service providers to use erasure coded storage as an alternative to replication. Refinements are required in various aspects of erasure coded storage systems to make it a real contender against data replication based storage systems. Streamlining huge bandwidth requirements during the recovery of failed nodes, inefficient update operations, effect of topology in recovery and consistency requirements of erasure coded storage systems, are some areas which need attention. This paper presents an in-depth study on the challenges faced, and research pursued in some of these areas. The survey shows that more research is required to improve erasure coded storage system from being bandwidth crunchers to efficient storage systems. Another challenge that has emerged from the study is the requirement of elaborate research for upgrading the erasure coded storage systems from being mere archival storage systems by providing better update methods. Provision of multiple level consistency in erasure coded storage is yet another research opportunity identified in this work. A brief introduction to open source libraries available for erasure coded storage is also presented in the paper.

Cloud computing is becoming increasingly important for provision of services and storage of data in the Internet. However there are several significant challenges in securing cloud infrastructures from different types of attacks. The focus of this paper is on the security services that a cloud provider can offer as part of its infrastructure to its customers (tenants) to counteract these attacks. Our main contribution is a security architecture that provides a flexible security as a service model that a cloud provider can offer to its tenants and customers of its tenants. Our security as a service model while offering a baseline security to the provider to protect its own cloud infrastructure also provides flexibility to tenants to have additional security functionalities that suit their security requirements. The paper describes the design of the security architecture and discusses how different types of attacks are counteracted by the proposed architecture. We have implemented the security architecture and the paper discusses analysis and performance evaluation results.

Cloud computing is pervasive, but cloud service outages still take place. One might say that the computing forecast for tomorrow is “cloudy with a chance of failure.” One main reason why major outages still occur is that there are many unknown large-scale failure scenarios in which recovery might fail. We propose a new type of cloud service, Failure as a Service (FaaS), which allows cloud services to routinely perform large-scale failure drills in real deployments.

Internet-based applications and their resulting multitier distributed architectures have changed the focus of design for large-scale Internet computing. Internet server applications execute in a horizontally scalable topology across hundreds or thousands of commodity servers in Internet data centers. Increasing scale and power density significantly impacts the data center's thermal properties. Effective thermal management is essential to the robustness of mission-critical applications. Internet service architectures can address multisystem resource management as well as thermal management within data centers.

The performance unpredictability associated with migrating applications into cloud computing infrastructures has impeded this migration. For example, CPU contention between co-located applications has been shown to exhibit counter-intuitive behavior. In this paper, we investigate IO performance interference through the experimental study of consolidated n-tier applications leveraging the same disk. Surprisingly, we found that specifying a specific disk allocation, e.g., limiting the number of Input/Output Operations Per Second (IOPs) per VM, results in significantly lower performance than fully sharing disk across VMs. Moreover, we observe severe performance interference among VMs can not be totally eliminated even with a sharing strategy (e.g., response times for constant workloads still increase over 1,100%). By using a micro-benchmark (Filebench) and an n-tier application benchmark systems (RUBBoS), we demonstrate the existence of disk contention in consolidated environments, and how performance loss occurs when co-located database systems in order to maintain database consistency flush their logs from memory to disk. Potential solutions to these isolation issues are (1) to increase the log buffer size to amortize the disk IO cost (2) to decrease the number of write threads to alleviate disk contention. We validate these methods experimentally and find a 64% and 57% reduction in response time (or more generally, a reduction in performance interference) for constant and increasing workloads respectively.

Cloud federations have been formed to share the services, prompt and support cooperation, as well as interoperability among their already deployed cloud systems. However, the creation and management of the cloud federations lead to various security issues such as confidentially, integrity and availability of the data. Despite the access control policies in place, an attacker may compromise the communication channel processing the access requests and the decisions between the access control systems and the members(users) and vice-versa. In cloud federation, the rating of the services offered by different cloud members becomes integral to providing the users with the best quality services. Hence, we propose an innovative blockchainbased framework that on the one hand permits secure communication between the members of the federation and the access control systems, while on the other hand provides the quality services to the members by considering the service constraints imposed by them.

A cloud workflow system is a type of platform service which facilitates the automation of distributed applications based on the novel cloud infrastructure. Compared with grid environment, data transfer is a big overhead for cloud workflows due to the market-oriented business model in the cloud environments. In this paper, a Revised Discrete Particle Swarm Optimization (RDPSO) is proposed to schedule applications among cloud services that takes both data transmission cost and computation cost into account. Experiment is conducted with a set of workflow applications by varying their data communication costs and computation costs according to a cloud price model. Comparison is made on make span and cost optimization ratio and the cost savings with RDPSO, the standard PSO and BRS (Best Resource Selection) algorithm. Experimental results show that the proposed RDPSO algorithm can achieve much more cost savings and better performance on make span and cost optimization.

Cloud Computing has been envisioned as the next-generation architecture of IT Enterprise. It moves the application software and databases to the centralized large data centers, where the management of the data and services may not be fully trustworthy. This unique paradigm brings about many new security challenges, which have not been well understood. This work studies the problem of ensuring the integrity of data storage in Cloud Computing. In particular, we consider the task of allowing a third party auditor (TPA), on behalf of the cloud client, to verify the integrity of the dynamic data stored in the cloud. The introduction of TPA eliminates the involvement of client through the auditing of whether his data stored in the cloud is indeed intact, which can be important in achieving economies of scale for Cloud Computing. The support for data dynamics via the most general forms of data operation, such as block modification, insertion and deletion, is also a significant step toward practicality, since services in Cloud Computing are not limited to archive or backup data only. While prior works on ensuring remote data integrity often lacks the support of either public verifiability or dynamic data operations, this paper achieves both. We first identify the difficulties and potential security problems of direct extensions with fully dynamic data updates from prior works and then show how to construct an elegant verification scheme for seamless integration of these two salient features in our protocol design. In particular, to achieve efficient data dynamics, we improve the Proof of Retrievability model [1] by manipulating the classic Merkle Hash Tree (MHT) construction for block tag authentication. Extensive security and performance analysis show that the proposed scheme is highly efficient and provably secure.