Question: Utilizing an active stereo head for eye gaze tracking.

This paper introduces SUITOR, a sophisticated attentive information system designed to closely monitor user activities and infer their information needs. Attentive systems, in general, are characterized by their ability to observe user behavior, analyze interests, and predict actions or desires. Given the vast spectrum of attentive systems, from human attendants to sophisticated web platforms that profile user preferences, our focus has narrowed to attentive information systems specifically. These systems monitor interactions with information resources, model the user's informational state, and propose relevant information that could potentially assist users. SUITOR, our implemented solution, employs a multi-channel approach to track computer users, incorporating gaze tracking, web browsing patterns, and application focus to gauge user interests and fulfill information requirements. By continuously observing user actions and refining user profiles, SUITOR is capable of identifying and presenting pertinent information that is both timely and delivered in a manner that minimally interrupts the user's ongoing tasks. This system demonstrates the potential for technology to proactively support users in their information-seeking endeavors without becoming an intrusive presence.

Games are undoubtedly the most successful computer application yet it is not easy to attribute their success to any one particular feature. Nonetheless, when talking about games, gamers and reviewers frequently refer to the immersive experience of the game as an important aspect to be attained. However, it is not clear what immersion is and even if it is a comparable experience between different players and different games. This paper aims to develop more quantifiable and therefore objective measures of immersion. We describe a study into switching from an immersive gaming experience to another task, not in the game world. Though the degree of immersion does seem to have an impact on the ability to perform the task, the experimental approach is complex and possibly quite fragile. We therefore set out hypotheses for a similar experiment with the aim of exploring whether eyetracking and body motion provide better indicators of the degree of immersion.

Substantial literature exists regarding how eye-tracking systems can be used to measure cognitive load and how these measurements can be useful for adapting training in real time. Much of the published literature discusses the applications and limitations of the research and typically provides recommendations for improvement. This review assesses these articles collectively to provide a clearer solution for implementing eye-tracking systems into a training environment. Although limitations exist for using eye tracking as an interface tool, gaze and pupillary response have been successfully used to reflect changes in cognitive load and are starting to be incorporated into adaptive training systems, although issues are still present with differentiating pupil responses from simultaneous psychological effects. Additionally, current eye-tracking systems and data analysis software have proven accurate enough for general use, but issues including system cost and software integration prevent this technology from becoming commercialized for use in common instructional settings.

Clumsy intermediary devices constrain our interaction with computers and their applications. Glove-based input devices let us apply our manual dexterity to the task. We provide a basis for understanding the field by describing key hand-tracking technologies and applications using glove-based input. The bulk of development in glove-based input has taken place very recently, and not all of it is easily accessible in the literature. We present a cross-section of the field to date. Hand-tracking devices may use the following technologies: position tracking, optical tracking, marker systems, silhouette analysis, magnetic tracking or acoustic tracking. Actual glove technologies on the market include: Sayre glove, MIT LED glove, Digital Data Entry Glove, DataGlove, Dexterous HandMaster, Power Glove, CyberGlove and Space Glove. Various applications of glove technologies include projects into the pursuit of natural interfaces, systems for understanding signed languages, teleoperation and robotic control, computer-based puppetry, and musical performance.<<ETX>>

In immersive virtual environments (IVEs), users can control their virtual viewpoint by moving their tracked head and walking through the real world. Usually, movements in the real world are mapped one-to-one to virtual camera motions. With redirection techniques, the virtual camera is manipulated by applying gains to user motion so that the virtual world moves differently than the real world. Thus, users can walk through large-scale IVEs while physically remaining in a reasonably small workspace. In psychophysical experiments with a two-alternative forced-choice task, we have quantified how much humans can unknowingly be redirected on physical paths that are different from the visually perceived paths. We tested 12 subjects in three different experiments: (E1) discrimination between virtual and physical rotations, (E2) discrimination between virtual and physical straightforward movements, and (E3) discrimination of path curvature. In experiment E1, subjects performed rotations with different gains, and then had to choose whether the visually perceived rotation was smaller or greater than the physical rotation. In experiment E2, subjects chose whether the physical walk was shorter or longer than the visually perceived scaled travel distance. In experiment E3, subjects estimate the path curvature when walking a curved path in the real world while the visual display shows a straight path in the virtual world. Our results show that users can be turned physically about 49 percent more or 20 percent less than the perceived virtual rotation, distances can be downscaled by 14 percent and upscaled by 26 percent, and users can be redirected on a circular arc with a radius greater than 22 m while they believe that they are walking straight.

The Air Force Research Laboratory has implemented and evaluated two brain-computer interfaces (BCI's) that translate the steady-state visual evoked response into a control signal for operating a physical device or computer program. In one approach, operators self-regulate the brain response; the other approach uses multiple evoked responses.

The Iterative Closest Point (ICP) algorithm is widely used to register two roughly aligned surfaces. Its most expensive step is the search for the closest point. Many efficient variants have been proposed to speed up this step, however they do not guarantee that the closest point on a triangulated surface is found. Instead they might result in an approximation of the closest point. In this paper we present a method for the closest point search that is fast and exact. The method was implemented and used to evaluate the accuracy of head motion estimation using dense 3D data obtained from a stereo camera system. The results show that the accuracy of the estimated head motion is better than 1 mm for translational movements and better than 1 degree for rotations. Key-Words: Iterative Closest Point, 3D Registration, Stereo, Head Motion Estimation

Active vision systems are usually limited to either partial or static scene reconstructions. In this paper, we propose to acquire the entire 3D shape of a dynamic scene. This is performed using a multiple projectors and cameras system, that allows to recover the entire shape of the object within a single scan at each frame. Like previous approaches, a static and simple pattern is used to avoid interferences of multiple patterns projected on the same object. In this paper, we extend the technique to capture a dense entire shape of a moving object with accuracy and high video frame rate. To achieve this, we mainly propose two additional steps; one is checking the consistency between the multiple cameras and projectors, and the other is an algorithm for light sectioning based on a plane parameter optimization. In addition, we also propose efficient noise reduction and mesh generation algorithm which are necessary for practical applications. In the experiments, we show that we can successfully reconstruct dense entire shapes of moving objects. Results are illustrated on real data from a system composed of six projectors and six cameras that was actually built.

Volumetric 3D displays are frequently purported to lack the ability to reconstruct scenes with viewer-position-dependent effects such as occlusion. To counter these claims, a swept-screen 198-view horizontal-parallax-only 3D display is reported here that is capable of viewer-position-dependent effects. A digital projector illuminates a rotating vertical diffuser with a series of multiperspective 768 x 768 pixel renderings of a 3D scene. Evidence of near-far object occlusion is reported. The aggregate virtual screen surface for a stationary observer is described, as are guidelines to construct a full-parallax system and the theoretical ability of the present system to project imagery outside of the volume swept by the screen.

We constructed an acoustic, gesture-based recognition system called Multiwave, which leverages the Doppler Effect to translate multidimensional movements into user interface commands. Our system only requires the use of two speakers and a microphone to be operational. Since these components are already built in to most end user systems, our design makes gesture-based input more accessible to a wider range of end users. By generating a known high frequency tone from multiple speakers and detecting movement using changes in the sound waves, we are able to calculate a Euclidean representation of hand velocity that is then used for more natural gesture recognition and thus, more meaningful interaction mappings. We present the results of a user study of Multiwave to evaluate recognition rates for different gestures and report accuracy rates comparable to or better than the current state of the art. We also report subjective user feedback and some lessons learned from our system that provide additional insight for future applications of multidimensional gesture recognition.

Here we present a new method for automatic and objective monitoring of ingestive behaviors in comparison with other facial activities through load cells embedded in a pair of glasses, named GlasSense. Typically, activated by subtle contraction and relaxation of a temporalis muscle, there is a cyclic movement of the temporomandibular joint during mastication. However, such muscular signals are, in general, too weak to sense without amplification or an electromyographic analysis. To detect these oscillatory facial signals without any use of obtrusive device, we incorporated a load cell into each hinge which was used as a lever mechanism on both sides of the glasses. Thus, the signal measured at the load cells can detect the force amplified mechanically by the hinge. We demonstrated a proof-of-concept validation of the amplification by differentiating the force signals between the hinge and the temple. A pattern recognition was applied to extract statistical features and classify featured behavioral patterns, such as natural head movement, chewing, talking, and wink. The overall results showed that the average F1 score of the classification was about 94.0% and the accuracy above 89%. We believe this approach will be helpful for designing a non-intrusive and un-obtrusive eyewear-based ingestive behavior monitoring system.


Question: Eye gaze tracking employing an active stereo head apparatus.

In this paper, we introduce a high-speed real-time face tracking system, capable of operating at a rate of 30 frames per second on an HP-9000 workstation, equipped with a framegrabber and a Canon VC-CI camera. Our system excels at tracking a person's face, even as they move freely within a room, engaging in various activities like walking, jumping, sitting, or standing. The system's robust performance is attributed to the integration of three key models. Firstly, we employ a probabilistic model to accurately characterize the skin-color distributions of human faces. This model ensures reliable tracking across different poses and viewing angles, and it is dynamically adaptable to a wide range of individuals and lighting conditions in real-time. Secondly, a motion model is utilized to estimate and predict image motion, facilitating the efficient determination of search windows. Lastly, a camera model is integrated to account for and compensate camera motion, enhancing tracking accuracy. The system's capabilities make it suitable for a variety of applications, including teleconferencing and human-computer interaction (HCI) scenarios, such as lip-reading and gaze tracking. The foundational principles of our system can be extended to address other tracking challenges, such as tracking the human hand, demonstrating its versatility and potential for broader application.

In cricket, a batsman watches a fast bowler's ball come toward him at a high and unpredictable speed, bouncing off ground of uncertain hardness. Although he views the trajectory for little more than half a second, he can accurately judge where and when the ball will reach him. Batsmen's eye movements monitor the moment when the ball is released, make a predictive saccade to the place where they expect it to hit the ground, wait for it to bounce, and follow its trajectory for 100–200 ms after the bounce. We show how information provided by these fixations may allow precise prediction of the ball's timing and placement. Comparing players with different skill levels, we found that a short latency for the first saccade distinguished good from poor batsmen, and that a cricket player's eye movement strategy contributes to his skill in the game.

Stereo visual odometry and dense scene reconstruction depend critically on accurate calibration of the extrinsic (relative) stereo camera poses. We present an algorithm for continuous, online stereo extrinsic re-calibration operating only on sparse stereo correspondences on a per-frame basis. We obtain the 5 degree of freedom extrinsic pose for each frame, with a fixed baseline, making it possible to model time-dependent variations. The initial extrinsic estimates are found by minimizing epipolar errors, and are refined via a Kalman Filter (KF). Observation covariances are derived from the Cramer-Rao lower bound of the solution uncertainty. The algorithm operates at frame rate with unoptimized Matlab code with over 1000 correspondences per frame. We validate its performance using a variety of real stereo datasets and simulations.

This paper reviews the recent findings on working memory, attention and eye movements. We discuss the research that shows that many phenomena related to visual attention taking place when selecting relevant information from the environment are similar to processes needed to keep information active in working memory. We discuss new data that show that when retrieving information from working memory, people may allocate visual spatial attention to the empty location in space that used to contain the information that has to be retrieved. Moreover, we show that maintaining a location in working memory not only may involve attention rehearsal, but might also recruit the oculomotor system. Recent findings seem to suggest that remembering a location may involve attention-based rehearsal in higher brain areas, while at the same time there is inhibition of specific motor programs at lower brain areas. We discuss the possibility that working memory functions do not reside at a special area in the brain, but emerge from the selective recruitment of brain areas that are typically involved in spatial attention and motor control.

In this article a new method for the calibration of a vision system which consists of two (or more) cameras is presented. The proposed method, which uses simple properties of vanishing points, is divided into two steps. In the first step, the intrinsic parameters of each camera, that is, the focal length and the location of the intersection between the optical axis and the image plane, are recovered from a single image of a cube. In the second step, the extrinsic parameters of a pair of cameras, that is, the rotation matrix and the translation vector which describe the rigid motion between the coordinate systems fixed in the two cameras are estimated from an image stereo pair of a suitable planar pattern. Firstly, by matching the corresponding vanishing points in the two images the rotation matrix can be computed, then the translation vector is estimated by means of a simple triangulation. The robustness of the method against noise is discussed, and the conditions for optimal estimation of the rotation matrix are derived. Extensive experimentation shows that the precision that can be achieved with the proposed method is sufficient to efficiently perform machine vision tasks that require camera calibration, like depth from stereo and motion from image sequence.

Here we present a new method for automatic and objective monitoring of ingestive behaviors in comparison with other facial activities through load cells embedded in a pair of glasses, named GlasSense. Typically, activated by subtle contraction and relaxation of a temporalis muscle, there is a cyclic movement of the temporomandibular joint during mastication. However, such muscular signals are, in general, too weak to sense without amplification or an electromyographic analysis. To detect these oscillatory facial signals without any use of obtrusive device, we incorporated a load cell into each hinge which was used as a lever mechanism on both sides of the glasses. Thus, the signal measured at the load cells can detect the force amplified mechanically by the hinge. We demonstrated a proof-of-concept validation of the amplification by differentiating the force signals between the hinge and the temple. A pattern recognition was applied to extract statistical features and classify featured behavioral patterns, such as natural head movement, chewing, talking, and wink. The overall results showed that the average F1 score of the classification was about 94.0% and the accuracy above 89%. We believe this approach will be helpful for designing a non-intrusive and un-obtrusive eyewear-based ingestive behavior monitoring system.

In this paper, we present work in progress towards the generation of a ground truth data set for High Dynamic Range (HDR) stereo matching. The development and evaluation of novel stereo matching algorithms that are tailored to the characteristics of HDR images would greatly benefit from the availability of such reference data. We describe our laboratory setup and processing steps for acquiring multi-exposed stereo images along with a corresponding reference disparity map computed by a structured light approach. We discuss the special requirements and challenges which HDR test scenes impose on the computation of the reference disparities and show some preliminary results.

At the request of NSF's Interactive Systems Program, a two-day invitational workshop was held March 23-24, 1992 at UNC Chapel Hill to identify and recommend future research directions in the area of "virtual environments" (VE) *. Workshop participants included some 18 experts (plus 4 NSF officials) from universities, industry, and other leading technical organizations. The two-day schedule alternated between sessions of the entire group and sessions in the following specialty areas, around which the recommendations came to be organized: 1) Perception , 2) Human-Machine Software Interface , 3) Software, 4) Hardware, and 5) Applications. Also, two participants developed a taxonomy of VE applications that is included as an appendix to the report. 1. Collaborative science-technology development programs should be established at several sites around the country to encourage closer collaboration between developers and scientists. 2. Theoretical research should focus on development of metrics of performance and task demands in VE. 3. Paradigmatic applications and theoretical questions that illustrate the science-technology synergy need identification. Audition Spatial Sound 1.Theoretical research should emphasize the role of individual differences in Head-Related Transfer Functions (HRTF's), critical cues for distance and externalization, spectral cues for enhancing elevation and disambiguating the cone-of-confusion, head-motion, and intersensory interaction and adaptation in the accurate perception of virtual acoustic sources. The notion of artificially enhanced localization cues is also a promising area. 2. A fruitful area for joint basic and applied research is the development of perceptually-viable methods of simplifying the synthesis technique to maximize the efficiency of algorithms for complex room modeling. 3. Future effort should still be devoted to developing more realistic models of acoustic environments with implementation on more powerful hardware platforms. Nonspeech Audio 1. Theoretical research should focus on lower-level sensory and higher-level cognitive determinants of acoustic perceptual organization, with particular emphasis on how acoustic parameters interact to determine the identification, segregation, and localization of multiple, simultaneous sources. 2. Technology development should focus on hardware and software systems specifically aimed at real-time generation and control for acoustic information display. Haptics 1. Development should be encouraged of a variety of computer-controlled mechanical devices for either basic scientific investigation of the human haptic system or to serve as haptic interfaces for virtual environments and teleloperation. 2. Research programs should be initiated to encourage collaboration among engineers who are capable of building high precision robotic devices and scientists who can conduct biomechanical and perceptual experiments with the devices. 3. Research …

There are currently no solutions for enabling direct face-to-face interaction between virtual reality (VR) users wearing head-mounted displays (HMDs). The main challenge is that the headset obstructs a significant portion of a user's face, preventing effective facial capture with traditional techniques. To advance virtual reality as a next-generation communication platform, we develop a novel HMD that enables 3D facial performance-driven animation in real-time. Our wearable system uses ultra-thin flexible electronic materials that are mounted on the foam liner of the headset to measure surface strain signals corresponding to upper face expressions. These strain signals are combined with a head-mounted RGB-D camera to enhance the tracking in the mouth region and to account for inaccurate HMD placement. To map the input signals to a 3D face model, we perform a single-instance offline training session for each person. For reusable and accurate online operation, we propose a short calibration step to readjust the Gaussian mixture distribution of the mapping before each use. The resulting animations are visually on par with cutting-edge depth sensor-driven facial performance capture systems and hence, are suitable for social interactions in virtual worlds.

|We propose a reliable method to detect occluding contours and occlusion by active binocular stereo with an occluding contour model that describes the correspondence between the contour of curved objects and its projected image. Applying the image ow generated by moving one camera to the model, we can restrict possible matched points seen by the other camera. We detect occluding contours by tting these matched points to the model. This method can nd occluding contours and occlusion more reliably than conventional ones because stereo matching is performed by using the geometric constraint based on the occluding contour model instead of heuristic constraints. Experimental results of the proposed method applied to the actual scene are presented.

This study develops a framework for the detection and analysis of stress/anxiety emotional states through video-recorded facial cues. A thorough experimental protocol was established to induce systematic variability in affective states (neutral, relaxed and stressed/anxious) through a variety of external and internal stressors. The analysis was focused mainly on non-voluntary and semi-voluntary facial cues in order to estimate the emotion representation more objectively. Features under investigation included eye-related events, mouth activity, head motion parameters and heart rate estimated through camerabased photoplethysmography. A feature selection procedure was employed to select the most robust features followed by classification schemes discriminating between stress/anxiety and neutral states nxiety tress link rate ead motion with reference to a relaxed state in each experimental phase. In addition, a ranking transformation was proposed utilizing self reports in order to investigate the correlation of facial parameters with a participant perceived amount of stress/anxiety. The results indicated that, specific facial cues, derived from eye activity, mouth activity, head movements and camera based heart activity achieve good accuracy and are suitable as discriminative indicators of stress and anxiety.


Question: Eye gaze tracking employing an active stereo head mechanism.

This paper outlines a forward-thinking perspective on Human-Computer Interaction (HCI), coined as "Tangible Bits," which seeks to merge the digital and physical realms by enabling users to directly engage with digital information through physical objects and environmental surfaces. With Tangible Bits, users can intuitively handle digital data (bits) by linking them to tangible objects and architectural elements, thereby facilitating a seamless interaction. Additionally, the approach facilitates users' passive awareness of less immediate digital information through the use of ambient display media, such as light, sound, air currents, and water movement, in an enhanced physical space. The core objective of Tangible Bits is to harmonize the digital and physical worlds, as well as the primary and secondary aspects of human activities. The paper elaborates on three foundational aspects of Tangible Bits: interactive surfaces that respond to user actions, the integration of digital data with graspable physical items, and the use of ambient media to enhance peripheral awareness of background information. These concepts are exemplified through three prototype systems: the metaDESK, transBOARD, and ambientROOM, which are designed to pinpoint underlying research challenges and opportunities.

Both the original version of David Lowe's innuential and classic algorithm for tracking known objects and a reformulation of it implemented by Ishii et al. rely on (diierent) approximated imaging models. Removing their simplifying assumptions yields a fully pro-jective solution with signiicantly improved accuracy and convergence, and arguably better computation{ time properties.

This paper focuses on the integration of acoustic and visual information for people tracking. The system presented relies on a probabilistic framework within which information from multiple sources is integrated at an intermediate stage. An advantage of the method proposed is that of using a generative approach which supports easy and robust integration of multi source information by means of sampled projection instead of triangulation. The system described has been developed in the EU funded CHIL Project research activities. Experimental results from the CLEAR evaluation workshop are reported.

In this paper, we introduce the DEEP-SEE FACE framework, an assistive device designed to improve cognition, interaction, and communication of visually impaired (VI) people in social encounters. The proposed approach jointly exploits computer vision algorithms (region proposal networks, ATLAS tracking and global, and low-level image descriptors) and deep convolutional neural networks in order to detect, track, and recognize, in real-time, various persons existent in the video streams. The major contribution of the paper concerns a global, fixed-size face representation that takes into the account of various video frames while remaining independent of the length of the image sequence. To this purpose, we introduce an effective weight adaptation scheme that is able to determine the relevance assigned to each face instance, depending on the frame degree of motion/camera blur, scale variation, and compression artifacts. Another relevant contribution involves a hard negative mining stage that helps us differentiating between known and unknown face identities. The experimental results, carried out on a large-scale data set, validate the proposed methodology with an average accuracy and recognition rates superior to 92%. When tested in real life, indoor/outdoor scenarios, the DEEP-SEE FACE prototype proves to be effective and easy to use, allowing the VI people to access visual information during social events.

The accuracy of 3-D reconstructions depends substantially on the accuracy of active vision system calibration. In this work, the problem of video projector calibration is solved by inverting the standard camera calibration work flow. The calibration procedure requires a single camera, which does not need to be calibrated and which is used as the sensor whether projected dots and calibration pattern landmarks, such as the checkerboard corners, coincide. The method iteratively adjusts the projected dots to coincide with the landmarks and the final coordinates are used as inputs to a camera calibration method. The otherwise slow iterative adjustment is accelerated by estimating a plane homography between the detected landmarks and the projected dots, which makes the calibration method fast.

This paper presents an 6-axis optical force sensor which can be used in fMRI. Recently, fMRIs are widely used for studying human brain function. Simultaneous measurement of brain activity and peripheral information, such as grip force, enables more precise investigations in studies of motor function. However, conventional force sensors cannot be used in fMRI environment, since metal elements generate noise which severely contaminate the signals of fMRI. An optical 2-axis force sensor has been developed using photo sensors and optical fibers by Tada et. al.[1], that resolved these problems. The developed force sensor removed all magnetic components from the sensing part. It detected minute displacements by measure amount of light and light traveled through the optical fibers. However, there still remain several problems on this optical force sensor. Firstly, the accuracy is not high compared to the conventional force sensors. Secondly, the robustness is not enough against the contact force to the optical fibers. In this paper, the problems concerning to the accuracy and the sensor output stability has been improved by novel methods of fixing fibers and arithmetic circuit. Furthermore, an optical 6-axis force sensor is developed based on these improvements, and usefulness of our sensor for brain function analysis is confirmed in fMRI experimentations.

The Iterative Closest Point (ICP) algorithm is widely used to register two roughly aligned surfaces. Its most expensive step is the search for the closest point. Many efficient variants have been proposed to speed up this step, however they do not guarantee that the closest point on a triangulated surface is found. Instead they might result in an approximation of the closest point. In this paper we present a method for the closest point search that is fast and exact. The method was implemented and used to evaluate the accuracy of head motion estimation using dense 3D data obtained from a stereo camera system. The results show that the accuracy of the estimated head motion is better than 1 mm for translational movements and better than 1 degree for rotations. Key-Words: Iterative Closest Point, 3D Registration, Stereo, Head Motion Estimation

This paper sets the stage for a series of reviews dealing with the problems associated with the reconstruction and analysis of in vivo skeletal system kinematics using optoelectronic stereophotogrammetric data. Instantaneous bone position and orientation and joint kinematic variable estimations are addressed in the framework of rigid body mechanics. The conceptual background to these exercises is discussed. Focus is placed on the experimental and analytical problem of merging the information relative to movement and that relative to the morphology of the anatomical body parts of interest. The various global and local frames that may be used in this context are defined. Common anatomical and mathematical conventions that can be used to describe joint kinematics are illustrated in a comparative fashion. The authors believe that an effort to systematize the different theoretical and experimental approaches to the problems involved and related nomenclatures, as currently reported in the literature, is needed to facilitate data and knowledge sharing, and to provide renewed momentum for the advancement of human movement analysis.

Hand gestures are an important type of natural language used in many research areas such as human-computer interaction and computer vision. Hand gestures recognition requires the prior determination of the hand position through detection and tracking. One of the most efficient strategies for hand tracking is to use 2D visual information such as color and shape. However, visual-sensor-based hand tracking methods are very sensitive when tracking is performed under variable light conditions. Also, as hand movements are made in 3D space, the recognition performance of hand gestures using 2D information is inherently limited. In this article, we propose a novel real-time 3D hand tracking method in depth space using a 3D depth sensor and employing Kalman filter. We detect hand candidates using motion clusters and predefined wave motion, and track hand locations using Kalman filter. To verify the effectiveness of the proposed method, we compare the performance of the proposed method with the visual-based method. Experimental results show that the performance of the proposed method out performs visual-based method.

Recent technological advances in connected-speech recognition and position sensing in space have encouraged the notion that voice and gesture inputs at the graphics interface can converge to provide a concerted, natural user modality.
 The work described herein involves the user commanding simple shapes about a large-screen graphics display surface. Because voice can be augmented with simultaneous pointing, the free usage of pronouns becomes possible, with a corresponding gain in naturalness and economy of expression. Conversely, gesture aided by voice gains precision in its power to reference.

We present a mobile vision system for multi-person tracking in busy environments. Specifically, the system integrates continuous visual odometry computation with tracking-by-detection in order to track pedestrians in spite of frequent occlusions and egomotion of the camera rig. To achieve reliable performance under real-world conditions, it has long been advocated to extract and combine as much visual information as possible. We propose a way to closely integrate the vision modules for visual odometry, pedestrian detection, depth estimation, and tracking. The integration naturally leads to several cognitive feedback loops between the modules. Among others, we propose a novel feedback connection from the object detector to visual odometry which utilizes the semantic knowledge of detection to stabilize localization. Feedback loops always carry the danger that erroneous feedback from one module is amplified and causes the entire system to become instable. We therefore incorporate automatic failure detection and recovery, allowing the system to continue when a module becomes unreliable. The approach is experimentally evaluated on several long and difficult video sequences from busy inner-city locations. Our results show that the proposed integration makes it possible to deliver stable tracking performance in scenes of previously infeasible complexity.


Question: Utilizing an active stereo head for eye gaze tracking.

This article introduces an innovative technique for calibrating a vision system comprising two or more cameras. The method, which capitalizes on fundamental properties of vanishing points, unfolds in two distinct phases. Initially, the intrinsic parameters of each camera, encompassing the focal length and the position where the optical axis intersects the image plane, are deduced from a single image of a cube. Subsequently, the extrinsic parameters between a pair of cameras, including the rotation matrix and the translation vector that describe the rigid motion between their respective coordinate systems, are estimated using a stereo image pair of a suitable planar pattern. By first aligning corresponding vanishing points across the two images, the rotation matrix is calculated. Then, through a straightforward triangulation process, the translation vector is determined. The article discusses the method's resilience to noise and delineates the conditions for achieving optimal rotation matrix estimation. Extensive testing confirms that the proposed method attains a level of precision sufficient for executing machine vision tasks reliant on camera calibration, such as depth perception from stereo images and motion analysis from image sequences.

While intelligent environments are often cited as a reason for doing work on visual person-tracking, really making an intelligent environment exposes many realworld problems in visual tracking that must be solved to make the technology practical. In the context of our EasyLiving project in intelligent environments, we created a practical person-tracking system that solves most of the real-world problems. It uses two sets of color stereo cameras for tracking multiple people during live demonstrations in a living room. The stereo images are used for locating people, and the color images are used for maintaining their identities. The system runs quickly enough to make the room feel responsive, and it tracks multiple people standing, walking, sitting, occluding, and entering and leaving the

We explore the development of an experimental augmented reality application that provides benefits to professional mechanics performing maintenance and repair tasks in a field setting. We developed a prototype that supports military mechanics conducting routine maintenance tasks inside an armored vehicle turret, and evaluated it with a user study. Our prototype uses a tracked headworn display to augment a mechanic's natural view with text, labels, arrows, and animated sequences designed to facilitate task comprehension, localization, and execution. A within-subject controlled user study examined professional military mechanics using our system to complete 18 common tasks under field conditions. These tasks included installing and removing fasteners and indicator lights, and connecting cables, all within the cramped interior of an armored personnel carrier turret. An augmented reality condition was tested against two baseline conditions: the same headworn display providing untracked text and graphics and a fixed flat panel display representing an improved version of the laptop-based documentation currently employed in practice. The augmented reality condition allowed mechanics to locate tasks more quickly than when using either baseline, and in some instances, resulted in less overall head movement. A qualitative survey showed that mechanics found the augmented reality condition intuitive and satisfying for the tested sequence of tasks.

The Kinect has primarily been used as a gesture-driven device for motion-based controls. To date, Kinect-based research has predominantly focused on improving tracking and gesture recognition across a wide base of users. In this paper, we propose to use the Kinect for biometrics; rather than accommodating a wide range of users we exploit each user's uniqueness in terms of gestures. Unlike pure biometrics, such as iris scanners, face detectors, and fingerprint recognition which depend on irrevocable biometric data, the Kinect can provide additional revocable gesture information. We propose a dynamic time-warping (DTW) based framework applied to the Kinect's skeletal information for user access control. Our approach is validated in two scenarios: user identification, and user authentication on a dataset of 20 individuals performing 8 unique gestures. We obtain an overall 4.14%, and 1.89% Equal Error Rate (EER) in user identification, and user authentication, respectively, for a gesture and consistently outperform related work on this dataset. Given the natural noise present in the real-time depth sensor this yields promising results.

Optical see-through head-mounted displays (OST-HMD) feature an unhindered and instantaneous view of the surgery site and can enable a mixed reality experience for surgeons during procedures. In this paper, we present a systematic approach to identify the criteria for evaluation of OST-HMD technologies for specific clinical scenarios, which benefit from using an object-anchored 2D-display visualizing medical information. Criteria for evaluating the performance of OST-HMDs for visualization of medical information and its usage are identified and proposed. These include text readability, contrast perception, task load, frame rate, and system lag. We choose to compare three commercially available OST-HMDs, which are representatives of currently available head-mounted display technologies. A multi-user study and an offline experiment are conducted to evaluate their performance. Statistical analysis demonstrates that Microsoft HoloLens performs best among the three tested OST-HMDs, in terms of contrast perception, task load, and frame rate, while ODG R-7 offers similar text readability. The integration of indoor localization and fiducial tracking on the HoloLens provides significantly less system lag in a relatively motionless scenario. With ever more OST-HMDs appearing on the market, the proposed criteria could be used in the evaluation of their suitability for mixed reality surgical intervention. Currently, Microsoft HoloLens may be more suitable than ODG R-7 and Epson Moverio BT-200 for clinical usability in terms of the evaluated criteria. To the best of our knowledge, this is the first paper that presents a methodology and conducts experiments to evaluate and compare OST-HMDs for their use as object-anchored 2D-display during interventions.

Passive stereo vision is widely used as a range sensing technology in robots, but suffers from dropouts: areas of low texture where stereo matching fails. By supplementing a stereo system with a strong texture projector, dropouts can be eliminated or reduced. This paper develops a practical stereo projector system, first by finding good patterns to project in the ideal case, then by analyzing the effects of system blur and phase noise on these patterns, and finally by designing a compact projector that is capable of good performance out to 3m in indoor scenes. The system has been implemented and has excellent depth precision and resolution, especially in the range out to 1.5m.

In this paper we present an augmented reality system for automatic try-on of 3D virtual eyeglasses. The user can select from various virtual models of eyeglasses for trying-on and the system will automatically fit the selected virtual glasses on the user's face. The user can see his/her face as in a mirror with the 3D virtual glasses fitted on it. We also propose a method for handling the occlusion problem, to display only those parts of the glasses that are not occluded by the face. This system can be used for online shopping, or short listing a large set of available models to a few before physical try-on at a retailer's site.

In this paper, we address the challenging computer vision problem of obtaining a reliable facial expression analysis from a naturally interacting person. We propose a system that combines a 3D generic face model, 3D head tracking, and 2D tracker to track facial landmarks and recognize expressions. First, we extract facial landmarks from a neutral frontal face, and then we deform a 3D generic face to fit the input face. Next, we use our real-time 3D head tracking module to track a person’s head in 3D and predict facial landmark positions in 2D using the projection from the updated 3D face model. Finally, we use tracked 2D landmarks to update the 3D landmarks. This integrated tracking loop enables efficient tracking of the non-rigid parts of a face in the presence of large 3D head motion. We conducted experiments for facial expression recognition using both framebased and sequence-based approaches. Our method provides a 75.9% recognition rate in 8 subjects with 7 key expressions. Our approach provides a considerable step forward toward new applications including human-computer interactions, behavioral science, robotics, and game applications.

The Iterative Closest Point (ICP) algorithm is widely used to register two roughly aligned surfaces. Its most expensive step is the search for the closest point. Many efficient variants have been proposed to speed up this step, however they do not guarantee that the closest point on a triangulated surface is found. Instead they might result in an approximation of the closest point. In this paper we present a method for the closest point search that is fast and exact. The method was implemented and used to evaluate the accuracy of head motion estimation using dense 3D data obtained from a stereo camera system. The results show that the accuracy of the estimated head motion is better than 1 mm for translational movements and better than 1 degree for rotations. Key-Words: Iterative Closest Point, 3D Registration, Stereo, Head Motion Estimation

For a learning platform to be adapti ve, it needs to monitor the user in r ealtime. At the ScienceCampus Tübingen, we conduct empirical research on learning strategies, in particular with regard to system adaptivity. Our experimental Webbased adaptiv e multimedia learning environment is able to communicate with an eyetracking system in real-time, and l inks gaze data to browser-based learning content. By processing simple evaluation scripts in the browser, we can control the learning process by adapting the learning content in response to predefined conditions and user inputs. The experimental system serves as a basis for a series of psyc hologi al studies in the learning context. We have already conducted some studies with our experimental learning platform, but its devel opment is still ongoing. In particular, we are adding new feature s required for future studies. In this paper, we desc ribe the actual state of our experimental learning environment, lessons learned from its a pplication in the studies, and our plans for future devel opment.

In immersive virtual environments (IVEs), users can control their virtual viewpoint by moving their tracked head and walking through the real world. Usually, movements in the real world are mapped one-to-one to virtual camera motions. With redirection techniques, the virtual camera is manipulated by applying gains to user motion so that the virtual world moves differently than the real world. Thus, users can walk through large-scale IVEs while physically remaining in a reasonably small workspace. In psychophysical experiments with a two-alternative forced-choice task, we have quantified how much humans can unknowingly be redirected on physical paths that are different from the visually perceived paths. We tested 12 subjects in three different experiments: (E1) discrimination between virtual and physical rotations, (E2) discrimination between virtual and physical straightforward movements, and (E3) discrimination of path curvature. In experiment E1, subjects performed rotations with different gains, and then had to choose whether the visually perceived rotation was smaller or greater than the physical rotation. In experiment E2, subjects chose whether the physical walk was shorter or longer than the visually perceived scaled travel distance. In experiment E3, subjects estimate the path curvature when walking a curved path in the real world while the visual display shows a straight path in the virtual world. Our results show that users can be turned physically about 49 percent more or 20 percent less than the perceived virtual rotation, distances can be downscaled by 14 percent and upscaled by 26 percent, and users can be redirected on a circular arc with a radius greater than 22 m while they believe that they are walking straight.


Question: Presenting MVTec ITODD - A dataset designed for 3D object recognition in industrial settings.

This paper introduces a novel voting-based pose estimation algorithm specifically tailored for 3D sensors, which are rapidly overtaking 2D sensors in robotics, computer vision, and gaming industries due to their superior performance. Recent studies have highlighted the effectiveness of using a pair of oriented 3D points, characterized by points on the object surface with normals, within a voting framework for achieving fast and robust pose estimation. However, while these oriented surface points are highly discriminative for objects exhibiting significant curvature variations, they fall short in providing compact and discriminative information for many industrial and real-world objects that predominantly feature planar surfaces. Inspired by the pivotal role of edges in 2D registration, we recognize that depth discontinuities are equally critical in 3D scenarios. Consequently, we explore and develop a series of pose estimation algorithms that effectively leverage this boundary information. Beyond oriented surface points, our approach incorporates two additional primitives: boundary points with directions and boundary line segments. Our experimental findings reveal that these judiciously selected primitives not only encode a greater amount of information in a more compact manner but also lead to enhanced accuracy for a diverse range of industrial components. Moreover, they facilitate faster computational processing. To showcase the practical applicability of our proposed algorithm, we present a robotic bin-picking system that successfully integrates the algorithm with a 3D sensor, demonstrating its real-world effectiveness.

We present ShapeNet: a richly-annotated, large-scale repository of shapes represented by 3D CAD models of objects. ShapeNet contains 3D models from a multitude of semantic categories and organizes them under the WordNet taxonomy. It is a collection of datasets providing many semantic annotations for each 3D model such as consistent rigid alignments, parts and bilateral symmetry planes, physical sizes, keywords, as well as other planned annotations. Annotations are made available through a public web-based interface to enable data visualization of object attributes, promote data-driven geometric analysis, and provide a large-scale quantitative benchmark for research in computer graphics and vision. At the time of this technical report, ShapeNet has indexed more than 3,000,000 models, 220,000 models out of which are classified into 3,135 categories (WordNet synsets). In this report we describe the ShapeNet effort as a whole, provide details for all currently available datasets, and summarize future plans.

We propose a new and efficient method for 3D object localization and fine-grained 3D pose estimation from a single 2D image. Our approach follows the classical paradigm of matching a 3D model to the 2D observations. Our first contribution is a 3D object model composed of a set of 3D edge primitives learned from 2D object blueprints, which can be viewed as a 3D generalization of HOG features. This model is used to define a matching cost obtained by applying a rigid-body transformation to the 3D object model, projecting it onto the image plane, and matching the projected model to HOG features extracted from the input image. Our second contribution is a very efficient branch-and-bound algorithm for finding the 3D pose that maximizes the matching score. For this, 3D integral images of quantized HOGs are employed to evaluate in constant time the maximum attainable matching scores of individual model primitives. We applied our method to three different datasets of cars and achieved promising results with testing times as low as less than half a second.

In this paper we consider the problem of recovering the free space of an indoor scene from its single image. We show that exploiting the box like geometric structure of furniture and constraints provided by the scene, allows us to recover the extent of major furniture objects in 3D. Our “boxy” detector localizes box shaped objects oriented parallel to the scene across different scales and object types, and thus blocks out the occupied space in the scene. To localize the objects more accurately in 3D we introduce a set of specially designed features that capture the floor contact points of the objects. Image based metrics are not very indicative of performance in 3D. We make the first attempt to evaluate single view based occupancy estimates for 3D errors and propose several task driven performance measures towards it. On our dataset of 592 indoor images marked with full 3D geometry of the scene, we show that: (a) our detector works well using image based metrics; (b) our refinement method produces significant improvements in localization in 3D; and (c) if one evaluates using 3D metrics, our method offers major improvements over other single view based scene geometry estimation methods.

Deep learning has made significant breakthrough in the past decade. In certain application domain, its detection accuracy has surpassed human being in the same task, e.g., voice recognition and object detection. Various novel applications has been developed and achieved good performance by leveraging the latest advances in deep learning. In this paper, we propose to utilize deep learning based technique, specifically, Convolutional Neural Network (CNN), to develop an auto-counting system for supermarket scenario. Given a picture, the system can automatically detect the specified categories of goods (e.g., Head & Shoulders bottles) and their respective numbers. To improve detection accuracy of the system, we propose to combine hard example mining and multi-scale feature extraction to the Faster R-CNN framework. Experimental results demonstrate the efficacy of the proposed system. Specifically, our system achieves an mAP of 92.1%, which is better than the state-of-the-art, and the response time is about 250 ms per image, including all steps on a GTX 1080 GPU.

This paper presents a new coded structured light pattern which permits to solve the correspondence problem by a single shot and without using geometrical constraints. The pattern is composed by the projection of a grid made by coloured slits in such a way that each slit with its two neighbours appears only once in the pattern. The technique proposed permits a rapid and robust 3D scene measurement, even with moving objects. Ó 1998 Elsevier Science B.V. All rights reserved.

This work presents a Multi-View Stereo system for robust and efficient dense modeling from unstructured image collections. Our core contributions are the joint estimation of depth and normal information, pixelwise view selection using photometric and geometric priors, and a multi-view geometric consistency term for the simultaneous refinement and image-based depth and normal fusion. Experiments on benchmarks and large-scale Internet photo collections demonstrate stateof-the-art performance in terms of accuracy, completeness, and efficiency. Fig. 1. Reconstructions for Louvre, Todai-ji, Paris Opera, and Astronomical Clock.

In this paper we consider the problem of computing the 3D shape of an unknown, arbitrarily-shaped scene from multiple photographs taken at known but arbitrarily-distributed viewpoints. By studying the equivalence class of all 3D shapes that reproduce the input photographs, we prove the existence of a special member of this class, the photo hull, that (1) can be computed directly from photographs of the scene, and (2) subsumes all other members of this class. We then give a provably-correct algorithm, called Space Carving, for computing this shape and present experimental results on complex real-world scenes. The approach is designed to (1) capture photorealistic shapes that accurately model scene appearance from a wide range of viewpoints, and (2) account for the complex interactions between occlusion, parallax, shading, and their view-dependent effects on scene-appearance.

We explore the performance of a number of popular feature detectors and descriptors in matching 3D object features across viewpoints and lighting conditions. To this end we design a method, based on intersecting epipolar constraints, for providing ground truth correspondence automatically. These correspondences are based purely on geometric information, and do not rely on the choice of a specific feature appearance descriptor. We test detector-descriptor combinations on a database of 100 objects viewed from 144 calibrated viewpoints under three different lighting conditions. We find that the combination of Hessian-affine feature finder and SIFT features is most robust to viewpoint change. Harris-affine combined with SIFT and Hessian-affine combined with shape context descriptors were best respectively for lighting change and change in camera focal length. We also find that no detector-descriptor combination performs well with viewpoint changes of more than 25–30∘.

This paper presents a novel method for detecting and localizing objects of a visual category in cluttered real-world scenes. Our approach considers object categorization and figure-ground segmentation as two interleaved processes that closely collaborate towards a common goal. As shown in our work, the tight coupling between those two processes allows them to benefit from each other and improve the combined performance. The core part of our approach is a highly flexible learned representation for object shape that can combine the information observed on different training examples in a probabilistic extension of the Generalized Hough Transform. The resulting approach can detect categorical objects in novel images and automatically infer a probabilistic segmentation from the recognition result. This segmentation is then in turn used to again improve recognition by allowing the system to focus its efforts on object pixels and to discard misleading influences from the background. Moreover, the information from where in the image a hypothesis draws its support is employed in an MDL based hypothesis verification stage to resolve ambiguities between overlapping hypotheses and factor out the effects of partial occlusion. An extensive evaluation on several large data sets shows that the proposed system is applicable to a range of different object categories, including both rigid and articulated objects. In addition, its flexible representation allows it to achieve competitive object detection performance already from training sets that are between one and two orders of magnitude smaller than those used in comparable systems.

In this paper, we tackle the labeling problem for 3D point clouds. We introduce a 3D point cloud labeling scheme based on 3D Convolutional Neural Network. Our approach minimizes the prior knowledge of the labeling problem and does not require a segmentation step or hand-crafted features as most previous approaches did. Particularly, we present solutions for large data handling during the training and testing process. Experiments performed on the urban point cloud dataset containing 7 categories of objects show the robustness of our approach.


Question: Presenting MVTec ITODD - A dataset designed for 3D object recognition in industrial settings.

This paper delves into the process of constructing a comprehensive 3D model of a physical object, focusing specifically on the use of range images for direct access to three-dimensional data. Unlike approaches utilizing intensity images, range images offer a more direct pathway to creating accurate 3D representations. The primary challenge addressed here is the determination of the transformation between various views of the object, a task that has traditionally been either assumed to be known (an assumption that is often unrealistic for detailed models) or calculated using feature matching techniques, which can be insufficiently precise for effective integration. We propose a novel method that operates directly on range data, facilitating the accurate alignment of successive views with sufficient overlap. This alignment is achieved by minimizing a functional that does not necessitate point-to-point correspondences, a process that we detail in the context of both our registration methodology and modeling procedure. We demonstrate the effectiveness of our approach through the application to real-world range images of intricate objects. The creation of object models is a crucial component for both biological and machine vision systems, serving as a foundation for various tasks such as object recognition, pose estimation, and inspection. In scenarios where CAD models are either unavailable or impractical, the construction of models from the physical object becomes essential. While some researchers have opted for models comprised of multiple views, such an approach may not always be sufficient. For a fully comprehensive model, the following steps are typically required: data acquisition, view registration, and view integration. The registration process, which involves finding the transformation between different views, is a fundamental prerequisite for successful integration. This paper outlines a method for achieving accurate registration, addressing the correspondence problem that has been central to numerous previous research endeavors. Through our method, we aim to provide a more robust and precise solution to the challenge of 3D object modeling.

The Pascal Visual Object Classes (VOC) challenge is a benchmark in visual object category recognition and detection, providing the vision and machine learning communities with a standard dataset of images and annotation, and standard evaluation procedures. Organised annually from 2005 to present, the challenge and its associated dataset has become accepted as the benchmark for object detection. This paper describes the dataset and evaluation procedure. We review the state-of-the-art in evaluated methods for both classification and detection, analyse whether the methods are statistically different, what they are learning from the images (e.g. the object or its context), and what the methods find easy or confuse. The paper concludes with lessons learnt in the three year history of the challenge, and proposes directions for future improvement and extension.

As a core robotic and vision problem, camera and range sensor calibration have been researched intensely over the last decades. However, robotic research efforts still often get heavily delayed by the requirement of setting up a calibrated system consisting of multiple cameras and range measurement units. With regard to removing this burden, we present a toolbox with web interface for fully automatic camera-to-camera and camera-to-range calibration. Our system is easy to setup and recovers intrinsic and extrinsic camera parameters as well as the transformation between cameras and range sensors within one minute. In contrast to existing calibration approaches, which often require user intervention, the proposed method is robust to varying imaging conditions, fully automatic, and easy to use since a single image and range scan proves sufficient for most calibration scenarios. Experimentally, we demonstrate that the proposed checkerboard corner detector significantly outperforms current state-of-the-art. Furthermore, the proposed camera-to-range registration method is able to discover multiple solutions in the case of ambiguities. Experiments using a variety of sensors such as grayscale and color cameras, the Kinect 3D sensor and the Velodyne HDL-64 laser scanner show the robustness of our method in different indoor and outdoor settings and under various lighting conditions.

We present a Bayesian object observation model for complete probabilistic semantic SLAM. Recent studies on object detection and feature extraction have become important for scene understanding and 3D mapping. However, 3D shape of the object is too complex to formulate the probabilistic observation model; therefore, performing the Bayesian inference of the object-oriented features as well as their pose is less considered. Besides, when the robot equipped with an RGB mono camera only observes the projected single view of an object, a significant amount of the 3D shape information is abandoned. Due to these limitations, semantic SLAM and viewpoint-independent loop closure using volumetric 3D object shape is challenging. In order to enable the complete formulation of probabilistic semantic SLAM, we approximate the observation model of a 3D object with a tractable distribution. We also estimate the variational likelihood from the 2D image of the object to exploit its observed single view. In order to evaluate the proposed method, we perform pose and feature estimation, and demonstrate that the automatic loop closure works seamlessly without additional loop detector in various environments.

Generative 3D face models are a powerful tool in computer vision. They provide pose and illumination invariance by modeling the space of 3D faces and the imaging process. The power of these models comes at the cost of an expensive and tedious construction process, which has led the community to focus on more easily constructed but less powerful models. With this paper we publish a generative 3D shape and texture model, the Basel Face Model (BFM), and demonstrate its application to several face recognition task. We improve on previous models by offering higher shape and texture accuracy due to a better scanning device and less correspondence artifacts due to an improved registration algorithm. The same 3D face model can be fit to 2D or 3D images acquired under different situations and with different sensors using an analysis by synthesis method. The resulting model parameters separate pose, lighting, imaging and identity parameters, which facilitates invariant face recognition across sensors and data sets by comparing only the identity parameters. We hope that the availability of this registered face model will spur research in generative models. Together with the model we publish a set of detailed recognition and reconstruction results on standard databases to allow complete algorithm comparisons.

In this paper, we propose a Scale and Rotation Invariant Implicit Shape Model (SRIISM), and develop a local feature matching based system using the model to accurately locate and identify large numbers of object instances in an image. Due to repeated instances and cluttered background, conventional methods for multiple object instance identification suffer from poor identification results. In the proposed SRIISM, we model the joint distribution of object centers, scale, and orientation computed from local feature matches in Hough voting, which is not only invariant to scale changes and rotation of objects, but also robust to false feature matches. In the multiple object instance identification system using SRIISM, we apply a fast 4D bin search method in Hough space with complexity O(n), where n is the number of feature matches, in order to segment and locate each instance. Futhermore, we apply maximum likelihood estimation (MLE) for accurate object pose detection. In the evaluation, we created datasets simulating various industrial applications such as pick-and-place and inventory management. Experiment results on the datasets show that our method outperforms conventional methods in both accuracy (5%-30% gain) and speed (2x speed up).

Intrinsic image decomposition refers to recover the albedo and shading from images, which is an ill-posed problem in signal processing. As realistic labeled data are severely lacking, it is difficult to apply learning methods in this issue. In this letter, we propose using a synthesized dataset to facilitate the solving of this problem. A physically based renderer is used to generate color images and their underlying ground-truth albedo and shading from three-dimensional models. Additionally, we render a Kinect-like noisy depth map for each instance. We utilize this synthetic dataset to train a deep neural network for intrinsic image decomposition and further fine-tune it for real-world images. Our model supports both RGB and RGB-D as input, and it employs both high-level and low-level features to avoid blurry outputs. Experimental results verify the effectiveness of our model on realistic images.

We train a generative convolutional neural network which is able to generate images of objects given object type, viewpoint, and color. We train the network in a supervised manner on a dataset of rendered 3D chair models. Our experiments show that the network does not merely learn all images by heart, but rather finds a meaningful representation of a 3D chair model allowing it to assess the similarity of different chairs, interpolate between given viewpoints to generate the missing ones, or invent new chair styles by interpolating between chairs from the training set. We show that the network can be used to find correspondences between different chairs from the dataset, outperforming existing approaches on this task.

Over the last years, the robotics community has made substantial progress in detection and 3D pose estimation of known and unknown objects. However, the question of how to identify objects based on language descriptions has not been investigated in detail. While the computer vision community recently started to investigate the use of attributes for object recognition, these approaches do not consider the task settings typically observed in robotics, where a combination of appearance attributes and object names might be used in referral language to identify specific objects in a scene. In this paper, we introduce an approach for identifying objects based on natural language containing appearance and name attributes. To learn rich RGB-D features needed for attribute classification, we extend recently introduced sparse coding techniques so as to automatically learn attribute-dependent features. We introduce a large data set of attribute descriptions of objects in the RGB-D object dataset. Experiments on this data set demonstrate the strong performance of our approach to language based object identification. We also show that our attribute-dependent features provide significantly better generalization to previously unseen attribute values, thereby enabling more rapid learning of new attribute values.

Build accurate DNN models requires training on large labeled, context specific datasets, especially those matching the target scenario. We believe advances in wireless localization, working in unison with cameras, can produce automated annotation of targets on images and videos captured in the wild. Using pedestrian and vehicle detection as examples, we demonstrate the feasibility, benefits, and challenges of an automatic image annotation system. Our work calls for new technical development on passive localization, mobile data analytics, and error-resilient ML models, as well as design issues in user privacy policies.

We provide a large dataset containing RGB-D image sequences and the ground-truth camera trajectories with the goal to establish a benchmark for the evaluation of visual SLAM systems. Our dataset contains the color and depth images of a Microsoft Kinect sensor and the groundtruth trajectory of camera poses. The data was recorded at full frame rate (30 Hz) and sensor resolution (640x480). The ground-truth trajectory was obtained from a high-accuracy motion-capture system with eight high-speed tracking cameras (100 Hz). Further, we provide the accelerometer data from the Kinect. Finally, we propose an evaluation criterion for measuring the quality of the estimated camera trajectory of visual SLAM systems.


Question: Presenting MVTec ITODD - A dataset designed for 3D object recognition in industrial settings.

This paper outlines an effective 3D object recognition and pose estimation methodology, specifically tailored for the task of grasping objects in cluttered and partially occluded settings. Unlike conventional approaches that primarily depend on visual appearance, our technique exclusively utilizes 3D geometry data. The core of our approach is anchored in a resilient geometric descriptor, an efficient hashing mechanism, and a streamlined RANSAC-like sampling strategy that facilitates localized processing. We posit that each object is characterized by a model, represented by a collection of points along with their associated surface normals. Our method is capable of simultaneously identifying multiple instances of these models and accurately estimating their orientations within the scene. Extensive testing reveals the robustness of our proposed method, even under conditions of significant noise, clutter, and segmentation absence, where only a fraction of the object surfaces are visible. A key feature of our algorithm is its linear time complexity, enabling high-speed recognition and facilitating its seamless integration into ongoing manipulation tasks. The practical utility of our method is demonstrated through experimental validation with a high-precision, 7-degrees-of-freedom Cartesian impedance controlled robot, showcasing its efficacy in grasping objects from a chaotic, unorganized stack. This application highlights the synergy between advanced computer vision and soft robotics, leading to a robotic system that can adeptly operate in unstructured and occluded environments.

This paper addresses the problem of transparent object matting. Existing image matting approaches for transparent objects often require tedious capturing procedures and long processing time, which limit their practical use. In this paper, we first formulate transparent object matting as a refractive flow estimation problem. We then propose a deep learning framework, called TOM-Net, for learning the refractive flow. Our framework comprises two parts, namely a multi-scale encoder-decoder network for producing a coarse prediction, and a residual network for refinement. At test time, TOM-Net takes a single image as input, and outputs a matte (consisting of an object mask, an attenuation mask and a refractive flow field) in a fast feed-forward pass. As no off-the-shelf dataset is available for transparent object matting, we create a large-scale synthetic dataset consisting of 178K images of transparent objects rendered in front of images sampled from the Microsoft COCO dataset. We also collect a real dataset consisting of 876 samples using 14 transparent objects and 60 background images. Promising experimental results have been achieved on both synthetic and real data, which clearly demonstrate the effectiveness of our approach.

With the proliferation of cameras on mobile devices there is an increased desire to image document pages as an alternative to scanning. However, the quality of captured document images is often lower than its scanned equivalent due to hardware limitations and stability issues. In this context, automatic assessment of the quality of captured images is useful for many applications. Although there has been a lot of work on developing computational methods and creating standard datasets for natural scene image quality assessment, until recently quality estimation of camera captured document images has not been given much attention. One traditional quality indicator for document images is the Optical Character Recognition (OCR) accuracy. In this work, we present a dataset of camera captured document images containing varying levels of focal-blur introduced manually during capture. For each image we obtained the character level OCR accuracy. Our dataset can be used to evaluate methods for predicting OCR quality of captured documents as well as enhancements. In order to make the dataset publicly and freely available, originals from two existing datasets University of Washington dataset and Tobacco Database were selected. We present a case study with three recent methods for predicting the OCR quality of images on our dataset.

As a core robotic and vision problem, camera and range sensor calibration have been researched intensely over the last decades. However, robotic research efforts still often get heavily delayed by the requirement of setting up a calibrated system consisting of multiple cameras and range measurement units. With regard to removing this burden, we present a toolbox with web interface for fully automatic camera-to-camera and camera-to-range calibration. Our system is easy to setup and recovers intrinsic and extrinsic camera parameters as well as the transformation between cameras and range sensors within one minute. In contrast to existing calibration approaches, which often require user intervention, the proposed method is robust to varying imaging conditions, fully automatic, and easy to use since a single image and range scan proves sufficient for most calibration scenarios. Experimentally, we demonstrate that the proposed checkerboard corner detector significantly outperforms current state-of-the-art. Furthermore, the proposed camera-to-range registration method is able to discover multiple solutions in the case of ambiguities. Experiments using a variety of sensors such as grayscale and color cameras, the Kinect 3D sensor and the Velodyne HDL-64 laser scanner show the robustness of our method in different indoor and outdoor settings and under various lighting conditions.

Pedestrian detection is a highly debated issue in the scientific community due to its outstanding importance for a large number of applications, especially in the fields of automotive safety, robotics and surveillance. In spite of the widely varying methods developed in recent years, pedestrian detection is still an open challenge whose accuracy and robustness has to be improved. Therefore, in this paper, we focus on improving the classification component in the pedestrian detection task on the Daimler stereo vision data set by adopting two approaches: 1) by combining three image modalities (intensity, depth and flow) to feed a unique convolutional neural network (CNN) and 2) by fusing the results of three independent CNNs.

The primary problem dealt with in this paper is the following. Given some description of a visual object, find that object in an actual photograph. Part of the solution to this problem is the specification of a descriptive scheme, and a metric on which to base the decision of "goodness" of matching or detection.

The ability to classify objects is fundamental for robots. Besides knowledge about their visual appearance, captured by the RGB channel, robots heavily need also depth information to make sense of the world. While the use of deep networks on RGB robot images has benefited from the plethora of results obtained on databases like ImageNet, using convnets on depth images requires mapping them into three-dimensional channels. This transfer learning procedure makes them processable by pretrained deep architectures. Current mappings are based on heuristic assumptions over preprocessing steps and on what depth properties should be most preserved, resulting often in cumbersome data visualizations, and in suboptimal performance in terms of generality and recognition results. Here, we take an alternative route and we attempt instead to learn an optimal colorization mapping for any given pretrained architecture, using as training data a reference RGB-D database. We propose a deep network architecture, exploiting the residual paradigm, that learns how to map depth data to three channel images. A qualitative analysis of the images obtained with this approach clearly indicates that learning the optimal mapping preserves the richness of depth information better than current hand-crafted approaches. Experiments on the Washington, JHUIT-50 and BigBIRD public benchmark databases, using CaffeNet, VGG-16, GoogleNet, and ResNet50 clearly showcase the power of our approach, with gains in performance of up to 16% compared to state of the art competitors on the depth channel only, leading to top performances when dealing with RGB-D data.

In this paper, a robust, real-time object tracking approach capable of dealing with multiple symmetric and non-symmetric objects in a real-time requirement setting is proposed. The approach relies only on depth data to track multiple objects in a dynamic environment and uses random-forest based learning to deal with problems like object occlusion, motion-blur due to camera motion and clutter. We show that the relation between object motion and the corresponding change in its 3D point cloud data can be learned using only 6 random forests. A framework that unites object pose estimation and object pose tracking to efficiently track multiple objects in 3D space is presented. The approach is robust in tracking objects even in presence of motion blur that causes noisy depth data and is capable of real-time performance with 1.8ms per frame. The experimental evaluations demonstrate the performance of the approach against robustness, accuracy and speed and compare the approach with the state of the art. A publicly available dataset with real-world data is also provided for future benchmarking.

Dense surface acquisition is one of the most challenging tasks for optical 3-D measurement systems in applications such as inspection of industrial parts, reverse engineering, digitization of virtual reality models and robot guidance. In order to achieve high accuracy we need good hardware equipment, as well as sophisticated data processing methods. Since measurements should be feasible under real world conditions, accuracy, reliability and consistency are a major issue. Based on the experiences with a previous system, a detailed analysis of the performance was carried out, leading to a new hardware setup. On the software side, we improved our calibration procedure and replaced the phase shift technique previously used by a new processing scheme which we call line shift processing. This paper describes our new approach. Results are presented and compared to results derived from the previous system.

This paper presents a new coded structured light pattern which permits to solve the correspondence problem by a single shot and without using geometrical constraints. The pattern is composed by the projection of a grid made by coloured slits in such a way that each slit with its two neighbours appears only once in the pattern. The technique proposed permits a rapid and robust 3D scene measurement, even with moving objects. Ó 1998 Elsevier Science B.V. All rights reserved.

The “German Traffic Sign Recognition Benchmark” is a multi-category classification competition held at IJCNN 2011. Automatic recognition of traffic signs is required in advanced driver assistance systems and constitutes a challenging real-world computer vision and pattern recognition problem. A comprehensive, lifelike dataset of more than 50,000 traffic sign images has been collected. It reflects the strong variations in visual appearance of signs due to distance, illumination, weather conditions, partial occlusions, and rotations. The images are complemented by several precomputed feature sets to allow for applying machine learning algorithms without background knowledge in image processing. The dataset comprises 43 classes with unbalanced class frequencies. Participants have to classify two test sets of more than 12,500 images each. Here, the results on the first of these sets, which was used in the first evaluation stage of the two-fold challenge, are reported. The methods employed by the participants who achieved the best results are briefly described and compared to human traffic sign recognition performance and baseline results.


Question: Presenting MVTec ITODD - A dataset designed for 3D object recognition in industrial settings.

This document outlines a novel technique for recognizing instances of 3D objects within a single camera image and accurately determining their 3D poses, focusing on the object's geometric information without relying on texture or reflectance details. This method, applicable to a broad spectrum of industrial and robotic scenarios such as bin-picking, employs a hierarchical view-based strategy that overcomes limitations of prior approaches. It effectively manages true perspective, demonstrates resilience against noise, occlusions, and clutter, and maintains invariance to contrast variations. A key innovation is the introduction of a model image generation method that accounts for scale-space effects, complemented by a similarity-based aspect graph for deriving essential object views. By merging the thoroughness of exhaustive search with the efficiency of hierarchical search, this technique achieves notable precision in pose determination: a positional accuracy of up to 0.12 percent relative to object distance, and an angular accuracy of up to 0.35 degrees, as observed in testing. The duration of recognition is largely unaffected by object complexity but is influenced by the pose range within which the object may be observed by the camera. For practical efficiency, the method supports the customization of the pose range based on specific application requirements, with typical processing times measured in a few hundred milliseconds.

Nowadays computer vision technology is helping the visually impaired by recognizing objects in their surroundings. Unlike research of navigation and wayfinding, there are no camera-based systems available in the market to find personal items for the blind. This paper proposes an object recognition method to help blind people find missing items using Speeded-Up Robust Features (SURF). SURF features can extract distinctive invariant features that can be utilized to perform reliable matching between different images in multiple scenarios. These features are invariant to image scale, translation, rotation, illumination, and partial occlusion. The proposed recognition process begins by matching individual features of the user queried object to a database of features with different personal items which are saved in advance. Experiment results demonstrate the effectiveness and efficiency of the proposed method.

Recognizing object classes and their 3D viewpoints is an important problem in computer vision. Based on a part-based probabilistic representation [31], we propose a new 3D object class model that is capable of recognizing unseen views by pose estimation and synthesis. We achieve this by using a dense, multiview representation of the viewing sphere parameterized by a triangular mesh of viewpoints. Each triangle of viewpoints can be morphed to synthesize new viewpoints. By incorporating 3D geometrical constraints, our model establishes explicit correspondences among object parts across viewpoints. We propose an incremental learning algorithm to train the generative model. A cellphone video clip of an object is first used to initialize model learning. Then the model is updated by a set of unsorted training images without viewpoint labels. We demonstrate the robustness of our model on object detection, viewpoint classification and synthesis tasks. Our model performs superiorly to and on par with state-of-the-art algorithms on the Savarese et al. 2007 and PASCAL datasets in object detection. It outperforms all previous work in viewpoint classification and offers promising results in viewpoint synthesis.

Dense surface acquisition is one of the most challenging tasks for optical 3-D measurement systems in applications such as inspection of industrial parts, reverse engineering, digitization of virtual reality models and robot guidance. In order to achieve high accuracy we need good hardware equipment, as well as sophisticated data processing methods. Since measurements should be feasible under real world conditions, accuracy, reliability and consistency are a major issue. Based on the experiences with a previous system, a detailed analysis of the performance was carried out, leading to a new hardware setup. On the software side, we improved our calibration procedure and replaced the phase shift technique previously used by a new processing scheme which we call line shift processing. This paper describes our new approach. Results are presented and compared to results derived from the previous system.

We present a novel approach for model-based 6D pose refinement in color data. Building on the established idea of contour-based pose tracking, we teach a deep neural network to predict a translational and rotational update. At the core, we propose a new visual loss that drives the pose update by aligning object contours, thus avoiding the definition of any explicit appearance model. In contrast to previous work our method is correspondence-free, segmentation-free, can handle occlusion and is agnostic to geometrical symmetry as well as visual ambiguities. Additionally, we observe a strong robustness towards rough initialization. The approach can run in real-time and produces pose accuracies that come close to 3D ICP without the need for depth data. Furthermore, our networks are trained from purely synthetic data and will be published together with the refinement code at http://campar.in.tum. de/Main/FabianManhardt to ensure reproducibility.

Touch has become a central input modality for a wide variety of interactive devices, most of our mobile devices are operated using touch. In addition to interacting with digital artifacts, people touch and interact with many other objects in their daily lives. We provide a unique photo dataset containing all touched objects over the last 11 years. All photos were contributed by Alberto Frigo, who was involved early on in the "Quantified Self" movement. He takes photos of every object he touches with his dominant hand. We analyzed the 258,218 images with respect to the types objects, their distribution, and related activities.

We present a survey of the most significant techniques, used in the last few years, concerning the coded structured light methods employed to get 3D information. In fact, depth perception is one of the most important subjects in computer vision. Stereovision is an attractive and widely used method, but, it is rather limited to make 3D surface maps, due to the correspondence problem. The correspondence problem can be improved using a method based on structured light concept, projecting a given pattern on the measuring surfaces. However, some relations between the projected pattern and the reflected one must be solved. This relationship can be directly found codifying the projected light, so that, each imaged region of the projected pattern carries the needed information to solve the correspondence problem. ( 1998 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Pattern projection Correspondence problem Active stereo Depth perception Range data Computer vision.

3D object categorization is a non-trivial task in computer vision encompassing many real-world applications. We pose the problem of categorizing 3D polygon meshes as learning appearance evolution from multi-view 2D images. Given a corpus of 3D polygon meshes, we first render the corresponding RGB and depth images from multiple viewpoints on a uniform sphere. Using rank pooling, we propose two methods to learn the appearance evolution of the 2D views. Firstly, we train view-invariant models based on a deep convolutional neural network (CNN) using the rendered RGB-D images and learn to rank the first fully connected layer activations and, therefore, capture the evolution of these extracted features. The parameters learned during this process are used as the 3D shape representations. In the second method, we learn the aggregation of the views from the outset by employing the ranking machine to the rendered RGB- D images directly, which produces aggregated 2D images which we term as ``3D shape images". We then learn CNN models on this novel shape representation for both RGB and depth which encode salient geometrical structure of the polygon. Experiments on the ModelNet40 and ModelNet10 datasets show that the proposed method consistently outperforms existing state-of-the-art algorithms in 3D shape recognition.

Autonomous vehicles, surveillance systems, face detection systems lead to the development of accurate object detection system [1]. These systems recognize, classify and localize every object in an image by drawing bounding boxes around the object [2]. These systems use existing classification models as backbone for Object Detection purpose. Object detection is the process of finding instances of real-world objects such as human faces, animals and vehicles etc., in pictures, images or in videos. An Object detection algorithm uses extracted features and learning techniques to recognize the objects in an image. In this paper, various Object Detection techniques have been studied and some of them are implemented. As a part of this paper, three algorithms for object detection in an image were implemented and their results were compared. The algorithms are “Object Detection using Deep Learning Framework by OpenCV”, “Object Detection using Tensorflow” and “Object Detection using Keras models”.

We present ShapeNet: a richly-annotated, large-scale repository of shapes represented by 3D CAD models of objects. ShapeNet contains 3D models from a multitude of semantic categories and organizes them under the WordNet taxonomy. It is a collection of datasets providing many semantic annotations for each 3D model such as consistent rigid alignments, parts and bilateral symmetry planes, physical sizes, keywords, as well as other planned annotations. Annotations are made available through a public web-based interface to enable data visualization of object attributes, promote data-driven geometric analysis, and provide a large-scale quantitative benchmark for research in computer graphics and vision. At the time of this technical report, ShapeNet has indexed more than 3,000,000 models, 220,000 models out of which are classified into 3,135 categories (WordNet synsets). In this report we describe the ShapeNet effort as a whole, provide details for all currently available datasets, and summarize future plans.

Autonomous vehicles require knowledge of the surrounding road layout, which can be predicted by state-of-the-art CNNs. This work addresses the current lack of data for determining lane instances, which are needed for various driving manoeuvres. The main issue is the timeconsuming manual labelling process, typically applied per image. We notice that driving the car is itself a form of annotation. Therefore, we propose a semi-automated method that allows for efficient labelling of image sequences by utilising an estimated road plane in 3D based on where the car has driven and projecting labels from this plane into all images of the sequence. The average labelling time per image is reduced to 5 seconds and only an inexpensive dash-cam is required for data capture. We are releasing a dataset of 24,000 images and additionally show experimental semantic segmentation and instance segmentation results.