Question: Handling hardware device failures through software tolerance methods

This paper presents the C Intermediate Language (CIL), a sophisticated framework designed for the analysis and transformation of C programs. Unlike standard C, CIL features a reduced set of constructs, simplifying complex C elements into more manageable forms, thereby operating at a level below abstract-syntax trees. However, CIL stands out as more abstract than conventional intermediate languages typically employed in compilation, such as three-address code. This unique position enables seamless analysis and manipulation of C programs while preserving the original source's likeness in the output. Accompanying the language is a powerful front-end capable of translating not only ANSI C programs but also those utilizing Microsoft C or GNU C extensions into CIL. The paper delves into the intricacies of CIL's structure, with a particular emphasis on its mechanisms for clarifying the most perplexing aspects of C for program analysis and transformation. A notable inclusion is the description of a whole-program merger, based on structural type equality, which facilitates viewing an entire project as a single compilation unit. To illustrate the practical utility of CIL, the paper details a transformation aimed at fortifying code against stack-smashing attacks. Currently, CIL is an integral part of a system dedicated to analyzing and instrumenting C programs with runtime checks to guarantee type safety. Our experience with CIL in this endeavor has been highly positive, and we are confident in its potential for application in various other scenarios. In a related context, the development of robust software capable of withstanding hardware device failures presents a significant challenge. This requires not only advanced programming techniques but also a deep understanding of both software and hardware interactions. CIL, with its capacity for precise analysis and transformation, could potentially play a pivotal role in crafting software that is resilient to such failures. By enabling the creation of more secure and reliable code, CIL might offer a solution to the problem of hardware device failures in software, thereby enhancing system stability and performance.

Various kinds of failures may occur in electronic products because of their life cycle environmental conditions including temperature, humidity, shock and vibration. Failure mechanism models are available to estimate the time to failure for most of these failures. Hence if the life cycle environment of a product can be determined, it is possible to assess the amount of damage induced and predict when the product might fail. This paper presents a life consumption monitoring methodology to determine the remaining life of a product. A battery powered data recorder is used to monitor the temperature, shock and vibration loads on a printed circuit board assembly placed under the hood of an automobile. The recorded data is used in conjunction with physics-of-failure models to determine the damage accumulation in the solder joints due to temperature and vibration loading. The remaining life of the solder joints of the test board is then obtained from the damage accumulation information. 1. Reliability Prediction Reliability is defined as the ability of a product to perform as intended (i.e., without failure and within specified performance limits) for a specified time, in its life cycle application environment. Over time, technology improvements have lead to higher I/O counts in components and circuit cards of electronic products. This has resulted in significant downsizing of interconnect thicknesses, making them more vulnerable in field applications. At the same time, increase in warranties and severe liabilities of electronic product failure have compelled manufacturers to predict the reliability of their products in field applications. An efficient reliability prediction scheme can be used for many purposes including [1]: • Logistics support (e.g., forecast warranty and life cycle costs, spare parts provisioning, availability)

Software cache-based side channel attacks present serious threats to modern computer systems. Using caches as a side channel, these attacks are able to derive secret keys used in cryptographic operations through legitimate activities. Among existing countermeasures, software solutions are typically application specific and incur substantial performance overhead. Recent hardware proposals including the Partition-Locked cache (PLcache) and Random-Permutation cache (RPcache) [23], although very effective in reducing performance overhead while enhancing the security level, may still be vulnerable to advanced cache attacks. In this paper, we propose three hardware-software approaches to defend against software cache-based attacks - they present different tradeoffs between hardware complexity and performance overhead. First, we propose to use preloading to secure the PLcache. Second, we leverage informing loads, which is a lightweight architectural support originally proposed to improve memory performance, to protect the RPcache. Third, we propose novel software permutation to replace the random permutation hardware in the RPcache. This way, regular caches can be protected with hardware support for informing loads. In our experiments, we analyze various processor models for their vulnerability to cache attacks and demonstrate that even to the processor model that is most vulnerable to cache attacks, our proposed software-hardware integrated schemes provide strong security protection.

As the complexity of distributed computing systems increases, systems management tasks require significantly higher levels of automation; examples include diagnosis and prediction based on real-time streams of computer events, setting alarms, and performing continuous monitoring. The core of autonomic computing, a recently proposed initiative towards next-generation IT-systems capable of 'self-healing', is the ability to analyze data in real-time and to predict potential problems. The goal is to avoid catastrophic failures through prompt execution of remedial actions.This paper describes an attempt to build a proactive prediction and control system for large clusters. We collected event logs containing various system reliability, availability and serviceability (RAS) events, and system activity reports (SARs) from a 350-node cluster system for a period of one year. The 'raw' system health measurements contain a great deal of redundant event data, which is either repetitive in nature or misaligned with respect to time. We applied a filtering technique and modeled the data into a set of primary and derived variables. These variables used probabilistic networks for establishing event correlations through prediction algorithms. We also evaluated the role of time-series methods, rule-based classification algorithms and Bayesian network models in event prediction.Based on historical data, our results suggest that it is feasible to predict system performance parameters (SARs) with a high degree of accuracy using time-series models. Rule-based classification techniques can be used to extract machine-event signatures to predict critical events with up to 70% accuracy.

Virtual machines were developed by IBM in the 1960’s to provide concurrent, interactive access to a mainframe computer. Each virtual machine is a replica of the underlying physical machine and users are given the illusion of running directly on the physical machine. Virtual machines also provide benefits like isolation and resource sharing, and the ability to run multiple flavors and configurations of operating systems. VMware Workstation brings such mainframe-class virtual machine technology to PC-based desktop and workstation computers. This paper focuses on VMware Workstation’s approach to virtualizing I/O devices. PCs have a staggering variety of hardware, and are usually pre-installed with an operating system. Instead of replacing the pre-installed OS, VMware Workstation uses it to host a user-level application (VMApp) component, as well as to schedule a privileged virtual machine monitor (VMM) component. The VMM directly provides high-performance CPU virtualization while the VMApp uses the host OS to virtualize I/O devices and shield the VMM from the variety of devices. A crucial question is whether virtualizing devices via such a hosted architecture can meet the performance required of high throughput, low latency devices. To this end, this paper studies the virtualization and performance of an Ethernet adapter on VMware Workstation. Results indicate that with optimizations, VMware Workstation’s hosted virtualization architecture can match native I/O throughput on standard PCs. Although a straightforward hosted implementation is CPU-limited due to virtualization overhead on a 733 MHz Pentium R III system on a 100 Mb/s Ethernet, a series of optimizations targeted at reducing CPU utilization allows the system to match native network throughput. Further optimizations are discussed both within and outside a hosted architecture.

Mounting concerns over variability, defects, and noise motivate a new approach for digital circuitry: stochastic logic, that is to say, logic that operates on probabilistic signals and so can cope with errors and uncertainty. Techniques for probabilistic analysis of circuits and systems are well established. We advocate a strategy for synthesis. In prior work, we described a methodology for synthesizing stochastic logic, that is to say logic that operates on probabilistic bit streams. In this paper, we apply the concept of stochastic logic to a reconfigurable architecture that implements processing operations on a datapath. We analyze cost as well as the sources of error: approximation, quantization, and random fluctuations. We study the effectiveness of the architecture on a collection of benchmarks for image processing. The stochastic architecture requires less area than conventional hardware implementations. Moreover, it is much more tolerant of soft errors (bit flips) than these deterministic implementations. This fault tolerance scales gracefully to very large numbers of errors.

Modern vehicles are evolving with more electronic components than ever before (In this paper, “vehicle” means “automotive vehicle.” It is also equal to “car.”) One notable example is graphical processing unit (GPU), which is a key component to implement a digital cluster. To implement the digital cluster that displays all the meters (such as speed and fuel gauge) together with infotainment services (such as navigator and browser), the GPU needs to be virtualized; however, GPU virtualization for the digital cluster has not been addressed yet. This paper presents a Virtualized Automotive DIsplay (VADI) system to virtualize a GPU and its attached display device. VADI manages two execution domains: one for the automotive control software and the other for the in-vehicle infotainment (IVI) software. Through GPU virtualization, VADI provides GPU rendering to both execution domains, and it simultaneously displays their images on a digital cluster. In addition, VADI isolates GPU from the IVI software in order to protect it from potential failures of the IVI software. We implement VADI with Vivante GC2000 GPU and perform experiments to ensure requirements of International Standard Organization (ISO) safety standards. The results show that VADI guarantees 30 frames per second (fps), which is the minimum frame rate for digital cluster mandated by ISO safety standards even with the failure of the IVI software. It also achieves 60 fps in a synthetic workload.

This paper considers current paradigms in computing and outlines the most important aspects concerning their reliability. The Fog computing paradigm as a non-trivial extension of the Cloud is considered and the reliability of the networks of smart devices are discussed. Combining the reliability requirements of grid and cloud paradigms with the reliability requirements of networks of sensor and actuators it follows that designing a reliable Fog computing platform is feasible.

Ideally, a measure of the security of a system should capture quantitatively the intuitive notion of ‘the ability of the system to resist attack’. That is, it should be operational, reflecting the degree to which the system can be expected to remain free of security breaches under particular conditions of operation (including attack). Instead, current security levels at best merely reflect the extensiveness of safeguards introduced during the design and development of a system. Whilst we might expect a system developed to a higher level than another to exhibit ‘more secure behaviour’ in operation, this cannot be guaranteed; more particularly, we cannot infer what the actual security behaviour will be from knowledge of such a level. In the paper we discuss similarities between reliability and security with the intention of working towards measures of ‘operational security’ similar to those that we have for reliability of systems. Very informally, these measures could involve expressions such as the rate of occurrence of security breaches (cf rate of occurrence of failures in reliability), or the probability that a specified ‘mission’ can be accomplished without a security breach (cf reliability function). This new approach is based on the analogy between system failure and security breach. A number of other analogies to support this view are introduced. We examine this duality critically, and have identified a number of important open questions that need to be answered before this quantitative approach can be taken further. The work described here is therefore somewhat tentative, and one of our major intentions is to invite discussion about the plausibility and feasibility of this new approach. Towards Operational Measures of Computer Security 2 _____________________________________________________________________

When systems fail in the field, logged error or warning messages are frequently the only evidence available for assessing and diagnosing the underlying cause. Consequently, the efficacy of such logging—how often and how well error causes can be determined via postmortem log messages—is a matter of significant practical importance. However, there is little empirical data about how well existing logging practices work and how they can yet be improved. We describe a comprehensive study characterizing the efficacy of logging practices across five large and widely used software systems. Across 250 randomly sampled reported failures, we first identify that more than half of the failures could not be diagnosed well using existing log data. Surprisingly, we find that majority of these unreported failures are manifested via a common set of generic error patterns (e.g., system call return errors) that, if logged, can significantly ease the diagnosis of these unreported failure cases. We further mechanize this knowledge in a tool called Errlog , that proactively adds appropriate logging statements into source code while adding only 1.4% performance overhead. A controlled user study suggests that Errlog can reduce diagnosis time by 60.7%.

The paper describes architectural enhancements to Xilinx FPGAs that provide better support for the creation of dynamically reconfigurable designs. These are augmented by a new design methodology that uses pre-routed IP cores for communication between static and dynamic modules and permits static designs to route through regions otherwise reserved for dynamic modules. A new CAD tool flow to automate the methodology is also presented. The new tools initially target the Virtex-II, Virtex-II Pro and Virtex-4 families and are derived from Xilinx's commercial CAD tools


Question: Handling hardware device failures through software tolerance methods

This paper delves into the meticulous static analysis of device drivers to enhance software resilience against hardware failures. It is widely acknowledged that kernel-level device driver bugs account for a staggering 85% of system crashes in the Windows XP operating system [44]. A significant contributor to these bugs is the inherent complexity of the Windows driver API, which necessitates a deep understanding of its intricate rules for proper driver development. To address this challenge, we have engineered a sophisticated static analysis engine capable of detecting API usage errors in C programs. Our tool, the Static Driver Verifier (SDV), harnesses this engine to pinpoint kernel API misuse in device drivers. SDV is equipped with comprehensive models of the operating system and the driver's environmental context, alongside an extensive library of over sixty API usage guidelines. The tool is designed for seamless integration into the workflow of driver developers, requiring no additional user input for operation and ensuring a minimal rate of false error reports. We elucidate the methodologies employed in SDV to meet these stringent demands and present empirical findings from its application on a diverse array of over one hundred Windows device drivers, highlighting its efficacy in bolstering software's ability to withstand hardware failures.

Chain replication is a new approach to coordinating clusters of fail-stop storage servers. The approach is intended for supporting large-scale storage services that exhibit high throughput and availability without sacrificing strong consistency guarantees. Besides outlining the chain replication protocols themselves, simulation experiments explore the performance characteristics of a prototype implementation. Throughput, availability, and several objectplacement strategies (including schemes based on distributed hash table routing) are discussed.

This paper provides a step-by-step guideline for the assessment of an automotive safety microprocessor with ISO 26262 hardware requirements. ISO 26262 part 5 - Product development at the hardware level - specifies the safety activities during the phase of the automotive hardware development. In this phase, hardware safety design is derived (from the results of ISO 26262 part 3 and 4), implemented, integrated, and tested. To prove the compliance with ISO 26262 hardware development process, quantitative evaluations on the hardware are indispensable. These quantitative evaluations are known as hardware architecture metrics and probabilistic hardware metrics. The assessment results qualify a design with an automotive safety integrity level (ASIL) which ranges from ASIL-A (lowest) to ASIL-D (highest). In this paper, we implemented an exemplary safety microprocessor to demonstrate the ISO 26262 hardware assessment process. The derivation procedures of the ASIL level from the hardware architecture metrics and probabilistic hardware metrics are fully discussed. Based on the evaluation results, we also provide design suggestions for the ISO 26262 safety hardware design.

We describe a new approach to building high performance nonstop infrastructure for scalable AI and cloud computing. With AI as the technological driving force behind future smart cities and smart societies, we will need powerful new nonstop AI infrastructures at both the cloud level and at the edge.

This paper presents an ESD self-protection scheme for a 700V high-voltage laterally diffused metal-oxide-semiconductor (LDMOS) field effect transistor. The safe operating area (SOA) and breakdown failure mechanism of 700V LDMOS are analyzed using simulations and experimental results. The scalability of thermal failure current with LDMOS width is also demonstrated.

Defining a processor micro-architecture for a targeted productspace involves multi-dimensional optimization across performance, power and reliability axes. A key decision in sucha definition process is the circuit-and technology-driven parameterof the nominal (voltage, frequency) operating point. This is a challenging task, since optimizing individually orpair-wise amongst these metrics usually results in a designthat falls short of the specification in at least one of the threedimensions. Aided by academic research, industry has nowadopted early-stage definition methodologies that considerboth energy-and performance-related metrics. Reliabilityrelatedenhancements, on the other hand, tend to get factoredin via a separate thread of activity. This task is typically pursuedwithout thorough pre-silicon quantifications of the energyor even the performance cost. In the late-CMOS designera, reliability needs to move from a post-silicon afterthoughtor validation-only effort to a pre-silicon definitionprocess. In this paper, we present BRAVO, a methodologyfor such reliability-aware design space exploration. BRAVOis supported by a multi-core simulation framework that integratesperformance, power and reliability modeling capability. Errors induced by both soft and hard fault incidence arecaptured within the reliability models. We introduce the notionof the Balanced Reliability Metric (BRM), that we useto evaluate overall reliability of the processor across soft andhard error incidences. We demonstrate up to 79% improvementin reliability in terms of this metric, for only a 6% dropin overall energy efficiency over design points that maximizeenergy efficiency. We also demonstrate several real-life usecaseapplications of BRAVO in an industrial setting.

It has been suggested that there is more than one reason for a software development project to fail. However, most of the literature that discusses project failure tends to be rather general, supplying us with lists of risk and failure factors, and focusing on the negative business effects of the failure. Very little research has attempted an in-depth investigation of a number of failed projects to identify exactly what are the factors behind the failure. In this research we analyze data from 70 failed projects. This data provides us with practitionerspsila perspectives on 57 development and management factors for projects they considered were failures. Our results show that all projects we investigated suffered from numerous failure factors. For a single project the number of such factors ranges from 5 to 47. While there does not appear to be any overarching set of failure factors we discovered that all of the projects suffered from poor project management. Most projects additionally suffered from organizational factors outside the project managerpsilas control. We conclude with suggestions for minimizing the four most common failure factors.

Trading small amounts of output accuracy to obtain significant performance improvement or reduce power consumption is a promising research direction. Many modern applications from different domains such as image and signal processing, computer vision, data mining, machine learning and speech recognition are amenable to approximate computing. In many cases, their output is interpreted by humans, so infrequent variation in outputs are not noticeable by users. Moreover, the input of these applications is noisy which prevents strict constraints on the output of the system. Hence, these domains have significant potential for the aforementioned trade-offs, and these applications are ideal targets for approximate computing. Existing approaches for approximate computing include modifying the ISA [6], compiler [1], programming language [2, 4], underlying hardware [14] or the entire framework [3, 8, 10]. Approximation accelerators [5, 7, 13] use some of these methods to trade off accuracy for higher performance or energy savings. These accelerators require the programmer to annotate code sections which are amenable for approximation. Approximate parts are offloaded to the accelerator and other parts are executed by the CPU at run time. Esmaeilzadeh et. al. [7] proposed a Neural Processing Unit (NPU) as a programmable approximate accelerator. The key idea is to train a neural network (nn) to mimic an approximable region of original code and replace that with an efficient computation of the learned model. Although the proposed approximate accelerators such as NPU demonstrate acceptable results on the benchmarks from different domains, they have some shortcomings. Firstly, different invocations of a program might produce varying output qualities because the output quality is dependent on the input values. Consequently, using an NPU with fixed configuration for a wide range of inputs may produce outputs with poor accuracies. On the other hand, if the output quality drops below a determined threshold, one way to improve the quality is re-executing the whole program on the exact hardware. However, the overhead of this recovery process may offset the gains of approximation. Secondly, existing techniques usually measure the output quality by averaging errors in individual output elements, e.g., pixels in an image. Previous works in approximate computing [9, 11, 12] show that although most of the output elements have small errors, there exist a few output elements that have considerably large errors which may degrade the whole user experience. Therefore, some inputs might need more specialized methods to deal with the issue of output quality. Lastly, it is challenging to tune the output quality of an approximate hardware dynamically. If different NPUs with different configurations are available in the system, defining the number of active NPUs per invocation will be a reasonable knob for changing output quality based on users preferences. NPU Output Input Monolithic NPU

This work presents a novel design for input ESD protection. By replacing the protection resistor with an active switch that isolates the input transistors from the pad under ESD stress, the ESD robustness can be greatly improved. The proposed designs were designed and verified in a 40-nm CMOS process using only thin oxide devices, which can successfully pass the typical industry ESD-protection specifications of 2-kV HBM and 200-V MM ESD tests.

Cloud computing is an increasingly important solution for providing services deployed in dynamically scalable cloud networks. Services in the cloud computing networks may be virtualized with specific servers which host abstracted details. Some of the servers are active and available, while others are busy or heavy loaded, and the remaining are offline for various reasons. Users would expect the right and available servers to complete their application requirements. Therefore, in order to provide an effective control scheme with parameter guidance for cloud resource services, failure detection is essential to meet users' service expectations. It can resolve possible performance bottlenecks in providing the virtual service for the cloud computing networks. Most existing Failure Detector (FD) schemes do not automatically adjust their detection service parameters for the dynamic network conditions, thus they couldn't be used for actual application. This paper explores FD properties with relation to the actual and automatic fault-tolerant cloud computing networks, and find a general non-manual analysis method to self-tune the corresponding parameters to satisfy user requirements. Based on this general automatic method, we propose specific and dynamic Self-tuning Failure Detector, called SFD, as a major breakthrough in the existing schemes. We carry out actual and extensive experiments to compare the quality of service performance between the SFD and several other existing FDs. Our experimental results demonstrate that our scheme can automatically adjust SFD control parameters to obtain corresponding services and satisfy user requirements, while maintaining good performance. Such an SFD can be extensively applied to industrial and commercial usage, and it can also significantly benefit the cloud computing networks.

In flash storage, an FTL is a complex piece of code that resides completely inside the storage device and is provided by the manufacturer. Its principal virtue is providing interoperability with conventional HDDs. However, this virtue is also its biggest impediment in reaching the full performance of the underlying flash storage. We propose to refactor the flash storage architecture so that it relies on a new block I/O interface which does not permit overwriting of data without intervening erasures.We demonstrate how high-level applications, in particular file systems, can deal with this restriction efficiently by employing append-only segments. This refactoring dramatically reduces flash management overhead and improves performance of applications, such as file systems and databases, by permitting them to directly manage flash storage. Our experiments on a machine with the new block I/O interface show that DRAM in the flash controller is reduced by 128X and the performance of the file system improves by 80% over conventional SSDs.


Question: A review of methods for tracking hands during interaction with objects.

This paper presents a comprehensive review of mathematical models aimed at understanding the human hand's functionality and capabilities in grasping and manipulation. The focus is on developing a toolkit for researchers and practitioners to study and describe the human hand's performance in various applications, distinct from medical or surgical contexts. The review explores hand models at three distinct levels of analysis: (1) Kinematic level, where the focus is on the movement of the hand's bones without considering the forces involved; (2) Musculotendon level, which examines the components of the hand responsible for generating forces and initiating motion; and (3) The integration of both kinematic and musculotendon structures, leading to a dynamic model of the hand that includes neurocontrol mechanisms. The aim is not only to offer a broad survey of existing hand modelling techniques but also to integrate recent findings and data, providing a holistic and up-to-date perspective on human hand modelling.

Recent years have seen advances in many enabling augmented reality technologies. Furthermore, much research has been carried out on how augmented reality can be used to enhance existing applications. This paper describes our experiences with an AR-museum guide that combines some of the latest technologies. Amongst other technologies, markerless tracking, hybrid tracking, and an ultra-mobile-PC were used. Like existing audio guides, the AR-guide can be used by any museum visitor, during a six-month exhibition on Islamic art. We provide a detailed description of the museumpsilas motivation for using AR, of our experiences in developing the system, and the initial results of user surveys. Taking this information into account, we can derive possible system improvements.

This paper discusses the problem of recognizing interaction-level human activities from a first-person viewpoint. The goal is to enable an observer (e.g., a robot or a wearable camera) to understand 'what activity others are performing to it' from continuous video inputs. These include friendly interactions such as 'a person hugging the observer' as well as hostile interactions like 'punching the observer' or 'throwing objects to the observer', whose videos involve a large amount of camera ego-motion caused by physical interactions. The paper investigates multi-channel kernels to integrate global and local motion information, and presents a new activity learning/recognition methodology that explicitly considers temporal structures displayed in first-person activity videos. In our experiments, we not only show classification results with segmented videos, but also confirm that our new approach is able to detect activities from continuous videos reliably.

We present an augmented reality system based on Kinect for on-line handbag shopping. The users can virtually try on different handbags on a TV screen at home. They can interact with the virtual handbags naturally, such as sliding a handbag to different positions on their arms and rotating a handbag to see it from different angles. The users can also see how the handbags fit them in different virtual environments other than the current real background. We describe the technical details for building such a system and demonstrate the experimental results.

In this paper, a robust, real-time object tracking approach capable of dealing with multiple symmetric and non-symmetric objects in a real-time requirement setting is proposed. The approach relies only on depth data to track multiple objects in a dynamic environment and uses random-forest based learning to deal with problems like object occlusion, motion-blur due to camera motion and clutter. We show that the relation between object motion and the corresponding change in its 3D point cloud data can be learned using only 6 random forests. A framework that unites object pose estimation and object pose tracking to efficiently track multiple objects in 3D space is presented. The approach is robust in tracking objects even in presence of motion blur that causes noisy depth data and is capable of real-time performance with 1.8ms per frame. The experimental evaluations demonstrate the performance of the approach against robustness, accuracy and speed and compare the approach with the state of the art. A publicly available dataset with real-world data is also provided for future benchmarking.

This paper presents preliminary results from a virtual reality (VR)-based system for hand rehabilitation that uses a CyberGlove and a Rutgers Master II-ND haptic glove. This computerized system trains finger range of motion, finger flexion speed, independence of finger motion, and finger strength using specific VR simulation exercises. A remote Web-based monitoring station was developed to allow telerehabilitation interventions. The remote therapist observes simplified versions of the patient exercises that are updated in real time. Patient data is stored transparently in an Oracle database, which is also Web accessible through a portal GUI. Thus the remote therapist or attending physician can graph exercise outcomes and thus evaluate patient outcomes at a distance. Data from the VR simulations is complemented by clinical measurements of hand function and strength. Eight chronic post-stroke subjects participated in a pilot study of the above system. In keeping with variability in both their lesion size and site and in their initial upper extremity function, each subject showed improvement on a unique combination of movement parameters in VR training. Importantly, these improvements transferred to gains on clinical tests, as well as to significant reductions in task-completion times for the prehension of real objects. These results are indicative of the potential feasibility of this exercise system for rehabilitation in patients with hand dysfunction resulting from neurological impairment.

Current touch input technologies are best suited for small and flat applications, such as smartphones, tablets and kiosks. In general, they are too expensive to scale to large surfaces, such as walls and furniture, and cannot provide input on objects having irregular and complex geometries, such as tools and toys. We introduce Electrick, a low-cost and versatile sensing technique that enables touch input on a wide variety of objects and surfaces, whether small or large, flat or irregular. This is achieved by using electric field tomography in concert with an electrically conductive material, which can be easily and cheaply added to objects and surfaces. We show that our technique is compatible with commonplace manufacturing methods, such as spray/brush coating, vacuum forming, and casting/molding enabling a wide range of possible uses and outputs. Our technique can also bring touch interactivity to rapidly fabricated objects, including those that are laser cut or 3D printed. Through a series of studies and illustrative example uses, we show that Electrick can enable new interactive opportunities on a diverse set of objects and surfaces that were previously static.

In this paper, we propose the Perceptual Shape Decomposition (PSD) to detect fingers for a Kinect-based hand gesture recognition system. The PSD is formulated as a discrete optimization problem by removing all negative minima with minimum cost. Experiments show that our PSD is perceptually relevant and robust against distortion and hand variations, and thus improves the recognition system performance. key words: Kinect camerta, hand gesture recognition, perceptual decomposition, finger detection

In this study, we propose integrated control of a robotic hand and arm using only proximity sensing from the fingertips. An integrated control scheme for the fingers and for the arm enables quick control of the position and posture of the arm by placing the fingertips adjacent to the surface of an object to be grasped. The arm control scheme enables adjustments based on errors in hand position and posture that would be impossible to achieve by finger motions alone, thus allowing the fingers to grasp an object in a laterally symmetric grasp. This can prevent grasp failures such as a finger pushing the object out of the hand or knocking the object over. Proposed control of the arm and hand allowed correction of position errors on the order of several centimeters. For example, an object on a workbench that is in an uncertain positional relation with the robot, with an inexpensive optical sensor such as a Kinect, which only provides coarse image data, would be sufficient for grasping an object.

Gesture-based interaction, as a natural way for human-computer interaction, has a wide range of applications in ubiquitous computing environment. This paper presents an acceleration-based gesture recognition approach, called FDSVM (Frame-based Descriptor and multi-class SVM), which needs only a wearable 3-dimensional accelerometer. With FDSVM, firstly, the acceleration data of a gesture is collected and represented by a frame-based descriptor, to extract the discriminative information. Then a SVM-based multi-class gesture classifier is built for recognition in the nonlinear gesture feature space. Extensive experimental results on a data set with 3360 gesture samples of 12 gestures over weeks demonstrate that the proposed FDSVM approach significantly outperforms other four methods: DTW, Naïve Bayes, C4.5 and HMM. In the user-dependent case, FDSVM achieves the recognition rate of 99.38% for the 4 direction gestures and 95.21% for all the 12 gestures. In the user-independent case, it obtains the recognition rate of 98.93% for 4 gestures and 89.29% for 12 gestures. Compared to other accelerometer-based gesture recognition approaches reported in literature FDSVM gives the best resulrs for both user-dependent and user-independent cases.

In this paper we present a new solution for real-time 3D camera pose estimation for Augmented Reality (AR) applications. The tracking system does not require special engineering of the environment, such as placing markers or beacons. The required input data are a CAD model of the target object to be tracked, and a calibrated reference image of it. We consider the whole process of camera tracking, and developed both an autonomous initialization and a real-time tracking procedure. The system is robust to abrupt camera motions, strong changes of the lighting conditions and partial occlusions. To avoid typical jitter and drift problems the tracker performs feature matching not only in an iterative manner, but also against stable reference features, which are dynamically cached in case of high confidence. We present experimental results generated with help of synthetic ground truth, real off-line and on-line image sequences using different types of target objects.


Question: A review of methods for tracking hands during interaction with objects.

This paper introduces an advanced real-time hand tracking framework that utilizes a single depth camera to achieve precise reconstruction of intricate hand postures across diverse subjects. The system is designed to ensure robust tracking, capable of swiftly recovering from any transient errors. Its most distinctive feature is its remarkable flexibility, which sets it apart from previous methodologies that were predominantly focused on close-range, front-facing scenarios. This enhanced versatility paves the way for a broad spectrum of human-computer interaction applications, encompassing long-range tracking for controlling devices from a distance (such as television), compatibility with moving depth cameras for mobile environments, and the ability to operate with arbitrary camera orientations (ideal for virtual reality headsets). These capabilities are underpinned by an innovative pipeline that integrates a multi-tiered discriminative reinitialization approach for frame-by-frame pose estimation, complemented by a generative model-fitting phase. The paper offers a comprehensive technical breakdown and a meticulous qualitative and quantitative evaluation to substantiate the system's performance and reliability.

A computer vision system has been implemented that can recognize threedimensional objects from unknown viewpoints in single gray-scale images. Unlike most other approaches, the recognition is accomplished without any attempt to reconstruct depth information bottom-up from the visual input. Instead, three other mechanisms are used that can bridge the gap between the two-dimensional image and knowledge of three-dimensional objects. First, a process of perceptual organization is used to form groupings and structures in the image that are likely to be invariant over a wide range of viewpoints. Second, a probabilistic ranking method is used to reduce the size of the search space during model based matching. Finally, a process of spatial correspondence brings the projections of three-dimensional models into direct correspondence with the image by solving for unknown viewpoint and model parameters. A high level of robustness in the presence of occlusion and missing data can be achieved through full application of a viewpoint consistency constraint. It is argued that similar mechanisms and constraints form the basis for recognition in human vision. This paper has been published in Artificial Intelligence, 31, 3 (March 1987), pp. 355–395.

The idea described in this paper is to use the built-in cameras of consumer mobile phones as sensors for 2-dimensional visual codes. Such codes can be attached to physical objects in order to retrieve object-related information and functionality. They are also suitable for display on electronic screens. The proposed visual code system allows the simultaneous detection of multiple codes, introduces a position-independent coordinate system, and provides the phone’s orientation as a parameter. The ability to detect objects in the user’s vicinity offers a natural way of interaction and strengthens the role of mobile phones in a large number of application scenarios. We describe the hardware requirements, the design of a suitable visual code, a lightweight recognition algorithm, and present some example applications.

This paper proposes a static hand gesture recognition method with low computation and memory consumption for wearable applications. The hand contour is chosen as the hand gesture feature and support vector machine is used to classify the feature. A multiscale weighted histogram of contour direction-based direction normalization is proposed to ensure good recognition performance. In order to improve efficiency, the proposed histogram only counts the direction of the contour point to focus on the most significant hand feature in the first-person view of wearable devices. Based on the hand’s anatomy, the proposed histogram is weighted by considering each contour point’s position and direction jointly using the direction-angle map, to ensure robustness. Experimental results show that the proposed method can give a recognition accuracy of 97.1% with a frame rate of 30 fps on a PC.

BodyAvatar is a Kinect-based interactive system that allows users without professional skills to create freeform 3D avatars using body gestures. Unlike existing gesture-based 3D modeling tools, BodyAvatar centers around a first-person "you're the avatar" metaphor, where the user treats their own body as a physical proxy of the virtual avatar. Based on an intuitive body-centric mapping, the user performs gestures to their own body as if wanting to modify it, which in turn results in corresponding modifications to the avatar. BodyAvatar provides an intuitive, immersive, and playful creation experience for the user. We present a formative study that leads to the design of BodyAvatar, the system's interactions and underlying algorithms, and results from initial user trials.

Simultaneous tracking of multiple persons in real-world environments is an active research field and several approaches have been proposed, based on a variety of features and algorithms. Recently, there has been a growing interest in organizing systematic evaluations to compare the various techniques. Unfortunately, the lack of common metrics for measuring the performance of multiple object trackers still makes it hard to compare their results. In this work, we introduce two intuitive and general metrics to allow for objective comparison of tracker characteristics, focusing on their precision in estimating object locations, their accuracy in recognizing object configurations and their ability to consistently label objects over time. These metrics have been extensively used in two large-scale international evaluations, the 2006 and 2007 CLEAR evaluations, to measure and compare the performance of multiple object trackers for a wide variety of tracking tasks. Selected performance results are presented and the advantages and drawbacks of the presented metrics are discussed based on the experience gained during the evaluations.

This paper introduces a novel prosthetic hand control architecture using inertial information for grasp prediction in order to reduce the cognitive burden of amputees. A pair of inertial measurement sensors (IMUs) are fitted on the wrist and bicep to record arm trajectory when reaching to grasp an object. Each object class can be associated with different methods for grasping and manipulation. An observation experiment was conducted to find the most common grasping methods for generic object classes: Very Small (VS), Small (S), and Medium (M). A Cup (CP) class was also examined to find differences in grasping habits for pick and place, and drinking applications. The resulting grasps were used to test the discriminatory ability of inertial motion features in the upper limb for VS, S and CP object classes. Subject experiments demonstrated an average classification success rate of 90.8%, 69.2% and 88.1% for VS, S and CP classes respectively when using a k-nearest neighbors algorithm with a Euclidean distance metric. The results suggest that inertial motion features have great potential to predict the grasp pattern during reach, and to the authors' knowledge, is the first IMU-based control strategy to utilize natural motion that is aimed at hand prosthesis control.

Gesture-based interaction, as a natural way for human-computer interaction, has a wide range of applications in ubiquitous computing environment. This paper presents an acceleration-based gesture recognition approach, called FDSVM (Frame-based Descriptor and multi-class SVM), which needs only a wearable 3-dimensional accelerometer. With FDSVM, firstly, the acceleration data of a gesture is collected and represented by a frame-based descriptor, to extract the discriminative information. Then a SVM-based multi-class gesture classifier is built for recognition in the nonlinear gesture feature space. Extensive experimental results on a data set with 3360 gesture samples of 12 gestures over weeks demonstrate that the proposed FDSVM approach significantly outperforms other four methods: DTW, Naïve Bayes, C4.5 and HMM. In the user-dependent case, FDSVM achieves the recognition rate of 99.38% for the 4 direction gestures and 95.21% for all the 12 gestures. In the user-independent case, it obtains the recognition rate of 98.93% for 4 gestures and 89.29% for 12 gestures. Compared to other accelerometer-based gesture recognition approaches reported in literature FDSVM gives the best resulrs for both user-dependent and user-independent cases.

While intelligent environments are often cited as a reason for doing work on visual person-tracking, really making an intelligent environment exposes many realworld problems in visual tracking that must be solved to make the technology practical. In the context of our EasyLiving project in intelligent environments, we created a practical person-tracking system that solves most of the real-world problems. It uses two sets of color stereo cameras for tracking multiple people during live demonstrations in a living room. The stereo images are used for locating people, and the color images are used for maintaining their identities. The system runs quickly enough to make the room feel responsive, and it tracks multiple people standing, walking, sitting, occluding, and entering and leaving the

In this paper we have proposed and designed FindingHuMo (Finding Human Motion), a real-time user tracking system for Smart Environments. FindingHuMo can perform device-free tracking of multiple (unknown and variable number of) users in the Hallway Environments, just from non-invasive and anonymous (not user specific) binary motion sensor data stream. The significance of our designed system are as follows: (a) fast tracking of individual targets from binary motion data stream from a static wireless sensor network in the infrastructure. This needs to resolve unreliable node sequences, system noise and path ambiguity, (b) Scaling for multi-user tracking where user motion trajectories may crossover with each other in all possible ways. This needs to resolve path ambiguity to isolate overlapping trajectories, FindingHumo applies the following techniques on the collected motion data stream: (i) a proposed motion data driven adaptive order Hidden Markov Model with Viterbi decoding (called Adaptive-HMM), and then (ii) an innovative path disambiguation algorithm (called CPDA). Using this methodology the system accurately detects and isolates motion trajectories of individual users. The system performance is illustrated with results from real-time system deployment experience in a Smart Environment.

In this paper we present a novel type of sensing technology for hand and finger gesture recognition that utilizes light in the invisible spectrum to detect changes in position and form of body tissue like tendons and muscles. The proposed system can be easily integrated with existing wearable devices. Our approach not only enables gesture recognition but it could potentially double to perform a variety of health related monitoring tasks (e.g. heart rate, stress).


Question: A review of methods for tracking hands during interaction with objects.

This paper introduces an innovative method for real-time, accurate, and robust hand pose estimation using an egocentric RGB-D camera in cluttered real-world scenarios. Existing techniques often struggle with hand-object interactions and cluttered scenes captured from a first-person perspective, a common challenge in virtual and augmented reality settings. Our solution employs a two-stage approach using Convolutional Neural Networks (CNNs) to localize and estimate 3D joint positions of the hand. Initially, a CNN is utilized to pinpoint the 2D location of the hand's center in the input image, even when the hand is partially obscured by clutter or objects. The identified hand position and associated depth information are then used to create a normalized, cropped image, which is fed into a second CNN to regress the relative 3D locations of hand joints in real time. To enhance accuracy, robustness, and temporal consistency, the pose estimates are refined using a kinematic pose tracking energy. To train the CNNs effectively, we have developed a new, highly realistic dataset. This dataset is created using a merged reality technique that captures and synthesizes extensive, annotated data of natural hand interactions within cluttered environments. Through comprehensive evaluation, we demonstrate that our method is highly resilient to self-occlusion and occlusions caused by objects, especially in dynamic egocentric viewpoints.

Touch has become a central input modality for a wide variety of interactive devices, most of our mobile devices are operated using touch. In addition to interacting with digital artifacts, people touch and interact with many other objects in their daily lives. We provide a unique photo dataset containing all touched objects over the last 11 years. All photos were contributed by Alberto Frigo, who was involved early on in the "Quantified Self" movement. He takes photos of every object he touches with his dominant hand. We analyzed the 258,218 images with respect to the types objects, their distribution, and related activities.

Although Augmented Reality technology was first developed over forty years ago, there has been little survey work giving an overview of recent research in the field. This paper reviews the ten-year development of the work presented at the ISMAR conference and its predecessors with a particular focus on tracking, interaction and display research. It provides a roadmap for future augmented reality research which will be of great value to this relatively young field, and also for helping researchers decide which topics should be explored when they are beginning their own studies in the area.

In this paper we present a markerless augmented reality (MAR) approach based on real-time 3D reconstruction using a low-cost depth camera, the Kinect. We intend to use this MAR approach for 3D medical visualization. In the actual state of developing, our approach is based on the KinectFusion algorithm with some minor adaptations, as long as our goal is not reconstruct scenes but use the reconstructed model as a reference to MAR. Keywords-Augmented Reality; 3D Reconstruction; Kinect;

Standard presentation systems consisting of a laptop connected to a projector suffer from two problems: (1) the projected image appears distorted (keystoned) unless the projector is precisely aligned to the projection screen; (2) the speaker is forced to interact with the computer rather than the audience. This paper shows how the addition of an uncalibrated camera, aimed at the screen, solves both problems. Although the locations, orientations and optical parameters of the camera and projector are unknown, the projector-camera system calibrates itself by exploiting the homography between the projected slide and the camera image. Significant improvements are possible over passively calibrating systems since the projector actively manipulates the environment by placing feature points into the scene. For instance, using a low-resolution (160x120) camera, we can achieve an accuracy of ±3 pixels in a 1024x768 presentation slide. The camera-projector system infers models for the projector-to-camera and projector-to-screen mappings in order to provide two major benefits. First, images sent to the projector are pre-warped in such a way that the distortions induced by the arbitrary projector-screen geometry are precisely negated. This enables projectors to be mounted anywhere in the environment – for instance, at the side of the room, where the speaker is less likely to cast shadows on the screen, and where the projector does not occlude the audience’s view. Second, the system detects the position of the user’s laser pointer dot in the camera image at 20Hz, allowing the laser pointer to emulate the pointing actions of a mouse. This enables the user to activate virtual buttons in the presentation (such as “next slide”) and draw on the projected image. The camera-assisted presentation system requires no special hardware (aside from the cheap camera) Rahul Sukthankar (rahuls@cs.cmu.edu) is now affiliated with Compaq Cambridge Research Lab and Carnegie Mellon; Robert Stockton and Matthew Mullin ({rstock,mmullin}@whizbang.com) are now with WhizBang! Labs. and runs on a standard laptop as a Java application. It is now used by the authors for all of their conference presentations.

We describea markerlesscamera tracking systemfor augmentedreality thatoperatesin environmentswhich contain oneor more planes. This is a commonspecialcase, which we showsignificantlysimplifiestracking. The result is a practical, reliable, vision-basedtracker. Furthermore, thetrackedplaneimposesa natural referenceframe, sothat thealignmentof thereal andvirtual coordinatesystemsis rather simplerthan would be the casewith a general structure-and-motionsystem.Multiple planescan be tracked, and additional data such as 2D point tracks are easilyincorporated.

Online 3D reconstruction of real-world scenes has been attracting increasing interests from both the academia and industry, especially with the consumer-level depth cameras becoming widely available. Recent most online reconstruction systems take live depth data from a moving Kinect camera and incrementally fuse them to a single high-quality 3D model in real time. Although most real-world scenes have static environment, the daily objects in a scene often move dynamically, which are non-trivial to reconstruct especially when the camera is also not still. To solve this problem, we propose a single depth camera-based real-time approach for simultaneous reconstruction of dynamic object and static environment, and provide solutions for its key issues. In particular, we first introduce a robust optimization scheme which takes advantage of raycasted maps to segment moving object and background from the live depth map. The corresponding depth data are then fused to the volumes, respectively. These volumes are raycasted to extract views of the implicit surface which can be used as a consistent reference frame for the next iteration of segmentation and tracking. Particularly, in order to handle fast motion of dynamic object and handheld camera in the fusion stage, we propose a sequential 6D pose prediction method which largely increases the registration robustness and avoids registration failures occurred in conventional methods. Experimental results show that our approach can reconstruct moving object as well as static environment with rich details, and outperform conventional methods in multiple aspects.

Human activity recognition is an important area of computer vision research. Its applications include surveillance systems, patient monitoring systems, and a variety of systems that involve interactions between persons and electronic devices such as human-computer interfaces. Most of these applications require an automated recognition of high-level activities, composed of multiple simple (or atomic) actions of persons. This article provides a detailed overview of various state-of-the-art research papers on human activity recognition. We discuss both the methodologies developed for simple human actions and those for high-level activities. An approach-based taxonomy is chosen that compares the advantages and limitations of each approach.
 Recognition methodologies for an analysis of the simple actions of a single person are first presented in the article. Space-time volume approaches and sequential approaches that represent and recognize activities directly from input images are discussed. Next, hierarchical recognition methodologies for high-level activities are presented and compared. Statistical approaches, syntactic approaches, and description-based approaches for hierarchical recognition are discussed in the article. In addition, we further discuss the papers on the recognition of human-object interactions and group activities. Public datasets designed for the evaluation of the recognition methodologies are illustrated in our article as well, comparing the methodologies' performances. This review will provide the impetus for future research in more productive areas.

In the past decade, multi-touch-sensitive interactive surfaces have transitioned from pure research prototypes in the lab, to commercial products with wide-spread adoption. One of the longer term visions of this research follows the idea of ubiquitous computing, where everyday surfaces in our environment are made interactive. However, most of current interfaces remain firmly tied to the traditional flat rectangular displays of the today's computers and while they benefit from the directness and the ease of use, they are often not much more than touch-enabled standard desktop interfaces.
 In this paper, we argue for explorations that transcend the traditional notion of the flat display, and envision interfaces that are curved, three-dimensional, or that cross the boundary between the digital and physical world. In particular, we present two research directions that explore this idea: (a) exploring the three-dimensional interaction space above the display and (b) enabling gestural and touch interactions on curved devices for novel interaction possibilities. To illustrate both of these, we draw examples from our own work and the work of others, and guide the reader through several case studies that highlight the challenges and benefits of such novel interfaces. The implications on media requirements and collaboration aspects are discussed in detail, and, whenever possible, we highlight promising directions of future research. We believe that the compelling application design for future non-flat user interfaces will greatly depend on exploiting the unique characteristics of the given form factor.

Active vision systems are usually limited to either partial or static scene reconstructions. In this paper, we propose to acquire the entire 3D shape of a dynamic scene. This is performed using a multiple projectors and cameras system, that allows to recover the entire shape of the object within a single scan at each frame. Like previous approaches, a static and simple pattern is used to avoid interferences of multiple patterns projected on the same object. In this paper, we extend the technique to capture a dense entire shape of a moving object with accuracy and high video frame rate. To achieve this, we mainly propose two additional steps; one is checking the consistency between the multiple cameras and projectors, and the other is an algorithm for light sectioning based on a plane parameter optimization. In addition, we also propose efficient noise reduction and mesh generation algorithm which are necessary for practical applications. In the experiments, we show that we can successfully reconstruct dense entire shapes of moving objects. Results are illustrated on real data from a system composed of six projectors and six cameras that was actually built.

This paper addresses the problem of simulating deformations between objects and the hand of a synthetic character during a grasping process. A numerical method based on finite element theory allows us to take into account the active forces of the fingers on the object and the reactive forces of the object on the fingers. The method improves control of synthetic human behavior in a task level animation system because it provides information about the environment of a synthetic human and so can be compared to the sense of touch. Finite element theory currently used in engineering seems one of the best approaches for modeling both elastic and plastic deformation of objects, as well as shocks with or without penetration between deformable objects. We show that intrinsic properties of the method based on composition/decomposition of elements have an impact in computer animation. We also state that the use of the same method for modeling both objects and human bodies improves the modeling both objects and human bodies improves the modeling of the contacts between them. Moreover, it allows a realistic envelope deformation of the human fingers comparable to existing methods. To show what we can expect from the method, we apply it to the grasping and pressing of a ball. Our solution to the grasping problem is based on displacement commands instead of force commands used in robotics and human behavior.


Question: A review of methods for tracking hands during interaction with objects.

This paper presents an efficient model-based approach for the 3D tracking of hand articulations using Kinect data. The challenge of tracking the intricate movements of human hands is both theoretically intriguing and practically significant, given its wide array of applications [1]. Building on previous efforts that primarily focused on markerless visual data, we introduce a model-based solution (Fig. 1) that stands in contrast to appearance-based methods [1]. Our approach frames the problem as an optimization task aimed at minimizing the difference between the hypothesized 3D hand model's structure and appearance, and the actual visual input. To solve this optimization problem, we employ a modified version of Particle Swarm Optimization (PSO) [2], a stochastic optimization technique. To enhance computational efficiency, the most resource-intensive components of our system are designed to run on a GPU [3]. This innovative setup enables real-time hand articulation tracking at a rate of 15Hz, showcasing the system's capability to handle complex hand movements in interaction with objects effectively.

In this paper we present a markerless augmented reality (MAR) approach based on real-time 3D reconstruction using a low-cost depth camera, the Kinect. We intend to use this MAR approach for 3D medical visualization. In the actual state of developing, our approach is based on the KinectFusion algorithm with some minor adaptations, as long as our goal is not reconstruct scenes but use the reconstructed model as a reference to MAR. Keywords-Augmented Reality; 3D Reconstruction; Kinect;

3D object temporal trackers estimate the 3D rotation and 3D translation of a rigid object by propagating the transformation from one frame to the next. To confront this task, algorithms either learn the transformation between two consecutive frames or optimize an energy function to align the object to the scene. The motivation behind our approach stems from a consideration on the nature of learners and optimizers. Throughout the evaluation of different types of objects and working conditions, we observe their complementary nature — on one hand, learners are more robust when undergoing challenging scenarios, while optimizers are prone to tracking failures due to the entrapment at local minima; on the other, optimizers can converge to a better accuracy and minimize jitter. Therefore, we propose to bridge the gap between learners and optimizers to attain a robust and accurate RGB-D temporal tracker that runs at approximately 2 ms per frame using one CPU core. Our work is highly suitable for Augmented Reality (AR), Mixed Reality (MR) and Virtual Reality (VR) applications due to its robustness, accuracy, efficiency and low latency. Aiming at stepping beyond the simple scenarios used by current systems, often constrained by having a single object in the absence of clutter, averting to touch the object to prevent close-range partial occlusion or selecting brightly colored objects to easily segment them individually, we demonstrate the capacity to handle challenging cases under clutter, partial occlusion and varying lighting conditions.

In this paper we have proposed and designed FindingHuMo (Finding Human Motion), a real-time user tracking system for Smart Environments. FindingHuMo can perform device-free tracking of multiple (unknown and variable number of) users in the Hallway Environments, just from non-invasive and anonymous (not user specific) binary motion sensor data stream. The significance of our designed system are as follows: (a) fast tracking of individual targets from binary motion data stream from a static wireless sensor network in the infrastructure. This needs to resolve unreliable node sequences, system noise and path ambiguity, (b) Scaling for multi-user tracking where user motion trajectories may crossover with each other in all possible ways. This needs to resolve path ambiguity to isolate overlapping trajectories, FindingHumo applies the following techniques on the collected motion data stream: (i) a proposed motion data driven adaptive order Hidden Markov Model with Viterbi decoding (called Adaptive-HMM), and then (ii) an innovative path disambiguation algorithm (called CPDA). Using this methodology the system accurately detects and isolates motion trajectories of individual users. The system performance is illustrated with results from real-time system deployment experience in a Smart Environment.

The aim of this thesis is to find new approaches to Sign Language Recognition (SLR) which are suited to working with the limited corpora currently available. Data available for SLR is of limited quality; low resolution and frame rates make the task of recognition even more complex. The content is rarely natural, concentrating on isolated signs and filmed under labo ratory conditions. In addition, the amount of accurately labelled data is minimal. To this end, several contributions are made: Tracking the hands is eschewed in favour of detection based techniques more robust to noise; for both signs and for linguistically-motivated sign sub-units are investigated, to make best use of limited data sets. Finally, an algorithm is proposed to learn signs fiom the inset signers on TV, with the aid of the accompanying subtitles, thus increasing the corpus of data available. Tracking fast moving hands under laboratory conditions is a complex task, move this to real world data and the challenge is even greater. When using tracked data as a base for SLR, the er rors in the tracking are compounded at the classification stage. Proposed instead, is a novel sign detection method, which views space-time as a 3D volume and the sign within it as an object to be located. Features are combined into strong classifiers using a novel boosting implementation designed to create optimal classifiers over sparse datasets. Using boosted volumetric features, on a robust frame differenced input, average classification rates reach 71% on seen signers and 66% on a mixture of seen and unseen signers, with individual sign classification rates gaining 95%. Using a classifier per sign approach to SLR, means that data sets need to contain numerous examples of the signs to be learnt. Instead, this thesis proposes lear nt classifiers to detect the common sub-units of sign. The responses of these classifiers can then be combined for recognition at the sign level. This approach requires fewer examples per sign to be learnt, since the sub-unit detectors are trained on data from multiple signs. It is also faster at detection time since there are fewer classifiers to consult, the number of these being limited by the linguistics of sign and not the number of signs being detected. For this method, appearance based boosted classifiers are introduced to distinguish the sub-units of sign. Results show that when combined with temporal models, these novel sub-unit classifiers, can outperform similar classifiers learnt on tracked results. As an added side effect; since the sub-units are linguistically derived tliey can be used independently to help linguistic annotators. Since sign language data sets are costly to collect and annotate, there are not many publicly available. Those which are, tend to be constiained in content and often taken under laboratory conditions. However, in the UK, the British Broadcasting Corporation (BBC) regularly pro duces programs with an inset signer and corresponding subtitles. This provides a natural signer, covering a wide range of topics, in real world conditions. While it has no ground truth, it is proposed tliat the tr anslated subtitles can provide weak labels for learning signs. The final con tributions of this tliesis, lead to an innovative approach to learn signs from these co-occurring streams of data. Using a unique, temporally constr ained, version of the Apriori mining algo rithm, similar sections of video are identified as possible sign locations. These estimates are improved upon by introducing the concept of contextual negatives, removing contextually sim ilar" noise. Combined with an iterative honing process, to enhance the localisation of the target sign, 23 word/sign combinations are learnt from a 30 minute news broadcast, providing a novel rnetliod for automatic data set creation.

This paper addresses the problem of simulating deformations between objects and the hand of a synthetic character during a grasping process. A numerical method based on finite element theory allows us to take into account the active forces of the fingers on the object and the reactive forces of the object on the fingers. The method improves control of synthetic human behavior in a task level animation system because it provides information about the environment of a synthetic human and so can be compared to the sense of touch. Finite element theory currently used in engineering seems one of the best approaches for modeling both elastic and plastic deformation of objects, as well as shocks with or without penetration between deformable objects. We show that intrinsic properties of the method based on composition/decomposition of elements have an impact in computer animation. We also state that the use of the same method for modeling both objects and human bodies improves the modeling both objects and human bodies improves the modeling of the contacts between them. Moreover, it allows a realistic envelope deformation of the human fingers comparable to existing methods. To show what we can expect from the method, we apply it to the grasping and pressing of a ball. Our solution to the grasping problem is based on displacement commands instead of force commands used in robotics and human behavior.

This paper presents a camera tracking system for the spatial stabilization of Augmented Reality (AR) media. Our approach integrates both artificial landmarks (fiducials) and natural features for camera tracking. Artificial landmarks are used for system initialization and computation of initial camera pose. Robust and extendible tracking is achieved by dynamically calibrating the 3D positions of a priori uncalibrated natural features. Analysis and experimental results demonstrate the effectiveness of this approach for presenting stabilized AR media in long camera-motion video sequences.

Pfinder is a real-time system for tracking people and interpreting their behavior. It runs at 10Hz on a standard SGI Indy computer, and has performed reliably on thousands of people in many different physical locations. The system uses a multiclass statistical model of color and shape to obtain a 2D representation of head and hands in a wide range of viewing conditions. Pfinder has been successfully used in a wide range of applications including wireless interfaces, video databases, and low-bandwidth coding. Index Terms —Blobs, blob tracking, real-time, person tracking, 3D person tracking, segmentation, gesture recognition, mixture model, MDL.

The use of gesture as a natural interface serves as a motivating force for research in modeling, analyzing and recognition of gestures. In particular, human computer intelligent interaction needs vision-based gesture recognition, which involves many interdisciplinary studies. A survey on recent vision-based gesture recognition approaches is given in this paper. We shall review methods of static hand posture and temporal gesture recognition. Several application systems of gesture recognition are also described in this paper. We conclude with some thoughts about future research directions.

An explicit three-dimensional representation is constructed from feature-points extracted from a sequence of images taken by a moving camera. The points are tracked through the sequence, and their 3D locations accurately determined by use of Kalman Filters. The ego-motion of the camera is also solved for.

The recent introduction of depth cameras like Leap Motion Controller allows researchers to exploit the depth information to recognize hand gesture more robustly. This paper proposes a novel hand gesture recognition system with Leap Motion Controller. A series of features are extracted from Leap Motion tracking data, we feed these features along with HOG feature extracted from sensor images into a multiclass SVM classifier to recognize performed gesture, dimension reduction and feature weighted fusion are also discussed. Our results show that our model is much more accurate than previous work.


Question: A review of methods for tracking hands during interaction with objects.

This document presents a real-time joint tracking system for hands and objects during manipulation and interaction, utilizing RGB-D input. Real-time tracking of hands and objects in tandem holds significant potential for augmented reality, tangible computing, and wearable technology applications. However, the task is complicated by factors such as occlusion, rapid movements, and the uniform appearance of hands, making it more challenging than tracking either component independently. Previous methods often require sophisticated multi-camera configurations to mitigate occlusion issues and frequently involve costly segmentation and optimization processes that hinder real-time tracking. Our proposed solution, designed for real-time operation, relies solely on a single RGB-D camera commonly available. The foundation of our approach is a 3D articulated Gaussian mixture alignment method specifically adapted for hand-object tracking, enabling rapid pose optimization. This alignment energy incorporates innovative regularizers to tackle occlusions and hand-object contact scenarios. To enhance robustness, the optimization process is guided by part classification of the hand and segmentation of the object using discriminative methods. We have extensively tested our method on various established datasets and have introduced a new annotated dataset focusing on hand-object interactions. Our quantitative and qualitative outcomes highlight the method's key benefits: speed, accuracy, and resilience.

**Question**: A Comparative Study of Machine Learning Techniques for the Prediction of Diabetes Mellitus

**Documents**: An ensemble learning approach for diabetes prediction using machine learning
Diabetes mellitus is a chronic disease characterized by high levels of blood glucose resulting from defects in insulin secretion, insulin action, or both. The early detection of diabetes is crucial for effective management and treatment. In this paper, we propose an ensemble learning approach for the prediction of diabetes using machine learning techniques. We have used various machine learning algorithms such as decision trees, support vector machines, and neural networks, and combined their predictions using an ensemble method. The proposed ensemble approach has been tested on the Pima Indians Diabetes Dataset, which is a well-known dataset in the field of diabetes research. The results obtained using the proposed ensemble approach have been compared with the results obtained using individual machine learning algorithms. The experimental results show that the proposed ensemble approach outperforms the individual machine learning algorithms in terms of accuracy and other performance metrics. The proposed ensemble approach can be used for the early detection of diabetes and can aid healthcare professionals in making informed decisions regarding patient care.

**Similar Documents**: This paper introduces an innovative ensemble learning strategy for the prediction of diabetes mellitus, a chronic condition marked by elevated blood glucose levels due to issues with insulin secretion or function. The timely identification of diabetes is essential for managing and treating

Real Time Human-machine Interaction system using hand gesture Recognition to handle the mouse event , media player , image viewer .Users have to repeat same mouse and keyboard actions, inducing waste of time. Gestures have long been considered as an interaction technique that can potentially deliver more natural. A fast gesture recognition scheme is proposed to be an interface for the human-machine interaction (HMI) of systems. The system presents some lowcomplexity algorithms and gestures to reduce the gesture recognition complexity and be more suitable for controlling real-time computer systems. In this paper we use the webcam for capturing the image. After capturing the image it converts into the binary image. A gesture is a specific combination of hand position.

Automatic recognition of human actions from video has been studied for many years. Although still very difficult in uncontrolled scenarios, it has been successful in more restricted settings (e.g., fixed viewpoint, no occlusions) with recognition rates approaching 100%. However, the best-performing methods are complex and computationally-demanding and thus not well-suited for real-time deployments. This paper proposes to leverage the Kinect camera for close-range gesture recognition using two methods. Both methods use feature vectors that are derived from the skeleton model provided by the Kinect SDK in real-time. Although both methods perform nearest-neighbor classification, one method does this in the space of features using the Euclidean distance metric, while the other method does this in the space of feature covariances using a log-Euclidean metric. Both methods recognize 8 hand gestures in real time achieving correct-classification rates of over 99% on a dataset of 20 subjects but the method based on Euclidean distance requires feature-vector collections to be of the same size, is sensitive to temporal misalignment, and has higher computation and storage requirements.

In order to perform useful tasks, a service robot needs to manipulate objects in its environment. In this paper, we propose a method for experimental evaluation of the suitability of a robotic hand for grasping tasks in service robotics. The method is applied to the Schunk 5-Finger Gripping Hand, which is a mechatronic gripper designed for service robots. During evaluation, it is shown, that it is able to grasp various common household objects and execute the grasps from the well known Cutkosky grasp taxonomy [1]. The result is, that it is a suitable hand for service robot tasks.

We constructed an acoustic, gesture-based recognition system called Multiwave, which leverages the Doppler Effect to translate multidimensional movements into user interface commands. Our system only requires the use of two speakers and a microphone to be operational. Since these components are already built in to most end user systems, our design makes gesture-based input more accessible to a wider range of end users. By generating a known high frequency tone from multiple speakers and detecting movement using changes in the sound waves, we are able to calculate a Euclidean representation of hand velocity that is then used for more natural gesture recognition and thus, more meaningful interaction mappings. We present the results of a user study of Multiwave to evaluate recognition rates for different gestures and report accuracy rates comparable to or better than the current state of the art. We also report subjective user feedback and some lessons learned from our system that provide additional insight for future applications of multidimensional gesture recognition.

This paper presents a deep learning architecture for detecting the palm and fingertip positions of stable grasps directly from partial object views. The architecture is trained using RGBD image patches of fingertip and palm positions from grasps computed on complete object models using a grasping simulator. At runtime, the architecture is able to estimate grasp quality metrics without the need to explicitly calculate the given metric. This ability is useful as the exact calculation of these quality functions is impossible from an incomplete view of a novel object without any tactile feedback. This architecture for grasp quality prediction provides a framework for generalizing grasp experience from known to novel objects.

This paper presents a method of estimating camera pose in an unknown scene. While this has previously been attempted by adapting SLAM algorithms developed for robotic exploration, we propose a system specifically designed to track a hand-held camera in a small AR workspace. We propose to split tracking and mapping into two separate tasks, processed in parallel threads on a dual-core computer: one thread deals with the task of robustly tracking erratic hand-held motion, while the other produces a 3D map of point features from previously observed video frames. This allows the use of computationally expensive batch optimisation techniques not usually associated with real-time operation: The result is a system that produces detailed maps with thousands of landmarks which can be tracked at frame-rate, with an accuracy and robustness rivalling that of state-of-the-art model-based systems.

This paper presents the concept for a test setup to prototype control algorithms for a multi-finger gripper. The human-robot interface has to provide enough degrees of freedom (DOF) to intuitively control advanced gripper and to accomplish this task a simple sensor glove equipped with flex and force sensors has been prepared for this project. The software architecture has to support both real hardware and simulation, as well as flexible communication standards, therefore, ROS architecture was employed in this matter. Paper presents some preliminary results for using sensor glove and simulated model of the three finger gripper.

With increasing use of computers in our daily lives, lately there has been a rapid increase in the efforts to develop a better human computer interaction interface. The need of easy to use and advance types of human-computer interaction with natural interfaces is more than ever. In the present framework, the UI (User Interface) of a computer allows user to interact with electronic devices with graphical icons and visual indicators, which is still inconvenient and not suitable for working in virtual environments. An interface which allow user to communicate through gestures is the next step in the direction of advance human computer interface. In the present paper author explore different aspects of gesture recognition techniques.

A real-time solution for estimating and tracking the 3D pose of a rigid object is presented for image-based visual servo with natural landmarks. The many state-of-the-art technologies that are available for recognizing the 3D pose of an object in a natural setting are not suitable for real-time servo due to their time lags. This paper demonstrates that a real-time solution of 3D pose estimation become feasible by combining a fast tracker such as KLT [7] [8] with a method of determining the 3D coordinates of tracking points on an object at the time of SIFT based tracking point initiation, assuming that a 3D geometric model with SIFT description of an object is known a-priori. Keeping track of tracking points with KLT, removing the tracking point outliers automatically, and reinitiating the tracking points using SIFT once deteriorated, the 3D pose of an object can be estimated and tracked in real-time. This method can be applied to both mono and stereo camera based 3D pose estimation and tracking. The former guarantees higher frame rates with about 1 ms of local pose estimation, while the latter assures of more precise pose results but with about 16 ms of local pose estimation. The experimental investigations have shown the effectiveness of the proposed approach with real-time performance.

Both the original version of David Lowe's innuential and classic algorithm for tracking known objects and a reformulation of it implemented by Ishii et al. rely on (diierent) approximated imaging models. Removing their simplifying assumptions yields a fully pro-jective solution with signiicantly improved accuracy and convergence, and arguably better computation{ time properties.