Question: An energy-efficient middleware solution for computation offloading in real-time embedded systems.

Introducing MAUI: Enhancing Smartphone Battery Life through Intelligent Code Offloading

This research introduces MAUI, a system designed to optimize energy efficiency in smartphones by offloading energy-intensive code execution to external infrastructure. Unlike previous solutions that either demanded significant programmer intervention for application partitioning or employed coarse-grained methods like full process or virtual machine migration, MAUI leverages the advantages of managed code environments. This approach facilitates fine-grained code offloading, maximizing energy savings while minimizing the overhead for developers. MAUI's dynamic decision-making mechanism determines which code segments should be executed remotely, guided by an optimization engine that maximizes energy conservation under the constraints of the device's current connectivity. Through empirical evaluation, MAUI demonstrates its effectiveness in three scenarios: 1) significantly reducing energy consumption for a resource-heavy face recognition application by an order of magnitude, 2) doubling the refresh rate for a latency-sensitive arcade game by offloading computation, and 3) overcoming smartphone limitations for a voice-based language translation app by remotely executing unsupported components. These outcomes underscore MAUI's potential to substantially enhance the energy efficiency and performance of mobile applications.

The last decades’ tremendous advances in wireless communications have been driven mainly by personal communications. Radio resource allocation mechanisms for optimizing key metrics, such as average throughput and delay, for such traffic are by now rather well-developed. However, with the increased interest in wireless machine-to-machine communication, e.g. for industrial control or monitoring of large-scale infrastructures, new challenges emerge. The performance of an estimator or closed-loop control system that operates over an unreliable wireless network depends on the full latency and loss distributions and not only on their averages. For these applications, more suitable performance metrics are per-packet guarantees on latency and reliability (on-time delivery). This thesis studies optimal forwarding of deadline-constrained traffic over lossy multi-hop networks. We assume a routing topology in the form of a directed graph with packet loss on links described by finite-state Markov chains, and focus on a single transient packet scenario. Two problems are considered: maximizing the probability that packets are delivered within their specified deadline; and minimizing the expected energy cost with a guaranteed probability of on-time delivery. The first problem can be formulated as a finite-horizon Markov decision process (MDP), while the second problem is a finite-horizon constrained Markov decision process (CMDP). A general dynamic programming framework that solves a weighted sum of reliability and energy maximization problem is proposed. The maximum deadline-constrained reliability problem is solved by studying a simplification of the general dynamic programming. The minimum energy optimal policy is a random selection between two deterministic and computable forwarding policies, each of which can be found via a dynamic programming framework. Particular instances with Bernoulli and Gilbert-Elliot loss models that admit numerically efficient solutions are discussed. Finally, we show the application of the technique for a co-design of forwarding policies and controller for wireless control. Based on the recent result on the monotonicity of the optimal control loss, the problem of minimum control loss and the problem of minimum energy cost with a guaranteed control performance can be solved optimally.

We describe a new algorithm, called BERT, that can be used to schedule both best effort and realtime tasks on a multimedia workstation. BERT exploits two innovations. First, it is based on the virtual clock algorithm originally developed to schedule bandwidth on packet switches. Because this algorithm maintains a relationship between virtual time and real time, our algorithm allows us to simultaneously factor realtime deadlines into the scheduling decision, and to allocate some fraction of the processor to best effort tasks. Second, BERT includes a mechanism that allows one task to steal cycles from another. This mechanism is valuable for two reasons: it allows us to distinguish among tasks based on their importance, and it results in a robust algorithm that is not overly sensitive to the task having made a perfect reservation.

Both distributed healthcare systems and the Internet of Things (IoT) are currently hot topics. The latter is a new computing paradigm to enable advanced capabilities in engineering various applications, including those for healthcare. For such systems, the core social requirement is the privacy/security of the patient information along with the technical requirements (e.g., energy consumption) and capabilities for adaptability and personalization. Typically, the functionality of the systems is predefined by the patient’s data collected using sensor networks along with medical instrumentation; then, the data is transferred through the Internet for treatment and decision-making. Therefore, systems creation is indeed challenging. In this paper, we propose a model-driven framework to develop the IoT-based prototype and its reference architecture for personalized health monitoring (PHM) applications. The framework contains a multi-layered structure with feature-based modeling and feature model transformations at the top and the application software generation at the bottom. We have validated the framework using available tools and developed an experimental PHM to test some aspects of the functionality of the reference architecture in real time. The main contribution of the paper is the development of the model-driven computational framework with emphasis on the synergistic effect of security and energy issues.

Supporting real-time and mobile data services, fog computing has been considered as a promising technology to overcome long and unpredicted delay in cloud computing. However, as resources in FNs are owned by independent users or infrastructure providers, the ADSSs cannot connect and access data services from the FNs directly, but can only request data service from the DSOs in the cloud. Accordingly, in fog computing, the DSOs are required to communicate with FNs and allocate resources from the FNs to the ADSSs. The DSOs provide virtualized data services to the ADSSs, and the FNs, motivated by the DSOs, provide data services in the physical network. Nevertheless, with fog computing added as the intermediate layer between the cloud and users, there are challenges such as the resource allocation in the virtualized network between the DSOs and ADSSs, the asymmetric information problem between DSOs and ADSSs, and the resource matching from the FNs to the ADSSs in the physical network. In this article, we propose a three-layer hierarchical game framework to solve the challenges in fog computing networks. In the proposed framework, we apply the Stackelberg sub-game for the interaction between DSOs and ADSSs, moral hazard modeling for the interaction between DSOs and FNs, and the student project allocation matching sub-game for the interaction between FNs and ADSSs. The purpose is to obtain stable and optimal utilities for each DSO, FN, and ADSS in a distributed fashion.

Modern Cloud Data Centers exploit virtualization for efficient resource management to reduce cloud computational cost and energy budget. Virtualization empowered by virtual machine (VM) migration meets the ever increasing demands of dynamic workload by relocating VMs within Cloud Data Centers. VM migration helps successfully achieve various resource management objectives such as load balancing, power management, fault tolerance, and system maintenance. However, being resourceintensive, the VM migration process rigorously affects application performance unless attended by smart optimization methods. Furthermore, a Cloud Data Centre exploits server consolidation and DVFS methods to optimize energy consumption. This paper reviews state-of-the-art bandwidth optimization schemes, server consolidation frameworks, DVFS-enabled power optimization, and storage optimization methods over WAN links. Through a meticulous literature review of state-of-the-art live VM migration schemes, thematic taxonomies are proposed to categorize the reported literature. The critical aspects of virtual machine migration schemes are investigated through a comprehensive analysis of the existing schemes. The commonalties and differences among existing VM migration schemes are highlighted through a set of parameters derived from the literature. Finally, open research issues and trends in the VM migration domain that necessitate further consideration to develop optimal VM migration schemes

IoT is going to be an established part of life by extending the communication and networking anytime, anywhere. Security requirements for IoT will certainly underline the importance of properly formulated, implemented, and enforced security policies throughout their life-cycle. This paper gives a detailed survey and analysis of embedded security, especially in the area of IoT. Together with the conventional security solutions, the paper highlights the need to provide in-built security in the device itself to provide a flexible infrastructure for dynamic prevention, detection, diagnosis, isolation, and countermeasures against successful breaches. Based on this survey and analysis, the paper defines the security needs taking into account computational time, energy consumption and memory requirements of the devices. Finally, this paper proposes an embedded security framework as a feature of software/hardware co-design methodology.

In real-time systems, the timing behavior is an important property of each task. It has to be guaranteed that the execution of a task does not take longer than the specified amount of time. Thus, a knowledge about the maximum execution time of programs is of utmost importance. This paper discusses the problems for the calculation of the maximum execution time (MAXT... MAximum eXecution Time). It shows the preconditions which have to be met before the MAXT of a task can be calculated. Rules for the MAXT calculation are described. Triggered by the observation that in most cases the calculated MAXT far exceeds the actual execution time, new language constructs are introduced. These constructs allow programmers to put into their programs more information about the behavior of the algorithms implemented and help to improve the self checking property of programs. As a consequence, the quality of MAXT calculations is improved significantly. In a realistic example, an improvement fator of 11 has been achieved.

With the increase of energy consumption associated with IT infrastructures, energy management is becoming a priority in the design and operation of complex service-based systems. At the same time, service providers need to comply with Service Level Agreement (SLA) contracts which determine the revenues and penalties on the basis of the achieved performance level. This paper focuses on the resource allocation problem in multitier virtualized systems with the goal of maximizing the SLAs revenue while minimizing energy costs. The main novelty of our approach is to address-in a unifying framework-service centers resource management by exploiting as actuation mechanisms allocation of virtual machines (VMs) to servers, load balancing, capacity allocation, server power state tuning, and dynamic voltage/frequency scaling. Resource management is modeled as an NP-hard mixed integer nonlinear programming problem, and solved by a local search procedure. To validate its effectiveness, the proposed model is compared to top-performing state-of-the-art techniques. The evaluation is based on simulation and on real experiments performed in a prototype environment. Synthetic as well as realistic workloads and a number of different scenarios of interest are considered. Results show that we are able to yield significant revenue gains for the provider when compared to alternative methods (up to 45 percent). Moreover, solutions are robust to service time and workload variations.

In this paper, we describe how to establish an embedded framework for real-time top-view people counting. The development of our system consists of two parts, i.e. establishing an embedded signal processing platform and designing a people counting algorithm for the embedded system. For the hardware platform construction, we use Kinect as the camera and exploit NVIDIA Jetson TK1 board as the embedded processing platform. We describe how to build a channel to make Kinect for windows version 2.0 communicate with Jetson TK1. Based on the embedded system, we adapt a water filling based scheme for top-view people counting, which integrates head detection based on water drop, people tracking and counting. Gaussian Mixture Model is used to construct and update the background model. The moving people in each video frame are extracted using background subtraction method. Additionally, the water filling algorithm is used to segment head area as Region Of Interest(ROI). Tracking and counting people are performed by calculating the distance of ROI center point before and after the frame. The whole framework is flexible and practical for real-time application.

Energy accounting determines how much a software principal contributes to the total system energy consumption. It is the foundation for evaluating software and for operating system based energy management. While various energy accounting policies have been tried, there is no known way to evaluate them directly simply because it is hard to track all hardware usage by software in a heterogeneous multicore system like modern smartphones and tablets.
 In this work, we argue that energy accounting should be formulated as a cooperative game and that the Shapley value provides the ultimate ground truth for energy accounting policies. We reveal the important flaws of existing energy accounting policies based on the Shapley value theory and provide Shapley value-based energy accounting, a practical approximation of the Shapley value, for battery-powered mobile systems. We evaluate this approximation against existing energy accounting policies in two ways: (i) how well they identify the top energy consuming applications, and (ii) how effective they are in system energy management. Using a prototype based on Texas Instruments Pandaboard and smartphone workload, we experimentally demonstrate existing energy accounting policies can deviate by 400% in attributing energy consumption to running applications and can be up to 25% less effective in system energy management when compared to Shapley value-based energy accounting.


Question: An energy-efficient middleware solution for computation offloading in real-time embedded systems.

This work introduces PowerBooter, an innovative technique for the automated construction of power models for smartphones, and PowerTutor, a tool for online power estimation and battery behavior analysis. PowerBooter leverages built-in battery voltage sensors and an understanding of battery discharge characteristics to monitor and control the power management and activity states of individual components, eliminating the need for external measurement equipment. The power models generated by PowerBooter can be quickly and easily created for various smartphone models, each with distinct power consumption traits, enabling developers and users to tailor power models to specific devices. PowerTutor, utilizing the models created by PowerBooter, facilitates online power estimation and component power management, aiming to simplify the design and selection of energy-efficient software for embedded systems. Together, PowerBooter and PowerTutor aim to democratize power modeling and analysis for a wider range of smartphone models and their users, promoting energy efficiency in real-time embedded systems.

In this paper the scheduling and transmit power control are investigated to minimize the energy consumption for battery-driven devices deployed in LTE networks. To enable efficient scheduling for a massive number of machine-type subscribers, a novel distributed scheme is proposed to let machine nodes form local clusters and communicate with the base-station through the cluster-heads. Then, uplink scheduling and power control in LTE networks are introduced and lifetime-aware solutions are investigated to be used for the communication between cluster-heads and the base-station. Beside the exact solutions, low-complexity suboptimal solutions are presented in this work which can achieve near optimal performance with much lower computational complexity. The performance evaluation shows that the network lifetime is significantly extended using the proposed protocols.

Implementing E cient, Portable Computations for Machine Learning

A modern mobile application processor is a heterogeneous multi-core SoC which integrates CPU and application-specific accelerators such as GPU and DSP. It provides opportunity to accelerate other compute-intensive applications, yet mapping an algorithm to such a heterogeneous platform is not a straightforward task and has many design decisions to make. In this paper, we evaluate the performance and energy benefits of utilizing the integrated GPU and DSP cores to offload or share CPU's compute-intensive tasks. The evaluation is conducted on three representative mobile platforms, TI's OMAP3530, Qualcomn's Snapdragon S2, and Nvidia's Tegra2, using common computation tasks in mobile applications. We identify key factors that should be considered in energy-optimized mobile heterogeneous computing. Our evaluation results show that, by effectively utilizing all the computing cores concurrently, an average of 3.7X performance improvement can be achieved with the cost of 33% more power consumption, in comparison with the case of utilizing CPU only. This stands for 2.8X energy saving.

The widespread adoption of IEEE 802.11 wireless networks has brought its security paradigm under active research. One of the important research areas in this field is the realization of fast and secure implementations of cryptographic algorithms. Under this work, such an implementation has been done for Advanced Encryption Standard (AES) on fast, efficient and low power Field Programmable Gate Arrays (FPGAs) whereby computational intensive cryptographic processes are offloaded from the main processor thus results in achieving highspeed secure wireless connectivity. The dedicated resources of Spartan-3 FPGAs have been effectively utilized to develop wider logic function which minimizes the critical paths by confining logic to single Configurable Logic Block (CLB), thus improving the performance, density and power consumption of the design. The resultant design consumes only 4 Block RAMs and 487 Slices to fit both AES cores and its key scheduling.

Fog Computing is a paradigm that extends Cloud computing and services to the edge of the network. Similar to Cloud, Fog provides data, compute, storage, and application services to end-users. In this article, we elaborate the motivation and advantages of Fog computing, and analyse its applications in a series of real scenarios, such as Smart Grid, smart traffic lights in vehicular networks and software defined networks. We discuss the state-of-the-art of Fog computing and similar work under the same umbrella. Security and privacy issues are further disclosed according to current Fog computing paradigm. As an example, we study a typical attack, man-in-the-middle attack, for the discussion of security in Fog computing. We investigate the stealthy features of this attack by examining its CPU and memory consumption on Fog device.

This survey covers hard real-time scheduling algorithms and schedulability analysis techniques for homogeneous multiprocessor systems. It reviews the key results in this field from its origins in the late 1960s to the latest research published in late 2009. The survey outlines fundamental results about multiprocessor real-time scheduling that hold independent of the scheduling algorithms employed. It provides a taxonomy of the different scheduling methods, and considers the various performance metrics that can be used for comparison purposes. A detailed review is provided covering partitioned, global, and hybrid scheduling algorithms, approaches to resource sharing, and the latest results from empirical investigations. The survey identifies open issues, key research challenges, and likely productive research directions.

We introduce ROC, a Rank-switching, Open-row Controller for Double Data Rate Dynamic RAM (DDR DRAM). ROC is optimized for mixed-criticality multicore systems using modern DDR devices: compared to existing real-time memory controllers, it provides significantly lower worst case latency bounds for hard real-time tasks and supports throughput-oriented optimizations for soft real-time applications. The key to improved performance is an innovative rank-switching mechanism which hides the latency of write-read transitions in DRAM devices without requiring unpredictable request reordering. We further employ open row policy to take advantage of the data caching mechanism (row buffering) in each device. ROC provides complete timing isolation between hard and soft tasks and allows for compositional timing analysis over the number of cores and memory ranks in the system. We implemented and synthesized the ROC back end in Verilog RTL, and evaluated its performance on both synthetic tasks and a set of representative benchmarks.

Self-configuring wireless sensor networks can be invaluable in many civil and military applications for collecting, processing, and disseminating wide ranges of complex environmental data. Because of this, they have attracted considerable research attention in the last few years. The WINS [1] and SmartDust [2] projects, for instance, aim to integrate sensing, computing, and wireless communication capabilities into a small form factor to enable low-cost production of these tiny nodes in large numbers. Several other groups are investigating efficient hardware/software system architectures, signal processing algorithms, and network protocols for wireless sensor networks [3]-[5]. Sensor nodes are battery driven and hence operate on an extremely frugal energy budget. Further, they must have a lifetime on the order of months to years, since battery replacement is not an option for networks with thousands of physically embedded nodes. In some cases, these networks may be required to operate solely on energy scavenged from the environment through seismic, photovoltaic, or thermal conversion. This transforms energy consumption into the most important factor that determines sensor node lifetime. Conventional low-power design techniques [6] and hardware architectures only provide point solutions which are insufficient for these highly energy-constrained systems. Energy optimization, in the case of sensor networks, is much more complex, since it involves not only reducing the energy consumption of a single sensor node but also maximizing the lifetime of an entire network. The network lifetime can be maximized

In real-time systems, the timing behavior is an important property of each task. It has to be guaranteed that the execution of a task does not take longer than the specified amount of time. Thus, a knowledge about the maximum execution time of programs is of utmost importance. This paper discusses the problems for the calculation of the maximum execution time (MAXT... MAximum eXecution Time). It shows the preconditions which have to be met before the MAXT of a task can be calculated. Rules for the MAXT calculation are described. Triggered by the observation that in most cases the calculated MAXT far exceeds the actual execution time, new language constructs are introduced. These constructs allow programmers to put into their programs more information about the behavior of the algorithms implemented and help to improve the self checking property of programs. As a consequence, the quality of MAXT calculations is improved significantly. In a realistic example, an improvement fator of 11 has been achieved.

In a cloud computing environment, users prefer to migrate their locally processing workloads onto the cloud where more resources with better performance can be expected. ProtoGENI [1] and PlanetLab [17] have further improved the current Internet-based resource outsourcing by allowing end users to construct a virtual network system through virtualization and programmable networking technologies. However, to the best of our knowledge, there is no such general service or resource provisioning platform designated for mobile devices. In this paper, we present a new design and implementation of MobiCloud that is a geo-distributed mobile cloud computing platform. The discussions of the system components, infrastructure, management, implementation flow, and service scenarios are followed by an example on how to experience the MobiCloud system.


Question: An energy-efficient middleware solution for computation offloading in real-time embedded systems.

This paper introduces a groundbreaking encryption scheme designed to enable secure processing of data without compromising privacy. The concept revolves around the idea of "fully homomorphic" encryption, which facilitates the processing of encrypted data by a third party without granting them access to the plaintext. This scheme ensures that the data remains confidential, even when complex functions are applied to it by a worker who does not possess the decryption key. Essentially, it allows for the computation of encrypted data to yield another encrypted result, without the intermediary having any knowledge of the underlying data. This innovation is particularly relevant in the realm of cloud computing, where it can significantly enhance privacy and security while still enabling powerful data processing capabilities. By allowing third-party services to perform intricate operations on encrypted data without decryption, it bridges the gap between cloud computing and data privacy, ensuring that sensitive information remains protected throughout the processing cycle.

The next generation of embedded computing systems will have to meet new challenges. The systems are expected to act mainly autonomously, to dynamically adapt to changing environments and to interact with one another if necessary. Such systems are called organic. Organic Computing systems are similar to autonomic computing systems. In addition Organic Computing systems often behave life-like and are inspired by nature/biological phenomena. Design and construction of such systems brings new challenges for the software engineering process. In this paper we present a framework for design, construction and analysis of organic computing systems. It can facilitate design and construction as well as it can be used to (semi-)formally define organic properties like self-configuration or self-adaptation. We illustrate the framework on a real-world case study from production automation.

The functional scope of today's software-defined data centers (SDDC) has expanded to such an extent that servers face a growing amount of critical background operational tasks like load monitoring, logging, migration, and duplication, etc. These ancillary operations, which we refer to as management operations, often nibble the stringent data center power envelope and exert a tremendous amount of pressure on front-end user tasks. However, existing power capping, peak shaving, and time shifting mechanisms mainly focus on managing data center power demand at the "macro level" -- they do not distinguish ancillary background services from user tasks, and therefore often incur significant performance degradation and energy overhead.
 In this study we explore "micro-level" power management in SDDC: tuning a specific set of critical loads for the sake of overall system efficiency and performance. Specifically, we look at management operations that can often lead to resource contention and energy overhead in an IaaS SDDC. We assess the feasibility of this new power management paradigm by characterizing the resource and power impact of various management operations. We propose HOPE, a new system optimization framework for eliminating the potential efficiency bottleneck caused by the management operations in the SDDC. HOPE is implemented on a customized OpenStack cloud environment with heavily instrumented power infrastructure. We thoroughly validate HOPE models and optimization efficacy under various user workload scenarios. Our deployment experiences show that the proposed technique allows SDDC to reduce energy consumption by 19%, reduce management operation execution time by 25.4%, and in the meantime improve workload performance by 30%.

Falls can cause serious traumas such as brain injuries and bone fractures, especially among elderly people. Fear of falling might reduce physical activities resulting in declining social interactions and eventually causing depression. To lessen the effects of a fall, timely delivery of medical treatment can play a vital role. In a similar scenario, an IoT-based wearable system can pave the most promising way to mitigate serious consequences of a fall while providing the convenience of usage. However, to deliver sufficient degree of monitoring and reliability, wearable devices working at the core of fall detection systems are required to work for a prolonged period of time. In this work, we focus on energy efficiency of a wearable sensor node in an Internet-of-Things (IoT) based fall detection system. We propose the design of a tiny, lightweight, flexible and energy efficient wearable device. We investigate different parameters (e.g. sampling rate, communication bus interface, transmission protocol, and transmission rate) impacting on energy consumption of the wearable device. In addition, we provide a comprehensive analysis of energy consumption of the wearable in different configurations and operating conditions. Furthermore, we provide hints (hardware and software) for system designers implementing the optimal wearable device for IoT-based fall detection systems in terms of energy efficiency and high quality of service. The results clearly indicate that the proposed sensor node is novel and energy efficient. In a critical condition, the wearable device can be used continuously for 76 h with a 1000 mAh li-ion battery.

Modern Cloud Data Centers exploit virtualization for efficient resource management to reduce cloud computational cost and energy budget. Virtualization empowered by virtual machine (VM) migration meets the ever increasing demands of dynamic workload by relocating VMs within Cloud Data Centers. VM migration helps successfully achieve various resource management objectives such as load balancing, power management, fault tolerance, and system maintenance. However, being resourceintensive, the VM migration process rigorously affects application performance unless attended by smart optimization methods. Furthermore, a Cloud Data Centre exploits server consolidation and DVFS methods to optimize energy consumption. This paper reviews state-of-the-art bandwidth optimization schemes, server consolidation frameworks, DVFS-enabled power optimization, and storage optimization methods over WAN links. Through a meticulous literature review of state-of-the-art live VM migration schemes, thematic taxonomies are proposed to categorize the reported literature. The critical aspects of virtual machine migration schemes are investigated through a comprehensive analysis of the existing schemes. The commonalties and differences among existing VM migration schemes are highlighted through a set of parameters derived from the literature. Finally, open research issues and trends in the VM migration domain that necessitate further consideration to develop optimal VM migration schemes

Hardware/software codesign investigates the concurrent design of hardware and software components of complex electronic systems. It tries to exploit the synergy of hardware and software with the goal to optimize and/or satisfy design constraints such as cost, performance, and power of the final product. At the same time, it targets to reduce the time-to-market frame considerably. This paper presents major achievements of two decades of research on methods and tools for hardware/software codesign by starting with a historical survey of its roots, by highlighting its major research directions and achievements until today, and finally, by predicting in which direction research in codesign might evolve in the decades to come.

Battery lifetime is critical for wireless video sensors. To enable battery-powered wireless video sensors, low-power design is required. In this paper, we consider applying multi-view summarization to wireless video sensors to remove redundant contents such that the compression and transmission power can be reduced. A low-complexity online multi-view video summarization scheme is proposed. Experiments show that the proposed summarization method successfully reduces the video content while keeping important events. A power analysis of the system also shows that a significant amount of energy can be saved.

During the last decade, a lot of progress has happened in the field of autonomous embedded devices (also known as motes) and wireless sensor networks. In this context, we envision next-generation systems to have autonomous database management functionalities on each mote, for supporting easy management of data on flash disks, adequate querying capabilities, and ubiquitous computing with no-effort. Following this major vision, in this paper we describe application scenarios, middleware approach and data management algorithms of a novel system, called Stream Operation (StreamOp), which effectively and efficiently realizes the depicted challenges. In particular, StreamOp supports heterogeneity, i.e. It works on top of different platforms, efficiency, i.e. It returns responses quickly, and autonomy, i.e. It saves battery power. We show how StreamOp provides these features, along with some experimental results that clearly validate our main assumptions.

The energy usage of computer systems is becoming more important, especially for battery operated systems. Displays, disks, and cpus, in that order, use the most energy. Reducing the energy used by displays and disks has been studied elsewhere; this paper considers a new method for reducing the energy used by the cpu. We introduce a new metric for cpu energy performance, millions-of-instructions-per-joule (MIPJ). We examine a class of methods to reduce MIPJ that are characterized by dynamic control of system clock speed by the operating system scheduler. Reducing clock speed alone does not reduce MIPJ, since to do the same work the system must run longer. However, a number of methods are available for reducing energy with reduced clock-speed, such as reducing the voltage [Chandrakasan et al 1992][Horowitz 1993] or using reversible [Younis and Knight 1993] or adiabatic logic [Athas et al 1994].
 What are the right scheduling algorithms for taking advantage of reduced clock-speed, especially in the presence of applications demanding ever more instructions-per-second? We consider several methods for varying the clock speed dynamically under control of the operating system, and examine the performance of these methods against workstation traces. The primary result is that by adjusting the clock speed at a fine grain, substantial CPU energy can be saved with a limited impact on performance.

This paper presents an architecture of a low-power real-time object detection processor using Adaboost with Haar-Like features. We employ a register array based architecture, and introduce two architectural-level power optimization techniques; signal gating domain for integral image extraction, and low-power integral image update. The power efficiency of our proposed architecture including nine classifiers is estimated to be 0.64mW/fps when handling VGA(640 × 480) 70fps video.

Increasing demand for power-efficient, high-performance computing has spurred a growing number and diversity of hardware accelerators in mobile and server Systems on Chip (SoCs). This paper makes the case that the co-design of the accelerator microarchitecture with the system in which it belongs is critical to balanced, efficient accelerator microarchitectures. We find that data movement and coherence management for accelerators are significant yet often unaccounted components of total accelerator runtime, resulting in misleading performance predictions and inefficient accelerator designs. To explore the design space of accelerator-system co-design, we develop gem5-Aladdin, an SoC simulator that captures dynamic interactions between accelerators and the SoC platform, and validate it to within 6% against real hardware. Our co-design studies show that the optimal energy-delay-product (EDP) of an accelerator microarchitecture can improve by up to 7.4× when system-level effects are considered compared to optimizing accelerators in isolation.


Question: A convolutional neural network-based hand tracking system.

This paper describes the successful implementation of back-propagation networks for the recognition of handwritten digits. The system required minimal preprocessing of the input data, yet the network architecture was meticulously tailored for this specific task. Normalized images of isolated digits served as the network's input. The method achieved a commendable 1% error rate and a 9% reject rate when tested on digits from the U.S. Postal Service's zip code database. The network's design and performance highlight its capability in handling handwritten digit recognition with high accuracy, showcasing the potential of back-propagation networks in this domain.

3-D convolutional neural networks (3-D-convNets) have been very recently proposed for action recognition in videos, and promising results are achieved. However, existing 3-D-convNets has two “artificial” requirements that may reduce the quality of video analysis: 1) It requires a fixed-sized (e.g., 112  $\times$ 112) input video; and 2) most of the 3-D-convNets require a fixed-length input (i.e., video shots with fixed number of frames). To tackle these issues, we propose an end-to-end pipeline named Two-stream 3-D-convNet Fusion, which can recognize human actions in videos of arbitrary size and length using multiple features. Specifically, we decompose a video into spatial and temporal shots. By taking a sequence of shots as input, each stream is implemented using a spatial temporal pyramid pooling (STPP) convNet with a long short-term memory (LSTM) or CNN-E model, softmax scores of which are combined by a late fusion. We devise the STPP convNet to extract equal-dimensional descriptions for each variable-size shot, and we adopt the LSTM/CNN-E model to learn a global description for the input video using these time-varying descriptions. With these advantages, our method should improve all 3-D CNN-based video analysis methods. We empirically evaluate our method for action recognition in videos and the experimental results show that our method outperforms the state-of-the-art methods (both 2-D and 3-D based) on three standard benchmark datasets (UCF101, HMDB51 and ACT datasets).

Different from RGB videos, depth data in RGB-D videos provide key complementary information for tristimulus visual data which potentially could achieve accuracy improvement for action recognition. However, most of the existing action recognition models solely using RGB videos limit the performance capacity. Additionally, the state-of-the-art action recognition models, namely 3D convolutional neural networks (3D-CNNs) contain tremendous parameters suffering from computational inefficiency. In this paper, we propose a series of 3D lightweight architectures for action recognition based on RGB-D data. Compared with conventional 3D-CNN models, the proposed lightweight 3D-CNNs have considerably less parameters involving lower computation cost, while it results in favorable recognition performance. Experimental results on two public benchmark datasets show that our models can approximate or outperform the state-of-the-art approaches. Specifically, on the RGB+DNTU (NTU) dataset, we achieve 93.2% and 97.6% for crosssubject and cross-view measurement, and on the NorthwesternUCLA Multiview Action 3D (N-UCLA) dataset, we achieve 95.5% accuracy of cross-view.

We present a mobile vision system for multi-person tracking in busy environments. Specifically, the system integrates continuous visual odometry computation with tracking-by-detection in order to track pedestrians in spite of frequent occlusions and egomotion of the camera rig. To achieve reliable performance under real-world conditions, it has long been advocated to extract and combine as much visual information as possible. We propose a way to closely integrate the vision modules for visual odometry, pedestrian detection, depth estimation, and tracking. The integration naturally leads to several cognitive feedback loops between the modules. Among others, we propose a novel feedback connection from the object detector to visual odometry which utilizes the semantic knowledge of detection to stabilize localization. Feedback loops always carry the danger that erroneous feedback from one module is amplified and causes the entire system to become instable. We therefore incorporate automatic failure detection and recovery, allowing the system to continue when a module becomes unreliable. The approach is experimentally evaluated on several long and difficult video sequences from busy inner-city locations. Our results show that the proposed integration makes it possible to deliver stable tracking performance in scenes of previously infeasible complexity.

Model-based recognition and motion tracking depends upon the ability to solve for projection and model parameters that will best fit a 3-D model to matching 2-D image features. This paper extends current methods of parameter solving to handle objects with arbitrary curved surfaces and with any number of internal parameters representing articulations, variable dimensions, or surface deformations. Numerical stabilization methods are developed that take account of inherent inaccuracies in the image measurements and allow useful solutions to be determined even when there are fewer matches than unknown parameters. The LevenbergMarquardt method is used to always ensure convergence of the solution. These techniques allow model-based vision to be used for a much wider class of problems than was possible with previous methods. Their application is demonstrated for tracking the motion of curved, parameterized objects. This paper has been published in IEEE Transactions on Pattern Analysis and Machine Intelligence, 13, 5 (May 1991), pp. 441–450.

The use of gesture as a natural interface serves as a motivating force for research in modeling, analyzing and recognition of gestures. In particular, human computer intelligent interaction needs vision-based gesture recognition, which involves many interdisciplinary studies. A survey on recent vision-based gesture recognition approaches is given in this paper. We shall review methods of static hand posture and temporal gesture recognition. Several application systems of gesture recognition are also described in this paper. We conclude with some thoughts about future research directions.

During the recent years, correlation filters have shown dominant and spectacular results for visual object tracking. The types of the features that are employed in this family of trackers significantly affect the performance of visual tracking. The ultimate goal is to utilize the robust features invariant to any kind of appearance change of the object, while predicting the object location as properly as in the case of no appearance change. As the deep learning based methods have emerged, the study of learning features for specific tasks has accelerated. For instance, discriminative visual tracking methods based on deep architectures have been studied with promising performance. Nevertheless, correlation filter based (CFB) trackers confine themselves to use the pre-trained networks, which are trained for object classification problem. To this end, in this manuscript the problem of learning deep fully convolutional features for the CFB visual tracking is formulated. In order to learn the proposed model, a novel and efficient backpropagation algorithm is presented based on the loss function of the network. The proposed learning framework enables the network model to be flexible for a custom design. Moreover, it alleviates the dependency on the network trained for classification. Extensive performance analysis shows the efficacy of the proposed custom design in the CFB tracking framework. By fine-tuning the convolutional parts of a state-of-the-art network and integrating this model to a CFB tracker, which is the top performing one of VOT2016, 18% increase is achieved in terms of expected average overlap, and tracking failures are decreased by 25%, while maintaining the superiority over the state-of-the-art methods in OTB-2013 and OTB-2015 tracking datasets.

This paper proposes an effective segmentation-free approach using a hybrid neural network hidden Markov model (NN-HMM) for offline handwritten Chinese text recognition (HCTR). In the general Bayesian framework, the handwritten Chinese text line is sequentially modeled by HMMs with each representing one character class, while the NN-based classifier is adopted to calculate the posterior probability of all HMM states. The key issues in feature extraction, character modeling, and language modeling are comprehensively investigated to show the effectiveness of NN-HMM framework for offline HCTR. First, a conventional deep neural network (DNN) architecture is studied with a well-designed feature extractor. As for the training procedure, the label refinement using forced alignment and the sequence training can yield significant gains on top of the frame-level cross-entropy criterion. Second, a deep convolutional neural network (DCNN) with automatically learned discriminative features demonstrates its superiority to DNN in the HMM framework. Moreover, to solve the challenging problem of distinguishing quite confusing classes due to the large vocabulary of Chinese characters, NN-based classifier should output 19900 HMM states as the classification units via a high-resolution modeling within each character. On the ICDAR 2013 competition task of CASIA-HWDB database, DNN-HMM yields a promising character error rate (CER) of 5.24% by making a good trade-off between the computational complexity and recognition accuracy. To the best of our knowledge, DCNN-HMM can achieve a best published CER of 3.53%.

Convolutional neural networks (CNNs) have gained great success in various computer vision applications. However, state-of-the-art CNN models are computation-intensive and hence are mainly processed on high performance processors like server CPUs and GPUs. Owing to the advantages of high performance, energy efficiency and reconfigurability, Field-Programmable Gate Arrays (FPGAs) have been widely explored as CNN accelerators. In this paper, we propose parallel structures to exploit the inherent parallelism and efficient computation units to perform operations in convolutional and fully-connected layers. Further, an automatic generator is proposed to generate Verilog HDL source code automatically according to high-level hardware description language. Execution time, DSP consumption and performance are analytically modeled based on some critical design variables. We demonstrate the automatic methodology by implementing two representative CNNs (LeNet and AlexNet) and evaluate the execution time models by comparing estimated and measured values. Our results show that the proposed automatic methodology yields hardware design with good performance and saves much developing round time.

Security and safety applications such as surveillance or forensics demand face recognition in low-resolution video data. We propose a face recognition method based on a Convolutional Neural Network (CNN) with a manifold-based track comparison strategy for low-resolution video face recognition. The low-resolution domain is addressed by adjusting the network architecture to prevent bottlenecks or significant upscaling of face images. The CNN is trained with a combination of a large-scale self-collected video face dataset and large-scale public image face datasets resulting in about 1.4M training images. To handle large amounts of video data and for effective comparison, the CNN face descriptors are compared efficiently on track level by local patch means. Our setup achieves 80.3 percent accuracy on a 32×32 pixels low-resolution version of the YouTube Faces Database and outperforms local image descriptors as well as the state-of-the-art VGG-Face network [20] in this domain. The superior performance of the proposed method is confirmed on a self-collected in-the-wild surveillance dataset.

The work described in this paper represents the study and the attempt to make a contribution to one of the most stimulating and promising sectors in the field of emotion recognition, which is health care management. Multidisciplinary studies in artificial intelligence, augmented reality and psychology stressed out the importance of emotions in communication and awareness. The intent is to recognize human emotions, processing images streamed in real-time from a mobile device. The adopted techniques involve the use of open source libraries of visual recognition and machine learning approaches based on convolutional neural networks (CNN).


Question: A convolutional neural network-based hand tracking system.

This paper introduces a novel method, Optimal Brain Damage, which employs information-theoretic principles to develop practical and nearly optimal strategies for adjusting the size of a neural network. By selectively eliminating insignificant weights within the network, several benefits are anticipated, including improved generalization capabilities, reduced requirements for training examples, and enhanced speed in both learning and classification processes. The core concept behind Optimal Brain Damage is to leverage second-derivative information to strike a balance between network complexity and training set error. The effectiveness of these methods is substantiated through experimental validation on a real-world application, demonstrating their practical utility in neural network optimization.

We describe a system to detect objects in three-dimensional space using video and inertial sensors (accelerometer and gyrometer), ubiquitous in modern mobile platforms from phones to drones. Inertials afford the ability to impose class-specific scale priors for objects, and provide a global orientation reference. A minimal sufficient representation, the posterior of semantic (identity) and syntactic (pose) attributes of objects in space, can be decomposed into a geometric term, which can be maintained by a localization-and-mapping filter, and a likelihood function, which can be approximated by a discriminatively-trained convolutional neural network The resulting system can process the video stream causally in real time, and provides a representation of objects in the scene that is persistent: Confidence in the presence of objects grows with evidence, and objects previously seen are kept in memory even when temporarily occluded, with their return into view automatically predicted to prime re-detection.

In this paper, we introduce the DEEP-SEE FACE framework, an assistive device designed to improve cognition, interaction, and communication of visually impaired (VI) people in social encounters. The proposed approach jointly exploits computer vision algorithms (region proposal networks, ATLAS tracking and global, and low-level image descriptors) and deep convolutional neural networks in order to detect, track, and recognize, in real-time, various persons existent in the video streams. The major contribution of the paper concerns a global, fixed-size face representation that takes into the account of various video frames while remaining independent of the length of the image sequence. To this purpose, we introduce an effective weight adaptation scheme that is able to determine the relevance assigned to each face instance, depending on the frame degree of motion/camera blur, scale variation, and compression artifacts. Another relevant contribution involves a hard negative mining stage that helps us differentiating between known and unknown face identities. The experimental results, carried out on a large-scale data set, validate the proposed methodology with an average accuracy and recognition rates superior to 92%. When tested in real life, indoor/outdoor scenarios, the DEEP-SEE FACE prototype proves to be effective and easy to use, allowing the VI people to access visual information during social events.

Correlation Filters (CFs) have recently demonstrated excellent performance in terms of rapidly tracking objects under challenging photometric and geometric variations. The strength of the approach comes from its ability to efficiently learn - on the fly - how the object is changing over time. A fundamental drawback to CFs, however, is that the background of the target is not modeled over time which can result in suboptimal performance. Recent tracking algorithms have suggested to resolve this drawback by either learning CFs from more discriminative deep features (e.g. DeepSRDCF [9] and CCOT [11]) or learning complex deep trackers (e.g. MDNet [28] and FCNT [33]). While such methods have been shown to work well, they suffer from high complexity: extracting deep features or applying deep tracking frameworks is very computationally expensive. This limits the real-time performance of such methods, even on high-end GPUs. This work proposes a Background-Aware CF based on hand-crafted features (HOG [6]) that can efficiently model how both the foreground and background of the object varies over time. Our approach, like conventional CFs, is extremely computationally efficient- and extensive experiments over multiple tracking benchmarks demonstrate the superior accuracy and real-time performance of our method compared to the state-of-the-art trackers.

This paper presents a new design of a wireless sensor glove developed for American Sign Language finger spelling gesture recognition. Five contact sensors are installed on the glove, in addition to five flex sensors on the fingers and a 3D accelerometer on the back of the hand. Each pair of flex and contact sensors are combined into the same input channel on the BSN node in order to save the number of channels and the installation area. After which, the signal is analyzed and separated back into flex and contact features by software. With electrical contacts and wirings made of conductive fabric and threads, the glove design has become thinner and more flexible. For validation, ASL finger spelling gesture recognition experiments have been performed on signals collected from six speech-impaired subjects and a normal subject. With the new sensor glove design, the experimental results have shown a significant increase in classification accuracy.

In this paper a novel approach that adopts Convolutional Neural Networks (CNN) for the Beat Tracking task is proposed. The proposed architecture involves 2 convolutional layers with the CNN filter dimensions corresponding to time and band frequencies, in order to learn a Beat Activation Function (BAF) from a time-frequency representation. The output of each convolutional layer is computed only over the past values of the previous layer, to enable the computation of the BAF in an online fashion. The output of the CNN is post-processed by a dynamic programming algorithm in combination with a bank of resonators for calculating the salient rhythmic periodicities. The proposed method has been designed to be computational efficient in order to be embedded on a dancing NAO robot application, where the dance moves of the choreography are synchronized with the beat tracking output. The proposed system was submitted to the Signal Processing Cup Challenge 2017 and ranked among the top third algorithms.

The work described in this paper represents the study and the attempt to make a contribution to one of the most stimulating and promising sectors in the field of emotion recognition, which is health care management. Multidisciplinary studies in artificial intelligence, augmented reality and psychology stressed out the importance of emotions in communication and awareness. The intent is to recognize human emotions, processing images streamed in real-time from a mobile device. The adopted techniques involve the use of open source libraries of visual recognition and machine learning approaches based on convolutional neural networks (CNN).

Locating hands in sign language video is challenging due to a number of factors. Hand appearance varies widely across signers due to anthropometric variations and varying levels of signer proficiency. Video can be captured under varying illumination, camera resolutions, and levels of scene clutter, e.g., high-res video captured in a studio vs. low-res video gathered by a Web cam in a user's home. Moreover, the signers' clothing varies, e.g., skin-toned clothing vs. contrasting clothing, short-sleeved vs. long-sleeved shirts, etc. In this work, the hand detection problem is addressed in an appearance matching framework. The histogram of oriented gradient (HOG) based matching score function is reformulated to allow non-rigid alignment between pairs of images to account for hand shape variation. The resulting alignment score is used within a support vector machine hand/not-hand classifier for hand detection. The new matching score function yields improved performance (in ROC area and hand detection rate) over the vocabulary guided pyramid match kernel (VGPMK) and the traditional, rigid HOG distance on American Sign Language video gestured by expert signers. The proposed match score function is computationally less expensive (for training and testing), has fewer parameters and is less sensitive to parameter settings than VGPMK. The proposed detector works well on test sequences from an inexpert signer in a non-studio setting with cluttered background.

Hand gesture-to-gesture translation in the wild is a challenging task since hand gestures can have arbitrary poses, sizes, locations and self-occlusions. Therefore, this task requires a high-level understanding of the mapping between the input source gesture and the output target gesture. To tackle this problem, we propose a novel hand Gesture Generative Adversarial Network (GestureGAN). GestureGAN consists of a single generator G and a discriminator D, which takes as input a conditional hand image and a target hand skeleton image. GestureGAN utilizes the hand skeleton information explicitly, and learns the gesture-to-gesture mapping through two novel losses, the color loss and the cycle-consistency loss. The proposed color loss handles the issue of "channel pollution" while back-propagating the gradients. In addition, we present the Frechet ResNet Distance (FRD) to evaluate the quality of generated images. Extensive experiments on two widely used benchmark datasets demonstrate that the proposed GestureGAN achieves state-of-the-art performance on the unconstrained hand gesture-to-gesture translation task. Meanwhile, the generated images are in high-quality and are photo-realistic, allowing them to be used as data augmentation to improve the performance of a hand gesture classifier. Our model and code are available at https://github.com/Ha0Tang/GestureGAN.

This paper proposes a novel method for offline text-independent writer identification by using convolutional neural network (CNN) and joint Bayesian, which consists of two stages, i.e. feature extraction and writer identification. In the stage of feature extraction, since a large number of data is essential to train an effective CNN model with high generalizability and the amount of handwriting is limited in writer identification, a data augmentation technique is first developed to generate thousands of handwriting images for each writer. Then a deep CNN network is designed to extract discriminative features to represent the properties of different writing styles, which is trained by using the generated handwriting images. In the stage of writer identification, the training dataset is used to train the CNN model for feature extraction and the joint Bayesian technique is employed to accomplish the task of writer identification based on the extracted CNN features. The proposed method is tested on two standard benchmark datasets, i.e. ICDAR2013 and CVL dataset. Experimental results demonstrate that the proposed method gets the best performance compared to the state-of-the-art approaches.

[1] J. Xiao, S. Baker, I. Matthews, T. Kanade, “Real-Time Combined 2D + 3D Active Appearance Models,” CVPR, pp.535-542, 2004. [2] N. Gourier, D. Hall, J. L. Crowley, “Estimating Face Orientation from Robust Detection of Salient Facial Features”, Proceedings of Pointing 2004, ICPR, International Workshop on Visual Observation of Deictic Gestures, 2004. [3] G. Fanelli, M. Dantone, J. Gall, A. Jssati, L. Van Gool, “Random Forests for Real Time 3D Face Analysis,” IJCV, pp.437-458, 2013. [4] Y. LeCun, R. Bottou, Y. Bengio, P. Haffner, “Gradient-based Learning Applied to Document Recognition,” Proc. IEEE, vol. 86, no. 11, pp.2278-2324, 1998. [5] A. Krizhevsky, I. Sutskever, G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” NIPS, pp.1106-1114, 2012. This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIP) (No. 2010-0028680). Proposed Method  Conventional Head Pose Estimation Approaches


Question: A convolutional neural network-based hand tracking system.

This document presents the development of a sophisticated digit recognition system, adept at identifying hand-printed numbers, through the fusion of traditional methodologies and neural network approaches. Rigorously trained and evaluated on authentic data obtained from U.S. postal mail zip codes, the system exhibits an impressive performance. It judiciously flags a marginal portion of inputs as unclassifiable, yet maintains a remarkably low error rate on the classified items. This system's efficacy is comparable to other leading-edge recognizers in the field. While some strategies are uniquely tailored for digit recognition, the broader goal is to ensure the applicability of these techniques across a variety of recognition challenges.

We present a novel approach for model-based 6D pose refinement in color data. Building on the established idea of contour-based pose tracking, we teach a deep neural network to predict a translational and rotational update. At the core, we propose a new visual loss that drives the pose update by aligning object contours, thus avoiding the definition of any explicit appearance model. In contrast to previous work our method is correspondence-free, segmentation-free, can handle occlusion and is agnostic to geometrical symmetry as well as visual ambiguities. Additionally, we observe a strong robustness towards rough initialization. The approach can run in real-time and produces pose accuracies that come close to 3D ICP without the need for depth data. Furthermore, our networks are trained from purely synthetic data and will be published together with the refinement code at http://campar.in.tum. de/Main/FabianManhardt to ensure reproducibility.

We propose to combine recent Convolutional Neural Networks (CNN) models with depth imaging to obtain a reliable and fast multi-person pose estimation algorithm applicable to Human Robot Interaction (HRI) scenarios. Our hypothesis is that depth images contain less structures and are easier to process than RGB images while keeping the required information for human detection and pose inference, thus allowing the use of simpler networks for the task. Our contributions are threefold. (i) we propose a fast and efficient network based on residual blocks (called RPM) for body landmark localization from depth images; (ii) we created a public dataset DIH comprising more than 170k synthetic images of human bodies with various shapes and viewpoints as well as real (annotated) data for evaluation; (iii) we show that our model trained on synthetic data from scratch can perform well on real data, obtaining similar results to larger models initialized with pre-trained networks. It thus provides a good trade-off between performance and computation. Experiments on real data demonstrate the validity of our approach.

Traditional architectures for solving computer vision problems and the degree of success they enjoyed have been heavily reliant on hand-crafted features. However, of late, deep learning techniques have offered a compelling alternative – that of automatically learning problem-specific features. With this new paradigm, every problem in computer vision is now being re-examined from a deep learning perspective. Therefore, it has become important to understand what kind of deep networks are suitable for a given problem. Although general surveys of this fast-moving paradigm (i.e., deep-networks) exist, a survey specific to computer vision is missing. We specifically consider one form of deep networks widely used in computer vision – convolutional neural networks (CNNs). We start with “AlexNet” as our base CNN and then examine the broad variations proposed over time to suit different applications. We hope that our recipe-style survey will serve as a guide, particularly for novice practitioners intending to use deep-learning techniques for computer vision.

In this paper, we show that learning features with convolutional neural networks is better than using hand-crafted features for handwritten word recognition. We consider two kinds of systems: a grapheme based segmentation and a sliding window segmentation. In both cases, the combination of a convolutional neural network with a HMM outperform a state-of-the art HMM system based on explicit feature extraction. The experiments are conducted on the Rimes database. The systems obtained with the two kinds of segmentation are complementary: when they are combined, they outperform the systems in isolation. The system based on grapheme segmentation yields lower recognition rate but is very fast, which is suitable for specific applications such as document classification.

Deep learning is currently popular in the field of computer vision and pattern recognition. Deep learning models such as Convolutional Neural Networks (CNNs) have been shown to be extremely effective at solving computer vision problems with state-of-the-art results. This work uses the CNN model to achieve a high degree of accuracy on the problem of fingerprint image classification. It will also be shown that effective image preprocessing can greatly reduce the dimensionality of the problem, allowing fast training times without compromising classification accuracy, even in networks of moderate depth. The proposed approach has achieved 95.9% classification accuracy on the NIST-DB4 dataset with zero sample rejection.

Faces represent complex, multidimensional, meaningful visual stimuli an d developing a computational model for face recognition is difficult [42]. We present a hybrid neural network solution which compares favorably with other methods. The system combines local image sam pling, a self-organizing map neural network, and a convolutional neural network. The self-organizi ng map provides a quantization of the image samples into a topological space where inputs that are n e rby in the original space are also nearby in the output space, thereby providing dimensionality red uction and invariance to minor changes in the image sample, and the convolutional neural network prov ides for partial invariance to translation, rotation, scale, and deformation. The convolutional net work extracts successively larger features in a hierarchical set of layers. We present results using the Karhunen -Loève transform in place of the self-organizing map, and a multi-layer perceptron in place of the con volutional network. The Karhunen-Loève transform performs almost as well (5.3% error versus 3 .8%). The multi-layer perceptron performs very poorly (40% error versus 3.8%). The method is cap able of rapid classification, requires only fast, approximate normalization and preprocessing, and cons istently exhibits better classification performance than the eigenfaces approach [42] on the database consider ed as the number of images per person in the training database is varied from 1 to 5. With 5 imag es per person the proposed method and eigenfaces result in 3.8% and 10.5% error respectively. The recogni zer provides a measure of confidence in its output and classification error approaches zero when rejecting as few as 10% of the examples. We use a database of 400 images of 40 individuals which cont ains quite a high degree of variability in expression, pose, and facial details. We analyze computati onal complexity and discuss how new classes could be added to the trained recognizer.

We present a new motion tracking technique to robustly reconstruct non-rigid geometries and motions from a single view depth input recorded by a consumer depth sensor. The idea is based on the observation that most non-rigid motions (especially human-related motions) are intrinsically involved in articulate motion subspace. To take this advantage, we propose a novel <inline-formula><tex-math notation="LaTeX">$L_0$</tex-math><alternatives> <inline-graphic xlink:href="xu-ieq2-2688331.gif"/></alternatives></inline-formula> based motion regularizer with an iterative solver that implicitly constrains local deformations with articulate structures, leading to reduced solution space and physical plausible deformations. The <inline-formula><tex-math notation="LaTeX">$L_0$</tex-math> <alternatives><inline-graphic xlink:href="xu-ieq3-2688331.gif"/></alternatives></inline-formula> strategy is integrated into the available non-rigid motion tracking pipeline, and gradually extracts articulate joints information online with the tracking, which corrects the tracking errors in the results. The information of the articulate joints is used in the following tracking procedure to further improve the tracking accuracy and prevent tracking failures. Extensive experiments over complex human body motions with occlusions, facial and hand motions demonstrate that our approach substantially improves the robustness and accuracy in motion tracking.

We introduce our solution of the problem of clothing recognition and identification. The cause of taking convolutional neural network as a mathematic model for the further research was the result of an experimental survey of methods of keypoints detection and computing their descriptors. The convolutional neural network model was developed and trained on the Apparel Classification with Style clothing image dataset. The training image dataset was expanded with the help of augmentation methods. The computational experiment aimed at determination of neural network efficiency and stability proved its productivity. Moreover, we developed clothing color recognition, which let us perform clothing identification by 2 parameters: type and color.

We present the design and construction of a handheld mirror simulation device. The perception of the world reflected through a mirror depends on the viewer's position with respect to the mirror and the 3-D geometry of the world. In order to simulate a real mirror on a computer screen, images of the observed world, consistent with the viewer's position, must be synthesized and displayed in realtime. Our system is build around a LCD screen manipulated by the user, a single camera fixed on the screen, and a tracking device. The continuous input video stream and tracker data is used to synthesize, in real-time, a continuous video stream displayed on the LCD screen. The synthesized video stream is a close approximation of what the user would see on the screen surface if it were a real mirror. Our system provides a generic interface for applications involving rich, first-person interaction, such as the Virtual Daguerreotype.

This paper proposes a novel method for offline text-independent writer identification by using convolutional neural network (CNN) and joint Bayesian, which consists of two stages, i.e. feature extraction and writer identification. In the stage of feature extraction, since a large number of data is essential to train an effective CNN model with high generalizability and the amount of handwriting is limited in writer identification, a data augmentation technique is first developed to generate thousands of handwriting images for each writer. Then a deep CNN network is designed to extract discriminative features to represent the properties of different writing styles, which is trained by using the generated handwriting images. In the stage of writer identification, the training dataset is used to train the CNN model for feature extraction and the joint Bayesian technique is employed to accomplish the task of writer identification based on the extracted CNN features. The proposed method is tested on two standard benchmark datasets, i.e. ICDAR2013 and CVL dataset. Experimental results demonstrate that the proposed method gets the best performance compared to the state-of-the-art approaches.


Question: A convolutional neural network-based hand tracking system.

This paper explores the statistical framework for learning complex input-output relationships using layered neural networks, emphasizing the importance of consistency in probability inference. By aligning the error minimization process with likelihood maximization for network training, the authors establish a Gibbs distribution applicable to a canonical ensemble of networks sharing a similar architecture. This statistical approach facilitates the assessment of the probability of accurate predictions for independent examples, post-training on a designated dataset. Notably, the prediction probability exhibits a strong correlation with the network's generalization capability, as evaluated beyond the training set. This insight leads to a practical and versatile criterion for network training, focusing on the minimization of prediction errors. The effectiveness of this criterion in determining the optimal network architecture is illustrated through its application to the continuity problem. Theoretically, the paper delves into the concept of learning curves, offering an estimation of the requisite training size for achieving accurate generalization in a straightforward scenario. This work underscores the significance of statistical consistency in enhancing the predictive and generalization capacities of layered neural networks.

We present a real-time face tracker in this paper The system has achieved a rate of 30% frameshecond using an HP-9000 workstation with a framegrabber and a Canon VC-CI camera. It can track a person 'sface while the person moves freely (e.g., walks, jumps, sits down and stands up) in a room. Three types of models have been employed in developing the system. First, we present a stochastic model to characterize skin-color distributions of human faces. The information provided by the model is sufJicient for tracking a human face in various poses and views. This model is adaptable to different people and different lighting conditions in real-time. Second, a motion model e's used to estimate image motion and to predict search window. Third, a camera model is used toprediet and to compensate for camera motion. The system can be applied to tele-conferencing and many HCI applications including lip-reading and gaze tracking. The principle in developing this system can be extended to other tracking problems such as tracking the human hand.

Real Time Human-machine Interaction system using hand gesture Recognition to handle the mouse event , media player , image viewer .Users have to repeat same mouse and keyboard actions, inducing waste of time. Gestures have long been considered as an interaction technique that can potentially deliver more natural. A fast gesture recognition scheme is proposed to be an interface for the human-machine interaction (HMI) of systems. The system presents some lowcomplexity algorithms and gestures to reduce the gesture recognition complexity and be more suitable for controlling real-time computer systems. In this paper we use the webcam for capturing the image. After capturing the image it converts into the binary image. A gesture is a specific combination of hand position.

In Proc. of IEEE Conf. on CVPR’2000, Vol.II, pp.88-94, Hilton Head Island, SC, 2000 Since human hand is highly articulated and deformable, hand posture recognition is a challenging example in the research of view-independent object recognition. Due to the difficulties of the modelbased approach, the appearance-based learning approach is promising to handle large variation in visual inputs. However, the generalization of many proposed supervised learning methods to this problem often suffers from the insufficiency of labeled training data. This paper describes an approach to alleviate this difficulty by adding a large unlabeled training set. Combining supervised and unsupervised learning paradigms, a novel and powerful learning approach, the Discriminant-EM (D-EM) algorithm, is proposed in this paper to handle the case of small labeled training set. Experiments show that D-EM outperforms many other learning methods. Based on this approach, we implement a gesture interface to recognize a set of predefined gesture commands, and it is also extended to hand detection. This algorithm can also apply to other object recognition tasks.

In this paper we have proposed and designed FindingHuMo (Finding Human Motion), a real-time user tracking system for Smart Environments. FindingHuMo can perform device-free tracking of multiple (unknown and variable number of) users in the Hallway Environments, just from non-invasive and anonymous (not user specific) binary motion sensor data stream. The significance of our designed system are as follows: (a) fast tracking of individual targets from binary motion data stream from a static wireless sensor network in the infrastructure. This needs to resolve unreliable node sequences, system noise and path ambiguity, (b) Scaling for multi-user tracking where user motion trajectories may crossover with each other in all possible ways. This needs to resolve path ambiguity to isolate overlapping trajectories, FindingHumo applies the following techniques on the collected motion data stream: (i) a proposed motion data driven adaptive order Hidden Markov Model with Viterbi decoding (called Adaptive-HMM), and then (ii) an innovative path disambiguation algorithm (called CPDA). Using this methodology the system accurately detects and isolates motion trajectories of individual users. The system performance is illustrated with results from real-time system deployment experience in a Smart Environment.

In computer vision many tasks are solved using machine learning. In the past few years, state of the art results in computer vision have been achieved using deep learning. Deeper machine learning architectures are better capable in handling complex recognition tasks, compared to previous more shallow models. Many architectures for computer vision make use of convolutional neural networks which were modeled after the visual cortex. Currently deep convolutional neural networks are the state of the art in computer vision. Through a literature survey an overview is given of the components in deep convolutional neural networks. The role and design decisions for each of the components is presented, and the difficulties involved in training deep neural networks are given. To improve deep learning architectures an analysis is given of the activation values in four different architectures using various activation functions. Current state of the art classifiers use dropout, max-pooling as well as the maxout activation function. New components may further improve the architecture by providing a better solution for the diminishing gradient problem.

Short-range continuous-wave Doppler radar sensors have been mainly used for noncontact detection of various motions. In this paper, we investigate the feasibility to implement the function of a remote mouse, an input device of a computer, by recognizing human gestures based on a dual-channel Doppler radar sensor. Direct conversion architecture, symmetric subcarrier modulation, and bandpass sampling techniques are used to obtain a cost-effective solution. An arcsine algorithm and a motion imaging algorithm are proposed to linearly reconstruct the hand and finger motions in a 2-D plane from the demodulated Doppler phase shifts. Experimental results verified the effectiveness of the proposed architecture and algorithms. Different from the frequency-domain “micro-Doppler” approach, the proposed remote gesture recognition based on linear motion reconstruction is able to recognize definitive signatures for the corresponding motions, exhibiting promising potential in practical applications of human-computer interaction.

Automatic recognition of human actions from video has been studied for many years. Although still very difficult in uncontrolled scenarios, it has been successful in more restricted settings (e.g., fixed viewpoint, no occlusions) with recognition rates approaching 100%. However, the best-performing methods are complex and computationally-demanding and thus not well-suited for real-time deployments. This paper proposes to leverage the Kinect camera for close-range gesture recognition using two methods. Both methods use feature vectors that are derived from the skeleton model provided by the Kinect SDK in real-time. Although both methods perform nearest-neighbor classification, one method does this in the space of features using the Euclidean distance metric, while the other method does this in the space of feature covariances using a log-Euclidean metric. Both methods recognize 8 hand gestures in real time achieving correct-classification rates of over 99% on a dataset of 20 subjects but the method based on Euclidean distance requires feature-vector collections to be of the same size, is sensitive to temporal misalignment, and has higher computation and storage requirements.

During the recent years, correlation filters have shown dominant and spectacular results for visual object tracking. The types of the features that are employed in this family of trackers significantly affect the performance of visual tracking. The ultimate goal is to utilize the robust features invariant to any kind of appearance change of the object, while predicting the object location as properly as in the case of no appearance change. As the deep learning based methods have emerged, the study of learning features for specific tasks has accelerated. For instance, discriminative visual tracking methods based on deep architectures have been studied with promising performance. Nevertheless, correlation filter based (CFB) trackers confine themselves to use the pre-trained networks, which are trained for object classification problem. To this end, in this manuscript the problem of learning deep fully convolutional features for the CFB visual tracking is formulated. In order to learn the proposed model, a novel and efficient backpropagation algorithm is presented based on the loss function of the network. The proposed learning framework enables the network model to be flexible for a custom design. Moreover, it alleviates the dependency on the network trained for classification. Extensive performance analysis shows the efficacy of the proposed custom design in the CFB tracking framework. By fine-tuning the convolutional parts of a state-of-the-art network and integrating this model to a CFB tracker, which is the top performing one of VOT2016, 18% increase is achieved in terms of expected average overlap, and tracking failures are decreased by 25%, while maintaining the superiority over the state-of-the-art methods in OTB-2013 and OTB-2015 tracking datasets.

KinectFusion enables a user holding and moving a standard Kinect camera to rapidly create detailed 3D reconstructions of an indoor scene. Only the depth data from Kinect is used to track the 3D pose of the sensor and reconstruct, geometrically precise, 3D models of the physical scene in real-time. The capabilities of KinectFusion, as well as the novel GPU-based pipeline are described in full. Uses of the core system for low-cost handheld scanning, and geometry-aware augmented reality and physics-based interactions are shown. Novel extensions to the core GPU pipeline demonstrate object segmentation and user interaction directly in front of the sensor, without degrading camera tracking or reconstruction. These extensions are used to enable real-time multi-touch interactions anywhere, allowing any planar or non-planar reconstructed physical surface to be appropriated for touch.

We propose a novel end-to-end deep architecture for face landmark detection, based on a deep convolutional and deconvolutional network followed by carefully designed recurrent network structures. The pipeline of this architecture consists of three parts. Through the first part, we encode an input face image to resolution-preserved deconvolutional feature maps via a deep network with stacked convolutional and deconvolutional layers. Then, in the second part, we estimate the initial coordinates of the facial key points by an additional convolutional layer on top of these deconvolutional feature maps. In the last part, by using the deconvolutional feature maps and the initial facial key points as input, we refine the coordinates of the facial key points by a recurrent network that consists of multiple long short-term memory components. Extensive evaluations on several benchmark data sets show that the proposed deep architecture has superior performance against the state-of-the-art methods.


Question: A single-degree-of-freedom superimposed rigid origami structure with multiple configurations.

This comprehensive review explores the utilization of inflatable booms in the development of deployable space structures, focusing on two key aspects: packing strategies for compact storage and rigidization methods to ensure structural integrity post-deployment. Inflatable booms, particularly cylindrical in shape, are essential components for truss structures and support systems for solar sails in space applications. The review synthesizes extensive research to outline the current state of the art in packaging techniques that facilitate the efficient stowage of these lightweight structures while guaranteeing their seamless deployment in space. Additionally, it delves into the various rigidization approaches aimed at maintaining long-term structural rigidity, a critical requirement for the functionality and longevity of inflatable space structures. By consolidating knowledge from a wide array of studies, this review provides a detailed overview of the design considerations and technological advancements in inflatable cylindrical booms for space deployment.

In this paper, we develop a holonomic omni-directional mobile vehicle with step-climbing ability. This system realizes omni-directional motion on flat floor using special wheels and passes over the step in forward or backward direction using the passive linkage mechanism. Our key ideas are the following topics. First topic is new omnidirectional mobile mechanism which consists of special wheels and passive linkage mechanism. Second topic is new passive linkage mechanism which can change its body configuration easily when the vehicle passes over the step. Third topic is wheel control reference derivation based on the body configuration, which changes passively during step climbing for reducing wheel slippage. Last topic is wheel control method which keeps the rotation velocity coordination among the wheels for reducing wheel slippage and increasing the step-climbing performance. We verified the effectiveness of our proposed method by the computer simulations and experiments. Utilizing our proposed mechanism and control systems, the vehicle has both omnidirectional mobility and step overcoming function. Furthermore, our developing vehicle can pass over the 128[mm] height step using the wheel which radius is 66[mm]. Its performance is three times larger than one of general wheeled vehicle.

This paper proposes a new design for stair-climbing using a wheel that can split into segments and walk up stairs or surmount other obstacles often found where humans traverse, while still being able to retain a perfectly round shape for traveling on smooth ground. Using this change of configuration, staircases with a wide range of dimensions can be covered efficiently and safely. The design, named Umbrella Wheel, can consist of as many wheel segments as desired, and as few as two. A smaller or higher number of wheel segments has advantages and disadvantages depending on the specific situation. Modeling the trajectory of the wheel when as it ascends or descends stairs is given and the results are analyzed.

It is important for a wearable robot to be compact and sufficiently light for use as an assistive device. Since human fingers are arranged in a row in dense space, the concept of traditional wearable robots using a rigid frame and a pin joint result in size and complexity problems. A structure without a conventional pin joint, called a jointless structure, has the potential to be used as a wearable robotic hand because the human skeleton and joint can replace the robot's conventional structure. Another way to reduce the weight of the system is to use under-actuation. Under-actuation enables adaptive grasping with less number of actuators for robotic hands. Differential mechanisms are widely used for multi-finger under-actuation; however, they require additional working space. We propose a design with a jointless structure and a novel under-actuation mechanism to reduce the size and weight of a hand exoskeleton. Using these concepts, we developed a prototype that weighs only 80 grams. To evaluate the prototype, fingertip force and blocked force are measured. Fingertip force is the force that can be applied by the finger of the hand exoskeleton on the object surface. The fingertip force is about 18 N when actuated by a tension force of 35 N from the motor. 18 N is sufficient for simple pinch motion in daily activities. Another factor related to performance of the under-actuation mechanism is blocked force, which is a force required to stop one finger while the other finger keeps on moving. It is measured to be 0.5 N, which is sufficiently small. With these experiments, the feasibility of the new hand exoskeleton has been shown.

This work presents a soft hand capable of robustly grasping and identifying objects based on internal state measurements. A highly compliant hand allows for intrinsic robustness to grasping uncertainty, but the specific configuration of the hand and object is not known, leaving undetermined if a grasp was successful in picking up the right object. A soft finger was adapted and combined to form a three finger gripper that can easily be attached to existing robots, for example, to the wrist of the Baxter robot. Resistive bend sensors were added within each finger to provide a configuration estimate sufficient for distinguishing between a set of objects. With one data point from each finger, the object grasped by the gripper can be identified. A clustering algorithm to find the correspondence for each grasped object is presented for both enveloping grasps and pinch grasps. This hand is a first step towards robust proprioceptive soft grasping.

There is a strong demand in many fields for practical robots, such as a porter robot and a personal mobility robot, that can move over rough terrain while carrying a load horizontally. We have developed a robot, called RT-Mover, which shows adequate mobility performance on targeted types of rough terrain. It has four drivable wheels and two leg-like axles but only five active shafts. A strength of this robot is that it realizes both a leg mode and a wheel mode in a simple mechanism. In this paper, the mechanical design concept is discussed. With an emphasis on minimizing the number of drive shafts, a mechanism is designed for a four-wheeled mobile body that is widely used in practical locomotive machinery. Also, strategies for moving on rough terrain are proposed. The kinematics, stability, and control of RT-Mover are also described in detail. Some typical cases of rough terrain for wheel mode and leg mode are selected, and the robot’s ability of locomotion is assessed through simulations and experiments. In each case, the robot is able to move over rough terrain while maintaining the horizontal orientation of its platform.

Electromagnetic vortex imaging based on orbital angular momentum (OAM) modulation has the ability of azimuthal resolution without relative motion, which needs various OAM modes. However, the main-lobe directions of different OAM beams differ from each other so that the echo energy is limited. In this letter, several concentric circular arrays are employed to adjust the directivity, each array generating a beam carrying special OAM mode. With this scheme, the target can be illuminated by main lobes of all OAM beams simultaneously. Our proposal can benefit the image reconstruction in noisy environment. Moreover, this scheme can successfully reduce the artifacts caused by high sidelobes at the cost of reduction in azimuthal resolution. Simulation results demonstrate the effectiveness of this scheme.

This paper presents the design and development of a single motor actuated peristaltic worm robot with three segments using a bio-inspired method of locomotion with one actuator that achieves optimised worm like peristaltic motion. Each segment consists of two solid circular disks that have a tendon connected through to the drive mechanism using a Bowden cable and a soft rubber skin that deforms under the compression of the tendon. Our hypothesis that a tuned peristaltic waveform can achieve improved performance of locomotion distance and clamping strength is proven using an initial test platform capable of demonstrating varying waveform types with multiple actuators. Three experiments were undertaken: (i) moving along a flat surface, (ii) moving through a confined tunnel (iii) moving through a confined tunnel whilst pulling a payload. Results from these experiments have identified the optimal parameters for a smart gearbox capable of achieving the same optimal peristaltic waveform signal as the more complex test platform but with only one actuator driving all three worm segments. Unlike other examples of peristaltic worm robots, this example uses a control method embedded within the mechanics of the design removing excessive number of actuators which contributes to miniaturisation, reduces power consumption and simplifies the overall design.

There are many advantages to the use of spherical robot designs. The current status of the design of spherical rolling robots is reviewed.

Dragonfly wings are highly specialized locomotor systems, which are formed by a combination of several structural components. The wing components, also known as structural elements, are responsible for the various aspects of the wing functionality. Considering the complex interactions between the wing components, modelling of the wings as a whole is only possible with inevitable huge oversimplifications. In order to overcome this difficulty, we have recently proposed a new approach to model individual components of complex wings comparatively. Here, we use this approach to study nodus, a structural element of dragonfly wings which has been less studied to date. Using a combination of several imaging techniques including scanning electron microscopy (SEM), wide-field fluorescence microscopy (WFM), confocal laser scanning microscopy (CLSM) and micro-computed tomography (micro-CT) scanning, we aim to characterize the spatial morphology and material composition of fore- and hindwing nodi of the dragonfly Brachythemis contaminata. The microscopy results show the presence of resilin in the nodi, which is expected to help the deformability of the wings. The computational results based on three-dimensional (3D) structural data suggest that the specific geometry of the nodus restrains its displacements when subjected to pressure on the ventral side. This effect, resulting from an interlocking mechanism, is expected to contribute to the dorso-ventral asymmetry of wing deformation and to provide a higher resistance to aerodynamic forces during the downstroke. Our results provide an important step towards better understanding of the structure-property-function relationship in dragonfly wings.

STATEMENT OF SIGNIFICANCE
In this study, we investigate the wing nodus, a specialized wing component in dragonflies. Using a combination of modern imaging techniques, we demonstrate the presence of resilin in the nodus, which is expected to facilitate the wing deformability in flight. The specific geometry of the nodus, however, seems to restrain its displacements when subjected to pressure on the ventral side. This effect, resulting from an interlocking mechanism, is suggested to contribute to dorso-ventral asymmetry of wing deformations and to provide a higher resistance to aerodynamic forces during the downstroke. Our results provide an important step towards better understanding of the structure-property-function relationship in dragonfly wings and might help to design more efficient wings for biomimetic micro-air vehicles.

A new method, embedded-3D printing (e-3DP), is reported for fabricating strain sensors within highly conformal and extensible elastomeric matrices. e-3DP allows soft sensors to be created in nearly arbitrary planar and 3D motifs in a highly programmable and seamless manner. Several embodiments are demonstrated and sensor performance is characterized.