Question: Mobile device management for secure and manageable health data collection in resource-limited regions.

Introducing Open Data Kit (ODK): An Open-Source Platform for Information Services in Underprivileged Areas

This paper introduces Open Data Kit (ODK), an open-source, adaptable suite of tools aimed at facilitating information services in underprivileged regions. ODK comprises four core tools: Collect, Aggregate, Voice, and Build. Collect is a mobile application that facilitates data manipulation and executes application logic. Aggregate offers a server solution that can be deployed with a single click, supporting data storage and transfer in the cloud or on local servers. Voice renders application logic through phone prompts, allowing users to interact using keypad inputs. Build serves as an application designer, generating the logic for the other tools. These tools can be used in conjunction or independently, adhering to established open standards. ODK benefits from a vibrant open-source community that has contributed additional tools, enhancing its functionality. The paper details four case studies that illustrate how ODK's architectural decisions enable bidirectional information flow in developing areas, making it a valuable resource for secure and manageable health data collection.

The deployment of ODK in under-resourced areas has demonstrated its capability to streamline health data collection, ensuring security and manageability. By leveraging mobile devices and a robust server infrastructure, ODK facilitates the gathering and processing of health data. Its modular design allows for flexibility in deployment, catering to the specific needs of each region. The inclusion of Voice as a tool ensures accessibility for populations with varying levels of technological literacy, making health data collection more inclusive. ODK's open-source nature fosters a collaborative environment, enabling continuous improvement and adaptation to evolving needs. Through its innovative approach, ODK is transforming health data collection in underprivileged areas, paving the way for improved health outcomes.

In Europe, in a context of growing population and decreasing resources, ageing related diseases represent one of the most relevant challenges in terms of healthcare organization complexity, plus care levels and system financing balancing. Nowadays there are several researches that are studying the application of the IoT (Internet of Things) paradigm in the e-health field. This paper explains how a solution, built on the top of the VIRTUS IoT middleware, provides a valid alternative to current IoT solutions, which are mainly based on SOA (Service Oriented Architecture). VIRTUS leverage an Instant Messaging protocol (XMPP) to guarantee a (near) real-time, secure and reliable communication channel among heterogeneous devices. The presented development has been exploited in a healthcare case study: an implementation of a cost-savvy remote body movement monitoring system, aimed at classify daily patients' activities, designed as a modular architecture and deployed in a large scale scenario. The paper analyzes the features offered by the VIRTUS middleware, if used within a e-health solution, providing a comparison with other well-known systems.

BACKGROUND
Physicians have limited time for weight-loss counseling, and there is a lack of resources to which they can refer patients for assistance with weight loss. Weight-loss mobile applications (apps) have the potential to be a helpful tool, but the extent to which they include the behavioral strategies included in evidence-based interventions is unknown.

PURPOSE
The primary aims of the study were to determine the degree to which commercial weight-loss mobile apps include the behavioral strategies included in evidence-based weight-loss interventions, and to identify features that enhance behavioral strategies via technology.

METHODS
Thirty weight-loss mobile apps, available on iPhone and/or Android platforms, were coded for whether they included any of 20 behavioral strategies derived from an evidence-based weight-loss program (i.e., Diabetes Prevention Program). Data on available apps were collected in January 2012; data were analyzed in June 2012.

RESULTS
The apps included on average 18.83% (SD=13.24; range=0%-65%) of the 20 strategies. Seven of the strategies were not found in any app. The most common technology-enhanced features were barcode scanners (56.7%) and a social network (46.7%).

CONCLUSIONS
Weight-loss mobile apps typically included only a minority of the behavioral strategies found in evidence-based weight-loss interventions. Behavioral strategies that help improve motivation, reduce stress, and assist with problem solving were missing across apps. Inclusion of additional strategies could make apps more helpful to users who have motivational challenges.

Mobile phone sensing is a new paradigm which takes advantage of the pervasive smartphones to collect and analyze data beyond the scale of what was previously possible. In a mobile phone sensing system, the platform recruits smartphone users to provide sensing service. Existing mobile phone sensing applications and systems lack good incentive mechanisms that can attract more user participation. To address this issue, we design incentive mechanisms for mobile phone sensing. We consider two system models: the platform-centric model where the platform provides a reward shared by participating users, and the user-centric model where users have more control over the payment they will receive. For the platform-centric model, we design an incentive mechanism using a Stackelberg game, where the platform is the leader while the users are the followers. We show how to compute the unique Stackelberg Equilibrium, at which the utility of the platform is maximized, and none of the users can improve its utility by unilaterally deviating from its current strategy. For the user-centric model, we design an auction-based incentive mechanism, which is computationally efficient, individually rational, profitable, and truthful. Through extensive simulations, we evaluate the performance and validate the theoretical properties of our incentive mechanisms.

This article describes the initial period (1994–2001) of an ongoing action research project to develop health information systems to support district management in South Africa. The reconstruction of the health sector in postapartheid South Africa striving for equity in health service delivery and building of a decentralized structure based on health districts. In terms of information systems (IS) development, this reform process translates into standardization of health data in ways that inscribe the goals of the new South Africa by enhancing local control and integration of information handling. We describe our approach to action research and use concepts from actor-network and structuration theories in analyzing the case material. In the detailed description and analysis of the process of IS development provided, we focus on the need to balance standardization and local  exibility (localization); standardization is thus seen as bottom-up alignment of an array of heterogeneous actors. Building on a social system model of information systems, we conceptualize the IS design strategy developed and used as the cultivation of processes whereby these actors are translating and aligning their interests. We develop a modular hierarchy of global and local datasets as a framework within which the tensions between standardization and localization may be understood and addressed. Finally, we discuss the possible relevance of the results of the research in other countries.

Homes are rich with information about people's energy consumption, medical health, and personal or family functions. In this paper, we present our experiences deploying large-scale residential sensing systems in over 20 homes. Deploying small-scale systems in homes can be deceptively easy, but in our deployments we encountered a phase transition in which deployment effort increases dramatically as residential deployments scale up in terms of 1) the number of nodes, 2) the length of time, and 3) the number of houses. In this paper, we distill our experiences down to a set of guidelines and design principles to help future deployments avoid the potential pitfalls of large-scale sensing in homes.

Wireless body area networks (BANs) have drawn much attention from research community and industry in recent years. Multimedia healthcare services provided by BANs can be available to anyone, anywhere, and anytime seamlessly. A critical issue in BANs is how to preserve the integrity and privacy of a person's medical data over wireless environments in a resource efficient manner. This paper presents a novel key agreement scheme that allows neighboring nodes in BANs to share a common key generated by electrocardiogram (ECG) signals. The improved Jules Sudan (IJS) algorithm is proposed to set up the key agreement for the message authentication. The proposed ECG-IJS key agreement can secure data commnications over BANs in a plug-n-play manner without any key distribution overheads. Both the simulation and experimental results are presented, which demonstrate that the proposed ECG-IJS scheme can achieve better security performance in terms of serval performance metrics such as false acceptance rate (FAR) and false rejection rate (FRR) than other existing approaches. In addition, the power consumption analysis also shows that the proposed ECG-IJS scheme can achieve energy efficiency for BANs.

The present-day health data ecosystem comprises a wide array of complex heterogeneous data sources. A wide range of clinical, health care, social and other clinically relevant information are stored in these data sources. These data exist either as structured data or as free-text. These data are generally individual person-based records, but social care data are generally case based and less formal data sources may be shared by groups. The structured data may be organised in a proprietary way or be coded using one-of-many coding, classification or terminologies that have often evolved in isolation and designed to meet the needs of the context that they have been developed. This has resulted in a wide range of semantic interoperability issues that make the integration of data held on these different systems changing. We present semantic interoperability challenges and describe a classification of these. We propose a four-step process and a toolkit for those wishing to work more ontologically, progressing from the identification and specification of concepts to validating a final ontology. The four steps are: (1) the identification and specification of data sources; (2) the conceptualisation of semantic meaning; (3) defining to what extent routine data can be used as a measure of the process or outcome of care required in a particular study or audit and (4) the formalisation and validation of the final ontology. The toolkit is an extension of a previous schema created to formalise the development of ontologies related to chronic disease management. The extensions are focused on facilitating rapid building of ontologies for time-critical research studies.

Nowadays, new tools and technologies are emerging rapidly. They are often used cross-culturally before being tested for suitability and validity. However, they must be validated to ensure that they work with all users, not just part of them. Mobile banking (as a new technology tool) has been introduced assuming that it performs well concerning authentication, among all members of the society. Our research aimed to evaluate authentication mobile banking user acceptance, through Technology Acceptance Model (TAM), in Arabic countries, namely Yemen. The results confirm the previous studies that have shown the importance of perceived ease of use and perceived usefulness. Furthermore, perceived ease of use plays a determinant role. KeywordsTechnology acceptance models; Mobile Banking;

The ubiquity of smartphones and their on-board sensing capabilities motivates crowd-sensing, a capability that harnesses the power of crowds to collect sensor data from a large number of mobile phone users. Unlike previous work on wireless sensing, crowd-sensing poses several novel requirements: support for humans-in-the-loop to trigger sensing actions or review results, the need for incentives, as well as privacy and security. Beyond existing crowd-sourcing systems, crowd-sensing exploits sensing and processing capabilities of mobile devices. In this paper, we design and implement Medusa, a novel programming framework for crowd-sensing that satisfies these requirements. Medusa provides high-level abstractions for specifying the steps required to complete a crowd-sensing task, and employs a distributed runtime system that coordinates the execution of these tasks between smartphones and a cluster on the cloud. We have implemented ten crowd-sensing tasks on a prototype of Medusa. We find that Medusa task descriptions are two orders of magnitude smaller than standalone systems required to implement those crowd-sensing tasks, and the runtime has low overhead and is robust to dynamics and resource attacks.

The current evolution of the traditional medical model toward the participatory medicine can be boosted by the Internet of Things (IoT) paradigm involving sensors (environmental, wearable, and implanted) spread inside domestic environments with the purpose to monitor the user's health and activate remote assistance. RF identification (RFID) technology is now mature to provide part of the IoT physical layer for the personal healthcare in smart environments through low-cost, energy-autonomous, and disposable sensors. It is here presented a survey on the state-of-the-art of RFID for application to body centric systems and for gathering information (temperature, humidity, and other gases) about the user's living environment. Many available options are described up to the application level with some examples of RFID systems able to collect and process multichannel data about the human behavior in compliance with the power exposure and sanitary regulations. Open challenges and possible new research trends are finally discussed.


Question: A mobile device administration strategy to ensure secure and manageable health data collection in under-resourced regions.

Introducing MyExperience: A Comprehensive System for Real-Time Collection of User Feedback on Mobile Devices

This paper details MyExperience, a sophisticated system designed to capture both objective and subjective data in real-time regarding mobile computing activities. MyExperience integrates two key methods: 1) passive monitoring of device usage, user context, and environmental sensor data, and 2) active, context-driven user experience sampling to gather on-the-spot, subjective user feedback. Currently operational on mobile phones, MyExperience can log over 140 distinct event categories, encompassing: 1) device usage details like communication patterns, application engagement, and media capture, 2) user context information such as calendar events, and 3) environmental sensor data including Bluetooth and GPS readings. Notably, user experience sampling can be precisely targeted to moments of interest by initiating collection based on sensor readings. We showcase the efficacy of MyExperience through various field deployments on individuals' personal phones, demonstrating its potential to provide deep insights into how users interact with and perceive mobile technology, particularly relevant for secure and manageable health data collection in under-resourced areas.

According to recent articles in popular technology websites, some mobile applications function in an insecure manner when presented with untrusted SSL certificates. These non-browser based applications seem to, in the absence of a standard way of alerting a user of an SSL error, accept any certificate presented to it. This paper intends to research these claims and show whether or not an invisible proxy based SSL attack can indeed steal user's credentials from mobile applications, and which types applications are most likely to be vulnerable to this attack vector. To ensure coverage of the most popular platforms, applications on both Android 4.2 and iOS 6 are tested. The results of our study showed that stealing credentials is indeed possible using invisible proxy man in the middle attacks.

A log is a collection of record of events that occurs within an organization containing systems and network. These logs are very important for any organization, because log file will able to record all user activities. As this log files plays vital role and also it contains sensitive information , it should be maintained highly secure. So, management and securely maintenance of log records are very tedious task. However, deploying such a system for high security and privacy of log records is an overhead for an organization and also it requires additional cost. Many techniques have been design so far for security of log records. The alternative solution is to maintaining log records over a cloud database. Log files over cloud environment leads to challenges about privacy, confidentiality and integrity of log files. In this paper, we propose highly secured cloud rendered log management and also use of some cryptographic algorithms for dealing the issues to access a cloud based data storage. To the best of knowledge, this is the strong work to provide a complete solution to the cloud based secure log management problem.

In this paper, The mobile application field has been receiving astronomical attention from the past few years due to the growing number of mobile app downloads and withal due to the revenues being engendered .With the surge in the number of apps, the number of lamentable apps/failing apps has withal been growing.Interesting mobile app statistics are included in this paper which might avail the developers understand the concerns and merits of mobile apps.The authors have made an effort to integrate all the crucial factors that cause apps to fail which include negligence by the developers, technical issues, inadequate marketing efforts, and high prospects of the users/consumers.The paper provides suggestions to eschew failure of apps. As per the various surveys, the number of lamentable/failing apps is growing enormously, primarily because mobile app developers are not adopting a standard development life cycle for the development of apps. In this paper, we have developed a mobile application with the aid of traditional software development life cycle phases (Requirements, Design, Develop, Test, and, Maintenance) and we have used UML, M-UML, and mobile application development technologies.

PEIR, the Personal Environmental Impact Report, is a participatory sensing application that uses location data sampled from everyday mobile phones to calculate personalized estimates of environmental impact and exposure. It is an example of an important class of emerging mobile systems that combine the distributed processing capacity of the web with the personal reach of mobile technology. This paper documents and evaluates the running PEIR system, which includes mobile handset based GPS location data collection, and server-side processing stages such as HMM-based activity classification (to determine transportation mode); automatic location data segmentation into "trips''; lookup of traffic, weather, and other context data needed by the models; and environmental impact and exposure calculation using efficient implementations of established models. Additionally, we describe the user interface components of PEIR and present usage statistics from a two month snapshot of system use. The paper also outlines new algorithmic components developed based on experience with the system and undergoing testing for integration into PEIR, including: new map-matching and GSM-augmented activity classification techniques, and a selective hiding mechanism that generates believable proxy traces for times a user does not want their real location revealed.

Through a study of field biology practices, we observed that biology fieldwork generates a wealth of heterogeneous information, requiring substantial labor to coordinate and distill. To manage this data, biologists leverage a diverse set of tools, organizing their effort in paper notebooks. These observations motivated ButterflyNet, a mobile capture and access system that integrates paper notes with digital photographs captured during field research. Through ButterflyNet, the activity of leafing through a notebook expands to browsing all associated digital photos. ButterflyNet also facilitates the transfer of captured content to spreadsheets, enabling biologists to share their work. A first-use study with 14 biologists found this system to offer rich data capture and transformation, in a manner felicitous with current practice.

A major challenge in computer security is about establishing the trustworthiness of remote platforms. Remote attestation is the most common approach to this challenge. It allows a remote platform to measure and report its system state in a secure way to a third party. Unfortunately, existing attestation solutions either provide low security, as they rely on unrealistic assumptions, or are not applicable to commodity low-cost and resource-constrained devices, as they require custom secure hardware extensions that are difficult to adopt across IoT vendors. In this work, we propose a novel remote attestation scheme, named Boot Attestation, that is particularly optimized for low-cost and resource-constrained embedded devices. In Boot Attestation, software integrity measurements are immediately committed to during boot, thus relaxing the traditional requirement for secure storage and reporting. Our scheme is very light on cryptographic requirements and storage, allowing efficient implementations, even on the most low-end IoT platforms available today. We also describe extensions for more flexible management of ownership and third party (public-key) attestation that may be desired in fully Internet-enabled devices. Our scheme is supported by many existing off-the-shelf devices. To this end, we review the hardware protection capabilities for a number of popular device types and present implementation results for two such commercially available platforms.

This paper is meant to give the reader a general overview about the application of asymmetric cryptography in communication, particular in mobile devices. The basics principles of a cryptosystem are addressed, as well as the idea of symmetric and asymmetric cryptography. The functional principles of RSA encryption and the DiffieHellman key exchange scheme, as well as the general idea of digital signatures are shortly described. Furthermore some challenges and solution approaches for the application of asymmetric encryption in low power mobile devices are named. On completing, some the future developments in cryptography are shortly described.

The need for more security on mobile devices is increasing with new functionalities and features made available. To improve the device security we propose gait recognition as a protection mechanism. Unlike previous work on gait recognition, which was based on the use of video sources, floor sensors or dedicated high-grade accelerometers, this paper reports the performance when the data is collected with a commercially available mobile device containing low-grade accelerometers. To be more specific, the used mobile device is the Google G1 phone containing the AK8976A embedded accelerometer sensor. The mobile device was placed at the hip on each volunteer to collect gait data. Preproccesing, cycle detection and recognition-analysis were applied to the acceleration signal. The performance of the system was evaluated having 51 volunteers and resulted in an equal error rate (EER) of 20%.

Nowadays, new tools and technologies are emerging rapidly. They are often used cross-culturally before being tested for suitability and validity. However, they must be validated to ensure that they work with all users, not just part of them. Mobile banking (as a new technology tool) has been introduced assuming that it performs well concerning authentication, among all members of the society. Our research aimed to evaluate authentication mobile banking user acceptance, through Technology Acceptance Model (TAM), in Arabic countries, namely Yemen. The results confirm the previous studies that have shown the importance of perceived ease of use and perceived usefulness. Furthermore, perceived ease of use plays a determinant role. KeywordsTechnology acceptance models; Mobile Banking;

After manufacturing and retail marketing, healthcare is considered the next home for Radio Frequency Identification (RFID). Although in its infancy, RFID technology has great potential in healthcare to significantly reduce cost, and improve patient safety and medical services. However, the challenge will be to incorporate RFID into medical practice, especially when relevant experience is limited. To explore this issue, we conducted a case study that demonstrated RFID integration into the medical world at one Taiwan hospital. The project that was studied, the Location-based Medicare Service project, was partially subsidized by the Taiwan government and was aimed at containing SARS, a dangerous disease that struck Taiwan in 2003. Innovative and productive of several significant results, the project established an infrastructure and platform allowing for other applications. In this paper we describe the development strategy, design, and implementation of the project. We discuss our findings pertaining to the collaborative development strategy, device management, data management, and value generation. These findings have important implications for developing RFID applications in healthcare organizations.


Question: Predicting human actions in smart homes through the application of deep learning techniques.

LIBSVM: An Essential Library for Support Vector Machines in Smart Home Applications

This document focuses on LIBSVM, a comprehensive and widely-used library designed for support vector machines (SVMs). SVMs are a set of supervised learning methods used for classification, regression, and other related tasks. LIBSVM, in particular, offers a range of functionalities that make it suitable for a variety of applications, including those in the realm of smart homes. In the context of smart homes, SVMs can be employed for the prediction of human behavior, enabling the development of intelligent systems that can anticipate and respond to the needs of occupants. LIBSVM's capabilities include handling both linear and non-linear data, making it a versatile tool for analyzing the complex patterns often associated with human behavior. Its efficiency and accuracy have been demonstrated in various studies, making it a preferred choice for researchers and developers working on smart home technologies. Moreover, LIBSVM's user-friendly interface and extensive documentation facilitate its integration into smart home systems, streamlining the process of behavior prediction and enhancing the overall user experience.

Homes are rich with information about people's energy consumption, medical health, and personal or family functions. In this paper, we present our experiences deploying large-scale residential sensing systems in over 20 homes. Deploying small-scale systems in homes can be deceptively easy, but in our deployments we encountered a phase transition in which deployment effort increases dramatically as residential deployments scale up in terms of 1) the number of nodes, 2) the length of time, and 3) the number of houses. In this paper, we distill our experiences down to a set of guidelines and design principles to help future deployments avoid the potential pitfalls of large-scale sensing in homes.

Activity recognition is an important subject in different areas of research, like e-health and ubiquitous computing. In this paper we give an overview of recent work in the field of activity recognition from both body mounted sensors and ambient sensors. We discuss some of the differences among approaches and present guidelines for choosing a suitable approach under various circumstances.

Reliable human activity recognition with wearable devices enables the development of human-centric pervasive applications.We aim to develop a robust wearable-based activity recognition system for real life situations where the device position is up to the user or where a user is unable to collect initial training data. Consequently, in this work we focus on the problem of recognizing the on-body position of the wearable device ensued by comprehensive experiments concerning subject-specific and cross-subjects activity recognition approaches that rely on acceleration data. We introduce a device localization method that predicts the on-body position with an F-measure of 89% and a cross-subjects activity recognition approach that considers common physical characteristics. In this context, we present a real world data set that has been collected from 15 participants for 8 common activities where they carried 7 wearable devices in different on-body positions. Our results show that the detection of the device position consistently improves the result of activity recognition for common activities. Regarding cross-subjects models, we identified the waist as the most suitable device location at which the acceleration patterns for the same activity across several people are most similar. In this context, our results provide evidence for the reliability of physical characteristics based cross-subjectsmodels. © 2017 Elsevier B.V. All rights reserved.

Capabilities of inference and prediction are the significant components of visual systems. Visual path prediction is an important and challenging task among them, with the goal to infer the future path of a visual object in a static scene. This task is complicated as it needs high-level semantic understandings of both the scenes and underlying motion patterns in video sequences. In practice, cluttered situations have also raised higher demands on the effectiveness and robustness of models. Motivated by these observations, we propose a deep learning framework, which simultaneously performs deep feature learning for visual representation in conjunction with spatiotemporal context modeling. After that, a unified path-planning scheme is proposed to make accurate path prediction based on the analytic results returned by the deep context models. The highly effective visual representation and deep context models ensure that our framework makes a deep semantic understanding of the scenes and motion patterns, consequently improving the performance on visual path prediction task. In experiments, we extensively evaluate the model's performance by constructing two large benchmark datasets from the adaptation of video tracking datasets. The qualitative and quantitative experimental results show that our approach outperforms the state-of-the-art approaches and owns a better generalization capability.

Interpretation of images and videos containing humans interacting with different objects is a daunting task. It involves understanding scene or event, analyzing human movements, recognizing manipulable objects, and observing the effect of the human movement on those objects. While each of these perceptual tasks can be conducted independently, recognition rate improves when interactions between them are considered. Motivated by psychological studies of human perception, we present a Bayesian approach which integrates various perceptual tasks involved in understanding human-object interactions. Previous approaches to object and action recognition rely on static shape or appearance feature matching and motion analysis, respectively. Our approach goes beyond these traditional approaches and applies spatial and functional constraints on each of the perceptual elements for coherent semantic interpretation. Such constraints allow us to recognize objects and actions when the appearances are not discriminative enough. We also demonstrate the use of such constraints in recognition of actions from static images without using any motion information.

People counting has many applications in smart buildings. For example, adjusting HVAC systems based on the number of occupants in each room can save a significant amount of energy. In addition, security and safety of the building can be managed by determining the number and location of occupants. Different technologies and sensing platforms have proposed for accurate and efficient people counting. However, these solutions are expensive, hard to deploy, or privacy invasive. We investigate the possibility of placing an 8×8 IR array sensor at the doorways and counting the number of people inside rooms. Our solution is real-time, inexpensive, privacy preserving with much less deployment constraints compared to its competitors. The proposed solution deals with realistic and dynamic changes in the sensing environment by leveraging a combination of Otsus thresholding and modeling thermal noise distribution. We evaluated our solution via several controlled and uncontrolled real-world environments. The results show an average of 93% accuracy in estimating the number of occupants in rooms.

Human motion modelling is a classical problem at the intersection of graphics and computer vision, with applications spanning human-computer interaction, motion synthesis, and motion prediction for virtual and augmented reality. Following the success of deep learning methods in several computer vision tasks, recent work has focused on using deep recurrent neural networks (RNNs) to model human motion, with the goal of learning time-dependent representations that perform tasks such as short-term motion prediction and long-term human motion synthesis. We examine recent work, with a focus on the evaluation methodologies commonly used in the literature, and show that, surprisingly, state of the art performance can be achieved by a simple baseline that does not attempt to model motion at all. We investigate this result, and analyze recent RNN methods by looking at the architectures, loss functions, and training procedures used in state-of-the-art approaches. We propose three changes to the standard RNN models typically used for human motion, which results in a simple and scalable RNN architecture that obtains state-of-the-art performance on human motion prediction.

Data driven motion generation is a challenging problem. Typically the model is trained using motion capture (MOCAP) data, which is limited in scope and diversity. Solving this problem using motion data extracted from Youtube videos would enable motion generation pipelines to feed from a wider range of action data, allowing for greater variability in the actions generated. Human action is expressed through a sequence of bodily movements. Hence, in this work we explore three stages for the motion synthesis pipeline: first by extracting human pose from video data with Part Affinity Fields method by [1], and then by training a 3-Layer Recurrent Neural Network (RNN) that learns to generate human-like motion for a given class from annotated video data, and finally we explore a simple synthesis of motion from multiple classes. Ultimately, this approach could enable robust generation of a wide range of motions, capturing the the subtle and fine variations of human movement expressed in videos that populate the Internet to artificially produce life like motion.

We use multilayer Long Short Term Memory (LSTM) networks to learn representations of video sequences. Our model uses an encoder LSTM to map an input sequence into a fixed length representation. This representation is decoded using single or multiple decoder LSTMs to perform different tasks, such as reconstructing the input sequence, or predicting the future sequence. We experiment with two kinds of input sequences – patches of image pixels and high-level representations (“percepts”) of video frames extracted using a pretrained convolutional net. We explore different design choices such as whether the decoder LSTMs should condition on the generated output. We analyze the outputs of the model qualitatively to see how well the model can extrapolate the learned video representation into the future and into the past. We try to visualize and interpret the learned features. We stress test the model by running it on longer time scales and on out-of-domain data. We further evaluate the representations by finetuning them for a supervised learning problem – human action recognition on the UCF-101 and HMDB-51 datasets. We show that the representations help improve classification accuracy, especially when there are only a few training examples. Even models pretrained on unrelated datasets (300 hours of YouTube videos) can help action recognition performance.

As sensor and actuator networks mature, they become a core utility of smart homes like electricity and water and enable the running of many CPS applications. Like other Cyber-Physical Systems (CPSs), when a number of applications share physical world entities, it raises many systems of systems interdependency problems. Such problems arise in the cyber part mainly because each application has assumptions on the physical world entities without knowing how other applications work. In this work, we propose DepSys, a utility sensing and actuation infrastructure for smart homes that provides comprehensive strategies to specify, detect, and resolve conflicts in a home setting. Based on real home data, we demonstrate the severity of conflicts when multiple CPSs are integrated and the significant ability of detecting and resolving such conflicts using DepSys.


Question: Forecasting human actions in smart homes through the application of deep learning techniques.

This paper explores the feasibility of employing power line communication (PLC) networks as a foundational component for the development of integrated smart home systems. Historically, the inherent noise and instability of low voltage electrical wiring in residential settings were deemed incompatible with high-speed data transmission. However, the advent of cutting-edge communication techniques, modulation strategies, and advanced digital signal processing algorithms has facilitated the creation of novel media access control (MAC) and physical layer (PHY) protocols. These protocols enable PLC networks to achieve transmission rates on par with those of conventional wired local area networks (LANs).

The paper advocates for the utilization of power line LANs as a core infrastructure for smart homes, where information appliances (IA) encompassing a wide spectrum from basic control and monitoring devices to sophisticated multimedia entertainment systems are interconnected through the very wiring that supplies them with power. Through a combination of simulations and practical testing using prototype commercial powerline products adhering to the HomePlug standard, the study demonstrates that the HomePlug MAC and PHY layers can ensure quality of service (QoS) for real-time communications. This capability is crucial for supporting data streams that are sensitive to latency, making it a viable solution for smart home applications that demand timely and reliable data transmission.

The document you provided discusses the implementation of power line communication networks in smart homes. To maintain the length, complexity, and semantic style of the original while preserving its relationship to the question about human behavior prediction for smart homes using deep learning, a similar document could be:

A deep learning framework for human behavior prediction in smart homes
Smart home systems are increasingly incorporating sophisticated sensors and devices to monitor and manage various aspects of daily life. This paper introduces a novel deep learning framework designed to predict human behavior patterns within the context of smart homes. By leveraging the vast amounts of data generated by these systems, the framework aims to enhance the predictive capabilities of smart home technologies, thereby improving their responsiveness and personalization.

The proposed framework utilizes recurrent neural networks (RNNs), a type of deep learning model well-suited for processing sequential data, to analyze temporal patterns in sensor data. These patterns include activities of daily living (ADLs) and environmental conditions, which are crucial for understanding and predicting human behavior. The RNNs are trained on a large dataset of labeled sensor readings, enabling the model to learn the complex relationships between sensor data and human activities.

To validate the effectiveness of the framework, a series of experiments are conducted using real-world data from smart home environments. The results demonstrate that the deep learning model outperforms traditional machine learning

What machine learning (ML) technique is used for system intelligence implementation in ADAS (advanced driving assistance system)? This paper tries to answer this question. This paper analyzes ADAS and ML independently and then relate which ML technique is applicable to what ADAS component and why. The paper gives a good grasp of the current state-of-the-art. Sample works in supervised, unsupervised, deep and reinforcement learnings are presented; their strengths and rooms for improvements are also discussed. This forms part of the basics in understanding autonomous vehicle. This work is a contribution to the ongoing research in ML aimed at reducing road traffic accidents and fatalities, and the invocation of safe driving.

This paper presents an Internet of Things (IoT) approach to Human Activity Recognition (HAR) using remote monitoring of vital signs in the context of a healthcare system for self-managed chronic heart patients. Our goal is to create a HAR-IoT system using learning algorithms to infer the activity done within 4 categories (lie, sit, walk and run) as well as the time consumed performing these activities and, finally giving feedback during and after the activity. Alike in this work, we provide a comprehensive insight on the cloud-based system implemented and the conclusions after implementing two different learning algorithms and the results of the overall system for larger implementations.

Care issues and costs associated with an increasing elderly population are becoming a major concern for many countries. The use of assistive robots in “smart-home” environments has been suggested as a possible partial solution to these concerns. A challenge is the personalization of the robot to meet the changing needs of the elderly person over time. One approach is to allow the elderly person, or their carers or relatives, to make the robot learn activities in the smart home and teach it to carry out behaviors in response to these activities. The overriding premise being that such teaching is both intuitive and “nontechnical.” To evaluate these issues, a commercially available autonomous robot has been deployed in a fully sensorized but otherwise ordinary suburban house. We describe the design approach to the teaching, learning, robot, and smart home systems as an integrated unit and present results from an evaluation of the teaching component with 20 participants and a preliminary evaluation of the learning component with three participants in a human-robot interaction experiment. Participants reported findings using a system usability scale and ad-hoc Likert questionnaires. Results indicated that participants thought that this approach to robot personalization was easy to use, useful, and that they would be capable of using it in real-life situations both for themselves and for others.

Smartphones equipped with sensors can be used for monitoring simple daily activities (e.g., walking, climbing stairs etc.). The data collected via sensors need to be partially or completely annotated in order to be used in the training phase of learning algorithms. In order to ease the time consuming and burdensome annotation process, a mobile data collection and annotation application named Sensor Log is developed. The annotation feature and its ease of use make the developed application superior to existing systems. The application can be downloaded from the Android applications distribution platform.

Energy and environmental sustainability can benefit a lot from advances in data mining and machine learning techniques. However, those advances rely on the availability of relevant datasets required to develop, improve and validate new techniques. Only recently the first datasets were made publicly available for the energy and sustainability research community. In this paper we present a freely available dataset containing power usage and related information from 50 homes. Here we describe our dataset, the hardware and software setups used when collecting the data and how others can access it. We then discuss potential uses of this data in the future of energy ecofeedback and demand side management research.

Gesture recognition aims to recognize meaningful movements of human bodies, and is of utmost importance in intelligent human–computer/robot interactions. In this paper, we present a multimodal gesture recognition method based on 3-D convolution and convolutional long-short-term-memory (LSTM) networks. The proposed method first learns short-term spatiotemporal features of gestures through the 3-D convolutional neural network, and then learns long-term spatiotemporal features by convolutional LSTM networks based on the extracted short-term spatiotemporal features. In addition, fine-tuning among multimodal data is evaluated, and we find that it can be considered as an optional skill to prevent overfitting when no pre-trained models exist. The proposed method is verified on the ChaLearn LAP large-scale isolated gesture data set (IsoGD) and the Sheffield Kinect gesture (SKIG) data set. The results show that our proposed method can obtain the state-of-the-art recognition accuracy (51.02% on the validation set of IsoGD and 98.89% on SKIG).

Capabilities of inference and prediction are the significant components of visual systems. Visual path prediction is an important and challenging task among them, with the goal to infer the future path of a visual object in a static scene. This task is complicated as it needs high-level semantic understandings of both the scenes and underlying motion patterns in video sequences. In practice, cluttered situations have also raised higher demands on the effectiveness and robustness of models. Motivated by these observations, we propose a deep learning framework, which simultaneously performs deep feature learning for visual representation in conjunction with spatiotemporal context modeling. After that, a unified path-planning scheme is proposed to make accurate path prediction based on the analytic results returned by the deep context models. The highly effective visual representation and deep context models ensure that our framework makes a deep semantic understanding of the scenes and motion patterns, consequently improving the performance on visual path prediction task. In experiments, we extensively evaluate the model's performance by constructing two large benchmark datasets from the adaptation of video tracking datasets. The qualitative and quantitative experimental results show that our approach outperforms the state-of-the-art approaches and owns a better generalization capability.

This paper presents the current status of work-in-progress in developing Ghost Hunters that is targeted to explore the possibilities of incorporating gesture control and ambient lighting in a mobile augmented reality game played with smart glasses. We present the design and implementation of the current prototype, together with a small feasibility evaluation of battery consumption and gesture control.

Several cooking support systems have been studied to give users instructions based on the recipes stepby-step, using multi-media contents. These systems usually disturb users’ cooking process forcing them to provide information to the system in order to give them beneficial information. In this sense, these systems are considered to be ”system centric”. We propose a system called ”Smart Kitchen” considered to be in ”user centric”, in which a user can cook normally without being concerned about the system. Smart Kitchen can understand cooking processes, in other words, what the user is doing. In this paper, we discuss the design of the Smart Kitchen system and explain three essential modules of tracking food, recognizing food material, and recognizing cooking action.

Household smart meters that measure power consumption in real-time at fine granularities are the foundation of a future smart electricity grid. However, the widespread deployment of smart meters has serious privacy implications since they inadvertently leak detailed information about household activities. In this paper, we show that even without a priori knowledge of household activities or prior training, it is possible to extract complex usage patterns from smart meter data using off-the-shelf statistical methods. Our analysis uses two months of data from three homes, which we instrumented to log aggregate household power consumption every second. With the data from our small-scale deployment, we demonstrate the potential for power consumption patterns to reveal a range of information, such as how many people are in the home, sleeping routines, eating routines, etc. We then sketch out the design of a privacy-enhancing smart meter architecture that allows an electric utility to achieve its net metering goals without compromising the privacy of its customers.


Question: Forecasting human actions in smart homes through deep learning techniques.

This paper explores the enhancement of Deep Boltzmann Machine (DBM) learning through the application of adaptive Markov Chain Monte Carlo (MCMC) methods, addressing the challenge of modeling complex, high-dimensional data. The inherent nature of DBM distributions, characterized by a rough energy landscape with numerous local minima, often leads to the Gibbs sampler becoming entrapped in one mode, resulting in unstable learning dynamics and suboptimal parameter estimation. To overcome this limitation, we investigate the synergy between Fast Persistent Contrastive Divergence (PCD) and adaptive MCMC techniques. Initially, we highlight the parallels between Fast PCD and adaptive MCMC, establishing a foundational link. Subsequently, we introduce the Coupled Adaptive Simulated Tempering algorithm, designed to navigate the complex multimodal energy landscape more effectively. Our experimental findings illustrate the significant enhancement in parameter estimation accuracy, particularly when scaling up to large-scale DBM learning scenarios, achieved through the proposed adaptive MCMC algorithm. This approach holds promise for improving the predictive capabilities of DBMs in smart home applications by better capturing the nuances of human behavior patterns.

A convolutional neural network (CNN) has been designed to interpret player actions in ice hockey video. The hourglass network is employed as the base to generate player pose estimation and layers are added to this network to produce action recognition. As such, the unified architecture is referred to as action recognition hourglass network, or ARHN. ARHN has three components. The first component is the latent pose estimator, the second transforms latent features to a common frame of reference, and the third performs action recognition. Since no benchmark dataset for pose estimation or action recognition is available for hockey players, we generate such an annotated dataset. Experimental results show action recognition accuracy of 65% for four types of actions in hockey. When similar poses are merged to three and two classes, the accuracy rate increases to 71% and 78%, proving the efficacy of the methodology for automated action recognition in hockey.

Gesture recognition aims to recognize meaningful movements of human bodies, and is of utmost importance in intelligent human–computer/robot interactions. In this paper, we present a multimodal gesture recognition method based on 3-D convolution and convolutional long-short-term-memory (LSTM) networks. The proposed method first learns short-term spatiotemporal features of gestures through the 3-D convolutional neural network, and then learns long-term spatiotemporal features by convolutional LSTM networks based on the extracted short-term spatiotemporal features. In addition, fine-tuning among multimodal data is evaluated, and we find that it can be considered as an optional skill to prevent overfitting when no pre-trained models exist. The proposed method is verified on the ChaLearn LAP large-scale isolated gesture data set (IsoGD) and the Sheffield Kinect gesture (SKIG) data set. The results show that our proposed method can obtain the state-of-the-art recognition accuracy (51.02% on the validation set of IsoGD and 98.89% on SKIG).

Human interaction prediction, i.e., the recognition of an ongoing interaction activity before it is completely executed, has a wide range of applications such as human-robot interaction and the prevention of dangerous events. Due to the large variations in appearance and the evolution of scenes, interaction prediction at an early stage is a challenging task. In this paper, a novel structural feature is exploited as a complement together with the spatial and temporal information to improve the performance of interaction prediction. The proposed structural feature is captured by Long Short Term Memory (LSTM) networks, which process the global and local features associated to each frame and each optical flow image. A new ranking score fusion method is then introduced to combine the spatial, temporal and structural models. Experimental results demonstrate that the proposed method outperforms state-of-the-art methods for human interaction prediction on the BIT-Interaction, the TV Human Interaction and the UT-Interaction datasets.

This paper is concerned with constructing a prototype smart home environment which has been built in the research building of Korea Institute of Industrial Technology (KITECH) to demonstrate the practicability of a robot-assisted future home environment. Key functionalities that a home service robot must provide are localization, navigation, object recognition and object handling. A considerable amount of research has been conducted to make the service robot perform these operations with its own sensors, actuators and a knowledge database. With all heavy sensors, actuators and a database, the robot could have performed the given tasks in a limited environment or showed the limited capabilities in a natural environment. We initiated a smart home environment project for light-weight service robots to provide reliable services by interacting with the environment through the wireless sensor networks. This environment consists of the following three main components: smart objects with an radio frequency identification (RFID) tag and smart appliances with sensor network functionality; the home server that connects smart devices as well as maintains information for reliable services; and the service robots that perform tasks in collaboration with the environment. In this paper, we introduce various types of smart devices which are developed for assisting the robot by providing sensing and actuation, and present our approach on the integration of these devices to construct the smart home environment. Finally, we discuss the future directions of our project.

This paper presents ElectriSense, a new solution for automatically detecting and classifying the use of electronic devices in a home from a single point of sensing. ElectriSense relies on the fact that most modern consumer electronics and fluorescent lighting employ switch mode power supplies (SMPS) to achieve high efficiency. These power supplies continuously generate high frequency electromagnetic interference (EMI) during operation that propagates throughout a home's power wiring. We show both analytically and by in-home experimentation that EMI signals are stable and predictable based on the device's switching frequency characteristics. Unlike past transient noise-based solutions, this new approach provides the ability for EMI signatures to be applicable across homes while still being able to differentiate between similar devices in a home. We have evaluated our solution in seven homes, including one six-month deployment. Our results show that ElectriSense can identify and classify the usage of individual devices with a mean accuracy of 93.82%.

The ubiquity of smartphones together with their ever-growing computing, networking and sensing powers have been changing the landscape of people’s daily life. Among others, activity recoginition, which takes the raw sensor reading as inputs and predicts a user’s motion activity, has become an active research area in recent years. It is the core building block in many high-impact applications, ranging from health and fitness monitoring, personal biometric signature, urban computing, assistive technology and elder-care, to indoor localization and navigation, etc. This paper presents a comprehensive survey of the recent advances in activity recognition with smartphones’ sensors. We start with the basic concepts such as sensors, activity types, etc. We review the core data mining techniques behind the main stream activity recognition algorithms, analyze their major challenges and introduce a variety of real applications enabled by activity recognition. Professor: Dr. Hanghang Tong Signature: Date: 06-12-2014 Professor: Dr. Ping Ji Signature: Date: 06-12-2014 Professor: Dr. Ted Brown Signature: Date: 06-12-2014 Advisor: Dr. Hanghang Tong

Body-worn sensors in general and accelerometers in particular have been widely used in order to detect human movements and activities. The execution of each type of movement by each particular individual generates sequences of time series of sensed data from which specific movement related patterns can be assessed. Several machine learning algorithms have been used over windowed segments of sensed data in order to detect such patterns in activity recognition based on intermediate features (either hand-crafted or automatically learned from data). The underlying assumption is that the computed features will capture statistical differences that can properly classify different movements and activities after a training phase based on sensed data. In order to achieve high accuracy and recall rates (and guarantee the generalization of the system to new users), the training data have to contain enough information to characterize all possible ways of executing the activity or movement to be detected. This could imply large amounts of data and a complex and time-consuming training phase, which has been shown to be even more relevant when automatically learning the optimal features to be used. In this paper, we present a novel generative model that is able to generate sequences of time series for characterizing a particular movement based on the time elasticity properties of the sensed data. The model is used to train a stack of auto-encoders in order to learn the particular features able to detect human movements. The results of movement detection using a newly generated database with information on five users performing six different movements are presented. The generalization of results using an existing database is also presented in the paper. The results show that the proposed mechanism is able to obtain acceptable recognition rates (F = 0.77) even in the case of using different people executing a different sequence of movements and using different hardware.

A key challenge in scaling up robot learning to many skills and environments is removing the need for human supervision, so that robots can collect their own data and improve their own performance without being limited by the cost of requesting human feedback. Model-based reinforcement learning holds the promise of enabling an agent to learn to predict the effects of its actions, which could provide flexible predictive models for a wide range of tasks and environments, without detailed human supervision. We develop a method for combining deep action-conditioned video prediction models with model-predictive control that uses entirely unlabeled training data. Our approach does not require a calibrated camera, an instrumented training set-up, nor precise sensing and actuation. Our results show that our method enables a real robot to perform nonprehensile manipulation — pushing objects — and can handle novel objects not seen during training.

Skilled robot task learning is best implemented by predictive action policies due to the inherent latency of sensorimotor processes. However, training such predictive policies is challenging as it involves finding a trajectory of motor activations for the full duration of the action. We propose a data-efficient deep predictive policy training (DPPT) framework with a deep neural network policy architecture which maps an image observation to a sequence of motor activations. The architecture consists of three sub-networks referred to as the perception, policy and behavior super-layers. The perception and behavior super-layers force an abstraction of visual and motor data trained with synthetic and simulated training samples, respectively. The policy super-layer is a small subnetwork with fewer parameters that maps data in-between the abstracted manifolds. It is trained for each task using methods for policy search reinforcement learning. We demonstrate the suitability of the proposed architecture and learning framework by training predictive policies for skilled object grasping and ball throwing on a PR2 robot. The effectiveness of the method is illustrated by the fact that these tasks are trained using only about 180 real robot attempts with qualitative terminal rewards.

Nowadays, numerous applications of smart home systems provide recommendations for users, including reducing their energy consumption, warnings of defective devices, selecting reliable devices and software, diagnoses, etc [1]. The internet connected, dynamic and heterogeneous nature of the smart home environment creates new security, authentication, and privacy challenges [2]. To solve those challenges, an approach to data privacy in smart home using blockchain technology, which is called smart home based the IoT-Blockchain (SHIB), is proposed in this paper. In order to demonstrate the proposed architecture, an experimental scenario using Ganache, Remix, and web3. js is built among the user, service provider, and smart home to evaluate the performance of the smart contract in the SHIB. Based on the experiment results, the SHIB architecture brings the advantages like data privacy, trust access control, and high extension ability. In addition, the comparison between the proposed architecture and existing models in different parameters such as smart contract, the privacy of data, usage of tokens, updating the policies, and misbehavior judging are performed.


Question: Predicting human actions in smart homes utilizing deep learning methods.

This paper explores the potential of Restricted Boltzmann Machines (RBMs) as standalone classifiers for a range of learning tasks, particularly focusing on their application in discriminative scenarios. Traditionally, RBMs have been utilized as feature extractors or as a pre-training step for deep feed-forward neural networks, rather than as direct solutions to classification challenges. Here, we advocate for the use of RBMs as comprehensive tools for constructing competitive non-linear classifiers. To enhance their classification capabilities, we evaluate various learning algorithms designed to infuse a discriminative element into RBM training. This strategy simplifies the process by directly employing RBMs for classification, bypassing their conventional role as preparatory steps. Moreover, we illustrate the effectiveness of discriminative RBMs in semi-supervised learning environments, demonstrating their versatility and adaptability in diverse classification settings.

**Question**: Investigating the effects of climate change on coastal erosion

**Documents**: Coastal erosion in response to sea level rise and storms
Coastal erosion is a significant environmental concern, especially in the context of global climate change. This paper examines the impacts of sea level rise and increased storm frequency on coastal erosion. Using a combination of historical data and predictive models, we analyze the extent and rate of erosion in coastal regions under various scenarios of climate change. The results show that coastal erosion is likely to accelerate as sea levels rise and storms become more frequent and intense. We also identify the factors that exacerbate erosion, such as the loss of protective vegetation and the construction of hard coastal defenses. The study provides insights into the potential consequences of climate change on coastal environments and suggests strategies for mitigating the effects of erosion.

**Similar Documents**: This study delves into the escalating issue of coastal erosion, particularly in light of the ongoing and anticipated impacts of climate change. By scrutinizing the effects of sea level rise and the intensification of storm events, we aim to understand how these factors contribute to the degradation of coastal landscapes. Utilizing a blend of historical records and advanced predictive models, we assess the magnitude and velocity of coastal erosion under different climate change scenarios. Our findings indicate a pronounced increase in erosion rates, paralleling the rise in sea levels and the frequency of severe storms. We highlight critical drivers of erosion, including the depletion of natural barriers like vegetation and the reliance on rigid coastal structures. This research offers valuable perspectives on the repercussions of climate change for coastal zones and proposes mitigation measures to address the escalating erosion problem.

A key aspect of pervasive computing is using computers and sensor networks to effectively and unobtrusively infer users' behavior in their environment. This includes inferring which activity users are performing, how they're performing it, and its current stage. Recognizing and recording activities of daily living is a significant problem in elder care. A new paradigm for ADL inferencing leverages radio-frequency-identification technology, data mining, and a probabilistic inference engine to recognize ADLs, based on the objects people use. We propose an approach that addresses these challenges and shows promise in automating some types of ADL monitoring. Our key observation is that the sequence of objects a person uses while performing an ADL robustly characterizes both the ADL's identity and the quality of its execution. So, we have developed Proactive Activity Toolkit (PROACT).

In this paper, we present a real-time algorithm for automatic recognition of not only physical activities, but also, in some cases, their intensities, using five triaxial wireless accelerometers and a wireless heart rate monitor. The algorithm has been evaluated using datasets consisting of 30 physical gymnasium activities collected from a total of 21 people at two different labs. On these activities, we have obtained a recognition accuracy performance of 94.6% using subject-dependent training and 56.3% using subject-independent training. The addition of heart rate data improves subject-dependent recognition accuracy only by 1.2% and subject-independent recognition only by 2.1%. When recognizing activity type without differentiating intensity levels, we obtain a subject-independent performance of 80.6%. We discuss why heart rate data has such little discriminatory power.

Energy and environmental sustainability can benefit a lot from advances in data mining and machine learning techniques. However, those advances rely on the availability of relevant datasets required to develop, improve and validate new techniques. Only recently the first datasets were made publicly available for the energy and sustainability research community. In this paper we present a freely available dataset containing power usage and related information from 50 homes. Here we describe our dataset, the hardware and software setups used when collecting the data and how others can access it. We then discuss potential uses of this data in the future of energy ecofeedback and demand side management research.

This paper introduces a human posture tracking platform to identify the human postures of sitting, standing or lying down, based on a smartwatch. This work develops such a system as a proof-of-concept study to investigate a smartwatch's ability to be used in future remote health monitoring systems and applications. This work validates the smartwatches' ability to track the posture of users accurately in a laboratory setting while reducing the sampling rate to potentially improve battery life, the first steps in verifying that such a system would work in future clinical settings. The algorithm developed classifies the transitions between three posture states of sitting, standing and lying down, by identifying these transition movements, as well as other movements that might be mistaken for these transitions. The system is trained and developed on a Samsung Galaxy Gear smartwatch, and the algorithm was validated through a leave-one-subject-out cross-validation of 20 subjects. The system can identify the appropriate transitions at only 10 Hz with an F-score of 0.930, indicating its ability to effectively replace smart phones, if needed.

Energy disaggregation estimates appliance-by-appliance electricity consumption from a single meter that measures the whole home's electricity demand. Recently, deep neural networks have driven remarkable improvements in classification performance in neighbouring machine learning fields such as image classification and automatic speech recognition. In this paper, we adapt three deep neural network architectures to energy disaggregation: 1) a form of recurrent neural network called `long short-term memory' (LSTM); 2) denoising autoencoders; and 3) a network which regresses the start time, end time and average power demand of each appliance activation. We use seven metrics to test the performance of these algorithms on real aggregate power data from five appliances. Tests are performed against a house not seen during training and against houses seen during training. We find that all three neural nets achieve better F1 scores (averaged over all five appliances) than either combinatorial optimisation or factorial hidden Markov models and that our neural net algorithms generalise well to an unseen house.

Some pioneer WiFi signal based human activity recognition systems have been proposed. Their key limitation lies in the lack of a model that can quantitatively correlate CSI dynamics and human activities. In this paper, we propose CARM, a CSI based human Activity Recognition and Monitoring system. CARM has two theoretical underpinnings: a CSI-speed model, which quantifies the correlation between CSI value dynamics and human movement speeds, and a CSI-activity model, which quantifies the correlation between the movement speeds of different human body parts and a specific human activity. By these two models, we quantitatively build the correlation between CSI value dynamics and a specific human activity. CARM uses this correlation as the profiling mechanism and recognizes a given activity by matching it to the best-fit profile. We implemented CARM using commercial WiFi devices and evaluated it in several different environments. Our results show that CARM achieves an average accuracy of greater than 96%.

In this work, a system for recognizing activities in the home setting using a set of small and simple state-change sensors is introduced. The sensors are designed to be “tape on and forget” devices that can be quickly and ubiquitously installed in home environments. The proposed sensing system presents an alternative to sensors that are sometimes perceived as invasive, such as cameras and microphones. Unlike prior work, the system has been deployed in multiple residential environments with non-researcher occupants. Preliminary results on a small dataset show that it is possible to recognize activities of interest to medical professionals such as toileting, bathing, and grooming with detection accuracies ranging from 25% to 89% depending on the evaluation criteria used .

In this paper we introduce a template-based method for recognizing human actions called action MACH. Our approach is based on a maximum average correlation height (MACH) filter. A common limitation of template-based methods is their inability to generate a single template using a collection of examples. MACH is capable of capturing intra-class variability by synthesizing a single Action MACH filter for a given action class. We generalize the traditional MACH filter to video (3D spatiotemporal volume), and vector valued data. By analyzing the response of the filter in the frequency domain, we avoid the high computational cost commonly incurred in template-based approaches. Vector valued data is analyzed using the Clifford Fourier transform, a generalization of the Fourier transform intended for both scalar and vector-valued data. Finally, we perform an extensive set of experiments and compare our method with some of the most recent approaches in the field by using publicly available datasets, and two new annotated human action datasets which include actions performed in classic feature films and sports broadcast television.

Activity recognition fits within the bigger framework of context awareness. In this paper, we report on our efforts to recognize user activity from accelerometer data. Activity recognition is formulated as a classification problem. Performance of base-level classifiers and meta-level classifiers is compared. Plurality Voting is found to perform consistently well across different settings.

Fluent and safe interactions of humans and robots require both partners to anticipate the others’ actions. A common approach to human intention inference is to model specific trajectories towards known goals with supervised classifiers. However, these approaches do not take possible future movements into account nor do they make use of kinematic cues, such as legible and predictable motion. The bottleneck of these methods is the lack of an accurate model of general human motion. In this work, we present a conditional variational autoencoder that is trained to predict a window of future human motion given a window of past frames. Using skeletal data obtained from RGB depth images, we show how this unsupervised approach can be used for online motion prediction for up to 1660 ms. Additionally, we demonstrate online target prediction within the first 300-500 ms after motion onset without the use of target specific training data. The advantage of our probabilistic approach is the possibility to draw samples of possible future motions. Finally, we investigate how movements and kinematic cues are represented on the learned low dimensional manifold.


Question: Predicting human actions in smart homes through the application of deep learning techniques.

Deep Learning Models as Compact Universal Predictors for Human Behavior
Deep learning models, particularly deep belief networks (DBN), have emerged as powerful tools for approximating complex functions and patterns. Initially proposed by Hinton, Osindero, and Teh (2006), DBNs are characterized by their layered structure of hidden causal variables and a greedy layer-wise unsupervised learning algorithm. Inspired by the work of Le Roux and Bengio (2008) and Sutskever and Hinton (2008), we explore the compactness and universality of deep but narrow generative networks. Our findings confirm that these deep models do not necessarily require an extensive parameter space to achieve universal approximation capabilities, as compared to shallow networks. Extending this concept, we demonstrate that deep, narrow feedforward neural networks equipped with sigmoidal activation units can effectively model any Boolean expression, underscoring their versatility in representing intricate logical relationships. In the context of smart homes, these models can be employed to predict human behavior by learning from environmental data and user interactions, thereby enhancing the automation and personalization of smart home environments.

We introduce a novel Recurrent Neural Network-based algorithm for future video feature generation and action anticipation called feature mapping RNN . Our novel RNN architecture builds upon three effective principles of machine learning, namely parameter sharing, Radial Basis Function kernels and adversarial training. Using only some of the earliest frames of a video, the feature mapping RNN is able to generate future features with a fraction of the parameters needed in traditional RNN. By feeding these future features into a simple multilayer perceptron facilitated with an RBF kernel layer, we are able to accurately predict the action in the video. In our experiments, we obtain 18% improvement on JHMDB-21 dataset, 6% on UCF101-24 and 13% improvement on UT-Interaction datasets over prior stateof-the-art for action anticipation.

Published research on smart homes and their users is growing exponentially, yet a clear understanding of who these users are and how they might use smart home technologies is missing from a field being overwhelmingly pushed by technology developers. Through a systematic analysis of peer-reviewed literature on smart homes and their users, this paper takes stock of the dominant research themes and the linkages and disconnects between them. Key findings within each of nine themes are analysed, grouped into three: (1) views of the smart home—functional, instrumental, socio-technical; (2) users and the use of the smart home—prospective users, interactions and decisions, using technologies in the home; and (3) challenges for realising the smart home—hardware and software, design, domestication. These themes are integrated into an organising framework for future research that identifies the presence or absence of cross-cutting relationships between different understandings of smart homes and their users. The usefulness of the organising framework is illustrated in relation to two major concerns—privacy and control—that have been narrowly interpreted to date, precluding deeper insights and potential solutions. Future research on smart homes and their users can benefit by exploring and developing cross-cutting relationships between the research themes identified.

Humans have the remarkable ability of imagination, where within the human mind virtual simulations are done of scenarios whether visual, auditory or any other senses. These imaginations are based on the experiences during interaction with the real world, where human senses help the mind understand their surroundings. Such level of imagination has not yet been achieved using current algorithms, but a current trend in deep learning architectures known as Generative Adversarial Networks (GANs) have proven capable of generating new and interesting images or videos based on the training data. In that way, GANs can be used to mimic human imagination, where the resulting generated visuals of GANs are based on the data used during training. In this paper, we use a combination of Long Short-Term Memory (LSTM) Networks and 3D GANs to generate videos. We use a 3D Convolutional GAN to generate new human action videos based on trained data. The generated human action videos are used to generate longer videos consisting of a sequence of short actions combined creating longer and more complex activities. To generate the sequence of actions needed we use an LSTM network to translate a simple input description text into the required sequence of actions. The generated chunks are then concatenated using a motion interpolation scheme to form a single video consisting of many generated actions. Hence a visualization of the input text description is generated as a video of a subject performing the activity described.

Care issues and costs associated with an increasing elderly population are becoming a major concern for many countries. The use of assistive robots in “smart-home” environments has been suggested as a possible partial solution to these concerns. A challenge is the personalization of the robot to meet the changing needs of the elderly person over time. One approach is to allow the elderly person, or their carers or relatives, to make the robot learn activities in the smart home and teach it to carry out behaviors in response to these activities. The overriding premise being that such teaching is both intuitive and “nontechnical.” To evaluate these issues, a commercially available autonomous robot has been deployed in a fully sensorized but otherwise ordinary suburban house. We describe the design approach to the teaching, learning, robot, and smart home systems as an integrated unit and present results from an evaluation of the teaching component with 20 participants and a preliminary evaluation of the learning component with three participants in a human-robot interaction experiment. Participants reported findings using a system usability scale and ad-hoc Likert questionnaires. Results indicated that participants thought that this approach to robot personalization was easy to use, useful, and that they would be capable of using it in real-life situations both for themselves and for others.

We present a practical approach for processing mobile sensor time series data for continual deep learning predictions. The approach comprises data cleaning, normalization, capping, time-based compression, and finally classification with a recurrent neural network. We demonstrate the effectiveness of the approach in a case study with 279 participants. On the basis of sparse sensor events, the network continually predicts whether the participants would attend to a notification within 10 minutes. Compared to a random baseline, the classifier achieves a 40% performance increase (AUC of 0.702) on a withheld test set. This approach allows to forgo resource-intensive, domain-specific, error-prone feature engineering, which may drastically increase the applicability of machine learning to mobile phone sensor data.

In this paper, a user activities outlier detection system is introduced. The proposed system is implemented in a smart home environment equipped with appropriate sensory devices. An activity outlier detection system consist of a two-stage integration of Principal Component Analysis (PCA) and Fuzzy Rule-Based System (FRBS). In the first stage, the Hamming distance is used to measure the distances between the activities. PCA is then applied to the distance measures to find two indices of Hotelling's T2 and Squared Prediction Error (SPE). In the second stage of the process, the calculated indices are provided as inputs to FRBSs to model them heuristically. They are used to identify the outliers and classify them. Three case studies are reported to demonstrate the effectiveness of the proposed system. The proposed system successfully identifies the outliers and helps in distinguishing between the normal and abnormal behaviour patterns of the Activities of Daily Living (ADL).

Technology integration is becoming an impetus to everyday lives. This new interconnected world can be found from our most private spaces to the public ones. Smart homes, which is the use of Internet of Things (IoT) within a home, has become the utmost concern in the security and privacy domain. In this paper, we review the current literature from both research and practice and offer five research trends for security and privacy in smart homes: potential for remote security breaches, risks for smart home devices, privacy violations, infrastructure vulnerabilities, and digital forensics challenges. We integrated these five trends in a conceptual model showcasing their roles in the rapidly changing IoT landscape. Combining the trends and smart forensics, we elucidate the bimodal lenses in security and privacy: preventive and investigative. These discussions offer research directions and practitioners’ implications for further development of a safe and secured smart home.

The aim of this paper is to investigate the feasibility of an audio-based context recognition system. Here, context recognition refers to the automatic classification of the context or an environment around a device. A system is developed and compared to the accuracy of human listeners in the same task. Particular emphasis is placed on the computational complexity of the methods, since the application is of particular interest in resource-constrained portable devices. Simplistic low-dimensional feature vectors are evaluated against more standard spectral features. Using discriminative training, competitive recognition accuracies are achieved with very low-order hidden Markov models (1-3 Gaussian components). Slight improvement in recognition accuracy is observed when linear data-driven feature transformations are applied to mel-cepstral features. The recognition rate of the system as a function of the test sequence length appears to converge only after about 30 to 60 s. Some degree of accuracy can be achieved even with less than 1-s test sequence lengths. The average reaction time of the human listeners was 14 s, i.e., somewhat smaller, but of the same order as that of the system. The average recognition accuracy of the system was 58% against 69%, obtained in the listening tests in recognizing between 24 everyday contexts. The accuracies in recognizing six high-level classes were 82% for the system and 88% for the subjects.

Gesture recognition aims to recognize meaningful movements of human bodies, and is of utmost importance in intelligent human–computer/robot interactions. In this paper, we present a multimodal gesture recognition method based on 3-D convolution and convolutional long-short-term-memory (LSTM) networks. The proposed method first learns short-term spatiotemporal features of gestures through the 3-D convolutional neural network, and then learns long-term spatiotemporal features by convolutional LSTM networks based on the extracted short-term spatiotemporal features. In addition, fine-tuning among multimodal data is evaluated, and we find that it can be considered as an optional skill to prevent overfitting when no pre-trained models exist. The proposed method is verified on the ChaLearn LAP large-scale isolated gesture data set (IsoGD) and the Sheffield Kinect gesture (SKIG) data set. The results show that our proposed method can obtain the state-of-the-art recognition accuracy (51.02% on the validation set of IsoGD and 98.89% on SKIG).

We propose DeepHand to estimate the 3D pose of a hand using depth data from commercial 3D sensors. We discriminatively train convolutional neural networks to output a low dimensional activation feature given a depth map. This activation feature vector is representative of the global or local joint angle parameters of a hand pose. We efficiently identify 'spatial' nearest neighbors to the activation feature, from a database of features corresponding to synthetic depth maps, and store some 'temporal' neighbors from previous frames. Our matrix completion algorithm uses these 'spatio-temporal' activation features and the corresponding known pose parameter values to estimate the unknown pose parameters of the input feature vector. Our database of activation features supplements large viewpoint coverage and our hierarchical estimation of pose parameters is robust to occlusions. We show that our approach compares favorably to state-of-the-art methods while achieving real time performance (≈ 32 FPS) on a standard computer.