Question: A cloud-based method for migrating NoSQL databases.

This paper presents a novel interoperable data migration system designed specifically for NoSQL columnar databases. The increasing demand for managing vast amounts of user-generated data while ensuring fault tolerance, availability, and scalability has led to the popularity of NoSQL databases. However, the diversity in properties, data models, and architectures across different NoSQL solutions poses significant challenges for application development and data migration. To address this, the paper introduces an innovative migration system that aims to facilitate seamless transitions between various NoSQL columnar databases. Central to this system is a unique Metamodel, which is engineered to maintain both strong and weak consistency across data updates, secondary indexes, and a wide range of data types. This ensures that the integrity of the data remains intact during the migration process. Furthermore, the system's design is flexible, allowing developers to easily incorporate support for new databases, thus enhancing its versatility and adaptability in a rapidly evolving technological landscape.

Cloud SQL Server is a relational database system designed to scale-out to cloud computing workloads. It uses Microsoft SQL Server as its core. To scale out, it uses a partitioned database on a shared-nothing system architecture. Transactions are constrained to execute on one partition, to avoid the need for two-phase commit. The database is replicated for high availability using a custom primary-copy replication scheme. It currently serves as the storage engine for Microsoft's Exchange Hosted Archive and SQL Azure.

Clouds is a native operating system for a distributed environment. The Clouds operating system is built on top of a kernel called Ra. Ra ís a second generation kernel derived from our experience with the first version of the Clouds operating system. Rø is a minimal, flexible kernel that provides a framework for implementing a variety of distributed operating systems. This paper presents the Clouds paradigm and a brief overview of its first implementation. We then present the details of the Rø kernel, the rationale for its design, and the system services that constitute the Clouds operating system. This work was funded by NSF grant CCR-8619886. @ Computing Systems, Vol. 3 . No. I . Winter 1990 11

Recent advances and innovations in smart sensor technologies, energy storage, data communications, and distributed computing paradigms are enabling technological breakthroughs in very large sensor networks. There is an emerging surge of next-generation sensor-rich computers in consumer mobile devices as well as tailor-made field platforms wirelessly connected to the Internet. Billions of such sensor computers are posing both challenges and opportunities in relation to scalable and reliable management of the peta- and exa-scale time series being generated over time. This paper presents a Cloud-computing approach to this issue based on the two well-known data storage and processing paradigms: Bigtable and MapReduce.

Due to its high portability and simplicity, web application (app) based on HTML/JavaScript/CSS has been widely used for various smart-device platforms. To take advantage of its wide platform pool, a new idea called app migration has been proposed for the web platform. Web app migration is a framework to serialize a web app running on a device and restore it in another device to continue its execution. In JavaScript semantics, one of the language features that does not allow easy app migration is a closure. A JavaScript function can access variables defined in its outer function even if the execution of the outer function is terminated. It is allowed because the inner function is created as a closure such that it contains the outer function’s environment. This feature is widely used in web app development because it is the most common way to implement data encapsulation in web programming. Closures are not easy to serialize because environments can be shared by a number of closures and environments can be created in a nested way. In this paper, we propose a novel approach to fully serialize closures. We created mechanisms to extract information from a closure’s environment through the JavaScript engine and to serialize the information in a proper order so that the original relationship between closures and environments can be restored properly. We implemented our mechanism on the WebKit browser and successfully migrated Octane benchmarks and seven real web apps which heavily exploit closures. We also show that our mechanism works correctly even for some extreme, closure-heavy cases.

Clustering is one of the most important tasks performed in Data Mining applications. This paper presents an efficient SQL implementation of the EM algorithm to perform clustering in very large databases. Our version can effectively handle high dimensional data, a high number of clusters and more importantly, a very large number of data records. We present three strategies to implement EM in SQL: horizontal, vertical and a hybrid one. We expect this work to be useful for data mining programmers and users who want to cluster large data sets inside a relational DBMS.

High Performance Computing (HPC) employs fast interconnect technologies to provide low communication and synchronization latencies for tightly coupled parallel compute jobs. Contemporary HPC clusters have a fixed capacity and static runtime environments; they cannot elastically adapt to dynamic workloads, and provide a limited selection of applications, libraries, and system software. In contrast, a cloud model for HPC clusters promises more flexibility, as it provides elastic virtual clusters to be available on-demand. This is not possible with physically owned clusters.
 In this paper, we present an approach that makes it possible to use InfiniBand clusters for HPC cloud computing. We propose a performance-driven design of an HPC IaaS layer for InfiniBand, which provides throughput and latency-aware virtualization of nodes, networks, and network topologies, as well as an approach to an HPC-aware, multi-tenant cloud management system for elastic virtualized HPC compute clusters.

As companies of all shapes and sizes begin to adapt to cloud computing, this new technology is evolving like never before. Industry experts believe that this trend will only continue to grow and develop even further in the coming few years. While Cloud computing is undoubtedly beneficial for mid-size to large companies, it is not without its downsides, especially for smaller businesses. In this paper we are presenting a list of advantages and disadvantages of Cloud computing technology, with a view to helping enterprises fully understand and adopt the concept of Cloud computing. Also, in the last chapter we are presenting a cloud application for telemetry with a focus on monitoring hydro-energy, in order to demonstrate the advantages that cloud technology can have for this domain. We consider that the way to make cloud vastly benefit all types of businesses is to know very well it's ups and downs and adapt to them accordingly. Key-Words: Cloud Computing, Grid Computing, telemetry, architecture, advantages, disadvantages.

As association rules widely used, it needs to study many problems, one of which is the generally larger and multi-dimensional datasets, and the rapid growth of the mount of data. Single-processor's memory and CPU resources are very limited, which makes the algorithm performance inefficient. Recently the development of network and distributed technology makes cloud computing a reality in the implementation of association rules algorithm. In this paper we describe the improved Apriori algorithm based on MapReduce mode, which can handle massive datasets with a large number of nodes on Hadoop platform.

It is imperative for cloud storage systems to be able to provide deadline guaranteed services according to service level agreements (SLAs) for online services. In spite of many previous works on deadline aware solutions, most of them focus on scheduling work flows or resource reservation in datacenter networks but neglect the server overload problem in cloud storage systems that prevents providing the deadline guaranteed services. In this paper, we introduce a new form of SLAs, which enables each tenant to specify a percentage of its requests it wishes to serve within a specified deadline. We first identify the multiple objectives (i.e., traffic and latency minimization, resource utilization maximization) in developing schemes to satisfy the SLAs. To satisfy the SLAs while achieving the multi-objectives, we propose a Parallel Deadline Guaranteed (PDG) scheme, which schedules data reallocation (through load re-assignment and data replication) using a tree-based bottom-up parallel process. The observation from our model also motivates our deadline strictness clustered data allocation algorithm that maps tenants with the similar SLA strictness into the same server to enhance SLA guarantees. We further enhance PDG in supplying SLA guaranteed services through two algorithms: i) a prioritized data reallocation algorithm that deals with request arrival rate variation, and ii) an adaptive request retransmission algorithm that deals with SLA requirement variation. Our trace-driven experiments on a simulator and Amazon EC2 show the effectiveness of our schemes for guaranteeing the SLAs while achieving the multi-objectives.

Recently, Cloud Computing, Edge Computing, and the Internet of Things (IoT) have been converging ever stronger, sparking creation of very large-scale, geographically distributed systems [1, 2]. Such systems intensively exploit Cloud Computing models and technologies, predominantly by utilizing large and remote data centers, but also nearby Cloudlets [3, 4] to enhance resource-constrained Edge devices (e.g., in terms of computation offloading [5–7] and data staging [8]) or to provide an execution environment for cloud-centric IoT/Edge applications [9, 10]. Serverless computing is an emerging paradigm, typically referring to a software architecture where application is decomposed into “triggers” and “actions” (or functions), and there is a platform that provides seamless hosting and execution of developer-defined functions (FaaS), making it easy to develop, manage, scale, and operate them. This complexity mitigation is mainly achieved by incorporating sophisticated runtime mechanisms into serverless or FaaS platforms. Hence, such platforms are usually characterized by fully automating many of the management and operations processes. Therefore, serverless computing can be considered as the next step in the evolution of Cloud platforms, such as PaaS, or more generally of the utility computing. While originally designed for centralized cloud deployments, the benefits of serverless paradigm become especially evident in the context of Edge Computing [11]. This is mainly because in such systems, traditional infrastructure and application management solutions are tedious, ineffective, error-prone, and ultimately very costly. Luckily, some of the existing serverless techniques, such


Question: Cloud-based migration of NoSQL data stores.

This paper explores the architecture and functionality of Amazon's Dynamo, a highly resilient key-value store designed to support the company's global e-commerce operations. Ensuring reliability amidst the vast scale of Amazon's infrastructure, which spans tens of thousands of servers and network components across multiple data centers, poses significant challenges. The continuous failure of both small and large components necessitates robust management of persistent state to maintain system reliability and scalability.
Dynamo is engineered to offer high availability, a critical requirement for Amazon's core services that aim to provide customers with a seamless, always-on experience. To achieve this, Dynamo makes a strategic trade-off by compromising on consistency in certain failure scenarios. It employs sophisticated mechanisms such as object versioning and application-assisted conflict resolution, which collectively offer a unique interface for developers to utilize. This approach enables Dynamo to handle the complexities of managing vast amounts of data with reliability, making it a cornerstone of Amazon's infrastructure for managing large-scale, global operations.

Cloud storage has rapidly become a cornerstone of many IT infrastructures, constituting a seamless solution for the backup, synchronization, and sharing of large amounts of data. Putting user data in the direct control of cloud service providers, however, raises security and privacy concerns related to the integrity of outsourced data, the accidental or intentional leakage of sensitive information, the profiling of user activities and so on. Furthermore, even if the cloud provider is trusted, users having access to outsourced files might be malicious and misbehave. These concerns are particularly serious in sensitive applications like personal health records and credit score systems. To tackle this problem, we present GORAM, a cryptographic system that protects the secrecy and integrity of outsourced data with respect to both an untrusted server and malicious clients, guarantees the anonymity and unlinkability of accesses to such data, and allows the data owner to share outsourced data with other clients, selectively granting them read and write permissions. GORAM is the first system to achieve such a wide range of security and privacy properties for outsourced storage. In the process of designing an efficient construction, we developed two new, generally applicable cryptographic schemes, namely, batched zero-knowledge proofs of shuffle and an accountability technique based on chameleon signatures, which we consider of independent interest. We implemented GORAM in Amazon Elastic Compute Cloud (EC2) and ran a performance evaluation demonstrating the scalability and efficiency of our construction.

A workflow models a process as consisting of a series of steps that simplifies the complexity of execution and management of applications. Scientific workflows in domains such as high-energy physics and life sciences utilize distributed resources in order to access, manage, and process a large amount of data from a higher level. Processing and managing such large amounts of data require the use of a distributed collection of computation and storage facilities. These resources are often limited in supply and are shared among many competing users. The recent progress in virtualization technologies and the rapid growth of cloud computing services have opened a new paradigm in distributed computing for utilizing existing (and often cheaper) resource pools for ondemand and scalable scientific computing. Scientific Workflow Management Systems (WfMS) need to adapt to this new paradigm in order to leverage the benefits of cloud services. Cloud services vary in the levels of abstraction and hence the type of service they present to application users. Infrastructure virtualization enables providers such as Amazon to offer virtual hardware for use in computeand dataintensive workflow applications. Platform-as-a-Service (PaaS) clouds expose a higher-level development and runtime environment for building and deploying workflow applications on cloud infrastructures. Such services may also expose domain-specific concepts for rapid-application development. Further up in the cloud stack are Software-as-a-Service providers who offer end users with

Graph databases (GDB) are now a viable alternative to Relational Database Systems (RDBMS). Chemistry, biology, semantic web, social networking and recommendation engines are all examples of applications that can be represented in a much more natural form. Comparisons will be drawn between relational database systems (Oracle, MySQL) and graph databases (Neo4J) focusing on aspects such as data structures, data model features and query facilities. Additionally, several of the inherent and contemporary limitations of current offerings comparing and contrasting graph vs. relational database implementations will be explored.

Cloud computing has transformed the accessibility and usage of information technology resources, by offering them as services via the Internet. Cloud service provisioning spans across infrastructure, platform and software levels. The management of the services at these levels is based on monitoring. The analysis of monitoring data provides insight into Cloud operations in order to make informed decisions. Due to the emergence of numerous heterogenous Cloud platforms with proprietary APIs, service and monitoring data are being formatted using diverse and mostly incompatible data interchange formats. This results to interoperability issues and makes the analysis of monitoring data from multi-Cloud service deployments difficult to handle. The existing research efforts on data interchange formats have been mainly focused on general performance analyses. Little or no effort has been channelled towards a combination of multiple data interchange formats based on data type to achieve efficient serialisation that can facilitate interoperability in federated Clouds, and also reduce the size of data and bandwidth utilisation cost. This paper addresses these issues by presenting automated framework that is capable of automatically selecting the most suitable data interchange formats for achieving an efficient formatting and serialisation outcome. The goal of the framework is to enable robust and transparent communication within and between multiple Cloud deployments. Based on three use case scenarios, we evaluate the proposed framework to demonstrate its efficacy in formatting and serialising data.

This paper proposes a comprehensive critical survey on the issues of warehousing and protecting big data, which are recognized as critical challenges of emerging big data research. Indeed, both are critical aspects to be considered in order to build truly, high-performance and highly-flexible big data management systems. We report on state-of-the-art approaches, methodologies and trends, and finally conclude by providing open problems and challenging research directions to be considered by future efforts.

Data outsourcing and cloud computing have been emerging at an ever-growing rate as successful approaches for allowing users and companies to rely on external services for storing and managing data. As data and access to them are not under the control of the data owner, there is a clear need to provide proper confidentiality protection. Such requirements concern the confidentiality not only of the stored data (content) but also of the specific accesses (or patterns of them) that users make on such data.
 In this article, we address these issues and propose an approach for guaranteeing content, access, and pattern confidentiality in a data outsourcing scenario. The proposed solution is based on the definition of a shuffle index structure, which adapts traditional B +-trees and, by applying a combination of techniques (covers, caches, and shuffling), ensures confidentiality of the data and of queries over them, protecting each single access as well as sequences thereof. The proposed solution also supports update operations over the data, while making reads and writes not recognizable as such by the server. We show that the shuffle index exhibits a limited performance cost, thus resulting effectively usable in practice.

Public Clouds today provide a variety of services for data analysis such as Amazon Elastic MapReduce and Google BigQuery. Each service comes with a pricing model and service level agreement (SLA). Today's pricing models and SLAs are described at the level of compute resources (instance-hours or gigabytes processed). They are also different from one service to the next. Both conditions make it difficult for users to select a service, pick a configuration, and predict the actual analysis cost. To address this challenge, we propose a new abstraction, called a Personalized Service Level Agreement, where users are presented with what they can do with their data in terms of query capabilities, guaranteed query performance and fixed hourly prices.

With the rise of location-aware IoT devices, there is an increased desire to process and manage the stationary and moving trajectory data generated by these real-time sensors. There has been a corresponding evolution of distributed database and compute technology to handle the increasing data load. Here we describe challenges in managing this kind of data and survey the technologies that address those challenges. We evaluate two distributed database technologies against a simulated high volume sensor data source and discuss the performance of predominant moving trajectory access patterns.

Database scale-out is commonly implemented by partitioning data across several database instances. This approach, however, has several restrictions. In particular, partitioned databases are inflexible in large-scale deployments and assume a partition-friendly workload in order to scale. In this paper, we analyze an alternative architecture design for distributed relational databases that overcomes the limitations of partitioned databases. The architecture is based on two fundamental principles: We decouple query processing and transaction management from data storage, and we share data across query processing nodes. The combination of these design choices provides scalability, elasticity, and operational flexibility without making any assumptions on the workload. As a drawback, sharing data among multiple database nodes causes synchronization overhead. To address this limitation, we introduce techniques for scalable transaction processing in shared-data environments. Specifically, we describe mechanisms for efficient data access, concurrency control, and data buffering. In combination with new hardware trends, the techniques enable performance characteristics that top state-of-the-art partitioned databases.

The properties of clouds -- elasticity, pay-per-use, and standardization of the runtime infrastructure -- enable cloud providers and users alike to benefit from economies of scale, faster provisioning times, and reduced runtime costs. However, to achieve these benefits, application architects and developers have to respect the characteristics of the cloud environment.
 To reduce the complexity of cloud application architectures, we propose a pattern-based approach for cloud application design and development. We defined a pattern format to describe the principles of cloud computing, available cloud offerings, and cloud application architectures. Based on this format we developed an architectural pattern language of cloud-based applications: through interrelation of patterns for cloud offering descriptions and cloud application architectures, developers are guided during the identification of cloud environments and architecture patterns applicable to their problems. We cover the proceeding how we identified patterns in various information sources and existing productively used applications, give an overview of previously discovered patterns, and introduce one new pattern. Further, we propose a framework for the organizations of patterns and the guidance of developers during pattern instantiation.


Question: Cloud-based migration of NoSQL data stores.

Introducing EQL: An Enhanced English Query Language for Relational Databases

This paper unveils EQL, an advanced English query language designed for efficient data access in integrated relational databases. EQL simplifies data manipulation by avoiding complex constructs like bound variables and quantifiers, instead focusing on a set of fundamental operations applicable to tabular structures. These operations are as potent as those found in first-order predicate calculus, ensuring robust functionality. EQL provides a user-friendly interface, presenting a standardized set of keyword English templates that align with how individuals typically interact with tables to gather information. Notably, EQL empowers users to combine these basic templates in a structured way to craft more sophisticated queries, catering to the needs of both professional programmers and occasional database users alike. The goal of EQL is to serve as a versatile sublanguage for relational database management, enhancing accessibility and functionality for a broad user base.

Note: The original document discusses SEQUEL, a structured English query language for relational databases, and its features. The similar document, EQL (Enhanced English Query Language), maintains the essence of SEQUEL while introducing a new name and slightly rephrased capabilities, ensuring the response remains relevant to the original question about a cloud-based NoSQL data migration. However, as the original document does not directly address NoSQL or cloud-based migrations, the similar document also does not diverge from the topic of query languages for relational databases.

Unbounded, unordered, global-scale datasets are increasingly common in day-to-day business (e.g. Web logs, mobile usage statistics, and sensor networks). At the same time, consumers of these datasets have evolved sophisticated requirements, such as event-time ordering and windowing by features of the data themselves, in addition to an insatiable hunger for faster answers. Meanwhile, practicality dictates that one can never fully optimize along all dimensions of correctness, latency, and cost for these types of input. As a result, data processing practitioners are left with the quandary of how to reconcile the tensions between these seemingly competing propositions, often resulting in disparate implementations and systems. We propose that a fundamental shift of approach is necessary to deal with these evolved requirements in modern data processing. We as a field must stop trying to groom unbounded datasets into finite pools of information that eventually become complete, and instead live and breathe under the assumption that we will never know if or when we have seen all of our data, only that new data will arrive, old data may be retracted, and the only way to make this problem tractable is via principled abstractions that allow the practitioner the choice of appropriate tradeoffs along the axes of interest: correctness, latency, and cost. In this paper, we present one such approach, the Dataflow Model, along with a detailed examination of the semantics it enables, an overview of the core principles that guided its design, and a validation of the model itself via the real-world experiences that led to its development. We use the term “Dataflow Model” to describe the processing model of Google Cloud Dataflow [20], which is based upon technology from FlumeJava [12] and MillWheel [2]. This work is licensed under the Creative Commons AttributionNonCommercial-NoDerivs 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/. Obtain permission prior to any use beyond those covered by the license. Contact copyright holder by emailing info@vldb.org. Articles from this volume were invited to present their results at the 41st International Conference on Very Large Data Bases, August 31st September 4th 2015, Kohala Coast, Hawaii. Proceedings of the VLDB Endowment, Vol. 8, No. 12 Copyright 2015 VLDB Endowment 2150-8097/15/08.

The migration of computation and storage to cloud data centers has imposed new communication requirements and a need for fundamental changes in infrastructure. The cloud paradigm means that significant network communication will occur between servers in data centers. Furthermore, the migration of virtual machines and containers across physical servers means that in addition to traditional requirements of efficiency, scalability, and easy manageability, future data center networks must also support efficient mobility. To solve the problem, the paper proposes a radical redesign for data center networks, using a new architectural approach that we call DCnet. The DCnet approach changes the addressing and routing schemes at layers 2 and 3 completely. The new addressing mechanism allows an organization to assign addresses that span multiple data centers and permits VM mobility across data centers. Despite radical changes in addressing and routing, the DCnet architecture retains basics from Ethernet and IP, allowing existing hardware building blocks to be reused. The paper uses IPv6 in examples, but the new addressing model and address mobility approach can be applied to IPv4. By assigning each addressable entity a unique ID, and using the ID as the host suffix in an IP address, DCnet allows legacy operating systems and applications to be used without changes. The paper reports a simulation study that uses mininet and SDN to demonstrate the feasibility of the DCnet approach.

In previous papers [SC05, SBC+07], some of us predicted the end of “one size fits all” as a commercial relational DBMS paradigm. These papers presented reasons and experimental evidence that showed that the major RDBMS vendors can be outperformed by 1-2 orders of magnitude by specialized engines in the data warehouse, stream processing, text, and scientific database markets. Assuming that specialized engines dominate these markets over time, the current relational DBMS code lines will be left with the business data processing (OLTP) market and hybrid markets where more than one kind of capability is required. In this paper we show that current RDBMSs can be beaten by nearly two orders of magnitude in the OLTP market as well. The experimental evidence comes from comparing a new OLTP prototype, H-Store, which we have built at M.I.T., to a popular RDBMS on the standard transactional benchmark, TPC-C. We conclude that the current RDBMS code lines, while attempting to be a “one size fits all” solution, in fact, excel at nothing. Hence, they are 25 year old legacy code lines that should be retired in favor of a collection of “from scratch” specialized engines. The DBMS vendors (and the research community) should start with a clean sheet of paper and design systems for tomorrow’s requirements, not continue to push code lines and architectures designed for yesterday’s needs.

We present Stratosphere, an open-source software stack for parallel data analysis. Stratosphere brings together a unique set of features that allow the expressive, easy, and efficient programming of analytical applications at very large scale. Stratosphere’s features include “in situ” data processing, a declarative query language, treatment of user-defined functions as first-class citizens, automatic program parallelization and optimization, support for iterative programs, and a scalable and efficient execution engine. Stratosphere covers a variety of “Big Data” use cases, such as data warehousing, information extraction and integration, data cleansing, graph analysis, and statistical analysis applications. In this paper, we present the overall system architecture design decisions, introduce Stratosphere through example queries, and then dive into the internal workings of the system’s components that relate to extensibility, programming model, optimization, and query execution. We experimentally compare Stratosphere against popular open-source alternatives, and we conclude with a research outlook for the next years.

The field of data integration has expanded significantly over the years, from providing a uniform query and update interface to structured databases within an enterprise to the ability to search, ex- change, and even update, structured or unstructured data that are within or external to the enterprise. This paper describes the evolution in the landscape of data integration since the work on rewriting queries using views in the mid-1990's. In addition, we describe two important challenges for the field going forward. The first challenge is to develop good open-source tools for different components of data integration pipelines. The second challenge is to provide practitioners with viable solutions for the long-standing problem of systematically combining structured and unstructured data.

In the era of big data, data analytics, business intelligence database management plays a vital role from technical business management and research point of view. Over many decades, database management has been a topic of active research. There are different type of database management system have been proposed over a period of time but Relational Database Management System (RDBMS) is the one which has been most popularly used in academic research as well as industrial setup[1]. In recent years, graph databases regained interest among the researchers for certain obvious reasons. One of the most important reasons for such an interest in a graph database is because of the inherent property of graphs as a graph structure. Graphs are present everywhere in the data structure, which represents the strong connectivity within the data. Most of the graph database models are defined in which data-structure for schema and instances are modeled as graph or generalization of a graph. In such graph database models, data manipulations are expressed by graph-oriented operations and type constructors [9]. Now days, most of the real world applications can be modeled as a graph and one of the best real world examples is social or biological network. This paper gives an overview of the different type of graph databases, applications, and comparison between their models based on some properties.

Algorithms for Continuous Queries: A Geometric

HopsFS is an open-source, next generation distribution of the Apache Hadoop Distributed File System(HDFS) that replaces the main scalability bottleneck in HDFS, single node in-memory metadata service, with a no-sharedstate distributed system built on a NewSQL database. By removing the metadata bottleneck in Apache HDFS, HopsFS enables significantly larger cluster sizes, more than an order of magnitude higher throughput, and significantly lower clientlatencies for large clusters. In this paper, we detail the techniques and optimizations that enable HopsFS to surpass 1 million file system operations per second - at least 16 times higher throughput than HDFS. In particular, we discuss how we exploit recent high performance features from NewSQL databases, such as application defined partitioning, partition-pruned index scans, and distribution aware transactions. Together with more traditional techniques, such as batching and write-ahead caches, we show how many incremental optimizations have enabled a revolution in distributed hierarchical file system performance.

Cloud computing is getting popular and IT giants such as Google, Amazon, Microsoft, IBM have started their cloud computing infrastructure. However, current cloud implementations are often isolated from other cloud implementations. This paper gives an overview survey of current cloud computing architectures, discusses issues that current cloud computing implementations have and proposes a Service-Oriented Cloud Computing Architecture (SOCCA) so that clouds can interoperate with each other. Furthermore, the SOCCA also proposes high level designs to better support multi-tenancy feature of cloud computing.

Many public knowledge bases are represented and stored as RDF graphs, where users can issue structured queries on such graphs using SPARQL. With massive queries over large and constantly growing RDF data, it is imperative that an RDF graph store should provide low latency and high throughput for concurrent query processing. However, prior systems still experience high perquery latency over large datasets and most prior designs have poor resource utilization such that each query is processed in sequence. We present Wukong, a distributed graph-based RDF store that leverages RDMA-based graph exploration to provide highly concurrent and low-latency queries over large data sets. Wukong is novel in three ways. First, Wukong provides an RDMA-friendly distributed key/value store that provides differentiated encoding and fine-grained partitioning of graph data to reduce RDMA transfers. Second, Wukong leverages full-history pruning to avoid the cost of expensive final join operations, based on the observation that the cost of one-sided RDMA operations is largely oblivious to the payload size to a certain extent. Third, countering conventional wisdom of preferring migration of execution over data, Wukong seamlessly combines data migration for low latency and execution distribution for high throughput by leveraging the low latency and high throughput of onesided RDMA operations, and proposes a worker-obliger model for efficient load balancing. Evaluation on a 6-node RDMA-capable cluster shows that Wukong significantly outperforms state-of-the-art systems like TriAD and Trinity.RDF for both latency and throughput, usually at the scale of orders of magnitude. Short for Sun Wukong, who is known as the Monkey King and is a main character in the Chinese classical novel “Journey to the West”. Since Wukong is known for his extremely fast speed (21,675 kilometers in one somersault) and the ability to fork himself to do massive multi-tasking, we term our system as Wukong. The source code and a brief instruction on how to install Wukong is available at http://ipads.se.sjtu.edu.cn/projects/wukong.


Question: Cloud-based migration of NoSQL databases.

This paper revisits the CAP theorem a decade after its introduction, exploring how the principles governing networked shared-data systems have evolved. The CAP theorem posits that such systems can only possess two out of the three attributes: consistency, availability, and partition tolerance. Yet, through deliberate management of partitions, system designers can now achieve a balance that incorporates elements of all three properties, albeit with varying degrees of compromise. This article is complemented by a Web extra—a podcast episode from Software Engineering Radio—where the host engages in a detailed discussion with Dwight Merriman regarding the burgeoning NoSQL movement. The conversation delves into the three main categories of nonrelational data stores, the implications of Brewer's CAP theorem, and a wide array of related topics, providing insights into the contemporary landscape of data management and the strategies for optimizing data systems in cloud environments.

This paper describes MRJS, a JavaScript based MapReduce framework. MRJS runs on the open internet over HTTP rather than in a data center. Instead of using clusters of machines for processing power, it relies on volunteer clients in the form of JavaScript engines in web browsers. Users can specify, execute, and fetch data for arbitrary MapReduce jobs with HTTP requests. Our centralized job server acts as a coordinator. We have implemented MRJS and results show that it performs well for certain types of MapReduce computations.

In this paper, a method for transforming a relational database to a graph database model is described. In this approach, the dependency graphs for the entities in the system are transformed into star graphs. This star graph model is transformed into a hyper graph model for the relational database, which, in turn, can be used to develop the domain relationship model that can be converted in to a graph database model.

A cloud workflow system is a type of platform service which facilitates the automation of distributed applications based on the novel cloud infrastructure. Compared with grid environment, data transfer is a big overhead for cloud workflows due to the market-oriented business model in the cloud environments. In this paper, a Revised Discrete Particle Swarm Optimization (RDPSO) is proposed to schedule applications among cloud services that takes both data transmission cost and computation cost into account. Experiment is conducted with a set of workflow applications by varying their data communication costs and computation costs according to a cloud price model. Comparison is made on make span and cost optimization ratio and the cost savings with RDPSO, the standard PSO and BRS (Best Resource Selection) algorithm. Experimental results show that the proposed RDPSO algorithm can achieve much more cost savings and better performance on make span and cost optimization.

Cloud Adoption decisions tend to involve multiple, conflicting criteria (attributes) with incommensurable units of measurements, which must be compared among multiple alternatives using imprecise and incomplete available information. Multi-attribute Decision Making (MADM) has been shown to provide a rational basis to aid decision making in such scenarios. We present a MADMAC framework for cloud adoption, consisting of 3 Decision Areas (DA) referred to as the Cloud Switch, Cloud Type and Vendor Choice. It requires the definition of Attributes, Alternatives and Attribute Weights, to construct a Decision Matrix and arrive at a relative ranking to identify the optimal alternative. We also present a taxonomy organized in a two level hierarchy: Server-centric clouds, Client-centric clouds and Mobile-centric clouds, which further map to detailed, specific applications or workloads. DSS presented showing algorithms derived from MADMAC can compute and optimize CA decisions separately for the three stages, where the attributes differently influence CA decisions. A modified Wide-band Delphi method is proposed for assessing the relative weights for each attribute, by workload. Relative ranks are calculated using these weights, and the Simple Additive Weighting (SAW) method is used to generate value functions for all the alternatives, and rank the alternatives by their value to finally choose the best alternative. Results from application of the method to four different types of workloads show that the method converges on reasonable cloud adoption decisions. MADMAC's key advantage is its fully quantitative and iterative convergence approach based on proven multi-attribute decision methods, which enables decision makers to comparatively assess the relative robustness of alternative cloud adoption decisions in a defensible manner. Being amenable to automation, it can respond well to even complex arrays of decision criteria inputs, unlike human decision makers. It can be implemented as a web-based DSS to readily support cloud decision making world-wide, and improved further using fuzzy TOPSIS methods, to address concerns about preferential inter-dependence of attributes, insufficient input data or judgment expertise.

With the advancement of information and communication technology, data are being generated at an exponential rate via various instruments and collected at an unprecedented scale. Such large volume of data generated is referred to as big data, which now are revolutionizing all aspects of our life ranging from enterprises to individuals, from science communities to governments, as they exhibit great potentials to improve efficiency of enterprises and the quality of life. To obtain nontrivial patterns and derive valuable information from big data, a fundamental problem is how to properly place the collected data by different users to distributed clouds and to efficiently analyze the collected data to save user costs in data storage and processing, particularly the cost savings of users who share data. By doing so, it needs the close collaborations among the users, by sharing and utilizing the big data in distributed clouds due to the complexity and volume of big data. Since computing, storage and bandwidth resources in a distributed cloud usually are limited, and such resource provisioning typically is expensive, the collaborative users require to make use of the resources fairly. In this paper, we study a novel collaboration- and fairness-aware big data management problem in distributed cloud environments that aims to maximize the system throughout, while minimizing the operational cost of service providers to achieve the system throughput, subject to resource capacity and user fairness constraints. We first propose a novel optimization framework for the problem. We then devise a fast yet scalable approximation algorithm based on the built optimization framework. We also analyze the time complexity and approximation ratio of the proposed algorithm. We finally conduct experiments by simulations to evaluate the performance of the proposed algorithm. Experimental results demonstrate that the proposed algorithm is promising, and outperforms other heuristics.

Mobile computing continuously evolve through the sustained effort of many researchers. It seamlessly augments users' cognitive abilities via compute-intensive capabilities such as speech recognition, natural language processing, etc. By thus empowering mobile users, we could transform many areas of human activity. This article discusses the technical obstacles to these transformations and proposes a new architecture for overcoming them. In this architecture, a mobile user exploits virtual machine (VM) technology to rapidly instantiate customized service software on a nearby cloudlet and then uses that service over a wireless LAN; the mobile device typically functions as a thin client with respect to the service. A cloudlet is a trusted, resource-rich computer or cluster of computers that's well-connected to the Internet and available for use by nearby mobile devices. Our strategy of leveraging transiently customized proximate infrastructure as a mobile device moves with its user through the physical world is called cloudlet-based, resource-rich, mobile computing. Crisp interactive response, which is essential for seamless augmentation of human cognition, is easily achieved in this architecture because of the cloudlet's physical proximity and one-hop network latency. Using a cloudlet also simplifies the challenge of meeting the peak bandwidth demand of multiple users interactively generating and receiving media such as high-definition video and high-resolution images. Rapid customization of infrastructure for diverse applications emerges as a critical requirement, and our results from a proof-of-concept prototype suggest that VM technology can indeed help meet this requirement.

Now a day’s Peta-Bytes of data becomes the norm in industries. Handling, analyzing such big data is challenging task. Even frameworks like Hadoop (Open Source Implementation of MapReduce Paradigm) and NoSQL databases like Cassandra, HBase can be used to analyze and store such large data; heterogeneity of data is still an issue. Data centers usually have clusters formed using heterogeneous nodes. Ecosystem like Hadoop can be used to manage such types of cluster but it can not schedule jobs (Application) efficiently on this heterogeneous cluster when data is itself heterogeneous. Heterogeneity of data may be because of data format or because of the complexity. This paper is review of systems and algorithms for distributed management of huge size data, efficiency of these approaches.

Personal mobile devices (PMDs) have become the most used computing devices for many people. With the introduction of mobile cloud computing, we can augment the storage and computing capabilities of PMDs via cloud support. However, there are many challenges in developing mobile cloud applications (MCAs) that incorporate cloud computing efficiently, especially for developers targeting multiple mobile platforms. This paper presents Uniport, a uniform framework for developing MCAs. We introduce a uniform architecture for MCAs based on the Model-View-Controller (MVC) pattern and a set of programming primitives and runtime libraries. Not only can Uniport support the creation of new MCAs, it can also help transform existing mobile applications to MCAs efficiently. We demonstrate the applicability and flexibility of Uniport in a case study to transform three existing mobile applications on iOS, Android and Windows Phone, to their mobile cloud versions respectively. Evaluation results show that, with very few modifications, we can easily transform mobile applications to MCAs that can exploit the cloud support to improve performance by 3 - 7x and save more than half of their energy consumption.

Cloud computing is the delivers the computing services over the internet. Cloud services help individuals and organization to use data that are managed by third parties or another person at remote locations. Virtual Machine (VM) is an emulation of a particular computer system. In cloud computing, Virtual machine migration is a useful tool for migrating Operating System instances across multiple physical machines. It is used to load balancing, fault management, low-level system maintenance and reduce energy consumption. There are various techniques and parameters available for VM migration. This paper presents the various virtual machine migration techniques.

This article introduces the Follow-Me Cloud concept and proposes its framework. The proposed framework is aimed at smooth migration of all or only a required portion of an ongoing IP service between a data center and user equipment of a 3GPP mobile network to another optimal DC with no service disruption. The service migration and continuity is supported by replacing IP addressing with service identification. Indeed, an FMC service/application is identified, upon establishment, by a session/service ID, dynamically changing along with the service being delivered over the session; it consists of a unique identifier of UE within the 3GPP mobile network, an identifier of the cloud service, and dynamically changing characteristics of the cloud service. Service migration in FMC is triggered by change in the IP address of the UE due to a change of data anchor gateway in the mobile network, in turn due to UE mobility and/or for load balancing. An optimal DC is then selected based on the features of the new data anchor gateway. Smooth service migration and continuity are supported thanks to logic installed at UE and DCs that maps features of IP flows to the session/service ID.


Question: Auto-conditioned recurrent networks for generating prolonged intricate human motion synthesis.

This document explores the computational intricacies and efficiency of gradient-based learning algorithms tailored for recurrent neural networks (RNNs). Unlike feedforward networks, RNNs possess a unique architecture that allows for the processing of sequential data, making them particularly suited for tasks involving temporal dynamics. The focus here is on understanding the computational complexity associated with training these networks using gradient-based methods. These algorithms, including backpropagation through time (BPTT) and real-time recurrent learning (RTRL), are essential for optimizing the weights in RNNs to minimize prediction errors. The document delves into the mathematical underpinnings of these techniques, examining how they scale with the size of the network and the length of the input sequences. It also discusses the challenges posed by vanishing and exploding gradients, common issues in deep RNNs, and strategies to mitigate these problems. The goal is to provide insights into the computational demands of training RNNs, which is crucial for practitioners aiming to balance model complexity with computational resources when dealing with tasks such as speech recognition, natural language processing, and complex human motion synthesis.

**Question**: An Overview of Gradient Descent Algorithms for Training Recurrent Neural Networks

**Documents**: Learning long-term dependencies in recurrent neural networks with gradient-based methods

**Similar Documents**: This document offers a comprehensive analysis of gradient-based methods employed for training recurrent neural networks (RNNs) in the context of learning long-term dependencies. RNNs, with their inherent capability to process sequential data, are pivotal in various applications, from language translation to speech recognition. However, capturing long-term dependencies in sequences presents a significant challenge due to the vanishing gradient problem, where the influence of earlier time steps on the current state diminishes rapidly. The document explores the intricacies of gradient-based learning algorithms, such as backpropagation through time (BPTT), and their limitations in handling long-term dependencies. It also highlights techniques and modifications to these algorithms designed to overcome these limitations, including the use of gated architectures like Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRUs). Additionally, the document discusses the computational complexity associated with these methods and the trade-offs between model complexity and training efficiency. The aim is to provide a detailed understanding of how gradient-based methods can be effectively used to train RNNs for tasks that require the recognition and processing of long-term dependencies in data sequences.

Skilled robot task learning is best implemented by predictive action policies due to the inherent latency of sensorimotor processes. However, training such predictive policies is challenging as it involves finding a trajectory of motor activations for the full duration of the action. We propose a data-efficient deep predictive policy training (DPPT) framework with a deep neural network policy architecture which maps an image observation to a sequence of motor activations. The architecture consists of three sub-networks referred to as the perception, policy and behavior super-layers. The perception and behavior super-layers force an abstraction of visual and motor data trained with synthetic and simulated training samples, respectively. The policy super-layer is a small subnetwork with fewer parameters that maps data in-between the abstracted manifolds. It is trained for each task using methods for policy search reinforcement learning. We demonstrate the suitability of the proposed architecture and learning framework by training predictive policies for skilled object grasping and ball throwing on a PR2 robot. The effectiveness of the method is illustrated by the fact that these tasks are trained using only about 180 real robot attempts with qualitative terminal rewards.

We study probabilistic models of natural images and extend the autoregressive family of PixelCNN models by incorporating latent variables. Subsequently, we describe two new generative image models that exploit different image transformations as latent variables: a quantized grayscale view of the image or a multi-resolution image pyramid. The proposed models tackle two known shortcomings of existing PixelCNN models: 1) their tendency to focus on low-level image details, while largely ignoring high-level image information, such as object shapes, and 2) their computationally costly procedure for image sampling. We experimentally demonstrate benefits of our LatentPixelCNN models, in particular showing that they produce much more realistically looking image samples than previous stateof-the-art probabilistic models.

We advocate the use of Gaussian Process Dynamical Models (GPDMs) for learning human pose and motion priors for 3D people tracking. A GPDM provides a lowdimensional embedding of human motion data, with a density function that gives higher probability to poses and motions close to the training data. With Bayesian model averaging a GPDM can be learned from relatively small amounts of data, and it generalizes gracefully to motions outside the training set. Here we modify the GPDM to permit learning from motions with significant stylistic variation. The resulting priors are effective for tracking a range of human walking styles, despite weak and noisy image measurements and significant occlusions.

In this paper, we propose a novel approach to video captioning based on adversarial learning and long short-term memory (LSTM). With this solution concept, we aim at compensating for the deficiencies of LSTM-based video captioning methods that generally show potential to effectively handle temporal nature of video data when generating captions but also typically suffer from exponential error accumulation. Specifically, we adopt a standard generative adversarial network (GAN) architecture, characterized by an interplay of two competing processes: a “generator” that generates textual sentences given the visual content of a video and a “discriminator” that controls the accuracy of the generated sentences. The discriminator acts as an “adversary” toward the generator, and with its controlling mechanism, it helps the generator to become more accurate. For the generator module, we take an existing video captioning concept using LSTM network. For the discriminator, we propose a novel realization specifically tuned for the video captioning problem and taking both the sentences and video features as input. This leads to our proposed LSTM–GAN system architecture, for which we show experimentally to significantly outperform the existing methods on standard public datasets.

Recommender systems traditionally assume that user profiles and movie attributes are static. Temporal dynamics are purely reactive, that is, they are inferred after they are observed, e.g. after a user's taste has changed or based on hand-engineered temporal bias corrections for movies. We propose Recurrent Recommender Networks (RRN) that are able to predict future behavioral trajectories. This is achieved by endowing both users and movies with a Long Short-Term Memory (LSTM) autoregressive model that captures dynamics, in addition to a more traditional low-rank factorization. On multiple real-world datasets, our model offers excellent prediction accuracy and it is very compact, since we need not learn latent state but rather just the state transition function.

Steering a car through traffic is a complex task that is difficult to cast into algorithms. Therefore, researchers turn to training artificial neural networks from front-facing camera data stream along with the associated steering angles. Nevertheless, most existing solutions consider only the visual camera frames as input, thus ignoring the temporal relationship between frames. In this work, we propose a Convolutional Long Short-Term Memory Recurrent Neural Network (C-LSTM), that is end-to-end trainable, to learn both visual and dynamic temporal dependencies of driving. Additionally, We introduce posing the steering angle regression problem as classification while imposing a spatial relationship between the output layer neurons. Such method is based on learning a sinusoidal function that encodes steering angles. To train and validate our proposed methods, we used the publicly available Comma.ai dataset. Our solution improved steering root mean square error by 35% over recent methods, and led to a more stable steering by 87%.

We introduce AutoHair, the first fully automatic method for 3D hair modeling from a single portrait image, with no user interaction or parameter tuning. Our method efficiently generates complete and high-quality hair geometries, which are comparable to those generated by the state-of-the-art methods, where user interaction is required. The core components of our method are: a novel hierarchical deep neural network for automatic hair segmentation and hair growth direction estimation, trained over an annotated hair image database; and an efficient and automatic data-driven hair matching and modeling algorithm, based on a large set of 3D hair exemplars. We demonstrate the efficacy and robustness of our method on Internet photos, resulting in a database of around 50K 3D hair models and a corresponding hairstyle space that covers a wide variety of real-world hairstyles. We also show novel applications enabled by our method, including 3D hairstyle space navigation and hair-aware image retrieval.

Creating and animating realistic 3D human faces is an important element of virtual reality, video games, and other areas that involve interactive 3D graphics. In this paper, we propose a system to generate photorealistic 3D blendshape-based face models automatically using only a single consumer RGB-D sensor. The capture and processing requires no artistic expertise to operate, takes 15 seconds to capture and generate a single facial expression, and approximately 1 minute of processing time per expression to transform it into a blendshape model. Our main contributions include a complete end-to-end pipeline for capturing and generating photorealistic blendshape models automatically and a registration method that solves dense correspondences between two face scans by utilizing facial landmarks detection and optical flows. We demonstrate the effectiveness of the proposed method by capturing different human subjects with a variety of sensors and puppeteering their 3D faces with real-time facial performance retargeting. The rapid nature of our method allows for just-in-time construction of a digital face. To that end, we also integrated our pipeline with a virtual reality facial performance capture system that allows dynamic embodiment of the generated faces despite partial occlusion of the user’s real face by the head-mounted display.

We present a new method for generating a dynamic, concatenative, unit of visual speech that can generate realistic visual speech animation. We redefine visemes as temporal units that describe distinctive speech movements of the visual speech articulators. Traditionally visemes have been surmized as the set of static mouth shapes representing clusters of contrastive phonemes (e.g. /p, b, m/, and /f, v/). In this work, the motion of the visual speech articulators are used to generate discrete, dynamic visual speech gestures. These gestures are clustered, providing a finite set of movements that describe visual speech, the visemes. Dynamic visemes are applied to speech animation by simply concatenating viseme units. We compare to static visemes using subjective evaluation. We find that dynamic visemes are able to produce more accurate and visually pleasing speech animation given phonetically annotated audio, reducing the amount of time that an animator needs to spend manually refining the animation.

Up until recently Recurrent neural networks (RNNs) have been the standard go-to component when processing sequential data with neural networks. Issues relative to vanishing gradient have been partly addressed by Long short-term memory (LSTM) and gated recurrent unit (GRU), but in practice experiments show that very long terms dependencies (beyond 1000 time steps) are difficult to learn. We introduce IGLOO, a new neural network architecture which aims at being faster than both LSTM and GRU but also their respective CuDNN optimized versions when convergence happens and provide an alternative in the case when there is no convergence at all. IGLOO’s core idea is to use the relationships between patches sliced out of the features maps of convolution layers at different levels of granularity to build a representation for the sequence. We show that the model can deal with dependencies of more than 25,000 steps in a reasonable time frame. Beyond the well bench-marked copy-memory and addition problems which show good results, we also achieve best recorded accuracy on permuted MNIST (98.4%). IGLOO is also applied on the IMDB set and on the TUH Abnormal EEG Corpus.


Question: Auto-conditioned recurrent networks for generating prolonged intricate human motion synthesis.

Introducing the Professor Forcing Technique: Enhancing Recurrent Network Training for Improved Sequence Generation

Recurrent networks are often trained using the Teacher Forcing technique, which involves feeding observed sequence values as inputs during training while relying on the network's one-step-ahead predictions for multi-step sampling. However, this method can lead to discrepancies between the network's dynamics during training and sampling phases. To address this issue, we present the Professor Forcing technique, a novel algorithm that employs adversarial domain adaptation to ensure that the recurrent network's behavior remains consistent across training and sampling stages, even over extended time steps.

Professor Forcing has been applied to a variety of tasks, including language modeling, raw waveform vocal synthesis, handwriting generation, and image synthesis. Our empirical findings indicate that this technique acts as a powerful regularizer, enhancing test likelihood on character-level Penn Treebank and sequential MNIST datasets. Moreover, we observe a qualitative improvement in sample quality, particularly when dealing with long sequences, as supported by human evaluations.

The trade-offs between Professor Forcing and Scheduled Sampling are discussed in detail. Our analysis, supported by T-SNE visualizations, confirms that Professor Forcing effectively narrows the gap between the network's dynamics during training and sampling, leading to more coherent and realistic sequence generation.

We consider the problem of providing dense segmentation masks for object discovery in videos. We formulate the object discovery problem as foreground motion clustering, where the goal is to cluster foreground pixels in videos into different objects. We introduce a novel pixel-trajectory recurrent neural network that learns feature embeddings of foreground pixel trajectories linked in time. By clustering the pixel trajectories using the learned feature embeddings, our method establishes correspondences between foreground object masks across video frames. To demonstrate the effectiveness of our framework for object discovery, we conduct experiments on commonly used datasets for motion segmentation, where we achieve state-of-the-art performance.

We propose an application of SeqGAN, generative adversarial networks for discrete sequence generation, for creating polyphonic musical sequences. Instead of monophonic melody generation suggested in the original work, we present an efficient representation of polyphony MIDI file that captures chords and melodies with dynamic timings simultaneously. The network can create sequences that are musically coherent. We also report that careful tuning of reinforcement learning signals of the model are crucial for general application.

Deep learning has exploded in popularity in the past several years. Picking up from recent projects using recurrent neural networks to generate text, we attempted to use the same principles to generate jazz-like improvisations. We provide an RNN using torch-rnn, trained on freely available MIDI files of jazz transcriptions, which can generate music that captures rhythms, and to a lesser extend harmony and melody. Listener feedback shows that our generated music is easily recognizable as “jazz”, though lacking macro-level structure as can be expected for a basic RNN generating entirely sequential patterns. We also provide a new ASCII-based file format capable of storing music in a compact but not totally opaque way. Our format allows for close to a 7-times size decrease from existing MIDI-to-CSV converters. We show the musical principles behind the information we chose to include and leave out in converting MIDI to our new format.

Generative Adversarial Networks (GAN) have shown great promise in tasks like synthetic image generation, image inpainting, style transfer, and anomaly detection. However, generating discrete data is a challenge. This work presents an adversarial training based correlated discrete data (CDD) generation model. It also details an approach for conditional CDD generation. The results of our approach are presented over two datasets; job-seeking candidates skill set (private dataset) and MNIST (public dataset). From quantitative and qualitative analysis of these results, we show that our model performs better as it leverages inherent correlation in the data, than an existing model that overlooks correlation.

ÐA new view-based approach to the representation and recognition of human movement is presented. The basis of the representation is a temporal templateÐa static vector-image where the vector value at each point is a function of the motion properties at the corresponding spatial location in an image sequence. Using aerobics exercises as a test domain, we explore the representational power of a simple, two component version of the templates: The first value is a binary value indicating the presence of motion and the second value is a function of the recency of motion in a sequence. We then develop a recognition method matching temporal templates against stored instances of views of known actions. The method automatically performs temporal segmentation, is invariant to linear changes in speed, and runs in real-time on standard platforms. Index TermsÐMotion recognition, computer vision.

In a conversation or a dialogue process, attention and intention play intrinsic roles. This paper proposes a neural network based approach that models the attention and intention processes. It essentially consists of three recurrent networks. The encoder network is a word-level model representing source side sentences. The intention network is a recurrent network that models the dynamics of the intention process. The decoder network is a recurrent network that produces responses to the input from the source side. It is a language model that is dependent on the intention and has an attention mechanism to attend to particular source side words, when predicting a symbol in the response. The model is trained end-toend without labeling data. Experiments show that this model generates natural responses to user inputs.

Learned 3D representations of human faces are useful for computer vision problems such as 3D face tracking and reconstruction from images, as well as graphics applications such as character generation and animation. Traditional models learn a latent representation of a face using linear subspaces or higher-order tensor generalizations. Due to this linearity, they can not capture extreme deformations and nonlinear expressions. To address this, we introduce a versatile model that learns a non-linear representation of a face using spectral convolutions on a mesh surface. We introduce mesh sampling operations that enable a hierarchical mesh representation that captures non-linear variations in shape and expression at multiple scales within the model. In a variational setting, our model samples diverse realistic 3D faces from a multivariate Gaussian distribution. Our training data consists of 20,466 meshes of extreme expressions captured over 12 different subjects. Despite limited training data, our trained model outperforms state-of-the-art face models with 50% lower reconstruction error, while using 75% fewer parameters. We also show that, replacing the expression space of an existing state-ofthe-art face model with our autoencoder, achieves a lower reconstruction error. Our data, model and code are available at http://github.com/ anuragranj/coma.

Being able to model long-term dependencies in sequential data, such as text, has been among the long-standing challenges of recurrent neural networks (RNNs). This issue is strictly related to the absence of explicit planning in current RNN architectures, more explicitly, the network is trained to predict only the next token given previous ones. In this paper, we introduce a simple way of biasing the RNNs towards planning behavior. Particularly, we introduce an additional neural network which is trained to generate the sequence in reverse order, and we require closeness between the states of the forward RNN and backward RNN that predict the same token. At each step, the states of the forward RNN are required to match the future information contained in the backward states. We hypothesize that the approach eases modeling of long-term dependencies thus helping in generating more globally consistent samples. The model trained with conditional generation achieved 4% relative improvement (CER of 7.3 compared to a baseline of 7.6).

Facial expression synthesis has drawn much attention in the field of computer graphics and pattern recognition. It has been widely used in face animation and recognition. However, it is still challenging due to the high-level semantic presence of large and non-linear face geometry variations. This paper proposes a Geometry-Guided Generative Adversarial Network (G2-GAN) for continuously-adjusting and identity-preserving facial expression synthesis. We employ facial geometry (fiducial points) as a controllable condition to guide facial texture synthesis with specific expression. A pair of generative adversarial subnetworks is jointly trained towards opposite tasks: expression removal and expression synthesis. The paired networks form a mapping cycle between neutral expression and arbitrary expressions, with which the proposed approach can be conducted among unpaired data. The proposed paired networks also facilitate other applications such as face transfer, expression interpolation and expression-invariant face recognition. Experimental results on several facial expression databases show that our method can generate compelling perceptual results on different expression editing tasks.

Recent progress in deep generative models has been fueled by two paradigms – autoregressive and adversarial models. We propose a combination of both approaches with the goal of learning generative models of text. Our method first produces a high-level sentence outline and then generates words sequentially, conditioning on both the outline and the previous outputs. We generate outlines with an adversarial model trained to approximate the distribution of sentences in a latent space induced by general-purpose sentence encoders. This provides strong, informative conditioning for the autoregressive stage. Our quantitative evaluations suggests that conditioning information from generated outlines is able to guide the autoregressive model to produce realistic samples, comparable to maximum-likelihood trained language models, even at high temperatures with multinomial sampling. Qualitative results also demonstrate that this generative procedure yields natural-looking sentences and interpolations.


Question: Auto-conditioned recurrent neural networks for generating extended, intricate human motion sequences.

Introducing the Auto-Conditioned Attentive Synthesizer (ACAS): A Recurrent Neural Network for Advanced Human Motion Generation

This paper presents the Auto-Conditioned Attentive Synthesizer (ACAS), a novel recurrent neural network architecture designed for the sophisticated synthesis of human motion sequences. ACAS integrates an innovative temporal attention mechanism that emulates the selective focus of human observation, paired with a sequential generative framework that enables the progressive construction of intricate and extended motion patterns. The system significantly outperforms existing models on benchmark datasets for human motion generation, and when trained on complex dance routines, it produces sequences that are indistinguishable from real performances to the untrained eye, showcasing its exceptional capability in capturing and reproducing the nuanced dynamics of human movement.

This paper proposes the use of an end-to-end Convolutional Neural Network for direct reconstruction of the 3D geometry of humans via volumetric regression. The proposed method does not require the fitting of a shape model and can be trained to work from a variety of input types, whether it be landmarks, images or segmentation masks. Additionally, non-visible parts, either self-occluded or otherwise, are still reconstructed, which is not the case with depth map regression. We present results that show that our method can handle both pose variation and detailed reconstruction given appropriate datasets for training.

We present a new topic model that generates documents by sampling a topic for one whole sentence at a time, and generating the words in the sentence using an RNN decoder that is conditioned on the topic of the sentence. We argue that this novel formalism will help us not only visualize and model the topical discourse structure in a document better, but also potentially lead to more interpretable topics since we can now illustrate topics by sampling representative sentences instead of bag of words or phrases. We present a variational auto-encoder approach for learning in which we use a factorized variational encoder that independently models the posterior over topical mixture vectors of documents using a feedforward network, and the posterior over topic assignments to sentences using an RNN. Our preliminary experiments on two different datasets indicate early promise, but also expose many challenges that remain to be addressed.

Researchers have recently started investigating deep neural networks for dialogue applications. In particular, generative sequence-to-sequence (Seq2Seq) models have shown promising results for unstructured tasks, such as word-level dialogue response generation. The hope is that such models will be able to leverage massive amounts of data to learn meaningful natural language representations and response generation strategies, while requiring a minimum amount of domain knowledge and hand-crafting. An important challenge is to develop models that can effectively incorporate dialogue context and generate meaningful and diverse responses. In support of this goal, we review recently proposed models based on generative encoder-decoder neural network architectures, and show that these models have better ability to incorporate long-term dialogue history, to model uncertainty and ambiguity in dialogue, and to generate responses with high-level compositional structure.

We describe a complete system for animating realistic clothing on synthetic bodies of any shape and pose without manual intervention. The key component of the method is a model of clothing called DRAPE (DRessing Any PErson) that is learned from a physics-based simulation of clothing on bodies of different shapes and poses. The DRAPE model has the desirable property of "factoring" clothing deformations due to body shape from those due to pose variation. This factorization provides an approximation to the physical clothing deformation and greatly simplifies clothing synthesis. Given a parameterized model of the human body with known shape and pose parameters, we describe an algorithm that dresses the body with a garment that is customized to fit and possesses realistic wrinkles. DRAPE can be used to dress static bodies or animated sequences with a learned model of the cloth dynamics. Since the method is fully automated, it is appropriate for dressing large numbers of virtual characters of varying shape. The method is significantly more efficient than physical simulation.

We study two fundamental issues about video-based human walking estimation, where the goal is to estimate 3D gait kinematics (i.e., joint positions) from 2D gait appearances (i.e., silhouettes). One is how to model the gait kinematics from different walking styles, and the other is how to represent the gait appearances captured under different views and from individuals of distinct walking styles and body shapes. Our research is conducted in three steps. First, we propose the idea of joint gait-pose manifold (JGPM), which represents gait kinematics by coupling two nonlinear variables, pose (a specific walking stage) and gait (a particular walking style) in a unified latent space. We extend the Gaussian process latent variable model (GPLVM) for JGPM learning, where two heuristic topological priors, a torus and a cylinder, are considered and several JGPMs of different degrees of freedom (DoFs) are introduced for comparative analysis. Second, we develop a validation technique and a series of benchmark tests to evaluate multiple JGPMs and recent GPLVMs in terms of their performance for gait motion modeling. It is shown that the toroidal prior is slightly better than the cylindrical one, and the JGPM of 4 DoFs that balances the toroidal prior with the intrinsic data structure achieves the best performance. Third, a JGPM-based visual gait generative model (JGPM-VGGM) is developed, where JGPM plays a central role to bridge the gap between the gait appearances and the gait kinematics. Our proposed JGPM-VGGM is learned from Carnegie Mellon University MoCap data and tested on the HumanEva-I and HumanEva-II data sets. Our experimental results demonstrate the effectiveness and competitiveness of our algorithms compared with existing algorithms.

Human activity understanding with 3D/depth sensors has received increasing attention in multimedia processing and interactions. This work targets on developing a novel deep model for automatic activity recognition from RGB-D videos. We represent each human activity as an ensemble of cubic-like video segments, and learn to discover the temporal structures for a category of activities, i.e. how the activities to be decomposed in terms of classification. Our model can be regarded as a structured deep architecture, as it extends the convolutional neural networks (CNNs) by incorporating structure alternatives. Specifically, we build the network consisting of 3D convolutions and max-pooling operators over the video segments, and introduce the latent variables in each convolutional layer manipulating the activation of neurons. Our model thus advances existing approaches in two aspects: (i) it acts directly on the raw inputs (grayscale-depth data) to conduct recognition instead of relying on hand-crafted features, and (ii) the model structure can be dynamically adjusted accounting for the temporal variations of human activities, i.e. the network configuration is allowed to be partially activated during inference. For model training, we propose an EM-type optimization method that iteratively (i) discovers the latent structure by determining the decomposed actions for each training example, and (ii) learns the network parameters by using the back-propagation algorithm. Our approach is validated in challenging scenarios, and outperforms state-of-the-art methods. A large human activity database of RGB-D videos is presented in addition.

Autonomous exploration of motor skills is a key capability of learning robotic systems. Learning motor skills can be formulated as inverse modeling problem, which targets at finding an inverse model that maps desired outcomes in some task space, e.g., via points of a motion, to appropriate actions, e.g., motion control policy parameters. In this paper, autonomous exploration of motor skills is achieved by incrementally learning inverse models starting from an initial demonstration. The algorithm is referred to as skill babbling, features sample-efficient learning, and scales to high-dimensional action spaces. Skill babbling extends ideas of goal-directed exploration, which organizes exploration in the space of goals. The proposed approach provides a modular framework for autonomous skill exploration by separating the learning of the inverse model from the exploration mechanism and a model of achievable targets, i.e. the workspace. The effectiveness of skill babbling is demonstrated for a range of motor tasks comprising the autonomous bootstrapping of inverse kinematics and parameterized motion primitives.

Markov chains are an enticing option for machine learned generation of platformer levels, but offer poor control for designers and are likely to produce unplayable levels. In this paper we present a method for guiding Markov chain generation using Monte Carlo Tree Search that we call Markov Chain Monte Carlo Tree Search (MCMCTS). We demonstrate an example use for this technique by creating levels trained on a corpus of levels from Super Mario Bros. We then present a player modeling study that was run with the hopes of using the data to better inform the generation of levels in future work.

Processing temporal sequences is central to a variety of applications in health care, and in particular multichannel Electrocardiogram (ECG) is a highly prevalent diagnostic modality that relies on robust sequence modeling. While Recurrent Neural Networks (RNNs) have led to significant advances in automated diagnosis with time-series data, they perform poorly when models are trained using a limited set of channels. A crucial limitation of existing solutions is that they rely solely on discriminative models, which tend to generalize poorly in such scenarios. In order to combat this limitation, we develop a generative modeling approach to limited channel ECG classification. This approach first uses a Seq2Seq model to implicitly generate the missing channel information, and then uses the latent representation to perform the actual supervisory task. This decoupling enables the use of unsupervised data and also provides highly robust metric spaces for subsequent discriminative learning. Our experiments with the Physionet dataset clearly evidence the effectiveness of our approach over standard RNNs in disease prediction.

Forecasting human activities from visual evidence is an emerging area of research which aims to allow computational systems to make predictions about unseen human actions. We explore the task of activity forecasting in the context of dual-agent interactions to understand how the actions of one person can be used to predict the actions of another. We model dual-agent interactions as an optimal control problem, where the actions of the initiating agent induce a cost topology over the space of reactive poses – a space in which the reactive agent plans an optimal pose trajectory. The technique developed in this work employs a kernel-based reinforcement learning approximation of the soft maximum value function to deal with the high-dimensional nature of human motion and applies a mean-shift procedure over a continuous cost function to infer a smooth reaction sequence. Experimental results show that our proposed method is able to properly model human interactions in a high dimensional space of human poses. When compared to several baseline models, results show that our method is able to generate highly plausible simulations of human interaction.


Question: Auto-regressive recurrent networks for prolonged intricate human motion generation.

In this work, we explore the application of Long Short-term Memory (LSTM) recurrent neural networks in the generation of intricate sequences that exhibit long-range structure, achieved through the sequential prediction of individual data points. This methodology is showcased in two domains: text, characterized by discrete data points, and online handwriting, which involves real-valued data. To further enhance the capability of our model, we introduce a conditioning mechanism that enables the network to tailor its predictions based on a given text sequence. This augmented system is capable of producing highly realistic cursive handwriting, demonstrating remarkable versatility in emulating diverse writing styles. This advancement in sequence generation, particularly in the context of human motion synthesis, highlights the potential of LSTM-based networks in capturing and reproducing complex patterns over extended periods. Specifically, in the realm of human motion, such models could be instrumental in synthesizing natural and fluid movements, essential for applications in animation, robotics, and virtual reality.

We study the use of feed-forward convolutional neural networks for the unsupervised problem of mining recurrent temporal patterns mixed in multivariate time series. Traditional convolutional autoencoders lack interpretability for two main reasons: the number of patterns corresponds to the manually-fixed number of convolution filters, and the patterns are often redundant and correlated. To recover clean patterns, we introduce different elements in the architecture, including an adaptive rectified linear unit function that improves patterns interpretability, and a group-lasso regularizer that helps automatically finding the relevant number of patterns. We illustrate the necessity of these elements on synthetic data and real data in the context of activity mining in videos.

This tutorial is a worked-out version of a 5-hour course originally held at AIS in September/October 2002. It has two distinct components. First, it contains a mathematically-oriented crash course on traditional training methods for recurrent neural networks, covering back-propagation through time (BPTT), real-time recurrent learning (RTRL), and extended Kalman filtering approaches (EKF). This material is covered in Sections 2 – 5. The remaining sections 1 and 6 – 9 are much more gentle, more detailed, and illustrated with simple examples. They are intended to be useful as a stand-alone tutorial for the echo state network (ESN) approach to recurrent neural network training.

Deep learning has exploded in popularity in the past several years. Picking up from recent projects using recurrent neural networks to generate text, we attempted to use the same principles to generate jazz-like improvisations. We provide an RNN using torch-rnn, trained on freely available MIDI files of jazz transcriptions, which can generate music that captures rhythms, and to a lesser extend harmony and melody. Listener feedback shows that our generated music is easily recognizable as “jazz”, though lacking macro-level structure as can be expected for a basic RNN generating entirely sequential patterns. We also provide a new ASCII-based file format capable of storing music in a compact but not totally opaque way. Our format allows for close to a 7-times size decrease from existing MIDI-to-CSV converters. We show the musical principles behind the information we chose to include and leave out in converting MIDI to our new format.

Up until recently Recurrent neural networks (RNNs) have been the standard go-to component when processing sequential data with neural networks. Issues relative to vanishing gradient have been partly addressed by Long short-term memory (LSTM) and gated recurrent unit (GRU), but in practice experiments show that very long terms dependencies (beyond 1000 time steps) are difficult to learn. We introduce IGLOO, a new neural network architecture which aims at being faster than both LSTM and GRU but also their respective CuDNN optimized versions when convergence happens and provide an alternative in the case when there is no convergence at all. IGLOO’s core idea is to use the relationships between patches sliced out of the features maps of convolution layers at different levels of granularity to build a representation for the sequence. We show that the model can deal with dependencies of more than 25,000 steps in a reasonable time frame. Beyond the well bench-marked copy-memory and addition problems which show good results, we also achieve best recorded accuracy on permuted MNIST (98.4%). IGLOO is also applied on the IMDB set and on the TUH Abnormal EEG Corpus.

We introduce a long short-term memory recurrent neural network (LSTM-RNN) approach for real-time facial animation, which automatically estimates head rotation and facial action unit activations of a speaker from just her speech. Specifically, the time-varying contextual non-linear mapping between audio stream and visual facial movements is realized by training a LSTM neural network on a large audio-visual data corpus. In this work, we extract a set of acoustic features from input audio, including Mel-scaled spectrogram, Mel frequency cepstral coefficients and chromagram that can effectively represent both contextual progression and emotional intensity of the speech. Output facial movements are characterized by 3D rotation and blending expression weights of a blendshape model, which can be used directly for animation. Thus, even though our model does not explicitly predict the affective states of the target speaker, her emotional manifestation is recreated via expression weights of the face model. Experiments on an evaluation dataset of different speakers across a wide range of affective states demonstrate promising results of our approach in real-time speech-driven facial animation.

High-level spinal cord injury (SCI) in humans causes paralysis below the neck. Functional electrical stimulation (FES) technology applies electrical current to nerves and muscles to restore movement, and controllers for upper extremity FES neuroprostheses calculate stimulation patterns to produce desired arm movement. However, currently available FES controllers have yet to restore natural movements. Reinforcement learning (RL) is a reward-driven control technique; it can employ user-generated rewards, and human preferences can be used in training. To test this concept with FES, we conducted simulation experiments using computer-generated “pseudo-human” rewards. Rewards with varying properties were used with an actor-critic RL controller for a planar two-degree-of-freedom biomechanical human arm model performing reaching movements. Results demonstrate that sparse, delayed pseudo-human rewards permit stable and effective RL controller learning. The frequency of reward is proportional to learning success, and human-scale sparse rewards permit greater learning than exclusively automated rewards. Diversity of training task sets did not affect learning. Long-term stability of trained controllers was observed. Using human-generated rewards to train RL controllers for upper-extremity FES systems may be useful. Our findings represent progress toward achieving human-machine teaming in control of upper-extremity FES systems for more natural arm movements based on human user preferences and RL algorithm learning capabilities.

The long short-term memory (LSTM) network trained by gradient descent solves difficult problems which traditional recurrent neural networks in general cannot. We have recently observed that the decoupled extended Kalman filter training algorithm allows for even better performance, reducing significantly the number of training steps when compared to the original gradient descent training algorithm. In this paper we present a set of experiments which are unsolvable by classical recurrent networks but which are solved elegantly and robustly and quickly by LSTM combined with Kalman filters.

Forecasting human activities from visual evidence is an emerging area of research which aims to allow computational systems to make predictions about unseen human actions. We explore the task of activity forecasting in the context of dual-agent interactions to understand how the actions of one person can be used to predict the actions of another. We model dual-agent interactions as an optimal control problem, where the actions of the initiating agent induce a cost topology over the space of reactive poses – a space in which the reactive agent plans an optimal pose trajectory. The technique developed in this work employs a kernel-based reinforcement learning approximation of the soft maximum value function to deal with the high-dimensional nature of human motion and applies a mean-shift procedure over a continuous cost function to infer a smooth reaction sequence. Experimental results show that our proposed method is able to properly model human interactions in a high dimensional space of human poses. When compared to several baseline models, results show that our method is able to generate highly plausible simulations of human interaction.

Training recurrent neural networks is hard. Recently it has however been discovered that it is possible to just construct a random recurrent topology, and only train a single linear readout layer. State-ofthe-art performance can easily be achieved with this setup, called Reservoir Computing. The idea can even be broadened by stating that any high dimensional, driven dynamic system, operated in the correct dynamic regime can be used as a temporal ‘kernel’ which makes it possible to solve complex tasks using just linear post-processing techniques. This tutorial will give an overview of current research on theory, application and implementations of Reservoir Computing.

One of the long standing goals of Artificial Intelligence (AI) is to build cognitive agents which can perform complex tasks from raw sensory inputs without explicit supervision (Lake et al. 2016). Recent progress in combining Reinforcement Learning objective functions and Deep Learning architectures has achieved promising results for such tasks. An important aspect of such sequential decision making problems, which has largely been neglected, is for the agent to decide on the duration of time for which to commit to actions. Such action repetition is important for computational efficiency, which is necessary for the agent to respond in real-time to events (in applications such as self-driving cars). Action Repetition arises naturally in real life as well as simulated environments. The time scale of executing an action enables an agent (both humans and AI) to decide the granularity of control during task execution. Current state of the art Deep Reinforcement Learning models, whether they are off-policy (Mnih et al. 2015; Wang et al. 2015) or on-policy (Mnih et al. 2016), consist of a framework with a static action repetition paradigm, wherein the action decided by the agent is repeated for a fixed number of time steps regardless of the contextual state while executing the task. In this paper, we propose a new framework Dynamic Action Repetition which changes Action Repetition Rate (the time scale of repeating an action) from a hyper-parameter of an algorithm to a dynamically learnable quantity. At every decision-making step, our models allow the agent to commit to an action and the time scale of executing the action. We show empirically that such a dynamic time scale mechanism improves the performance on relatively harder games in the Atari 2600 domain, independent of the underlying Deep Reinforcement Learning algorithm used.