Question: Insights Gained From Past SSL/TLS Assaults - A Concise Timeline of Vulnerabilities and Intrusions

This paper introduces a series of innovative hash function classes with advantageous properties and two new applications that leverage these functions for authentication and set equality testing. One of the classes features a compact set of functions that are nearly universal, significantly reducing the space needed for specifying a function within the class. For hashing n-bit names into m-bit indices, the specification of a function from this class requires only O((m + log log n) log n) bits, markedly less than the O(n) bits required by previous methods. This is particularly efficient for long names, with the space requirement being approximately m times greater than the theoretical minimum of m + log n - log m bits. An application of this class is a robust authentication mechanism for secure message transmission over untrusted networks. A second class of functions offers even stronger security properties than those provided by universal hashing. This class is employed for testing the equality of two sets, enabling operations such as "add member to set," "remove member from set," and "verify set equality" to be executed in constant expected time with an error probability that can be controlled below a specified threshold. The authentication method ensures the integrity and authenticity of messages, as even an adversary with unlimited computational resources cannot tamper with or forge messages without being detected. The set equality technique optimizes the process of comparing sets, thereby enhancing the efficiency of operations involving set membership and equality checks.

Software based side-channel attacks allow an unprivileged spy process to extract secret information from a victim (cryptosystem) process by exploiting some indirect leakage of “side-channel” information. It has been realized that some components of modern computer microarchitectures leak certain side-channel information and can create unforeseen security risks. An example of such MicroArchitectural Side-Channel Analysis is the Cache Attack — a group of attacks that exploit information leaks from cache latencies [4, 7, 13, 15, 17]. Public awareness of Cache Attack vulnerabilities lead software writers of OpenSSL (version 0.9.8a and subsequent versions) to incorporate countermeasures for preventing these attacks. In this paper, we present a new and yet unforeseen side channel attack that is enabled by the recently published Simple Branch Prediction Analysis (SBPA) which is another type of MicroArchitectural Analysis, cf. [2, 3]. We show that modular inversion — a critical primitive in public key cryptography — is a natural target of SBPA attacks because it typically uses the Binary Extended Euclidean algorithm whose nature is an input-centric sequence of conditional branches. Our results show that SBPA can be used to extract secret parameters during the execution of the Binary Extended Euclidean algorithm. This poses a new potential risk to crypto-applications such as OpenSSL, which already employs Cache Attack countermeasures. Thus, it is necessary to develop new software mitigation techniques for BPA and incorporate them with cache analysis countermeasures in security applications. To mitigate this new risk in full generality, we apply a security-aware algorithm design methodology and propose some changes to the CRT-RSA algorithm flow. These changes either avoid some of the steps that require modular inversion, or remove the critical information leak from this procedure. In addition, we also show by example that, independently of the required changes in the algorithms, careful software analysis is also required in order to assure that the software implementation does not inadvertently introduce branches that may expose the application to SBPA attacks. These offer several simple ways for modifying OpenSSL in order to mitigate Branch Prediction Attacks.

Our growing reliance on online services accessible on the Internet demands highly-available systems that provide correct service without interruptions. Byzantine faults such as software bugs, operator mistakes, and malicious attacks are the major cause of service interruptions. This thesis describes a new replication algorithm, BFT, that can be used to build highly-available systems that tolerate Byzantine faults. It shows, for the first time, how to build Byzantine-fault-tolerant systems that can be used in practice to implement real services because they do not rely on unrealistic assumptions and they perform well. BFT works in asynchronous environments like the Internet, it incorporates mechanisms to defend against Byzantine-faulty clients, and it recovers replicas proactively. The recovery mechanism allows the algorithm to tolerate any number of faults over the lifetime of the system provided fewer than 1/3 of the replicas become faulty within a small window of vulnerability. The window may increase under a denial-of-service attack but the algorithm can detect and respond to such attacks and it can also detect when the state of a replica is corrupted by an attacker. BFT has been implemented as a generic program library with a simple interface. The BFT library provides a complete solution to the problem of building real services that tolerate Byzantine faults. We used the library to implement the first Byzantine-fault-tolerant NFS file system, BFS. The BFT library and BFS perform well because the library incorporates several important optimizations. The most important optimization is the use of symmetric cryptography to authenticate messages. Public-key cryptography, which was the major bottleneck in previous systems, is used only to exchange the symmetric keys. The performance results show that BFS performs 2% faster to 24% slower than production implementations of the NFS protocol that are not replicated. Therefore, we believe that the BFT library can be used to build practical systems that tolerate Byzantine faults. Thesis Supervisor: Barbara H. Liskov Title: Ford Professor of Engineering

This paper sheds light on crucial weaknesses in the design of hidden services that allow us to break the anonymity of hidden service clients and operators passively. In particular, we show that the circuits, paths established through the Tor network, used to communicate with hidden services exhibit a very different behavior compared to a general circuit. We propose two attacks, under two slightly different threat models, that could identify a hidden service client or operator using these weaknesses. We found that we can identify the users’ involvement with hidden services with more than 98% true positive rate and less than 0.1% false positive rate with the first attack, and 99% true positive rate and 0.07% false positive rate with the second. We then revisit the threat model of previous website fingerprinting attacks, and show that previous results are directly applicable, with greater efficiency, in the realm of hidden services. Indeed, we show that we can correctly determine which of the 50 monitored pages the client is visiting with 88% true positive rate and false positive rate as low as 2.9%, and correctly deanonymize 50 monitored hidden service servers with true positive rate of 88% and false positive rate of 7.8% in an open world setting.

Despite many research and development efforts in the area of data communications security, the importance of internal Ž . local area network LAN security is still underestimated. This paper discusses why many traditional approaches to network Ž . security e.g. firewalls, the modern IPSec or various application level protocols are not so effective in local networks and proposes a prospective solution for building of secure encrypted Ethernet LANs. The architecture presented allows for employment of the existing office network infrastructure, does not require changes to workstations’ software, and provides high level of protection. The core idea is to apply security measures in the network interface devices connecting individual Ž Ž .. computers to the network i.e. network interface cards NICs . This eliminates the physical possibility of sending unprotected information through the network. Implementation details are given for data and key exchange, network management, and interoperability issues. An in-depth security analysis of the proposed architecture is presented and some conclusions are drawn. q 2000 Elsevier Science B.V. All rights reserved.

SQL Injection vulnerabilities still exist even after almost two decades that it first appeared. In spite of numerous prevention methodologies being used today, web applications still tend to be vulnerable to SQL injection attacks. Technology has improved drastically over the past few years and computers have certainly brought a great impact on our lifestyle. The computer applications and their usage over the web are myriad. It is quite evident that the in the near future, the usage of computers would relatively be higher than what we are witnessing today. A wide variety of data such as credit information, military data, human communication data, and countless types of data is shared over the far-flung computer networks. As the usage and reliability on computers increase, the threat to sensitive data likewise increases. The challenges with the cyber security when dealing with sensitive information is now a nightmare. To help understand the threats and the severity of exploits deployed, the paper provides proof of concepts for exploits carried out to compromise web applications and how the databases are exploited using the SQL injection methodologies. The SQL injection vulnerabilities in the web applications are surprisingly very vast and this is definitely is a huge security threat to personal data of people that is stored on web. In this paper, the methods used in information gathering, how the security is breached, and how payloads are used to exploit web applications are explained using the Kali Linux. In addition, an analysis is carried out on how the websites are comprised. Advanced methods on how to defend SQL injections are briefly justified. For the readers to understand better, a real time scenario of a penetration tester and a database server is set up with a few suppositions, and the commands that dodge the security characteristics and manipulate the databases are explicated.

We often send our confidential information such as passport, credit card, social security numbers over email without concern about the security of email services. Existing network security mechanisms provide adequate security from external malicious adversaries and eavesdroppers, but they don't guarantee that the email service providers (ESPs) wouldn't or can't access our email data themselves, which in some cases could be highly confidential. One of the ways to protect email data from ESPs is to use Pretty Good Privacy (PGP) that has many limitations including key storage problem and dependability on third party services, making it cumbersome to use in practice. In this paper, we present S3Email method that provides email security against ESPs. The proposed method uses a cryptographic secret sharing technique in a novel way and encrypts the email metadata, body and attachments before the email is sent. In the proposed solution, the email sender and receiver must have at least two email accounts on the existing ESPs, which is not unusual today. Experiments and analysis show that the S3Email method provides information theoretic security with minimal computational overhead.

This report describes an analysis of the Fitbit Flex ecosystem. Our objectives are to describe (1) the data Fitbit collects from its users, (2) the data Fitbit provides to its users, and (3) methods of recovering data not made available to device owners. Our analysis covers four distinct attack vectors. First, we analyze the security and privacy properties of the Fitbit device itself. Next, we observe the Bluetooth traffic sent between the Fitbit device and a smartphone or personal computer during synchronization. Third, we analyze the security of the Fitbit Android app. Finally, we study the security properties of the network traffic between the Fitbit smartphone or computer application and the Fitbit web service. We provide evidence that Fitbit unnecessarily obtains information about nearby Flex devices under certain circumstances. We further show that Fitbit does not provide device owners with all of the data collected. In fact, we find evidence of per-minute activity data that is sent to the Fitbit web service but not provided to the owner. We also discovered that MAC addresses on Fitbit devices are never changed, enabling usercorrelation attacks. BTLE credentials are also exposed on the network during device pairing over TLS, which might be intercepted by MITM attacks. Finally, we demonstrate that actual user activity data is authenticated and not provided in plaintext on an end-to-end basis from the device to the Fitbit web service.

We present CacheAudit, a versatile framework for the automatic, static analysis of cache side channels. CacheAudit takes as input a program binary and a cache configuration and derives formal, quantitative security guarantees for a comprehensive set of side-channel adversaries, namely, those based on observing cache states, traces of hits and misses, and execution times. Our technical contributions include novel abstractions to efficiently compute precise overapproximations of the possible side-channel observations for each of these adversaries. These approximations then yield upper bounds on the amount of information that is revealed.
 In case studies, we apply CacheAudit to binary executables of algorithms for sorting and encryption, including the AES implementation from the PolarSSL library, and the reference implementations of the finalists of the eSTREAM stream cipher competition. The results we obtain exhibit the influence of cache size, line size, associativity, replacement policy, and coding style on the security of the executables and include the first formal proofs of security for implementations with countermeasures such as preloading and data-independent memory access patterns.

Website fingerprinting allows a local, passive observer monitoring a web-browsing client’s encrypted channel to determine her web activity. Previous attacks have shown that website fingerprinting could be a threat to anonymity networks such as Tor under laboratory conditions. However, there are significant differences between laboratory conditions and realistic conditions. First, the training data set is very similar to the testing data set under laboratory conditions, but the attacker may not be able to guarantee similarity realistically. Second, laboratory packet sequences correspond to a single page each, but for realistic packet sequences the split between pages is not obvious. Third, packet sequences may include noise, which may adversely affect website fingerprinting, but this effect has not been studied. In this paper, we tackle these three problems to bridge the gap between laboratory and realistic conditions for website fingerprinting. We show that we can maintain a fresh training set with minimal resources. We demonstrate several classification-based techniques that allow us to split full packet sequences effectively into sequences corresponding to a single page each. Although we were not able to remove noise effectively, we will show that it is difficult for users to generate sufficient background noise to disrupt website fingerprinting on Tor. With our techniques, we are able to build the first website fingerprinting system that can operate on packet sequences collected in the wild.

We initiate the study of multi-user (mu) security of authenticated encryption (AE) schemes as a way to rigorously formulate, and answer, questions about the “randomized nonce” mechanism proposed for the use of the AE scheme GCM in TLS 1.3. We (1) Give definitions of mu ind (indistinguishability) and mu kr (key recovery) security for AE (2) Characterize the intent of nonce randomization as being improved mu security as a defense against mass surveillance (3) Cast the method as a (new) AE scheme RGCM (4) Analyze and compare the mu security of both GCM and RGCM in the model where the underlying block cipher is ideal, showing that the mu security of the latter is indeed superior in many practical contexts to that of the former, and (5) Propose an alternative AE scheme XGCM having the same efficiency as RGCM but better mu security and a more simple and modular design.


Question: A method for achieving communication efficiency in distributed machine learning using a parameter server.

This paper introduces Shotgun, a novel parallel coordinate descent algorithm designed for the minimization of L1-regularized losses. Despite the seemingly sequential nature of coordinate descent, Shotgun is shown to achieve near-linear speedups, with bounds that are dependent on the specific problem at hand. We conduct an extensive empirical analysis of Shotgun for both Lasso and sparse logistic regression tasks. The theoretical predictions regarding the algorithm's parallelism potential are found to be highly accurate when applied to real-world datasets. Shotgun's performance surpasses that of existing solvers across a variety of large-scale problems, solidifying its position as one of the most scalable algorithms for L1-regularized loss minimization.

**Question**: A Direct Search Method to solve Economic Dispatch Problem with Valve-Point Effect

**Documents**: A hybrid of genetic algorithm and particle swarm optimization for recurrent network design

**Similar Documents**: This work presents a pioneering methodology, the Hybrid Genetic Algorithm-Particle Swarm Optimization (HGAPSO), aimed at automating the design of recurrent neural/fuzzy networks. HGAPSO innovates by merging genetic algorithm (GA) principles with particle swarm optimization (PSO) techniques, creating a novel evolutionary learning algorithm. Unlike conventional GA approaches, HGAPSO's new generations are not solely produced through crossover and mutation operations; they also benefit from PSO's influence. The elite strategy is a central concept in HGAPSO, where the highest-performing individuals in a population are identified as elites and undergo an enhancement process. These elites are treated as a swarm, with each individual corresponding to a particle. Through PSO, the elites are optimized, reflecting a maturation process observed in nature. In the subsequent generation, half of the population consists of these enhanced elites, while the other half is created by applying crossover and mutation to the refined elites. The efficacy of HGAPSO is demonstrated in recurrent neural/fuzzy network design. For recurrent neural networks, a fully connected design is implemented to address temporal sequence production tasks. In the context of recurrent fuzzy network design, a Takagi-Sugeno-Kang-type network is employed for dynamic plant control applications. HGAPSO's performance is compared against traditional GA and PSO methods, underscoring its superior capabilities in recurrent network design challenges.

Graph partition quality affects the overall performance of parallel graph computation systems. The quality of a graph partition is measured by the balance factor and edge cut ratio. A balanced graph partition with small edge cut ratio is generally preferred since it reduces the expensive network communication cost. However, according to an empirical study on Giraph, the performance over well partitioned graph might be even two times worse than simple random partitions. This is because these systems only optimize for the simple partition strategies and cannot efficiently handle the increasing workload of local message processing when a high quality graph partition is used. In this paper, we propose a novel partition aware graph computation engine named PAGE, which equips a new message processor and a dynamic concurrency control model. The new message processor concurrently processes local and remote messages in a unified way. The dynamic model adaptively adjusts the concurrency of the processor based on the online statistics. The experimental evaluation demonstrates the superiority of PAGE over the graph partitions with various qualities.

As applications such as Apple Siri, Google Now, Microsoft Cortana, and Amazon Echo continue to gain traction, web-service companies are adopting large deep neural networks (DNN) for machine learning challenges such as image processing, speech recognition, natural language processing, among others. A number of open questions arise as to the design of a server platform specialized for DNN and how modern warehouse scale computers (WSCs) should be outfitted to provide DNN as a service for these applications.
 In this paper, we present DjiNN, an open infrastructure for DNN as a service in WSCs, and Tonic Suite, a suite of 7 end-to-end applications that span image, speech, and language processing. We use DjiNN to design a high throughput DNN system based on massive GPU server designs and provide insights as to the varying characteristics across applications. After studying the throughput, bandwidth, and power properties of DjiNN and Tonic Suite, we investigate several design points for future WSC architectures. We investigate the total cost of ownership implications of having a WSC with a disaggregated GPU pool versus a WSC composed of homogeneous integrated GPU servers. We improve DNN throughput by over 120x for all but one application (40x for Facial Recognition) on an NVIDIA K40 GPU. On a GPU server composed of 8 NVIDIA K40s, we achieve near-linear scaling (around 1000x throughput improvement) for 3 of the 7 applications. Through our analysis, we also find that GPU-enabled WSCs improve total cost of ownership over CPU-only designs by 4-20x, depending on the composition of the workload

In recent years, Distributed Data Mining (DDM) has evolved in large space aiming at minimizing computation cost and memory overhead in processing huge geographically distributed data. There are two approaches of DDM -- homogeneous and heterogeneous classifier approach. This paper presents implementation of four homogeneous classifier techniques for DDM with different Electronic Health Records (EHRs) homogeneous datasets namely diabetes, hepatitis, hypothyroid and further analyzing results based on metric evaluation.

The next generation of AI applications will continuously interact with the environment and learn from these interactions. These applications impose new and demanding systems requirements, both in terms of performance and flexibility. In this paper, we consider these requirements and present Ray—a distributed system to address them. Ray implements a unified interface that can express both task-parallel and actor-based computations, supported by a single dynamic execution engine. To meet the performance requirements, Ray employs a distributed scheduler and a distributed and fault-tolerant store to manage the system’s control state. In our experiments, we demonstrate scaling beyond 1.8 million tasks per second and better performance than existing specialized systems for several challenging reinforcement learning applications.

Collaborative Filtering(CF) algorithms are widely used in a lot of recommender systems, however, the computational complexity of CF is high thus hinder their use in large scale systems. In this paper, we implement user-based CF algorithm on a cloud computing platform, namely Hadoop, to solve the scalability problem of CF. Experimental results show that a simple method that partition users into groups according to two basic principles, i.e., tidy arrangement of mapper number to overcome the initiation of mapper and partition task equally such that all processors finish task at the same time, can achieve linear speedup.

Implementing E cient, Portable Computations for Machine Learning

Large-scale datacenters (DCs) host tens of thousands of diverse applications each day. However, interference between colocated workloads and the difficulty to match applications to one of the many hardware platforms available can degrade performance, violating the quality of service (QoS) guarantees that many cloud workloads require. While previous work has identified the impact of heterogeneity and interference, existing solutions are computationally intensive, cannot be applied online and do not scale beyond few applications.
 We present Paragon, an online and scalable DC scheduler that is heterogeneity and interference-aware. Paragon is derived from robust analytical methods and instead of profiling each application in detail, it leverages information the system already has about applications it has previously seen. It uses collaborative filtering techniques to quickly and accurately classify an unknown, incoming workload with respect to heterogeneity and interference in multiple shared resources, by identifying similarities to previously scheduled applications. The classification allows Paragon to greedily schedule applications in a manner that minimizes interference and maximizes server utilization. Paragon scales to tens of thousands of servers with marginal scheduling overheads in terms of time or state.
 We evaluate Paragon with a wide range of workload scenarios, on both small and large-scale systems, including 1,000 servers on EC2. For a 2,500-workload scenario, Paragon enforces performance guarantees for 91% of applications, while significantly improving utilization. In comparison, heterogeneity-oblivious, interference-oblivious and least-loaded schedulers only provide similar guarantees for 14%, 11% and 3% of workloads. The differences are more striking in oversubscribed scenarios where resource efficiency is more critical.

In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We evaluate the effect of various RPU device features/non-idealities and system parameters on performance in order to derive the device and system level specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30, 000 × compared to state-of-the-art microprocessors while providing power efficiency of 84, 000 GigaOps∕s∕W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisting of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration, and analysis of multimodal sensory data flows from a massive number of IoT (Internet of Things) sensors.

Now a days people are enjoying the world of data because size and amount of the data has tremendously increased which acts like an invitation to Big data. But some of the classifier techniques like Support Vector Machine (SVM) is not able to handle the huge amount of data due to it's excessive memory requirement and unreasonable complexity in algorithm tough it is one of the most popularly used classifier in machine learning field. Hence a new technique comes into picture which performs parallel algorithm in a efficient way to work data having large scale called as PSVM. In this paper we are going to discuss a PSVM model for risk analysis which is based on map-reduce, and can easily handle a huge amount of data in a distributed manner.

We study distributed graph algorithms that adopt an iterati ve vertexcentric framework for graph processing, popularized by Goo gle’s Pregel system. Since then, there are several attempts to imp lement many graph algorithms in a vertex-centric framework, a s well as efforts to design optimization techniques for improving the efficiency. However, to the best of our knowledge, there has not been any systematic study to compare these vertex-centric i mplementations with their sequential counterparts. Our paper a dd esses this gap in two ways. (1) We analyze the computational complexity of such implementations with the notion of time-pro cessor product, and benchmark several vertex-centric graph algor ithms whether they perform more work with respect to their best-kn ow sequential solutions. (2) Employing the concept of balance d practical Pregel algorithms, we study if these implementations suffer from imbalanced workload and large number of iterations. Ou r findings illustrate that with the exception of Euler tour tre e algorithm, all other algorithms either perform asymptotically more work than their best-known sequential approach, or suffer from i mbalanced workload/ large number of iterations, or even both. We also emphasize on graph algorithms that are fundamentally diffic ult to be expressed in vertex-centric frameworks, and conclude by discussing the road ahead for distributed graph processing.


Question: A method for enhancing communication efficiency in distributed machine learning using a parameter server.

This paper introduces a distributed large-scale graph factorization framework specifically tailored for natural graphs, such as social networks and instant messaging networks, which are prevalent due to the internet's global reach. These graphs are colossal, often encompassing hundreds of millions of nodes and billions of edges, making their analysis challenging due to their sheer size and complex data characteristics.
To tackle the scale of these graphs, the proposed framework is designed to be distributed, enabling the data to be divided across a cluster of shared-nothing machines. A unique factorization technique is presented, which focuses on minimizing the number of neighboring vertices, rather than edges, that are split across different partitions. This novel approach is based on a streaming algorithm that is sensitive to the network topology of the underlying hardware, optimizing the decomposition for the specific computational resources available.
The framework utilizes local copies of variables and an efficient asynchronous communication protocol to synchronize the replicated values, allowing for the majority of the computation to be performed without the overhead of network communication. This results in significant computational gains, as demonstrated on a massive graph derived from an email communication network, comprising 200 million vertices and 10 billion edges. The algorithm exhibits strong convergence properties while achieving near-linear scalability in the number of computers used, showcasing its effectiveness in handling large-scale natural graph factorization problems.

Recently, there have been increasing demands to construct compact deep architectures to remove unnecessary redundancy and to improve the inference speed. While many recent works focus on reducing the redundancy by eliminating unneeded weight parameters, it is not possible to apply a single deep network for multiple devices with different resources. When a new device or circumstantial condition requires a new deep architecture, it is necessary to construct and train a new network from scratch. In this work, we propose a novel deep learning framework, called a nested sparse network, which exploits an n-in-1-type nested structure in a neural network. A nested sparse network consists of multiple levels of networks with a different sparsity ratio associated with each level, and higher level networks share parameters with lower level networks to enable stable nested learning. The proposed framework realizes a resource-aware versatile architecture as the same network can meet diverse resource requirements, i.e., anytime property. Moreover, the proposed nested network can learn different forms of knowledge in its internal networks at different levels, enabling multiple tasks using a single network, such as coarse-to-fine hierarchical classification. In order to train the proposed nested network, we propose efficient weight connection learning and channel and layer scheduling strategies. We evaluate our network in multiple tasks, including adaptive deep compression, knowledge distillation, and learning class hierarchy, and demonstrate that nested sparse networks perform competitively, but more efficiently, compared to existing methods.

A growing number of commercial and enterprise systems increasingly rely on compute-intensive Machine Learning (ML) algorithms. While the demand for these compute-intensive applications is growing, the performance benefits from general-purpose platforms are diminishing. Field Programmable Gate Arrays (FPGAs) provide a promising path forward to accommodate the needs of machine learning algorithms and represent an intermediate point between the efficiency of ASICs and the programmability of general-purpose processors. However, acceleration with FPGAs still requires long development cycles and extensive expertise in hardware design. To tackle this challenge, instead of designing an accelerator for a machine learning algorithm, we present TABLA, a framework that generates accelerators for a class of machine learning algorithms. The key is to identify the commonalities across a wide range of machine learning algorithms and utilize this commonality to provide a high-level abstraction for programmers. TABLA leverages the insight that many learning algorithms can be expressed as a stochastic optimization problem. Therefore, learning becomes solving an optimization problem using stochastic gradient descent that minimizes an objective function over the training data. The gradient descent solver is fixed while the objective function changes for different learning algorithms. TABLA provides a template-based framework to accelerate this class of learning algorithms. Therefore, a developer can specify the learning task by only expressing the gradient of the objective function using our high-level language. Tabla then automatically generates the synthesizable implementation of the accelerator for FPGA realization using a set of hand-optimized templates. We use Tabla to generate accelerators for ten different learning tasks targeted at a Xilinx Zynq FPGA platform. We rigorously compare the benefits of FPGA acceleration to multi-core CPUs (ARM Cortex A15 and Xeon E3) and many-core GPUs (Tegra K1, GTX 650 Ti, and Tesla K40) using real hardware measurements. TABLA-generated accelerators provide 19.4x and 2.9x average speedup over the ARM and Xeon processors, respectively. These accelerators provide 17.57x, 20.2x, and 33.4x higher Performance-per-Watt in comparison to Tegra, GTX 650 Ti and Tesla, respectively. These benefits are achieved while the programmers write less than 50 lines of code.

The Distributed Rate Limiting (DRL) paradigm is a recently proposed mechanism for decentralized control of cloud-based services. DRL is a simple and efficient approach to resolve the issues of pricing and resource control/engineering of cloud based services. The existing DRL schemes focus on very specific performance metrics (such as loss rate and fair-share) and their design heavily depends on the assumption that the traffic is generated by elastic TCP sources. In this paper we tackle the DRL problem for general workloads and performance metrics and propose an analytic framework for the design of stable DRL algorithms. The closed-form nature of our results allows simple design rules which, together with extremely low communication overhead, makes the presented algorithms practical and easy to deploy with guaranteed convergence properties under a wide range of possible scenarios.

Datacenter workloads demand high computational capabilities, flexibility, power efficiency, and low cost. It is challenging to improve all of these factors simultaneously. To advance datacenter capabilities beyond what commodity server designs can provide, we designed and built a composable, reconfigurable hardware fabric based on field programmable gate arrays (FPGA). Each server in the fabric contains one FPGA, and all FPGAs within a 48-server rack are interconnected over a low-latency, high-bandwidth network.
 We describe a medium-scale deployment of this fabric on a bed of 1632 servers, and measure its effectiveness in accelerating the ranking component of the Bing web search engine. We describe the requirements and architecture of the system, detail the critical engineering challenges and solutions needed to make the system robust in the presence of failures, and measure the performance, power, and resilience of the system. Under high load, the large-scale reconfigurable fabric improves the ranking throughput of each server by 95% at a desirable latency distribution or reduces tail latency by 29% at a fixed throughput. In other words, the reconfigurable fabric enables the same throughput using only half the number of servers.

Natural graphs, such as social networks, email graphs, or instant messaging patterns, have become pervasive through the internet. These graphs are massive, often containing hundreds of millions of nodes and billions of edges. While some theoretical models have been proposed to study such graphs, their analysis is still difficult due to the scale and nature of the data.
 We propose a framework for large-scale graph decomposition and inference. To resolve the scale, our framework is distributed so that the data are partitioned over a shared-nothing set of machines. We propose a novel factorization technique that relies on partitioning a graph so as to minimize the number of neighboring vertices rather than edges across partitions. Our decomposition is based on a streaming algorithm. It is network-aware as it adapts to the network topology of the underlying computational hardware. We use local copies of the variables and an efficient asynchronous communication protocol to synchronize the replicated values in order to perform most of the computation without having to incur the cost of network communication. On a graph of 200 million vertices and 10 billion edges, derived from an email communication network, our algorithm retains convergence properties while allowing for almost linear scalability in the number of computers.

Deep learning has become a promising approach for automated medical diagnoses. When medical data samples are limited, collaboration among multiple institutions is necessary to achieve high algorithm performance. However, sharing patient data often has limitations due to technical, legal, or ethical concerns. In such cases, sharing a deep learning model is a more attractive alternative. The best method of performing such a task is unclear, however. In this study, we simulate the dissemination of learning deep learning network models across four institutions using various heuristics and compare the results with a deep learning model trained on centrally hosted patient data. The heuristics investigated include ensembling single institution models, single weight transfer, and cyclical weight transfer. We evaluated these approaches for image classification in three independent image collections (retinal fundus photos, mammography, and ImageNet). We find that cyclical weight transfer resulted in a performance (testing accuracy = 77.3%) that was closest to that of centrally hosted patient data (testing accuracy = 78.7%). We also found that there is an improvement in the performance of cyclical weight transfer heuristic with high frequency of weight transfer.

We study a distributed computation model for optimizing a sum of convex objective functions corresponding to multiple agents. For solving this (not necessarily smooth) optimization problem, we consider a subgradient method that is distributed among the agents. The method involves every agent minimizing his/her own objective function while exchanging information locally with other agents in the network over a time-varying topology. We provide convergence results and convergence rate estimates for the subgradient method. Our convergence rate results explicitly characterize the tradeoff between a desired accuracy of the generated approximate optimal solutions and the number of iterations needed to achieve the accuracy.

In this paper, we consider the design of a system in which Internet-connected mobile users contribute sensor data as training samples, and collaborate on building a model for classification tasks such as activity or context recognition. Constructing the model can naturally be performed by a service running in the cloud, but users may be more inclined to contribute training samples if the privacy of these data could be ensured. Thus, in this paper, we focus on privacy-preserving collaborative learning for the mobile setting, which addresses several competing challenges not previously considered in the literature: supporting complex classification methods like support vector machines, respecting mobile computing and communication constraints, and enabling user-determined privacy levels. Our approach, Pickle, ensures classification accuracy even in the presence of significantly perturbed training samples, is robust to methods that attempt to infer the original data or poison the model, and imposes minimal costs. We validate these claims using a user study, many real-world datasets and two different implementations of Pickle.

Management and analysis of big data are systematically associated with a data distributed architecture in the Hadoop and now Spark frameworks. This article offers an introduction for statisticians to these technologies by comparing the performance obtained by the direct use of three reference environments : R, Python Scikit-learn, ∗Université de Toulouse – INSA, Institut de Mathématiques, UMR CNRS 5219 †Institut de Mathématiques, UMR CNRS 5219 ‡Université de Toulouse – UT3, Institut de Mathématiques, UMR CNRS 5219 1 ar X iv :1 60 9. 09 61 9v 1 [ st at .A P] 3 0 Se p 20 16 Spark MLlib on three public use cases : character recognition, recommending films, categorizing products. As main result, it appears that, if Spark is very efficient for data munging and recommendation by collaborative filtering (non-negative factorization), current implementations of conventional learning methods (logistic regression, random forests) in MLlib or SparkML do not ou poorly compete habitual use of these methods (R, Python Scikit-learn) in an integrated or undistributed architecture.

The alternating direction method of multipliers (ADMM) is a powerful optimization solver in machine learning. Recently, stochastic ADMM has been integrated with variance reduction methods for stochastic gradient, leading to SAG-ADMM and SDCA-ADMM that have fast convergence rates and low iteration complexities. However, their space requirements can still be high. In this paper, we propose an integration of ADMM with the method of stochastic variance reduced gradient (SVRG). Unlike another recent integration attempt called SCAS-ADMM, the proposed algorithm retains the fast convergence benefits of SAG-ADMM and SDCA-ADMM, but is more advantageous in that its storage requirement is very low, even independent of the sample size n. Experimental results demonstrate that it is as fast as SAG-ADMM and SDCA-ADMM, much faster than SCAS-ADMM, and can be used on much bigger data sets.


Question: A method for achieving communication-efficient distributed machine learning using a parameter server.

In this acknowledgment section, the authors of the "The Matrix Cookbook" express their gratitude to a multitude of individuals for their contributions and suggestions. Among those recognized are Bill Baxter, Brian Templeton, Christian Rishøj, Christian Schröppel, Douglas L. Theobald, Esben Hoegh-Rasmussen, Glynne Casteel, Jan Larsen, Jun Bin Gao, Jürgen Struckmeier, Kamil Dedecius, Korbinian Strimmer, Lars Christiansen, Lars Kai Hansen, Leland Wilkinson, Liguo He, Loic Thibaut, Miguel Barão, Ole Winther, Pavel Sakov, Stephan Hattinger, Vasile Sima, Vincent Rabaud, and Zhaoshui He. The authors also extend their appreciation to The Oticon Foundation for financing their doctoral studies. This section highlights the collaborative effort and support that went into the creation of the cookbook, acknowledging the diverse expertise and financial backing that contributed to its development. However, it does not directly pertain to the topic of communication-efficient distributed machine learning with the parameter server, which focuses on the optimization and coordination of learning algorithms in a distributed environment. The acknowledgement section instead serves as a testament to the collaborative nature of scientific and academic endeavors.

When Google first introduced the Map/Reduce paradigm in 2004, no comparable system had been available to the general public. The situation has changed since then. The Map/Reduce paradigm has become increasingly popular and there is no shortage of Map/Reduce implementations in today's computing world. The predominant solution is currently Apache Hadoop, started by Yahoo. Besides employing custom Map/Reduce installations, customers of cloud services can now exploit ready-made made installations (e.g. the Elastic Map/Reduce System). In the mean time, other, second generation frameworks have started to appear. They either fine tune the Map/Reduce model for specific scenarios, or change the paradigm altogether, such as Google's Pregel. In this paper, we present a comparison between these second generation frameworks and the current de-facto standard Hadoop, by focusing on a specific scenario: large-scale graph analysis. We analyze the different means of fine-tuning those systems by exploiting their unique features. We base our analysis on the k-core decomposition problem, whose goal is to compute the centrality of each node in a given graph; we tested our implementation in a cluster of Amazon EC2 nodes with realistic datasets made publicly available by the SNAP project.

One measure of usefulness of a general-purpose distributed computing system is the system’s ability to provide a level of performance commensurate to the degree of multiplicity of resources present in the system. Many different approaches and metrics of performance have been proposed in an attempt to achieve this goal in existing systems. In addition, analogous problem formulations exist in other fields such as control theory, operations research, and production management. However, due to the wide variety of approaches to this problem, it is difficult to meaningfully compare different systems since there is no uniform means for qualitatively or quantitatively evaluating them. It is difficult to successfully build upon existing work or identify areas worthy of additional effort without some understanding of the relationships between past efforts. In this paper, a taxonomy of approaches to the resource management problem is presented in an attempt to provide a common terminology and classification mechanism necessary in addressing this problem. The taxonomy, while presented and discussed in terms of distributed scheduling, is also applicable to most types of resource management. As an illustration of the usefulness of the taxonomy an annotated bibliography is given which classifies a large number of distributed scheduling approaches according to the taxonomy.

Data warehouse systems, like Apache Hive, have been widely used in the distributed computing field. However, current generation data warehouse systems have not fully embraced High Performance Computing (HPC) technologies even though the trend of converging Big Data and HPC is emerging. For example, in traditional HPC field, Message Passing Interface (MPI) libraries have been optimized for HPC applications during last decades to deliver ultra-high data movement performance. Recent studies, like DataMPI, are extending MPI for Big Data applications to bridge these two fields. This trend motivates us to explore whether MPI can benefit data warehouse systems, such as Apache Hive. In this paper, we propose a novel design to accelerate Apache Hive by utilizing DataMPI. We further optimize the DataMPI engine by introducing enhanced non-blocking communication and parallelism mechanisms for typical Hive workloads based on their communication characteristics. Our design can fully and transparently support Hive workloads like Intel HiBench and TPC-H with high productivity. Performance evaluation with Intel HiBench shows that with the help of light-weight DataMPI library design, efficient job start up and data movement mechanisms, Hive on DataMPI performs 30% faster than Hive on Hadoop averagely. And the experiments on TPC-H with ORCFile show that the performance of Hive on DataMPI can improve 32% averagely and 53% at most more than that of Hive on Hadoop. To the best of our knowledge, Hive on DataMPI is the first attempt to propose a general design for fully supporting and accelerating data warehouse systems with MPI.

While Deep Neural Networks (DNNs) have achieved tremendous success for large vocabulary continuous speech recognition (LVCSR) tasks, training of these networks is slow. One reason is that DNNs are trained with a large number of training parameters (i.e., 10-50 million). Because networks are trained with a large number of output targets to achieve good performance, the majority of these parameters are in the final weight layer. In this paper, we propose a low-rank matrix factorization of the final weight layer. We apply this low-rank technique to DNNs for both acoustic modeling and language modeling. We show on three different LVCSR tasks ranging between 50-400 hrs, that a low-rank factorization reduces the number of parameters of the network by 30-50%. This results in roughly an equivalent reduction in training time, without a significant loss in final recognition accuracy, compared to a full-rank representation.

Large-scale learning problems require a plethora of labels that can be efficiently collected from crowdsourcing services at low cost. However, labels annotated by crowdsourced workers are often noisy, which inevitably degrades the performance of large-scale optimizations including the prevalent stochastic gradient descent (SGD). Specifically, these noisy labels adversely affect updates of the primal variable in conventional SGD. To solve this challenge, we propose a robust SGD mechanism called progressive stochastic learning (POSTAL), which naturally integrates the learning regime of curriculum learning (CL) with the update process of vanilla SGD. Our inspiration comes from the progressive learning process of CL, namely learning from “easy” tasks to “complex” tasks. Through the robust learning process of CL, POSTAL aims to yield robust updates of the primal variable on an ordered label sequence, namely, from “reliable” labels to “noisy” labels. To realize POSTAL mechanism, we design a cluster of “screening losses,” which sorts all labels from the reliable region to the noisy region. To sum up, POSTAL using screening losses ensures robust updates of the primal variable on reliable labels first, then on noisy labels incrementally until convergence. In theory, we derive the convergence rate of POSTAL realized by screening losses. Meanwhile, we provide the robustness analysis of representative screening losses. Experimental results on UCI1 simulated and Amazon Mechanical Turk crowdsourcing data sets show that the POSTAL using screening losses is more effective and robust than several existing baselines.1UCI is the abbreviation of University of California Irvine.

Parameter-specific adaptive learning rate methods are computationally efficient ways to reduce the ill-conditioning problems encountered when training large deep networks. Following recent work that strongly suggests that most of the critical points encountered when training such networks are saddle points, we find how considering the presence of negative eigenvalues of the Hessian could help us design better suited adaptive learning rate schemes. We show that the popular Jacobi preconditioner has undesirable behavior in the presence of both positive and negative curvature, and present theoretical and empirical evidence that the socalled equilibration preconditioner is comparatively better suited to non-convex problems. We introduce a novel adaptive learning rate scheme, called ESGD, based on the equilibration preconditioner. Our experiments show that ESGD performs as well or better than RMSProp in terms of convergence speed, always clearly improving over plain stochastic gradient descent.

The predominant methodology in training deep learning advocates the use of stochastic gradient descent methods (SGDs). Despite its ease of implementation, SGDs are difficult to tune and parallelize. These problems make it challenging to develop, debug and scale up deep learning algorithms with SGDs. In this paper, we show that more sophisticated off-the-shelf optimization methods such as Limited memory BFGS (L-BFGS) and Conjugate gradient (CG) with line search can significantly simplify and speed up the process of pretraining deep algorithms. In our experiments, the difference between LBFGS/CG and SGDs are more pronounced if we consider algorithmic extensions (e.g., sparsity regularization) and hardware extensions (e.g., GPUs or computer clusters). Our experiments with distributed optimization support the use of L-BFGS with locally connected networks and convolutional neural networks. Using L-BFGS, our convolutional network model achieves 0.69% on the standard MNIST dataset. This is a state-of-theart result on MNIST among algorithms that do not use distortions or pretraining.

To reduce information exchange requirements in smart grids, an event-triggered communication-based distributed optimization is proposed for economic dispatch. In this work, the θ-logarithmic barrier-based method is employed to reformulate the economic dispatch problem, and the consensus-based approach is considered for developing fully distributed technology-enabled algorithms. Specifically, a novel distributed algorithm utilizes the minimum connected dominating set (CDS), which efficiently allocates the task of balancing supply and demand for the entire power network at the beginning of economic dispatch. Further, an event-triggered communication-based method for the incremental cost of each generator is able to reach a consensus, coinciding with the global optimality of the objective function. In addition, a fast gradient-based distributed optimization method is also designed to accelerate the convergence rate of the event-triggered distributed optimization. Simulations based on the IEEE 57-bus test system demonstrate the effectiveness and good performance of proposed algorithms.

This survey shows that heterogeneous server clusters can be made more efficient by conserving power and energy while exploiting information from the service level, such as request priorities established by service-level agreements.

Natural graphs, such as social networks, email graphs, or instant messaging patterns, have become pervasive through the internet. These graphs are massive, often containing hundreds of millions of nodes and billions of edges. While some theoretical models have been proposed to study such graphs, their analysis is still difficult due to the scale and nature of the data.
 We propose a framework for large-scale graph decomposition and inference. To resolve the scale, our framework is distributed so that the data are partitioned over a shared-nothing set of machines. We propose a novel factorization technique that relies on partitioning a graph so as to minimize the number of neighboring vertices rather than edges across partitions. Our decomposition is based on a streaming algorithm. It is network-aware as it adapts to the network topology of the underlying computational hardware. We use local copies of the variables and an efficient asynchronous communication protocol to synchronize the replicated values in order to perform most of the computation without having to incur the cost of network communication. On a graph of 200 million vertices and 10 billion edges, derived from an email communication network, our algorithm retains convergence properties while allowing for almost linear scalability in the number of computers.


Question: A methodology for achieving communication efficiency in distributed machine learning using a parameter server.

In the realm of Independent Component Analysis (ICA) for feature learning, an approach that integrates a reconstruction cost has been proposed to enhance efficiency in overcomplete feature sets. This method, which does not require data whitening, essentially transforms the reconstruction cost into a weighted linear regression problem. This transformation occurs in a space that has been rotated by the eigenvectors and scaled by the eigenvalues of the data's covariance matrix, Cx. The result is a cost function heavily weighted towards principal component directions, with lesser emphasis on other directions. Notably, the matrix D^(-1/2)E, where D is a diagonal matrix of eigenvalues and E is the matrix of eigenvectors of Cx, represents the PCA whitening matrix. The optimization process inherently incorporates an inverse whitening matrix, compelling the weight matrix W to learn this whitening matrix as part of its adjustments, thereby achieving a whitening effect as a byproduct of the optimization process.

**Question**: A Hybrid Method for Power System State Estimation with Uncertain Data

**Documents**: Optimal Control of a Power System Considering Uncertainty in Wind Power and Load
In this paper, we present a new optimal control strategy for a power system that includes wind power and load uncertainty. The control strategy is based on a stochastic model predictive control (MPC) approach, which is formulated as a mixed-integer linear programming (MILP) problem. The MILP problem is solved using a branch-and-cut algorithm, which allows us to handle the discrete decisions in the control strategy. The proposed control strategy is applied to a test system consisting of a thermal power plant, a wind farm, and a load. The results show that the proposed control strategy can effectively handle the uncertainty in wind power and load, and can achieve a lower cost than a deterministic control strategy.

**Similar Documents**: This work introduces an innovative optimal control methodology for power systems incorporating wind power and load uncertainty. The proposed strategy is grounded in stochastic model predictive control (MPC), formulated as a mixed-integer linear programming (MILP) problem. To tackle the discrete decisions inherent in this control framework, a branch-and-cut algorithm is employed. The effectiveness of the proposed strategy is demonstrated through its application to a test scenario featuring a thermal power plant, a wind farm, and a load. The outcomes reveal that this control strategy adeptly manages the uncertainties associated with wind power and load, achieving superior cost efficiency compared to deterministic control strategies.

Management and analysis of big data are systematically associated with a data distributed architecture in the Hadoop and now Spark frameworks. This article offers an introduction for statisticians to these technologies by comparing the performance obtained by the direct use of three reference environments : R, Python Scikit-learn, ∗Université de Toulouse – INSA, Institut de Mathématiques, UMR CNRS 5219 †Institut de Mathématiques, UMR CNRS 5219 ‡Université de Toulouse – UT3, Institut de Mathématiques, UMR CNRS 5219 1 ar X iv :1 60 9. 09 61 9v 1 [ st at .A P] 3 0 Se p 20 16 Spark MLlib on three public use cases : character recognition, recommending films, categorizing products. As main result, it appears that, if Spark is very efficient for data munging and recommendation by collaborative filtering (non-negative factorization), current implementations of conventional learning methods (logistic regression, random forests) in MLlib or SparkML do not ou poorly compete habitual use of these methods (R, Python Scikit-learn) in an integrated or undistributed architecture.

We present Resilient Distributed Datasets (RDDs), a distributed memory abstraction that allows programmers to perform in-memory computations on large clusters while retaining the fault tolerance of data flow models like MapReduce. RDDs are motivated by two types of applications that current data flow systems handle inefficiently: iterative algorithms, which are common in graph applications and machine learning, and interactive data mining tools. In both cases, keeping data in memory can improve performance by an order of magnitude. To achieve fault tolerance efficiently, RDDs provide a highly restricted form of shared memory: they are read-only datasets that can only be constructed through bulk operations on other RDDs. However, we show that RDDs are expressive enough to capture a wide class of computations, including MapReduce and specialized programming models for iterative jobs such as Pregel. Our implementation of RDDs can outperform Hadoop by 20× for iterative jobs and can be used interactively to search a 1 TB dataset with latencies of 5–7 seconds.

In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We evaluate the effect of various RPU device features/non-idealities and system parameters on performance in order to derive the device and system level specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30, 000 × compared to state-of-the-art microprocessors while providing power efficiency of 84, 000 GigaOps∕s∕W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisting of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration, and analysis of multimodal sensory data flows from a massive number of IoT (Internet of Things) sensors.

Stream-processing workloads and modern shared cluster environments exhibit high variability and unpredictability. Combined with the large parameter space and the diverse set of user SLOs, this makes modern streaming systems very challenging to statically configure and tune. To address these issues, in this paper we investigate a novel control-plane design, Chi, which supports continuous monitoring and feedback, and enables dynamic re-configuration. Chi leverages the key insight of embedding control-plane messages in the data-plane channels to achieve a low-latency and flexible control plane for stream-processing systems. Chi introduces a new reactive programming model and design mechanisms to asynchronously execute control policies, thus avoiding global synchronization. We show how this allows us to easily implement a wide spectrum of control policies targeting different use cases observed in production. Large-scale experiments using production workloads from a popular cloud provider demonstrate the flexibility and efficiency of our approach. PVLDB Reference Format: Luo Mai, Kai Zeng, Rahul Potharaju, Le Xu, Steve Suh, Shivaram Venkataraman, Paolo Costa, Terry Kim, Saravanan Muthukrishnan, Vamsi Kuppa, Sudheer Dhulipalla, Sriram Rao. Chi: A Scalable and Programmable Control Plane for Distributed Stream Processing Systems. PVLDB, 11 (10): xxxx-yyyy, 2018. DOI: https://doi.org/10.14778/3231751.3231765

Many recent deep learning platforms rely on thirdparty libraries (such as cuBLAS) to utilize the computing power of modern hardware accelerators (such as GPUs). However, we observe that they may achieve suboptimal performance because the library functions are not used appropriately. In this paper, we target at optimizing the operations of multiplying a matrix with the transpose of another matrix (referred to as NT operation hereafter), which contribute half of the training time of fully connected deep neural networks. Rather than directly calling the library function, we propose a supervised learning based algorithm selection approach named MTNN, which uses a gradient boosted decision tree to select one from two alternative NT implementations intelligently: (1) calling the cuBLAS library function; (2) calling our proposed algorithm TNN that uses an efficient out-of-place matrix transpose. We evaluate the performance of MTNN on two modern GPUs: NVIDIA GTX 1080 and NVIDIA Titan X Pascal. MTNN can achieve 96% of prediction accuracy with very low computational overhead, which results in an average of 54% performance improvement on a range of NT operations. To further evaluate the impact of MTNN on the training process of deep neural networks, we have integrated MTNN into a popular deep learning platform Caffe. Our experimental results show that the revised Caffe can outperform the original one by an average of 28%. Both MTNN and the revised Caffe are open-source.

|As both the number of processors and the complexity of problems to be solved increase, programming multiprocessing systems becomes more di cult and errorprone. This paper discusses programming assistance and automation concepts and their application to a program development tool for message-passing systems called Hypertool. It performs scheduling and handles the communication primitive insertion automatically. Two algorithms, based on the critical-path method, are presented for scheduling processes statically. Hypertool also generates the performance estimates and other program quality measures to help programmers in improving their algorithms and programs.

We present Cloud Haskell, a domain-specific language for developing programs for a distributed computing environment. Implemented as a shallow embedding in Haskell, it provides a message-passing communication model, inspired by Erlang, without introducing incompatibility with Haskell's established shared-memory concurrency. A key contribution is a method for serializing function closures for transmission across the network. Cloud Haskell has been implemented; we present example code and some preliminary performance measurements.

Matrix operations are a fundamental building block of many computational tasks in fields as diverse as scientific computing, machine learning, and data mining. Matrix inversion is an important matrix operation, but it is difficult to implement in today's popular parallel dataflow programming systems, such as MapReduce. The reason is that each element in the inverse of a matrix depends on multiple elements in the input matrix, so the computation is not easily partitionable. In this paper, we present a scalable and efficient technique for matrix inversion in MapReduce. Our technique relies on computing the LU decomposition of the input matrix and using that decomposition to compute the required matrix inverse. We present a technique for computing the LU decomposition and the matrix inverse using a pipeline of MapReduce jobs. We also present optimizations of this technique in the context of Hadoop. To the best of our knowledge, our technique is the first matrix inversion technique using MapReduce. We show experimentally that our technique has good scalability, enabling us to invert a 10^5 x 10^5 matrix in 5 hours on Amazon EC2. We also show that our technique outperforms ScaLAPACK, a state-of-the-art linear algebra package that uses MPI.

Implementing E cient, Portable Computations for Machine Learning

As association rules widely used, it needs to study many problems, one of which is the generally larger and multi-dimensional datasets, and the rapid growth of the mount of data. Single-processor's memory and CPU resources are very limited, which makes the algorithm performance inefficient. Recently the development of network and distributed technology makes cloud computing a reality in the implementation of association rules algorithm. In this paper we describe the improved Apriori algorithm based on MapReduce mode, which can handle massive datasets with a large number of nodes on Hadoop platform.


Question: A method for achieving communication-efficient distributed machine learning using a parameter server.

This paper presents a novel technique called Chain Replication, designed to enhance the coordination and performance of clusters of fail-stop storage servers. Aimed at facilitating large-scale storage services, this method prioritizes achieving high throughput and availability while maintaining robust consistency guarantees. The paper not only details the foundational protocols of chain replication but also delves into a series of simulation experiments that scrutinize the performance aspects of a prototype implementation. Key performance indicators such as throughput, availability, and various object-placement strategies are thoroughly analyzed, including those based on distributed hash table routing. This comprehensive approach to evaluating chain replication underscores its potential to significantly impact the efficiency and reliability of storage systems.

**Question**: A Direct Search Method to solve Economic Dispatch Problem with Valve-Point Effect

**Documents**: An Enhanced Harmony Search Algorithm for Power Economic Dispatch Problem
An enhanced harmony search algorithm (EHS) is proposed to solve the economic dispatch (ED) problem with valve-point effect. The valve-point effect is the nonlinear cost function with multi-valleys. The proposed algorithm is an improved version of the original harmony search (HS) algorithm. It has been developed by implementing the concepts of dynamic harmony memory consideration rate, dynamic harmony memory bandwidth, and a new pitch adjustment operation. The effectiveness of the proposed algorithm is demonstrated on three test systems with different characteristics. The simulation results show that the proposed EHS algorithm has the ability to obtain high-quality solutions and is efficient for solving the ED problem.

**Similar Documents**: This work introduces an advanced version of the Harmony Search Algorithm, termed Enhanced Harmony Search (EHS), specifically tailored to address the economic dispatch (ED) problem in the presence of valve-point effects. Valve-point effects introduce complexities in the form of nonlinear cost functions with multiple valleys. The EHS algorithm, an evolutionary step from the original Harmony Search, integrates dynamic harmony memory consideration rates, dynamic harmony memory bandwidths, and an innovative pitch adjustment mechanism. Its efficacy is showcased through simulations on three distinct test systems, each presenting unique challenges. The outcomes highlight the EHS algorithm's proficiency in yielding high-caliber solutions, affirming its suitability and efficiency for tackling ED problems with valve-point effects.

Data-intensive computing has become one of the major workloads on traditional high-performance computing (HPC) clusters. Currently, deploying data-intensive computing software framework on HPC clusters still faces performance and scalability issues. In this paper, we develop a new two-level storage system by integrating Tachyon, an in-memory file system with OrangeFS, a parallel file system. We model the I/O throughputs of four storage structures: HDFS, OrangeFS, Tachyon and two-level storage. We conduct computational experiments to characterize I/O throughput behavior of two-level storage and compare its performance to that of HDFS and OrangeFS, using TeraSort benchmark. Theoretical models and experimental tests both show that the two-level storage system can increase the aggregate I/O throughputs. This work lays a solid foundation for future work in designing and building HPC systems that can provide a better support on I/O intensive workloads with preserving existing computing resources.

Training deep neural networks (DNNs) requires an increasing amount of computation resources, and it becomes typical to use a mixture of GPU and CPU devices. Due to the heterogeneity of these devices, a recent challenge is how each operation in a neural network can be optimally placed on these devices, so that the training process can take the shortest amount of time possible. The current state-of-the-art solution uses reinforcement learning based on the policy gradient method, and it suffers from suboptimal training times. In this paper, we propose Spotlight, a new reinforcement learning algorithm based on proximal policy optimization, designed specifically for finding an optimal device placement for training DNNs. The design of our new algorithm relies upon a new model of the device placement problem: by modeling it as a Markov decision process with multiple stages, we are able to prove that Spotlight achieves a theoretical guarantee on performance improvements. We have implemented Spotlight in the CIFAR-10 benchmark and deployed it on the Google Cloud platform. Extensive experiments have demonstrated that the training time with placements recommended by Spotlight is 60.9% of that recommended by the policy gradient method.

Modern large-scale heterogeneous computers incorporating GPUs offer impressive processing capabilities. It is desirable to fully utilize such systems for serving multiple users concurrently to visualize large data at interactive rates. However, as the disparity between data transfer speed and compute speed continues to increase in heterogeneous systems, data locality becomes crucial for performance. We present a new job scheduling design to support multi-user exploration of large data in a heterogeneous computing environment, achieving near optimal data locality and minimizing I/O overhead. The targeted application is a parallel visualization system which allows multiple users to render large volumetric data sets in both interactive mode and batch mode. We present a cost model to assess the performance of parallel volume rendering and quantify the efficiency of job scheduling. We have tested our job scheduling scheme on two heterogeneous systems with different configurations. The largest test volume data used in our study has over two billion grid points. The timing results demonstrate that our design effectively improves data locality for complex multi-user job scheduling problems, leading to better overall performance of the service.

This paper describes MRJS, a JavaScript based MapReduce framework. MRJS runs on the open internet over HTTP rather than in a data center. Instead of using clusters of machines for processing power, it relies on volunteer clients in the form of JavaScript engines in web browsers. Users can specify, execute, and fetch data for arbitrary MapReduce jobs with HTTP requests. Our centralized job server acts as a coordinator. We have implemented MRJS and results show that it performs well for certain types of MapReduce computations.

Data centers are widely used for big data analytics, which often involve data-parallel jobs, including query and web service. Meanwhile, cluster frameworks are rapidly developed for data-intensive applications in data center networks (DCNs). To promote the performance of these frameworks, many efforts have been paid to improve scheduling strategies and resource allocation algorithms. With the deployment of geo-distributed data centers and data-intensive applications, the optimization in DCNs regains pervasive attention in both industry and academia. Many solutions, such as the coflow-aware scheduling and speculative execution, have been proposed to meet various requirements. Therefore, we present a solid starting ground and comprehensive overview in this area to help readers quickly understand state-of-the-art technologies and research progress. We observe that algorithms in cluster frameworks are implemented with different guidelines and can be classified according to scheduling granularity, controller management, and prior-knowledge requirement. In addition, mechanisms for conquering crucial challenges in DCNs are discussed, including providing low latency and minimizing job completion time. Moreover, we analyze desirable properties of fault tolerance and scalability to illuminate the design principles of distributed systems. We hope that this paper will shed light on this promising land and serve as a guide for further researches.

Large-scale datacenters (DCs) host tens of thousands of diverse applications each day. However, interference between colocated workloads and the difficulty to match applications to one of the many hardware platforms available can degrade performance, violating the quality of service (QoS) guarantees that many cloud workloads require. While previous work has identified the impact of heterogeneity and interference, existing solutions are computationally intensive, cannot be applied online and do not scale beyond few applications.
 We present Paragon, an online and scalable DC scheduler that is heterogeneity and interference-aware. Paragon is derived from robust analytical methods and instead of profiling each application in detail, it leverages information the system already has about applications it has previously seen. It uses collaborative filtering techniques to quickly and accurately classify an unknown, incoming workload with respect to heterogeneity and interference in multiple shared resources, by identifying similarities to previously scheduled applications. The classification allows Paragon to greedily schedule applications in a manner that minimizes interference and maximizes server utilization. Paragon scales to tens of thousands of servers with marginal scheduling overheads in terms of time or state.
 We evaluate Paragon with a wide range of workload scenarios, on both small and large-scale systems, including 1,000 servers on EC2. For a 2,500-workload scenario, Paragon enforces performance guarantees for 91% of applications, while significantly improving utilization. In comparison, heterogeneity-oblivious, interference-oblivious and least-loaded schedulers only provide similar guarantees for 14%, 11% and 3% of workloads. The differences are more striking in oversubscribed scenarios where resource efficiency is more critical.

Emerging big data analytics applications require a significant amount of server computational power. As chips are hitting power limits, computing systems are moving away from general-purpose designs and toward greater specialization. Hardware acceleration through specialization has received renewed interest in recent years, mainly due to the dark silicon challenge. To address the computing requirements of big data, and based on the benchmarking and characterization results, we envision a data-driven heterogeneous architecture for next generation big data server platforms that leverage the power of field-programmable gate array (FPGA) to build custom accelerators in a Hadoop MapReduce framework. Unlike a full and dedicated implementation of Hadoop MapReduce algorithm on FPGA, we propose the hardware/software (HW/SW) co-design of the algorithm, which trades some speedup at a benefit of less hardware. Considering communication overhead with FPGA and other overheads involved in Hadoop MapReduce environment such as compression and decompression, shuffling and sorting, our experimental results show significant potential for accelerating Hadoop MapReduce machine learning kernels using HW/SW co-design methodology.

Parameter-specific adaptive learning rate methods are computationally efficient ways to reduce the ill-conditioning problems encountered when training large deep networks. Following recent work that strongly suggests that most of the critical points encountered when training such networks are saddle points, we find how considering the presence of negative eigenvalues of the Hessian could help us design better suited adaptive learning rate schemes. We show that the popular Jacobi preconditioner has undesirable behavior in the presence of both positive and negative curvature, and present theoretical and empirical evidence that the socalled equilibration preconditioner is comparatively better suited to non-convex problems. We introduce a novel adaptive learning rate scheme, called ESGD, based on the equilibration preconditioner. Our experiments show that ESGD performs as well or better than RMSProp in terms of convergence speed, always clearly improving over plain stochastic gradient descent.

Traditionally, MPI runtimes have been designed for clusters with a large number of nodes. However, with the advent of MPI+CUDA applications and dense multi-GPU systems, it has become important to design efficient communication schemes. This coupled with new application workloads brought forward by Deep Learning frameworks like Caffe and Microsoft CNTK pose additional design constraints due to very large message communication of GPU buffers during the training phase. In this context, special-purpose libraries like NCCL have been proposed. In this paper, we propose a pipelined chain (ring) design for the MPI_Bcast collective operation along with an enhanced collective tuning framework in MVAPICH2-GDR that enables efficient intra-/internode multi-GPU communication. We present an in-depth performance landscape for the proposed MPI_Bcast schemes along with a comparative analysis of NCCL Broadcast and NCCL-based MPI_Bcast. The proposed designs for MVAPICH2-GDR enable up to 14X and 16.6X improvement, compared to NCCL-based solutions, for intra- and internode broadcast latency, respectively. In addition, the proposed designs provide up to 7% improvement over NCCL-based solutions for data parallel training of the VGG network on 128 GPUs using Microsoft CNTK. The proposed solutions outperform the recently introduced NCCL2 library for small and medium message sizes and offer comparable/better performance for very large message sizes.

The use of large-scale data mining and machine learning has proliferated through the adoption of technologies such as Hadoop, with its simple programming semantics and rich and active ecosystem. This paper presents LinkedIn's Hadoop-based analytics stack, which allows data scientists and machine learning researchers to extract insights and build product features from massive amounts of data. In particular, we present our solutions to the ``last mile'' issues in providing a rich developer ecosystem. This includes easy ingress from and egress to online systems, and managing workflows as production processes. A key characteristic of our solution is that these distributed system concerns are completely abstracted away from researchers. For example, deploying data back into the online system is simply a 1-line Pig command that a data scientist can add to the end of their script. We also present case studies on how this ecosystem is used to solve problems ranging from recommendations to news feed updates to email digesting to descriptive analytical dashboards for our members.


Question: Gated network systems: a comprehensive inventory.

In the realm of artificial intelligence, deep learning has emerged as a transformative force, revolutionizing various sectors and applications. From enhancing web search capabilities and content curation on social media to refining recommendations on e-commerce platforms, and its integration into everyday devices like cameras and smartphones, deep learning is reshaping the landscape of machine learning. Traditionally, machine learning was constrained in processing raw, natural data, necessitating intricate engineering and specialized knowledge to design feature extractors that could transform raw inputs into a format amenable to pattern recognition. However, representation learning, and more specifically, deep learning, have overcome these limitations. Deep learning employs multiple layers of representation, each derived from the previous layer through non-linear transformations, allowing for the automatic discovery of complex patterns within data. This hierarchical representation enables the learning of increasingly abstract features, with higher layers focusing on crucial aspects for classification while filtering out irrelevant variations. For instance, in image recognition, initial layers detect edges, followed by motifs, parts of objects, and eventually, complete objects. These representations are not handcrafted but learned from data, showcasing the power of deep learning in automatically discovering intricate structures in high-dimensional data. This capability has been instrumental in advancing various domains, from setting new benchmarks in image and speech recognition to outperforming traditional methods in drug molecule activity prediction, particle physics data analysis, and genomics. Surprisingly, deep learning has also excelled in natural language understanding tasks such as topic classification, sentiment analysis, question answering, and language translation. With its minimal reliance on manual engineering, deep learning is poised for further success, capitalizing on advancements in computational resources and data availability. Ongoing research in deep neural networks and learning algorithms will likely accelerate this progress, further solidifying deep learning's position as a cornerstone in artificial intelligence and beyond.

The state machine approach is a general method for implementing fault-tolerant services in distributed systems. This paper reviews the approach and describes protocols for two different failure models—Byzantine and fail stop. Systems reconfiguration techniques for removing faulty components and integrating repaired components are also discussed.

ion-Based Livelock/Deadlock Checking for Hardware Verification In-Ho Moon and Kevin Harer Synopsys Inc. {mooni, kevinh}@synopsys.com

Declaration This dissertation is the result of my own work done under the guidance of my supervisor, and includes nothing which is the outcome of work done in collaboration except where specifically indicated in the text. This dissertation is not substantially the same as any that I have submitted or will be submitting for a degree or diploma or other qualification at this or any other University. This dissertation does not exceed the regulation length of 60,000 words, including tables and footnotes. 5 Summary We introduce two abstract notions of equational algebraic system, called Equational System (ES) and Term Equational System (TES), in order to achieve sufficient expressivity as needed in modern applications in computer science. These generalize the classical concept of (enriched) algebraic theory of Kelly and Power [1993]. We also develop a theory for constructing free algebras for ESs and a theory of equational reasoning for TESs. In Part I, we introduce the general abstract, yet practical, concept of equational system and develop finitary and transfinitary conditions under which we give an explicit construction of free algebras for ESs. This free construction extends the well-known construction of free algebras for ω-cocontinuous endofunctors to an equational setting, capturing the intuition that free algebras consist of freely constructed terms quotiented by given equations and congruence rules. We further show the monadicity and cocom-pleteness of categories of algebras for ESs under the finitary and transfinitary conditions. To illustrate the expressivity of equational systems, we exhibit various examples including two modern applications, the Σ-monoids of Fiore et al. [1999] and the π-algebras of Stark [2005]. In Part II, we introduce the more concrete notion of term equational system, which is obtained by specializing the concept of equational system, but remains more general than that of enriched algebraic theory. We first develop a sound logical deduction system, called Term Equational Logic (TEL), for equational reasoning about algebras of TESs. Then, to pursue a complete logic, we give an internal completeness result, from which together with the explicit construction of free algebras one can typically synthesize sound and complete rewriting-style equational logics. To exemplify this scenario, we give two applications: multi-sorted algebraic theories and nominal equational theories of Clouston and Pitts [2007] and of Gabbay and Mathijssen [2007]. 7 Acknowledgements First and foremost, I am deeply grateful to my Ph.D. supervisor Marcelo Fiore for his patient guidance and support, without which this thesis would …

Bayesian networks provide a modeling language and associated inference algorithm for stochastic domains. They have been successfully applied in a variety of medium-scale applications. However, when faced with a large complex domain, the task of modeling using Bayesian networks begins to resemble the task of programming using logical circuits. In this paper, we describe an object-oriented Bayesian network (OOBN) language, which allows complex domains to be described in terms of inter-related objects. We use a Bayesian network fragment to describe the probabilistic relations between the attributes of an object. These attributes can themselves be objects, providing a natural framework for encoding part-of hierarchies. Classes are used to provide a reusable probabilistic model which can be applied to multiple similar objects. Classes also support inheritance of model fragments from a class to a subclass, allowing the common aspects of related classes to be defined only once. Our language has clear declarative semantics: an OOBN can be interpreted as a stochastic functional program, so that it uniquely specifies a probabilistic model. We provide an inference algorithm for OOBNs, and show that much of the structural information encoded by an OOBN—particularly the encapsulation of variables within an object and the reuse of model fragments in different contexts—can also be used to speed up the inference process.

A general-purpose, fully interconnected neural-net chip was used to perform computationally intensive tasks for handwritten digit recognition. The chip has nearly 3000 programmable connections, which can be set for template matching. The templates can be reprogrammed as needed during the recognition sequence. The recognition process proceeds in four major steps. First, the image is captured using a TV camera and a digital framegrab. This image is converted, using a digital computer, to either black or white pixels and scaled to fill a 16*16-pixel frame. Next, using the neural-net chip, the image is skeletonized, i.e. the image is thinned to a backbone one pixel wide. Then, the chip is programmed, and a feature map is created by template-matching stored primitive patterns on the chip with regions on the skeletonized image. Finally, recognition, based on the feature map, is achieved using any one of a variety of statistical and heuristic techniques on a digital computer. Best scores range between 90% and 99% correct classification, depending on the quality of the original handwritten digits.<<ETX>>

Technological progress in integrated, low-power, CMOS communication devices and sensors makes a rich design space of networked sensors viable. They can be deeply embedded in the physical world and spread throughout our environment like smart dust. The missing elements are an overall system architecture and a methodology for systematic advance. To this end, we identify key requirements, develop a small device that is representative of the class, design a tiny event-driven operating system, and show that it provides support for efficient modularity and concurrency-intensive operation. Our operating system fits in 178 bytes of memory, propagates events in the time it takes to copy 1.25 bytes of memory, context switches in the time it takes to copy 6 bytes of memory and supports two level scheduling. The analysis lays a groundwork for future architectural advances.

The use of networks for communications between the electronic control units (ECU) of a vehicle in production cars dates from the beginning of the 1990s. The specific requirements of the different car domains have led to the development of a large number of automotive networks such as Local Interconnect Network, J1850, CAN, TTP/C, FlexRay, media-oriented system transport, IDB1394, etc. This paper first introduces the context of in-vehicle embedded systems and, in particular, the requirements imposed on the communication systems. Then, a comprehensive review of the most widely used automotive networks, as well as the emerging ones, is given. Next, the current efforts of the automotive industry on middleware technologies, which may be of great help in mastering the heterogeneity, are reviewed. Finally, we highlight future trends in the development of automotive communication systems.

We identify two families of canonical representations called ROBDD[∧î]C and ROBDD[∧T̂ ,i]T by augmenting ROBDD with two types of conjunctive decompositions. These representations cover the three existing languages ROBDD, ROBDD with as many implied literals as possible (ROBDDL∞), and AND/OR BDD. We introduce a new time efficiency criterion called rapidity which reflects the idea that exponential operations may be preferable if the language can be exponentially more succinct. Then we demonstrate that the expressivity, succinctness and operation rapidity do not decrease from ROBDD[∧T̂ ,i]T to ROBDD[∧î]C , and then to ROBDD[∧ î+1 ]C . We also demonstrate that ROBDD[∧î]C (i > 1) and ROBDD[∧T̂ ,i]T are not less tractable than ROBDD-L∞ and ROBDD, respectively. Finally, we develop a compiler for ROBDD[∧∞̂]C which significantly advances the compiling efficiency of canonical representations.