Question: A probability-based security metric derived from attack graphs.

This paper introduces a method for constructing attack scenarios by correlating intrusion alerts, addressing a critical gap in traditional intrusion detection systems (IDSs). These systems often focus on low-level attacks or anomalies, generating alerts independently without considering potential logical interconnections. This approach can lead to a mix of real and false alerts, particularly in environments with heavy intrusion activity, overwhelming human users or automated response systems with an unmanageable volume of alerts. To tackle this challenge, the paper proposes a practical technique for correlating alerts based on the prerequisites and consequences of intrusions. The prerequisite of an intrusion is defined as the essential condition for its successful execution, whereas the consequence refers to the potential outcomes. The proposed method correlates alerts by matching (partially or fully) the consequences of earlier alerts with the prerequisites of subsequent ones. This paper's contributions encompass a formal framework for alert correlation, the development of an offline alert correlator based on this framework, and the evaluation of the method using the 2000 DARPA intrusion detection scenario-specific datasets. The results affirm the method's potential and superiority over competing approaches, demonstrating its effectiveness in managing and interpreting intrusion alerts more efficiently.

**Question**: A novel feature extraction method for text categorization

**Documents**: Text categorization using feature selection based on word co-occurrence
In this paper, we propose a novel feature selection method for text categorization based on word co-occurrence. The method is aimed at selecting a subset of features that can effectively represent the text documents while minimizing the dimensionality of the feature space. The proposed method utilizes word co-occurrence statistics to measure the relevance of a feature to a category. The relevance score of a feature is calculated by comparing the frequency of its co-occurrence with words in the category to the frequency of its co-occurrence with words in the rest of the documents. A feature is considered relevant if it co-occurs more frequently with words in the category than with words in the rest of the documents. The proposed method is evaluated on several benchmark datasets and compared with other feature selection methods. The experimental results show that the proposed method can achieve higher classification accuracy than other methods while reducing the dimensionality of the feature space. The proposed method can be used as a pre-processing step for text categorization tasks to improve the efficiency and accuracy of the classification.

**Similar Documents**: This paper presents a new feature selection technique for text categorization that leverages word co-occurrence statistics to optimize the representation of text documents. The goal is to identify a subset of

This paper investigates two fundamental characteristics of a wireless multi -hop network: its minimum node degree and its k--connectivity. Both topology attributes depend on the spatial distribution of the nodes and their transmission range. Using typical modeling assumptions :--- :a random uniform distribution of the nodes and a simple link model :--- :we derive an analytical expression that enables the determination of the required range r0 that creates, for a given node density ρ, an almost surely k--connected network. Equivalently, if the maximum r0 of the nodes is given, we can find out how many nodes are needed to cover a certain area with a k--connected network. We also investigate these questions by various simulations and thereby verify our analytical expressions. Finally, the impact of mobility is discussed.The results of this paper are of practical value for researchers in this area, e.g., if they set the parameters in a network--level simulation of a mobile ad hoc network or if they design a wireless sensor network.

Man-At-The-End (MATE) attacks and fortifications are difficult to analyze, model, and evaluate predominantly for three reasons: firstly, the attacker is human and, therefore, utilizes motivation, creativity, and ingenuity. Secondly, the attacker has limitless and authorized access to the target. Thirdly, all major protections stand up to a determined attacker till a certain period of time. Digital assets range from business to personal use, from consumer devices to home networks, the public Internet, the cloud, and the Internet of Things – where traditional computer and network security are inadequate to address MATE attacks. MATE is fundamentally a hard problem. Much of the extant focus to deal with MATE attacks is purely technical; though security is more than just a technical issue. The main objective of the paper is to mitigate the consequences of MATE attacks through the human element of security and highlight the need for this element to form a part of a holistic security strategy alongside the necessary techniques and technologies. This paper contributes by taking software protection (SP) research to a new realm of challenges. Moreover, the paper elaborates the concept of MATE attacks, the different forms, and the analysis of MATE versus insider threats to present a thematic taxonomy of a MATE attack. The ensuing paper also highlights the fundamental concept of digital assets, and the core protection mechanisms and their qualitative comparison against MATE attacks. Finally, we present state-of-the-art trends and cutting-edge future research directions by taking into account only the human aspects for young researchers and professionals. & 2014 Elsevier Ltd. All rights reserved.

A distributed detection method is proposed to detect single stage multi-point (SSMP) attacks on a Cyber Physical System (CPS). Such attacks aim at compromising two or more sensors or actuators at any one stage of a CPS and could totally compromise a controller and prevent it from detecting the attack. However, as demonstrated in this work, using the flow properties of water from one stage to the other, a neighboring controller was found effective in detecting such attacks. The method is based on physical invariants derived for each stage of the CPS from its design. The attack detection effectiveness of the method was evaluated experimentally against an operational water treatment testbed containing 42 sensors and actuators. Results from the experiments point to high effectiveness of the method in detecting a variety of SSMP attacks but also point to its limitations. Distributing the attack detection code among various controllers adds to the scalability of the proposed method.

Besides a large set of malware categories such as worms and Trojan horses, Advanced Persistent Threat (APT) is another more sophisticated attack entity emerging in the cyber threats environment. In this paper we propose a model of the APT detection problem as well as a methodology to implement it on a generic organization network. From our knowledge, the proposed method is the first to address the problem of modeling an APT and to provide a possible detection framework.

The pervasiveness of Cyber-Physical Systems (CPS) in various aspects of the modern society grows rapidly. This makes CPS to increasingly attractive targets for various kinds of attacks. We consider cyber-security as an integral part of CPS security. Additionally, the necessity exists to investigate the CPS-specific aspects which are out of scope of cyber-security. Most importantly, attacks capable to cross the cyber-physical domain boundary should be analyzed. The vulnerability of CPS to such cross-domain attacks has been practically proven by numerous examples, e.g., by the currently most famous Stuxnet attack. In this paper, we propose taxonomy for description of attacks on CPS. The proposed taxonomy is capable of representing both conventional cyber-attacks as well as cross-domain attacks on CPS. Furthermore, based on the proposed taxonomy, we define the attack categorization. Several possible application areas of the proposed taxonomy are extensively discussed. Among others, it can be used to establish a knowledge base about attacks on CPS known in the literature. Furthermore, the proposed description structure will foster the quantitative and qualitative analysis of these attacks, both of which are necessarily to improve CPS security.

Lee, Stolfo, and Mok have previously reported the use of association rules and frequency episodes for mining audit data to gain knowledge for intrusion detection. The integration of association rules and frequency episodes with fuzzy logic can produce more abstract and flexible patterns for intrusion detection, since many quantitative features are involved in intrusion detection and security itself is fuzzy. We present a modification of a previously reported algorithm for mining fuzzy association rules, define the concept of fuzzy frequency episodes, and present an original algorithm for mining fuzzy frequency episodes. We add a normalization step to the procedure for mining fuzzy association rules in order to prevent one data instance from contributing more than others. We also modify the procedure for mining frequency episodes to learn fuzzy frequency episodes. Experimental results show the utility of fuzzy association rules and fuzzy frequency episodes in intrusion detection. Draft: Updated version published in the International Journal of Intelligent Systems, Volume 15, No. I, August 2000 3

http://dx.doi.org/10.1016/j.infsof.2014.09.012 0950-5849/ 2014 Elsevier B.V. All rights reserved. ⇑ Corresponding author at: Center of Informatics, Federal University of Pernambuco (CIn/UFPE), Av. Jornalista Anibal Fernandes, S/N, Cidade Universitária, 50. Recife, PE, Brazil. Tel.: +55 81 8250 0470. E-mail addresses: fss4@cin.ufpe.br (F. Selleri Silva), fsfs@cin.ufpe.br (F.S.F. Soares), alp5@cin.ufpe.br (A.L. Peres), ima2@cin.ufpe.br (I.M.d. Azevedo), aplfv@ci (A.P.L.F. Vasconcelos), fkk@cin.ufpe.br (F.K. Kamei), srlm@cin.ufpe.br (S.R.d.L. Meira). Fernando Selleri Silva a,b,⇑, Felipe Santana Furtado Soares , Angela Lima Peres , Ivanildo Monteiro de Azevedo , Ana Paula L.F. Vasconcelos , Fernando Kenji Kamei , Silvio Romero de Lemos Meira a,c

Protecting the privacy of individuals in graph structured data while making accurate versions of the data available is one of the most challenging problems in data privacy. Most efforts to date to perform this data release end up mired in complexity, overwhelm the signal with noise, and are not effective for use in practice. In this paper, we introduce a new method which guarantees differential privacy. It specifies a probability distribution over possible outputs that is carefully defined to maximize the utility for the given input, while still providing the required privacy level. The distribution is designed to form a 'ladder', so that each output achieves the highest 'rung' (maximum probability) compared to less preferable outputs. We show how our ladder framework can be applied to problems of counting the number of occurrences of subgraphs, a vital objective in graph analysis, and give algorithms whose cost is comparable to that of computing the count exactly. Our experimental study confirms that our method outperforms existing methods for counting triangles and stars in terms of accuracy, and provides solutions for some problems for which no effective method was previously known. The results of our algorithms can be used to estimate the parameters of suitable graph models, allowing synthetic graphs to be sampled.

This paper introduces a new similarity measure, the covering similarity, which we formally define for evaluating the similarity between a symbolic sequence and a set of symbolic sequences. A pairwise similarity can also be directly derived from the covering similarity to compare two symbolic sequences. An efficient implementation to compute the covering similarity is proposed which uses a suffix-tree data structure, but other implementations, based on suffix array for instance, are possible and are possibly necessary for handling very large-scale problems. We have used this similarity to isolate attack sequences from normal sequences in the scope of host-based intrusion detection. We have assessed the covering similarity on two well-known benchmarks in the field. In view of the results reported on these two datasets for the state-of-the-art methods, according to the comparative study, we have carried out based on three challenging similarity measures commonly used for string processing, or in bioinformatics, we show that the covering similarity is particularly relevant to address the detection of anomalies in sequences of system calls.

JavaScript is used by web developers to enhance the interactivity of their sites, offload work to the users' browsers and improve their sites' responsiveness and user-friendliness, making web pages feel and behave like traditional desktop applications. An important feature of JavaScript, is the ability to combine multiple libraries from local and remote sources into the same page, under the same namespace. While this enables the creation of more advanced web applications, it also allows for a malicious JavaScript provider to steal data from other scripts and from the page itself. Today, when developers include remote JavaScript libraries, they trust that the remote providers will not abuse the power bestowed upon them.
 In this paper, we report on a large-scale crawl of more than three million pages of the top 10,000 Alexa sites, and identify the trust relationships of these sites with their library providers. We show the evolution of JavaScript inclusions over time and develop a set of metrics in order to assess the maintenance-quality of each JavaScript provider, showing that in some cases, top Internet sites trust remote providers that could be successfully compromised by determined attackers and subsequently serve malicious JavaScript. In this process, we identify four, previously unknown, types of vulnerabilities that attackers could use to attack popular web sites. Lastly, we review some proposed ways of protecting a web application from malicious remote scripts and show that some of them may not be as effective as previously thought.


Question: A probabilistic security metric based on attack graphs.

This paper discusses the development of the COPS (Computerized Oracle and Password System) Security Checker System, a tool designed to aid system administrators in identifying and mitigating common security vulnerabilities, particularly those affecting UNIX systems. Over the years, numerous security issues have been repeatedly highlighted in literature, including issues with set user ID (setuid) programs, inappropriate file permissions, and weak passwords. These problems are notable for two reasons: they are generally straightforward to rectify once discovered, and they are relatively easy to detect. Given that these vulnerabilities are prevalent across various systems, there is a clear need for a dedicated tool to assist in their detection. The COPS system is presented as a response to this need, offering a freely available solution to help administrators pinpoint potential security flaws. By automating the process of identifying common security weaknesses, COPS serves as a valuable asset in enhancing the security posture of UNIX-based systems.

The discovery of vulnerabilities in source code is a key for securing computer systems. While specific types of security flaws can be identified automatically, in the general case the process of finding vulnerabilities cannot be automated and vulnerabilities are mainly discovered by manual analysis. In this paper, we propose a method for assisting a security analyst during auditing of source code. Our method proceeds by extracting abstract syntax trees from the code and determining structural patterns in these trees, such that each function in the code can be described as a mixture of these patterns. This representation enables us to decompose a known vulnerability and extrapolate it to a code base, such that functions potentially suffering from the same flaw can be suggested to the analyst. We evaluate our method on the source code of four popular open-source projects: LibTIFF, FFmpeg, Pidgin and Asterisk. For three of these projects, we are able to identify zero-day vulnerabilities by inspecting only a small fraction of the code bases.

Situation awareness depends on a reliable perception of the environment and comprehension of its semantic structures. In this respect, cyberspace presents a unique challenge to the situation awareness of users and analysts, since it is a unique combination of human and machine elements, whose complex interactions occur in a global communication network. Accordingly, we outline the underpinnings of an ontology of secure operations in cyberspace, presenting the ontology framework and providing two modeling examples. We make the case for adopting a rigorous semantic model of cyber security to overcome the current limits of the state of the art. Keywords— cyber security, ontology, situation awareness,

Efforts to build "secure" computer systems have now been underway for more than a decade. Many designs have been proposed, some prototypes have been constructed, and a few systems are approaching the production stage. A small number of systems are even operating in what the Department of Defense calls the "multilevel" mode some information contained m these computer systems may have a clasmfication higher than the clearance of some of the users of those systems. This paper revmws the need for formal security models, describes the structure and operation of military security controls, considers how automation has affected security problems, surveys models that have been proposed and applied to date, and suggests possible d~rectlons for future models

Due to their distributed management, smart grids can be vulnerable to malicious attacks that undermine their cyber security. An adversary can take control of few nodes in the network and spread digital attacks like an infection, whose diffusion is facilitated by the lack of centralized supervision within the smart grid. In this paper, we propose to investigate these phenomena by means of epidemic models applied to cellular automata. We show that the common key parameters of epidemic models, such as the basic reproductive ratio, are also useful in this context to understand the extent of the grid portion that can be compromised. At the same time, the lack of mobility of individuals limits the spreading of the infection. In particular, we evaluate the role of the grid connectivity degree in both containing the epidemics and avoiding its spreading on the entire network, and also increasing the number of nodes that do not get any contact with the cyber attacks.

We analyze the concrete security and key sizes of theoretically sound lattice-based encryption schemes based on the “learning with errors” (LWE) problem. Our main contributions are: (1) a new lattice attack on LWE that combines basis reduction with an enumeration algorithm admitting a time/success tradeoff, which performs better than the simple distinguishing attack considered in prior analyses; (2) concrete parameters and security estimates for an LWE-based cryptosystem that is more compact and efficient than the well-known schemes from the literature. Our new key sizes are up to 10 times smaller than prior examples, while providing even stronger concrete security levels.

Jimmy Lin, Adam Roegiest, Luchen Tan, Richard McCreadie, Ellen Voorhees, and Fernando Diaz 1 David R. Cheriton School of Computer Science, University of Waterloo, Ontario, Canada 2 School of Computing Science, University of Glasgow, Scotland, the United Kingdom 3 National Institute for Standards and Technology, Maryland, USA 4 Microsoft Research, New York, USA {jimmylin, aroegies, luchen.tan}@uwaterloo.ca richard.mccreadie@glasgow.ac.uk, ellen.voorhees@nist.gov, fdiaz@microsoft.com

This paper focuses on cyber-security simulations in networks modeled as a Markov game with incomplete information and stochastic elements. The resulting game is an adversarial sequential decision making problem played with two agents, the attacker and defender. The two agents pit one reinforcement learning technique, like neural networks, Monte Carlo learning and Q-learning, against each other and examine their effectiveness against learning opponents. The results showed that Monte Carlo learning with the Softmax exploration strategy is most effective in performing the defender role and also for learning attacking strategies.

Network-based attacks have become common and sophisticated. For this reason, intrusion detection systems are now shifting their focus from the hosts and their operating systems to the network itself. Network-based intrusion detection is challenging because network auditing produces large amounts of data, and different events related to a single intrusion may be visible in different places on the network. This paper presents NetSTAT, a new approach to network intrusion detection. By using a formal model of both the network and the attacks, NetSTAT is able to determine which network events have to be monitored and where they can be monitored.

Security equipment such as intrusion prevention system is an important supplementary for security management. They reduce the difficulty of network management by giving alarms corresponding to different attacks instead of raw traffic packet inspection. But there are many false alarms due to their running mechanism, which greatly reduces its usability. In this paper, we develop a hierarchical framework to mine high threating alarms from the massive alarm logs, and aim to provide fundamental and useful information for administrators to design efficient management policy. First, the alarms are divided into two parts based on their attributes, the first part mainly includes several kinds of famous attacks which are critical for security management, we proposed a similar alarm mining method based on Choquet integral to cluster and rank the frequently occurred attacks. The rest alarms constitute the second part, which are caused by the potential threats attacks, also include many false alarms. To reduce the effect of false alarms and rank the potential threats, we employ the frequent pattern mining algorithm to mine correlation rules and then filter false alarms. Following, we proposed a self-adapting threat degree calculation method to qualify the threat degree of these alarms after filtering. To verity the methods developed, an experimental platform is constructed in the campus network of Xi’an Jiaotong University. Experimental results based on the data collected verify the efficiency of the developed methods. For the first kind of alarms, the similar alarms mining accuracy is higher than 97% and the alarms are ranked with different processing urgencies. For the rest alarms, the proposed methods have filtering accuracy above 80% and can rank the potential threats. Based on the ranking results, administrators can deal with the high threats with their limited time and energy, in turn, keep the network under control.

http://dx.doi.org/10.1016/j.infsof.2014.09.012 0950-5849/ 2014 Elsevier B.V. All rights reserved. ⇑ Corresponding author at: Center of Informatics, Federal University of Pernambuco (CIn/UFPE), Av. Jornalista Anibal Fernandes, S/N, Cidade Universitária, 50. Recife, PE, Brazil. Tel.: +55 81 8250 0470. E-mail addresses: fss4@cin.ufpe.br (F. Selleri Silva), fsfs@cin.ufpe.br (F.S.F. Soares), alp5@cin.ufpe.br (A.L. Peres), ima2@cin.ufpe.br (I.M.d. Azevedo), aplfv@ci (A.P.L.F. Vasconcelos), fkk@cin.ufpe.br (F.K. Kamei), srlm@cin.ufpe.br (S.R.d.L. Meira). Fernando Selleri Silva a,b,⇑, Felipe Santana Furtado Soares , Angela Lima Peres , Ivanildo Monteiro de Azevedo , Ana Paula L.F. Vasconcelos , Fernando Kenji Kamei , Silvio Romero de Lemos Meira a,c


Question: A high-performance industrial big-data processing engine.

In the realm of business intelligence and analytics, the transition from managing big data to leveraging it for significant impact has become a pivotal concern for industries. This evolution necessitates an efficient industrial big-data engine capable of processing vast volumes of data to extract actionable insights. The focus has shifted from mere data collection to the strategic use of analytics to drive business decisions, improve operational efficiency, and enhance customer experiences. Key to this transformation is the development of advanced analytical tools and platforms that can handle the complexity and scale of industrial data. These systems must be capable of real-time analysis, predictive modeling, and machine learning to unlock the full potential of big data. By integrating these capabilities, businesses can achieve a competitive edge, optimize resource allocation, and respond proactively to market trends and customer needs.

**Question**: A Direct Search Method to solve Economic Dispatch Problem with Valve-Point Effect

**Documents**: A novel hybrid optimization approach for economic dispatch problems

**Similar Documents**: This document discusses a cutting-edge hybrid optimization methodology designed to address economic dispatch problems, particularly those complicated by the valve-point effect. The proposed approach integrates the strengths of multiple optimization techniques to achieve a more efficient and effective solution. It aims to minimize the total cost of generating electricity while adhering to operational constraints and considering the non-linear characteristics introduced by the valve-point effect. The method is designed to navigate the complex landscape of the dispatch problem, where the objective is to determine the optimal power output for each generating unit to meet the system load demand at the lowest cost. By combining diverse optimization algorithms, the proposed methodology can better explore the solution space, avoid local optima, and find a global minimum. This approach not only enhances the accuracy of the economic dispatch solution but also improves the computational efficiency, making it a promising strategy for real-world power system operations.

With almost everything now online, organizations look at the Big Data collected to gain insights for improving their services. In the analytics process, derivation of such insights requires experimenting-with and integrating different analytics techniques, while handling the Big Data high arrival velocity and large volumes. Existing solutions cover bits-and-pieces of the analytics process, leaving it to organizations to assemble their own ecosystem or buy an off-the-shelf ecosystem that can have unnecessary components to them. We build on this point by dividing the Big Data Analytics problem into six main pillars. We characterize and show examples of solutions designed for each of these pillars. We then integrate these six pillars into a taxonomy to provide an overview of the possible state-of-the-art analytics ecosystems. In the process, we highlight a number of ecosystems to meet organizations different needs. Finally, we identify possible areas of research for building future Big Data Analytics Ecosystems.

Resource Description Framework (RDF) represents a flexible and concise model for representing the metadata of resources on the web. Over the past years, with the increasing amount of RDF data, efficient and scalable RDF data management has become a fundamental challenge to achieve the Semantic Web vision. However, multiple approaches for RDF storage have been suggested, ranging from simple triple stores to more advanced techniques like vertical partitioning on the predicates or centralized approaches. Unfortunately, it is still a challenge to store a huge quantity of RDF quads due, in part, to the query processing for RDF data. This paper proposes a scalable solution for RDF data management that uses Apache Accumulo. We focus on introducing storage methods and indexing techniques that scale to billions of quads across multiple nodes, while providing fast and easy access to the data through conventional query mechanisms such as SPARQL. Our performance evaluation shows that in most cases our approach works well against large RDF datasets.

Silo is a new in-memory database that achieves excellent performance and scalability on modern multicore machines. Silo was designed from the ground up to use system memory and caches efficiently. For instance, it avoids all centralized contention points, including that of centralized transaction ID assignment. Silo's key contribution is a commit protocol based on optimistic concurrency control that provides serializability while avoiding all shared-memory writes for records that were only read. Though this might seem to complicate the enforcement of a serial order, correct logging and recovery is provided by linking periodically-updated epochs with the commit protocol. Silo provides the same guarantees as any serializable database without unnecessary scalability bottlenecks or much additional latency. Silo achieves almost 700,000 transactions per second on a standard TPC-C workload mix on a 32-core machine, as well as near-linear scalability. Considered per core, this is several times higher than previously reported results.

In many applications involving continuous data streams, data arrival is bursty and data rate fluctuates over time. Systems that seek to give rapid or real-time query responses in such an environment must be prepared to deal gracefully with bursts in data arrival without compromising system performance. We discuss one strategy for processing bursty streams --- adaptive, load-aware scheduling of query operators to minimize resource consumption during times of peak load. We show that the choice of an operator scheduling strategy can have significant impact on the run-time system memory usage. We then present Chain scheduling, an operator scheduling strategy for data stream systems that is near-optimal in minimizing run-time memory usage for any collection of single-stream queries involving selections, projections, and foreign-key joins with stored relations. Chain scheduling also performs well for queries with sliding-window joins over multiple streams, and multiple queries of the above types. A thorough experimental evaluation is provided where we demonstrate the potential benefits of Chain scheduling, compare it with competing scheduling strategies, and validate our analytical conclusions.

Yellow elephants are slow. A major reason is that they consume their inputs entirely before responding to an elephant rider’s orders. Some clever riders have trained their yellow elephants to only consume parts of the inputs before responding. However, the teaching time to make an elephant do that is high. So high that the teaching lessons often do not pay off. We take a different approach. We make elephants aggressive; only this will make them very fast. We propose HAIL (Hadoop Aggressive Indexing Library), an enhancement of HDFS and Hadoop MapReduce that dramatically improves runtimes of several classes of MapReduce jobs. HAIL changes the upload pipeline of HDFS in order to create different clustered indexes on each data block replica. An interesting feature of HAIL is that we typically create a win-win situation: we improve both data upload to HDFS and the runtime of the actual Hadoop MapReduce job. In terms of data upload, HAIL improves over HDFS by up to 60% with the default replication factor of three. In terms of query execution, we demonstrate that HAIL runs up to 68x faster than Hadoop. In our experiments, we use six clusters including physical and EC2 clusters of up to 100 nodes. A series of scalability experiments also demonstrates the superiority of HAIL.

This paper describes novel aspects of the column store implemented in the MemSQL database engine and describes the design choices made to support real-time streaming workloads. Column stores have traditionally been restricted to data warehouse scenarios where low latency queries are a secondary goal, and where restricting data ingestion to be offline, batched, append-only, or some combination thereof is acceptable. In contrast, the MemSQL column store implementation treats low latency queries and ongoing writes as first class citizens, with a focus on avoiding interference between read, ingest, update, and storage optimization workloads through the use of fragmented snapshot transactions and optimistic storage reordering. This implementation broadens the range of serviceable column store workloads to include those with more stringent demands on query and data latency, such as those backing operational systems used by adtech, financial services, fraud detection and other real-time or data streaming applications.

Big Data has shown lot of potential in real world industry and research community. We support the power and potential of it in solving real world problems. However, it is imperative to understand Big Data through the lens of 7 V's. 7th V as `Value' is desired output for industry challenges and issues. We provide a brief survey study of 7 Vs of Big Data in order to understand Big Data and extract Value concept in general. Finally we conclude by showing our vision of improved healthcare, a product of Big Data Utilization, as a future work for researchers and students, while moving forward.

Hyracks is a new partitioned-parallel software platform designed to run data-intensive computations on large shared-nothing clusters of computers. Hyracks allows users to express a computation as a DAG of data operators and connectors. Operators operate on partitions of input data and produce partitions of output data, while connectors repartition operators' outputs to make the newly produced partitions available at the consuming operators. We describe the Hyracks end user model, for authors of dataflow jobs, and the extension model for users who wish to augment Hyracks' built-in library with new operator and/or connector types. We also describe our initial Hyracks implementation. Since Hyracks is in roughly the same space as the open source Hadoop platform, we compare Hyracks with Hadoop experimentally for several different kinds of use cases. The initial results demonstrate that Hyracks has significant promise as a next-generation platform for data-intensive applications.

Distributed computing is essential for handling very large datasets. Online learning is also promising for learning from rapid data streams. However, it is still an unresolved problem how to combine them for scalable learning and prediction on big data streams. We propose a general computational framework called loose model sharing for online and distributed machine learning. The key is to share only models rather than data between distributed servers. We also introduce Jubatus, an open source software platform based on the framework. Finally, we describe the details of implementing classifier and nearest neighbor algorithms, and discuss our experimental evaluations.

Many stream-based applications are naturally distributed. Applications are often embedded in an environment with numerous connected computing devices with heterogeneous capabilities. As data travels from its point of origin (e.g., sensors) downstream to applications, it passes through many computing devices, each of which is a potential target of computation. Furthermore, to cope with time-varying load spikes and changing demand, many servers would be brought to bear on the problem. In both cases, distributed computation is the norm. Abstract


Question: A high-performance industrial big-data processing engine.

Innovative Big Data Analytics: The Current Landscape, Emerging Challenges, and Research Directions

The ubiquity of the term "big data" in today's discourse is undeniable, permeating a wide spectrum of domains—from the mundane aspects of daily life to the intricacies of traditional research areas such as geography, transportation, biology, chemistry, medicine, and rehabilitation. The proliferation of diverse networks has significantly expanded the scope of big data in terms of its types, associated issues, and potential solutions. This paper provides an exhaustive review of the recent advancements in big data research, focusing on data categorization, storage paradigms, privacy concerns, security measures, analytical techniques, and practical applications, particularly in the context of network-related big data. We also highlight the prevailing challenges and the evolution of big data, offering insights into the current trends and future trajectories that are likely to shape the field.

**Question**: A Novel Algorithm for Solving the Travelling Salesman Problem

**Documents**: A New Hybrid Genetic Algorithm for the Traveling Salesman Problem
In this paper, we present a new hybrid genetic algorithm (HGA) for the traveling salesman problem (TSP). The proposed algorithm combines genetic operators with local search heuristics to improve the quality of the solutions. We conduct extensive computational experiments on a variety of TSP instances from the TSPLIB library to evaluate the performance of the proposed algorithm. The experimental results show that the proposed algorithm can find high-quality solutions for the TSP in a relatively short time.

**Similar Documents**:

An Enhanced Hybrid Genetic Algorithm for the Traveling Salesman Challenge

This study introduces an enhanced hybrid genetic algorithm (EHGA) designed to tackle the traveling salesman problem (TSP) more efficiently. The algorithm integrates traditional genetic operators with sophisticated local search heuristics to refine the solution space. To assess the efficacy of the EHGA, we performed comprehensive computational experiments using a diverse set of TSP instances sourced from the TSPLIB repository. Our findings indicate that the EHGA is capable of generating high-quality solutions for the TSP within a significantly reduced timeframe, showcasing its superiority in solving complex routing problems.

This paper presents the design of an MPI-based parallel and distributed machine learning platform on large-scale HPC clusters. Researchers and practitioners can implement easily a class of parallelizable machine learning algorithms on the platform, or port quickly an existing non-parallel implementation of a parallelizable algorithm to the platform with only minor modifications. Complicated functions in parallel programming such as scheduling, caching and load balancing are handled automatically by the platform. The platform performance was evaluated in a series of stress tests by using a k-means clustering task on 7,500 hours of speech data (about 2.7 billion 52dimensional feature vectors). Good scalability is demonstrated on an HPC cluster with thousands of CPU cores.

Data-driven applications are required to adapt to the everincreasing volume, velocity and veracity of data generated by a variety of sources including the Web and Internet of Things devices. At the same time, an event-driven computational paradigm is emerging as the core of modern systems designed for both database queries, data analytics and on-demand applications. MapReduce has been generalized to Map Collective and shown to be very effective in machine learning. However one often uses a dataflow computing model, which has been adopted by most major big data processing runtimes. The HPC community has also developed several asynchronous many tasks (AMT) systems according to the dataflow model. From a different point of view, the services community is moving to an increasingly event-driven model where (micro)services are composed of small functions driven by events in the form of Function as a Service(Faas) and serverless computing. Such designs allow the applications to scale quickly as well as be cost effective in cloud environments. An event-driven runtime designed for data processing consists of well-understood components such as communication, scheduling, and fault tolerance. One can make different design decisions for these components that will determine the type of applications a system can support efficiently. We find that modern systems are designed in a monolithic approach with a fixed set of choices that cannot be changed easily afterwards. Because of these design choices their functionality is limited to specific sets of applications. In this paper we study existing systems (candidate event-driven runtimes), the design choices they have made for each component, and how this affects the type of applications they can support. Further we propose a loosely coupled component-based approach for designing a big data toolkit where each component can have different implementations to support various applications. We believe such a polymorphic design would allow services and data analytics to be integrated seamlessly and expand from edge to cloud to high performance computing environments.

Mainstream parallel algorithms for mining frequent itemsets (patterns) were designed by implementing FP-Growth or Apriori algorithms on MapReduce (MR) framework. Existing MR FP-Growth algorithms can not distribute data equally among nodes, and MR Apriori algorithms utilize multiple map/reduce procedures and generate too many key-value pairs with value of 1; these disadvantages hinder their performance. This paper proposes an algorithm FIMMR: it firstly mines local frequent itemsets for each data chunk as candidates, applies prune strategies to the candidates, and then identifies global frequent itemsets from candidates. Experimental results show that the time efficiency of FIMMR outperforms PFP and SPC significantly; and under small minimum support threshold, FIMMR can achieve one order of magnitude improvement than the other two algorithms; meanwhile, the speedup of FIMMR is also satisfactory.

High-speed data transfer is vital to data-intensive computing that often requires moving large data volumes efficiently within a local data center and among geographically dispersed facilities. Effective utilization of the abundant resources in modern multicore environments for data transfer remains a persistent challenge, particularly, for Non-Uniform Memory Access (NUMA) systems wherein the locality of data accessing is an important factor. This requires rethinking how to exploit parallel access to data and to optimize the storage and network I/Os. We address this challenge and present a novel design of asynchronous processing and resource-aware task scheduling in the context of high-throughput data replication. Our software allocates multiple sets of threads to different stages of the processing pipeline, including storage I/O and network communication, based on their capacities. Threads belonging to each stage follow an asynchronous model, and attain high performance via multiple locality-aware and peer-aware mechanisms, such as task grouping, buffer sharing, affinity control and communication protocols. Our design also integrates high performance features to enhance the scalability of data transfer in several scenarios, e.g., file-level sorting, block-level asynchrony, and thread-level pipelining. Our experiments confirm the advantages of our software under different types of workloads and dynamic environments with contention for shared resources, including a 28-160 percent increase in bandwidth for transferring large files, 1.7-66 times speed-up for small files, and up to 108 percent larger throughput for mixed workloads compared with three state of the art alternatives, GridFTP , BBCP and Aspera.

Google's Borg system is a cluster manager that runs hundreds of thousands of jobs, from many thousands of different applications, across a number of clusters each with up to tens of thousands of machines.
 It achieves high utilization by combining admission control, efficient task-packing, over-commitment, and machine sharing with process-level performance isolation. It supports high-availability applications with runtime features that minimize fault-recovery time, and scheduling policies that reduce the probability of correlated failures. Borg simplifies life for its users by offering a declarative job specification language, name service integration, real-time job monitoring, and tools to analyze and simulate system behavior.
 We present a summary of the Borg system architecture and features, important design decisions, a quantitative analysis of some of its policy decisions, and a qualitative examination of lessons learned from a decade of operational experience with it.

The LogicBlox system aims to reduce the complexity of software development for modern applications which enhance and automate decision-making and enable their users to evolve their capabilities via a ``self-service'' model. Our perspective in this area is informed by over twenty years of experience building dozens of mission-critical enterprise applications that are in use by hundreds of large enterprises across industries such as retail, telecommunications, banking, and government. We designed and built LogicBlox to be the system we wished we had when developing those applications.
 In this paper, we discuss the design considerations behind the LogicBlox system and give an overview of its implementation, highlighting innovative aspects. These include: LogiQL, a unified and declarative language based on Datalog; the use of purely functional data structures; novel join processing strategies; advanced incremental maintenance and live programming facilities; a novel concurrency control scheme; and built-in support for prescriptive and predictive analytics.

Over the last two decades, manufacturing across the globe has evolved to be more intel-ligent and data driven. In the age of industrial Internet of Things, a smart production unit can be perceived as a large connected industrial system of materials, parts, machines, tools, inventory, and logistics that can relay data and communicate with each other. While, traditionally, the focus has been on machine health and predictive maintenance, the manufacturing industry has also started focusing on analyzing data from the entire production line. These applications bring a new set of analytics challenges. Unlike tradi-tional data mining analysis, which consists of lean datasets (that is, datasets with few fea-tures), manufacturing has fat datasets. In addition, previous approaches to manufacturing analytics restricted themselves to small time periods of data. The latest advances in big data analytics allows researchers to do a deep dive into years of data. Bosch collects and utilizes all available information about its products to increase its understanding of complex linear and nonlinear relationships between parts, machines, and assembly lines. This helps in use cases such as the discovery of the root cause of internal defects. This article presents a case study and provides detail about challenges and approaches in data extraction, modeling, and visualization.

As association rules widely used, it needs to study many problems, one of which is the generally larger and multi-dimensional datasets, and the rapid growth of the mount of data. Single-processor's memory and CPU resources are very limited, which makes the algorithm performance inefficient. Recently the development of network and distributed technology makes cloud computing a reality in the implementation of association rules algorithm. In this paper we describe the improved Apriori algorithm based on MapReduce mode, which can handle massive datasets with a large number of nodes on Hadoop platform.

This paper proposes a comprehensive critical survey on the issues of warehousing and protecting big data, which are recognized as critical challenges of emerging big data research. Indeed, both are critical aspects to be considered in order to build truly, high-performance and highly-flexible big data management systems. We report on state-of-the-art approaches, methodologies and trends, and finally conclude by providing open problems and challenging research directions to be considered by future efforts.

Big Data has shown lot of potential in real world industry and research community. We support the power and potential of it in solving real world problems. However, it is imperative to understand Big Data through the lens of 7 V's. 7th V as `Value' is desired output for industry challenges and issues. We provide a brief survey study of 7 Vs of Big Data in order to understand Big Data and extract Value concept in general. Finally we conclude by showing our vision of improved healthcare, a product of Big Data Utilization, as a future work for researchers and students, while moving forward.


Question: A high-performance industrial big-data processing engine.

This paper introduces Resilient Distributed Datasets (RDDs), a novel distributed memory abstraction designed to enable in-memory computations on large-scale clusters while maintaining the fault tolerance typically associated with data flow models such as MapReduce. RDDs are particularly well-suited for two classes of applications that are often poorly served by existing data flow systems: iterative algorithms, which are prevalent in graph processing and machine learning, and interactive data mining tasks. For these applications, the ability to keep data in memory can result in significant performance gains, often by an order of magnitude. To ensure efficient fault tolerance, RDDs offer a tightly controlled form of shared memory: they are immutable datasets that can only be created through bulk operations on other RDDs. Despite these restrictions, we demonstrate that RDDs are sufficiently versatile to encompass a broad range of computational tasks, including MapReduce and specialized models for iterative computations akin to Pregel. Our implementation of RDDs shows a remarkable improvement over Hadoop, achieving up to 20 times faster performance for iterative jobs, and it supports interactive queries on a 1 TB dataset with response times as low as 5 to 7 seconds. This showcases the RDD abstraction's ability to enhance efficiency and interactivity in large-scale data processing.

Mayo Clinic (MC) healthcare generates a large number of HL7 V2 messages—0.7–1.1 million on weekends and 1.7–2.2 million on business days at present. With multiple RDBMS-based systems, such a large volume of HL7 messages still cannot be real-time or near-real-time stored, analyzed, and retrieved for enterprise-level clinic and nonclinic usage. To determine if Big Data technology coupled with ElasticSearch technology can satisfy MC daily healthcare needs for HL7 message processing, a BigData platform was developed to contain two identical Hadoop clusters (TDH1.3.2 version)—each containing an ElasticSearch cluster and instances of a storm topology—MayoTopology for processing HL7 messages on MC ESB queues into an ElasticSearch index and the HDFS. The implemented BigData platform can process 62 ± 4 million HL7 messages per day while the ElasticSearch index can provide ultrafast free-text searching at a speed level of 0.2-s per query on an index containing a dataset of 25 million HL7-derived-JSON-documents. The results suggest that the implemented BigData platform exceeds MC enterprise-level patient-care needs.

The Internet of Things (IoT) has provided a promising opportunity to build powerful industrial systems and applications by leveraging the growing ubiquity of Radio Frequency IDentification (RFID) and wireless sensors devices. Benefiting from RFID and sensor network technology, common physical objects can be connected, and are able to be monitored and managed by a single system. Such a network brings a series of challenges for data storage and processing in a cloud platform. IoT data can be generated quite rapidly, the volume of data can be huge and the types of data can be various. In order to address these potential problems, this paper proposes a data storage framework not only enabling efficient storing of massive IoT data, but also integrating both structured and unstructured data. This data storage framework is able to combine and extend multiple databases and Hadoop to store and manage diverse types of data collected by sensors and RFID readers. In addition, some components are developed to extend the Hadoop to realize a distributed file repository, which is able to process massive unstructured files efficiently. A prototype system based on the proposed framework is also developed to illustrate the framework's effectiveness.

In this paper, we characterize Google applications, based on a one-month Google trace with over 650k jobs running across over 12000 heterogeneous hosts from a Google data center. On one hand, we carefully compute the valuable statistics about task events and resource utilization for Google applications, based on various types of resources (such as CPU, memory) and execution types (e.g., whether they can run batch tasks or not). Resource utilization per application is observed with an extremely typical Pareto principle. On the other hand, we classify applications via a K-means clustering algorithm with optimized number of sets, based on task events and resource usage. The number of applications in the K-means clustering sets follows a Pareto-similar distribution. We believe our work is very interesting and valuable for the further investigation of Cloud environment.

Cloud computing promises a number of advantages for the deployment of data-intensive applications. One important promise is reduced cost with a pay-as-you-go business model. Another promise is (virtually) unlimited throughput by adding servers if the workload increases. This paper lists alternative architectures to effect cloud computing for database applications and reports on the results of a comprehensive evaluation of existing commercial cloud services that have adopted these architectures. The focus of this work is on transaction processing (i.e., read and update workloads), rather than analytics or OLAP workloads, which have recently gained a great deal of attention. The results are surprising in several ways. Most importantly, it seems that all major vendors have adopted a different architecture for their cloud services. As a result, the cost and performance of the services vary significantly depending on the workload.

Let's stop trying to be 'sexy' – preparing managers for the (big) data-driven business era Kevin Daniel André Carillo, Article information: To cite this document: Kevin Daniel André Carillo, (2017) "Let's stop trying to be 'sexy' – preparing managers for the (big) data-driven business era", Business Process Management Journal, Vol. 23 Issue: 3, doi: 10.1108/BPMJ-09-2016-0188 Permanent link to this document: http://dx.doi.org/10.1108/BPMJ-09-2016-0188

Traditional database systems offer rich query interfaces (SQL) and efficient query execution for data that they store. Recent years have seen the rise of Big Data analytics platforms offering query-based access to "raw" external data, e.g., file-resident data (often in HDFS). In this paper, we describe techniques to achieve the qualities offered by DBMSs when accessing external data. This work has been built into Apache AsterixDB, an open source Big Data Management System. We describe how we build distributed indexes over external data, partition external indexes, provide query consistency across access paths, and manage external indexes amidst concurrent activities. We compare the performance of this new AsterixDB capability to an external-only solution (Hive) and to its internally managed data and indexes.

The modern enterprise software stack—a collection of applications supporting bookkeeping, analytics, planning, and forecasting for enterprise data—is in danger of collapsing under its own weight. The task of building and maintaining enterprise software is tedious and laborious; applications are cumbersome for end-users; and adapting to new computing hardware and infrastructures is difficult. We believe that much of the complexity in today’s architecture is accidental, rather than inherent. This tutorial provides an overview of the LogicBlox platform, a ambitious redesign of the enterprise software stack centered around a unified declarative programming model, based on an extended version of Datalog. 1 The Enterprise Hairball Modern enterprise applications involve an enormously complex technology stack composed of many disparate systems programmed in a hodgepodge of programming languages. We refer to this stack, depicted in Figure 1, as “the enterprise hairball.” First, there is an online transaction processing (OLTP) layer that performs bookkeeping of the core business data for an enterprise. Such data could include the current product catalog, recent sales figures, current outstanding invoices, customer account balances, and so forth. This OLTP layer typically includes a relational DBMS—programmed in a combination of a query language (SQL), a stored procedure language (like PL/SQL or TSQL), and a batch programming language like Pro*C—an application server, such as Oracle WebLogic [35], IBM WebSphere [36], or SAP NetWeaver [26]—programmed in an object-oriented language like Java, C#, or ABAP—and a web browser front-end, programmed using HTML and Javascript. In order to track the performance of the enterprise over time, a second business intelligence (BI) layer typically holds five to ten years of historical information that was originally recorded in the OLTP layer and performs read-only analyses on this information. This layer typically includes another DBMS (or, more commonly, a BI variant like Teradata [33] or IBM Netezza [25]) along with a BI application server such as Microstrategy [23], SAP BusinessObjects [5], or IBM Cognos [7], programmed using a vendor-specific declarative language. Fig. 1: Enterprise software components and technology stack example. Data is moved from the transaction layer to the BI layer via so-called extracttransform-load (ETL) tools that come with their own tool-specific programming language. Finally, in order to plan the future actions of an enterprise, there is a planning layer, which supports a class of read-write use cases for which the BI layer is unsuitable. This layer typically includes a planning application server, like Oracle Hyperion [13] or IBM Cognos TM1 [34], that are programmed using a vendor-specific declarative language or a standard language like MDX [21], and spreadsheets like Microsoft Excel that are programmed in a vendor-specific formula language (e.g. A1 = B17 D12) and optionally a scripting language like VBA. In order to enhance or automate decisions made in the planning layer, statistical predictive models are often prototyped using modeling tools like SAS, Matlab, SPSS, or R, and then rewritten for production in C++ or Java so they can be embedded in the OLTP or OLAP layers. In summary, enterprise software developed according to the hairball model must be programmed using a dozen or so different programming languages, running in almost as many system-scale components. Some of the languages are imperative (both object-oriented and not) and some are declarative (every possible flavor). Most of the languages used are vendor-specific and tied to the component in which they run (e.g. ABAP, Excel, R, OPL, etc.). Even the languages that are based on open standards are not easily ported from one component to another because of significant variations in performance or because of vendor specific extensions (e.g., the same SQL query on Oracle will perform very differently on DB2). Recent innovations in infrastructure technology for supporting much larger numbers of users (Web applications) and big data (predictive analytics), including NoSQL [28], NewSQL [27] and analytic databases, have introduced even more components into the technology stack described above, and have not helped reduce its overall complexity. This complexity makes enterprise software hard to build, hard to implement, and hard to change. Simple changes—like extending a product identifier by 3 characters or an employee identifier by 1 digit—often necessitate modifications to most of the different components in the stack, requiring thousands of days of person effort and costing many millions of dollars. An extreme example of the problems caused by this accidental complexity occurred in the lead-up to the year 2000, where the simple problem of adding two digits to the year field (the Y2K problem) cost humanity over $300 billion dollars [24]. Moreover, as a result of the time required for such changes, individuals within an enterprise often resort to ad hoc, error-prone methods to perform the calculations needed in order to make timely business decisions. For example, they often roll their own extensions using spreadsheets, where they incompletely or incorrectly implement such calculations based on copies of the data that is not kept in sync with the OLTP database and thus may no longer accurately reflect the state of their enterprise.

Over the last two decades, manufacturing across the globe has evolved to be more intel-ligent and data driven. In the age of industrial Internet of Things, a smart production unit can be perceived as a large connected industrial system of materials, parts, machines, tools, inventory, and logistics that can relay data and communicate with each other. While, traditionally, the focus has been on machine health and predictive maintenance, the manufacturing industry has also started focusing on analyzing data from the entire production line. These applications bring a new set of analytics challenges. Unlike tradi-tional data mining analysis, which consists of lean datasets (that is, datasets with few fea-tures), manufacturing has fat datasets. In addition, previous approaches to manufacturing analytics restricted themselves to small time periods of data. The latest advances in big data analytics allows researchers to do a deep dive into years of data. Bosch collects and utilizes all available information about its products to increase its understanding of complex linear and nonlinear relationships between parts, machines, and assembly lines. This helps in use cases such as the discovery of the root cause of internal defects. This article presents a case study and provides detail about challenges and approaches in data extraction, modeling, and visualization.

Nowadays, cloud-based storage services are rapidly growing and becoming an emerging trend in data storage field. There are many problems when designing an efficient storage engine for cloud-based systems with some requirements such as big-file processing, lightweight meta-data, low latency, parallel I/O, deduplication, distributed, high scalability. Key-value stores played an important role and showed many advantages when solving those problems. This paper presents about Big File Cloud (BFC) with its algorithms and architecture to handle most of problems in a big-file cloud storage system based on key-value store. It is done by proposing low-complicated, fixed-size meta-data design, which supports fast and highly-concurrent, distributed file I/O, several algorithms for resumable upload, download and simple data deduplication method for static data. This research applied the advantages of ZDB - an in-house key-value store which was optimized with auto-increment integer keys for solving big-file storage problems efficiently. The results can be used for building scalable distributed data cloud storage that support big-file with size up to several terabytes.

We describe the design and initial implementation of Chukwa, a data collection system for monitoring and analyzing large distributed systems. Chukwa is built on top of Hadoop, an open source distributed filesystem and MapReduce implementation, and inherits Hadoop’s scalability and robustness. Chukwa also includes a flexible and powerful toolkit for displaying monitoring and analysis results, in order to make the best use of this collected data.


Question: A high-performance industrial big-data processing engine.

This paper introduces BigBench, a comprehensive benchmark proposal aimed at establishing an industry-standard for evaluating and comparing the performance of big data storage and processing systems. Given the growing interest in big data from academia, industry, and a wide user base, and the proliferation of commercial and open-source products designed to support big data operations, there is a pressing need for a benchmark that can objectively assess these systems' capabilities.
BigBench is an end-to-end benchmark that encompasses a data model and a synthetic data generator tailored to address the three key aspects of big data systems: variety, velocity, and volume. It includes structured, semi-structured, and unstructured data components. The structured data model is based on the TPC-DS benchmark, enhanced with additional semi-structured and unstructured data elements. The semi-structured data represents user interactions on a retailer's website, capturing both registered and guest user clicks. The unstructured data comprises online product reviews submitted by customers. The data generator for BigBench is designed to produce scalable volumes of raw data based on a scale factor, allowing for the creation of datasets of varying sizes.
The BigBench workload is centered around a series of queries executed against the data model. These queries are designed to cover the various categories of big data analytics outlined by McKinsey, encompassing business and technical perspectives. From a technical standpoint, the queries span three dimensions: data sources, query processing types, and analytic techniques.
To demonstrate the viability of BigBench, the benchmark has been implemented on the Teradata Aster Database. The test involved generating and loading a 200 Gigabyte BigBench dataset and executing the benchmark queries (written using Teradata Aster SQL-MR) to measure their response times. This implementation showcases the potential of BigBench as a standard benchmark for big data analytics, capable of providing valuable insights into the performance of big data systems.

The modern enterprise software stack—a collection of applications supporting bookkeeping, analytics, planning, and forecasting for enterprise data—is in danger of collapsing under its own weight. The task of building and maintaining enterprise software is tedious and laborious; applications are cumbersome for end-users; and adapting to new computing hardware and infrastructures is difficult. We believe that much of the complexity in today’s architecture is accidental, rather than inherent. This tutorial provides an overview of the LogicBlox platform, a ambitious redesign of the enterprise software stack centered around a unified declarative programming model, based on an extended version of Datalog. 1 The Enterprise Hairball Modern enterprise applications involve an enormously complex technology stack composed of many disparate systems programmed in a hodgepodge of programming languages. We refer to this stack, depicted in Figure 1, as “the enterprise hairball.” First, there is an online transaction processing (OLTP) layer that performs bookkeeping of the core business data for an enterprise. Such data could include the current product catalog, recent sales figures, current outstanding invoices, customer account balances, and so forth. This OLTP layer typically includes a relational DBMS—programmed in a combination of a query language (SQL), a stored procedure language (like PL/SQL or TSQL), and a batch programming language like Pro*C—an application server, such as Oracle WebLogic [35], IBM WebSphere [36], or SAP NetWeaver [26]—programmed in an object-oriented language like Java, C#, or ABAP—and a web browser front-end, programmed using HTML and Javascript. In order to track the performance of the enterprise over time, a second business intelligence (BI) layer typically holds five to ten years of historical information that was originally recorded in the OLTP layer and performs read-only analyses on this information. This layer typically includes another DBMS (or, more commonly, a BI variant like Teradata [33] or IBM Netezza [25]) along with a BI application server such as Microstrategy [23], SAP BusinessObjects [5], or IBM Cognos [7], programmed using a vendor-specific declarative language. Fig. 1: Enterprise software components and technology stack example. Data is moved from the transaction layer to the BI layer via so-called extracttransform-load (ETL) tools that come with their own tool-specific programming language. Finally, in order to plan the future actions of an enterprise, there is a planning layer, which supports a class of read-write use cases for which the BI layer is unsuitable. This layer typically includes a planning application server, like Oracle Hyperion [13] or IBM Cognos TM1 [34], that are programmed using a vendor-specific declarative language or a standard language like MDX [21], and spreadsheets like Microsoft Excel that are programmed in a vendor-specific formula language (e.g. A1 = B17 D12) and optionally a scripting language like VBA. In order to enhance or automate decisions made in the planning layer, statistical predictive models are often prototyped using modeling tools like SAS, Matlab, SPSS, or R, and then rewritten for production in C++ or Java so they can be embedded in the OLTP or OLAP layers. In summary, enterprise software developed according to the hairball model must be programmed using a dozen or so different programming languages, running in almost as many system-scale components. Some of the languages are imperative (both object-oriented and not) and some are declarative (every possible flavor). Most of the languages used are vendor-specific and tied to the component in which they run (e.g. ABAP, Excel, R, OPL, etc.). Even the languages that are based on open standards are not easily ported from one component to another because of significant variations in performance or because of vendor specific extensions (e.g., the same SQL query on Oracle will perform very differently on DB2). Recent innovations in infrastructure technology for supporting much larger numbers of users (Web applications) and big data (predictive analytics), including NoSQL [28], NewSQL [27] and analytic databases, have introduced even more components into the technology stack described above, and have not helped reduce its overall complexity. This complexity makes enterprise software hard to build, hard to implement, and hard to change. Simple changes—like extending a product identifier by 3 characters or an employee identifier by 1 digit—often necessitate modifications to most of the different components in the stack, requiring thousands of days of person effort and costing many millions of dollars. An extreme example of the problems caused by this accidental complexity occurred in the lead-up to the year 2000, where the simple problem of adding two digits to the year field (the Y2K problem) cost humanity over $300 billion dollars [24]. Moreover, as a result of the time required for such changes, individuals within an enterprise often resort to ad hoc, error-prone methods to perform the calculations needed in order to make timely business decisions. For example, they often roll their own extensions using spreadsheets, where they incompletely or incorrectly implement such calculations based on copies of the data that is not kept in sync with the OLTP database and thus may no longer accurately reflect the state of their enterprise.

Data cleaning has become an indispensable part of data analysis due to the increasing amount of dirty data. Data scientists spend most of their time preparing dirty data before it can be used for data analysis. At the same time, the existing tools that attempt to automate the data cleaning procedure typically focus on a specific use case and operation. Still, even such specialized tools exhibit long running times or fail to process large datasets. Therefore, from a user’s perspective, one is forced to use a different, potentially inefficient tool for each category of errors. This paper addresses the coverage and efficiency problems of data cleaning. It introduces CleanM (pronounced clean’em), a language which can express multiple types of cleaning operations. CleanM goes through a three-level translation process for optimization purposes; a different family of optimizations is applied in each abstraction level. Thus, CleanM can express complex data cleaning tasks, optimize them in a unified way, and deploy them in a scaleout fashion. We validate the applicability of CleanM by using it on top of CleanDB, a newly designed and implemented framework which can query heterogeneous data. When compared to existing data cleaning solutions, CleanDB a) covers more data corruption cases, b) scales better, and can handle cases for which its competitors are unable to terminate, and c) uses a single interface for querying and for data cleaning.

Advances in sensing and computing capabilities are making it possible to embed increasing computing power in small devices. This has enabled the sensing devices not just to passively capture data at very high resolution but also to take sophisticated actions in response. Combined with advances in communication, this is resulting in an ecosystem of highly interconnected devices referred to as the Internet of Things IoT. In conjunction, the advances in machine learning have allowed building models on this ever increasing amounts of data. Consequently, devices all the way from heavy assets such as aircraft engines to wearables such as health monitors can all now not only generate massive amounts of data but can draw back on aggregate analytics to “improve” their performance over time. Big data analytics has been identified as a key enabler for the IoT. In this chapter, we discuss various avenues of the IoT where big data analytics either is already making a significant impact or is on the cusp of doing so. We also discuss social implications and areas of concern.

Developers use cloud computing platforms to process a large quantity of data in parallel when developing big data analytics. Debugging the massive parallel computations that run in today's datacenters is time consuming and error-prone. To address this challenge, we design a set of interactive, real-time debugging primitives for big data processing in Apache Spark, the next generation data-intensive scalable cloud computing platform. This requires rethinking the notion of step-through debugging in a traditional debugger such as gdb, because pausing the entire computation across distributed worker nodes causes significant delay and naively inspecting millions of records using a watchpoint is too time consuming for an end user.
 First, BigDebug's simulated breakpoints and on-demand watchpoints allow users to selectively examine distributed, intermediate data on the cloud with little overhead. Second, a user can also pinpoint a crash-inducing record and selectively resume relevant sub-computations after a quick fix. Third, a user can determine the root causes of errors (or delays) at the level of individual records through a fine-grained data provenance capability. Our evaluation shows that BigDebug scales to terabytes and its record-level tracing incurs less than 25% overhead on average. It determines crash culprits orders of magnitude more accurately and provides up to 100% time saving compared to the baseline replay debugger. The results show that BigDebug supports debugging at interactive speeds with minimal performance impact.

Cloud computing promises a number of advantages for the deployment of data-intensive applications. One important promise is reduced cost with a pay-as-you-go business model. Another promise is (virtually) unlimited throughput by adding servers if the workload increases. This paper lists alternative architectures to effect cloud computing for database applications and reports on the results of a comprehensive evaluation of existing commercial cloud services that have adopted these architectures. The focus of this work is on transaction processing (i.e., read and update workloads), rather than analytics or OLAP workloads, which have recently gained a great deal of attention. The results are surprising in several ways. Most importantly, it seems that all major vendors have adopted a different architecture for their cloud services. As a result, the cost and performance of the services vary significantly depending on the workload.

Effective processing of extremely large volumes of spatial data has led to many organizations employing distributed processing frameworks. Apache Spark is one such open-source framework that is enjoying widespread adoption. Within this data space, it is important to note that most of the observational data (i.e., data collected by sensors, either moving or stationary) has a temporal component, or timestamp. In order to perform advanced analytics and gain insights, the temporal component becomes equally important as the spatial and attribute components. In this paper, we detail several variants of a spatial join operation that addresses both spatial, temporal, and attribute-based joins. Our spatial join technique differs from other approaches in that it combines spatial, temporal, and attribute predicates in the join operator.
 In addition, our spatio-temporal join algorithm and implementation differs from others in that it runs in commercial off-the-shelf (COTS) application. The users of this functionality are assumed to be GIS analysts with little if any knowledge of the implementation details of spatio-temporal joins or distributed processing. They are comfortable using simple tools that do not provide the ability to tweak the configuration of the

In many graph applications, the structure of the graph changes dynamically over time and may require real time analysis. However, constructing a large graph is expensive, and most studies for large graphs have not focused on a dynamic graph data structure, but rather a static one. To address this issue, we propose DegAwareRHH, a high performance dynamic graph data store designed for scaling out to store large, scale-free graphs by leveraging compact hash tables with high data locality. We extend DegAwareRHH for multiple processes and distributed memory, and perform dynamic graph construction on large scale-free graphs using emerging 'Big Data HPC' systems such as the Catalyst cluster at LLNL. We demonstrate that DegAwareRHH processes a request stream 206.5x faster than a state-of-the-art shared-memory dynamic graph processing framework, when both implementations use 24 threads/processes to construct a graph with 1 billion edge insertion requests and 54 million edge deletion requests. DegAwareRHH also achieves a processing rate of over 2 billion edge insertion requests per second using 128 compute nodes to construct a large-scale web graph, containing 128 billion edges, the largest open-source real graph dataset to our knowledge.

Hadoop is a MAD system for data analytics  Magnetism: attracts all sources of data  Agility: adapts in sync with rapid data evolution  Depth: supports complex analytics needs Starfish makes Hadoop MADDER and Self-Tuning  Data-lifecycle-awareness: achieves good performance throughout data lifecycle  Elasticity: adjusts resources and operational costs  Robustness: provides availability and predictability Just-in-Time Job Optimization

We found that interactive services at Bing have highly variable datacenter-side processing latencies because their processing consists of many sequential stages, parallelization across 10s-1000s of servers and aggregation of responses across the network. To improve the tail latency of such services, we use a few building blocks: reissuing laggards elsewhere in the cluster, new policies to return incomplete results and speeding up laggards by giving them more resources. Combining these building blocks to reduce the overall latency is non-trivial because for the same amount of resource (e.g., number of reissues), different stages improve their latency by different amounts. We present Kwiken, a framework that takes an end-to-end view of latency improvements and costs. It decomposes the problem of minimizing latency over a general processing DAG into a manageable optimization over individual stages. Through simulations with production traces, we show sizable gains; the 99th percentile of latency improves by over 50% when just 0.1% of the responses are allowed to have partial results and by over 40% for 25% of the services when just 5% extra resources are used for reissues.

Several domain-specific languages (DSLs) for parallel graph analytics have been proposed recently. In this paper, we argue that existing DSLs can be implemented on top of a general-purpose infrastructure that (i) supports very fine-grain tasks, (ii) implements autonomous, speculative execution of these tasks, and (iii) allows application-specific control of task scheduling policies. To support this claim, we describe such an implementation called the Galois system.
 We demonstrate the capabilities of this infrastructure in three ways. First, we implement more sophisticated algorithms for some of the graph analytics problems tackled by previous DSLs and show that end-to-end performance can be improved by orders of magnitude even on power-law graphs, thanks to the better algorithms facilitated by a more general programming model. Second, we show that, even when an algorithm can be expressed in existing DSLs, the implementation of that algorithm in the more general system can be orders of magnitude faster when the input graphs are road networks and similar graphs with high diameter, thanks to more sophisticated scheduling. Third, we implement the APIs of three existing graph DSLs on top of the common infrastructure in a few hundred lines of code and show that even for power-law graphs, the performance of the resulting implementations often exceeds that of the original DSL systems, thanks to the lightweight infrastructure.


Question: A high-performance engine for managing industrial big data.

Exploring the Potential of Big Data Analytics in Healthcare: Challenges and Opportunities

OBJECTIVE
This paper aims to outline the potential and emerging trends of big data analytics within the healthcare sector, emphasizing its transformative impact on patient outcomes and cost efficiency.

METHODS
The paper provides a comprehensive review of the nascent field of big data analytics in healthcare. It highlights the benefits, proposes a conceptual architectural framework, and discusses methodologies for data analysis. Additionally, it presents case studies from existing literature, briefly touches upon the challenges, and concludes with insights on future directions.

RESULTS
The paper offers a detailed understanding of big data analytics for healthcare professionals and researchers, showcasing how it can significantly improve decision-making processes and patient care.

CONCLUSIONS
Big data analytics in healthcare is rapidly becoming a pivotal tool for extracting valuable insights from vast datasets, leading to improved healthcare outcomes and reduced costs. Despite its immense potential, the field faces significant challenges that need to be addressed to fully harness its capabilities.

Big data analytics associated with database searching, mining, and analysis can be seen as an innovative IT capability that can improve firm performance. Even though some leading companies are actively adopting big data analytics to strengthen market competition and to open up new business opportunities, many firms are still in the early stage of the adoption curve due to lack of understanding of and experience with big data. Hence, it is interesting and timely to understand issues relevant to big data adoption. In this study, a research model is proposed to explain the acquisition intention of big data analytics

Massive data sequence collections exist in virtually every scientific and social domain, and have to be analyzed to extract useful knowledge. However, no existing data management solution (such as relational databases, column stores, array databases, and time series management systems) can offer native support for sequences and the corresponding operators necessary for complex analytics. We argue for the need to study the theory and foundations for sequence management of big data sequences, and to build corresponding systems that will enable scalable management and analysis of very large sequence collections. To this effect, we need to develop novel techniques to efficiently support a wide range of sequence queries and mining operations, while leveraging modern hardware. The overall goal is to allow analysts across domains to tap in the goldmine of the massive and ever-growing sequence collections they (already) have.

Datacenter workloads demand high computational capabilities, flexibility, power efficiency, and low cost. It is challenging to improve all of these factors simultaneously. To advance datacenter capabilities beyond what commodity server designs can provide, we designed and built a composable, reconfigurable hardware fabric based on field programmable gate arrays (FPGA). Each server in the fabric contains one FPGA, and all FPGAs within a 48-server rack are interconnected over a low-latency, high-bandwidth network.
 We describe a medium-scale deployment of this fabric on a bed of 1632 servers, and measure its effectiveness in accelerating the ranking component of the Bing web search engine. We describe the requirements and architecture of the system, detail the critical engineering challenges and solutions needed to make the system robust in the presence of failures, and measure the performance, power, and resilience of the system. Under high load, the large-scale reconfigurable fabric improves the ranking throughput of each server by 95% at a desirable latency distribution or reduces tail latency by 29% at a fixed throughput. In other words, the reconfigurable fabric enables the same throughput using only half the number of servers.

An evaluation of recent industrial and societal trends revealed three key factors driving software engineering's future: speed, data, and ecosystems. These factors' implications have led to guidelines for companies to evolve their software engineering practices. This article is part of a special issue on the Future of Software Engineering.

During the last decade, a lot of progress has happened in the field of autonomous embedded devices (also known as motes) and wireless sensor networks. In this context, we envision next-generation systems to have autonomous database management functionalities on each mote, for supporting easy management of data on flash disks, adequate querying capabilities, and ubiquitous computing with no-effort. Following this major vision, in this paper we describe application scenarios, middleware approach and data management algorithms of a novel system, called Stream Operation (StreamOp), which effectively and efficiently realizes the depicted challenges. In particular, StreamOp supports heterogeneity, i.e. It works on top of different platforms, efficiency, i.e. It returns responses quickly, and autonomy, i.e. It saves battery power. We show how StreamOp provides these features, along with some experimental results that clearly validate our main assumptions.

Unbounded, unordered, global-scale datasets are increasingly common in day-to-day business (e.g. Web logs, mobile usage statistics, and sensor networks). At the same time, consumers of these datasets have evolved sophisticated requirements, such as event-time ordering and windowing by features of the data themselves, in addition to an insatiable hunger for faster answers. Meanwhile, practicality dictates that one can never fully optimize along all dimensions of correctness, latency, and cost for these types of input. As a result, data processing practitioners are left with the quandary of how to reconcile the tensions between these seemingly competing propositions, often resulting in disparate implementations and systems. We propose that a fundamental shift of approach is necessary to deal with these evolved requirements in modern data processing. We as a field must stop trying to groom unbounded datasets into finite pools of information that eventually become complete, and instead live and breathe under the assumption that we will never know if or when we have seen all of our data, only that new data will arrive, old data may be retracted, and the only way to make this problem tractable is via principled abstractions that allow the practitioner the choice of appropriate tradeoffs along the axes of interest: correctness, latency, and cost. In this paper, we present one such approach, the Dataflow Model, along with a detailed examination of the semantics it enables, an overview of the core principles that guided its design, and a validation of the model itself via the real-world experiences that led to its development. We use the term “Dataflow Model” to describe the processing model of Google Cloud Dataflow [20], which is based upon technology from FlumeJava [12] and MillWheel [2]. This work is licensed under the Creative Commons AttributionNonCommercial-NoDerivs 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/. Obtain permission prior to any use beyond those covered by the license. Contact copyright holder by emailing info@vldb.org. Articles from this volume were invited to present their results at the 41st International Conference on Very Large Data Bases, August 31st September 4th 2015, Kohala Coast, Hawaii. Proceedings of the VLDB Endowment, Vol. 8, No. 12 Copyright 2015 VLDB Endowment 2150-8097/15/08.

Today, we live in a digital universe in which information and technology are not only around us but also play important roles in dictating the quality of our lives. As we delve deeper into this digital universe, we're witnessing explosive growth in the variety, velocity, and volume of data1,2 being transmitted over the Internet. A zetabyte of data passed through the Internet in the past year; IDC predicts that this digital universe will explode to an unimaginable eight Zbytes by 2015. These data are and will be generated mainly from Internet search, social media, mobile devices, the Internet of Things, business transactions, next-generation radio astronomy telescopes, high-energy physics synchrotron, and content distribution. Government and business organizations are now overflowing with data, easily aggregating to terabytes or even petabytes of information.

The modern enterprise software stack—a collection of applications supporting bookkeeping, analytics, planning, and forecasting for enterprise data—is in danger of collapsing under its own weight. The task of building and maintaining enterprise software is tedious and laborious; applications are cumbersome for end-users; and adapting to new computing hardware and infrastructures is difficult. We believe that much of the complexity in today’s architecture is accidental, rather than inherent. This tutorial provides an overview of the LogicBlox platform, a ambitious redesign of the enterprise software stack centered around a unified declarative programming model, based on an extended version of Datalog. 1 The Enterprise Hairball Modern enterprise applications involve an enormously complex technology stack composed of many disparate systems programmed in a hodgepodge of programming languages. We refer to this stack, depicted in Figure 1, as “the enterprise hairball.” First, there is an online transaction processing (OLTP) layer that performs bookkeeping of the core business data for an enterprise. Such data could include the current product catalog, recent sales figures, current outstanding invoices, customer account balances, and so forth. This OLTP layer typically includes a relational DBMS—programmed in a combination of a query language (SQL), a stored procedure language (like PL/SQL or TSQL), and a batch programming language like Pro*C—an application server, such as Oracle WebLogic [35], IBM WebSphere [36], or SAP NetWeaver [26]—programmed in an object-oriented language like Java, C#, or ABAP—and a web browser front-end, programmed using HTML and Javascript. In order to track the performance of the enterprise over time, a second business intelligence (BI) layer typically holds five to ten years of historical information that was originally recorded in the OLTP layer and performs read-only analyses on this information. This layer typically includes another DBMS (or, more commonly, a BI variant like Teradata [33] or IBM Netezza [25]) along with a BI application server such as Microstrategy [23], SAP BusinessObjects [5], or IBM Cognos [7], programmed using a vendor-specific declarative language. Fig. 1: Enterprise software components and technology stack example. Data is moved from the transaction layer to the BI layer via so-called extracttransform-load (ETL) tools that come with their own tool-specific programming language. Finally, in order to plan the future actions of an enterprise, there is a planning layer, which supports a class of read-write use cases for which the BI layer is unsuitable. This layer typically includes a planning application server, like Oracle Hyperion [13] or IBM Cognos TM1 [34], that are programmed using a vendor-specific declarative language or a standard language like MDX [21], and spreadsheets like Microsoft Excel that are programmed in a vendor-specific formula language (e.g. A1 = B17 D12) and optionally a scripting language like VBA. In order to enhance or automate decisions made in the planning layer, statistical predictive models are often prototyped using modeling tools like SAS, Matlab, SPSS, or R, and then rewritten for production in C++ or Java so they can be embedded in the OLTP or OLAP layers. In summary, enterprise software developed according to the hairball model must be programmed using a dozen or so different programming languages, running in almost as many system-scale components. Some of the languages are imperative (both object-oriented and not) and some are declarative (every possible flavor). Most of the languages used are vendor-specific and tied to the component in which they run (e.g. ABAP, Excel, R, OPL, etc.). Even the languages that are based on open standards are not easily ported from one component to another because of significant variations in performance or because of vendor specific extensions (e.g., the same SQL query on Oracle will perform very differently on DB2). Recent innovations in infrastructure technology for supporting much larger numbers of users (Web applications) and big data (predictive analytics), including NoSQL [28], NewSQL [27] and analytic databases, have introduced even more components into the technology stack described above, and have not helped reduce its overall complexity. This complexity makes enterprise software hard to build, hard to implement, and hard to change. Simple changes—like extending a product identifier by 3 characters or an employee identifier by 1 digit—often necessitate modifications to most of the different components in the stack, requiring thousands of days of person effort and costing many millions of dollars. An extreme example of the problems caused by this accidental complexity occurred in the lead-up to the year 2000, where the simple problem of adding two digits to the year field (the Y2K problem) cost humanity over $300 billion dollars [24]. Moreover, as a result of the time required for such changes, individuals within an enterprise often resort to ad hoc, error-prone methods to perform the calculations needed in order to make timely business decisions. For example, they often roll their own extensions using spreadsheets, where they incompletely or incorrectly implement such calculations based on copies of the data that is not kept in sync with the OLTP database and thus may no longer accurately reflect the state of their enterprise.

RDF is a data model for schema-free structured information that is gaining momentum in the context of Semantic-Web data, life sciences, and also Web 2.0 platforms. The “pay-as-you-go” nature of RDF and the flexible pattern-matching capabilities of its query language SPARQL entail efficiency and scalability challenges for complex queries including long join paths. This paper presents the RDF-3X engine, an implementation of SPARQL that achieves excellent performance by pursuing a RISC-style architecture with streamlined indexing and query processing. The physical design is identical for all RDF-3X databases regardless of their workloads, and completely eliminates the need for index tuning by exhaustive indexes for all permutations of subject-property-object triples and their binary and unary projections. These indexes are highly compressed, and the query processor can aggressively leverage fast merge joins with excellent performance of processor caches. The query optimizer is able to choose optimal join orders even for complex queries, with a cost model that includes statistical synopses for entire join paths. Although RDF-3X is optimized for queries, it also provides good support for efficient online updates by means of a staging architecture: direct updates to the main database indexes are deferred, and instead applied to compact differential indexes which are later merged into the main indexes in a batched manner. Experimental studies with several large-scale datasets with more than 50 million RDF triples and benchmark queries that include pattern matching, manyway star-joins, and long path-joins demonstrate that RDF-3X can outperform the previously best alternatives by one or two orders of magnitude.

In recent years, progress in hardware technology has resulted in the possibility of monitoring many events in real time. The volume of incoming data may be so large, that monitoring all individual data might be intractable. Revisiting any particular record can also be impossible in this environment. Therefore, many database schemes, such as aggregation, join, frequent pattern mining, and indexing, become more challenging in this context. This paper surveys the previous efforts to resolve these issues in processing data streams. The emphasis is on specifying and processing sliding window queries, which are supported in many stream processing engines. We also review the related work on stream query processing, including synopsis structures, plan sharing, operator scheduling, load shedding, and disorder control. Category: Ubiquitous computing