Question: GHT: a geographic hash table for storage in data-centric systems.

This paper delves into Directed Diffusion, a robust and scalable communication model tailored for sensor networks. Enabled by advancements in processor, memory, and radio technologies, miniature and cost-effective nodes are now capable of sensing, communicating, and computing. These nodes, when interconnected, can collaboratively monitor environmental conditions. Directed Diffusion facilitates this coordination by adopting a data-centric communication approach, where all exchanges are focused on named data. Unlike traditional networks, every node in a Directed Diffusion network is application-aware, which empowers the system to conserve energy through the selection of efficient paths, in-network caching, and processing of data. The paper investigates and assesses the application of Directed Diffusion within a rudimentary remote-surveillance sensor network, demonstrating its capabilities and benefits.

The bloom filter is a probabilistic data structure that provides a compact representation of a set of elements. To keep false positive probabilities low, the size of the bloom filter must be dimensioned a priori to be linear in the maximum number of keys inserted, with the linearity constant ranging typically from one to few bytes. A bloom filter is most commonly used as an in memory data structure, hence its size is limited by the availability of RAM space on the machine. As datasets have grown over time to Internet scale, so have the RAM space requirements of bloom filters. If sufficient RAM space is not available, we advocate that flash memory may serve as a suitable medium for storing bloom filters, since it is about one-tenth the cost of RAM per GB while still providing access times orders of magnitude faster than hard disk. We present BLOOMFLASH, a bloom filter designed for flash memory based storage, that provides a new dimension of trade off with bloom filter access times to reduce RAM space usage (and hence system cost). The simple design of a single flat bloom filter on flash suffers from many performance bottlenecks, including in-place bit updates that are inefficient on flash and multiple reads and random writes spread out across many flash pages for a single lookup or insert operation. To mitigate these performance bottlenecks, BLOOMFLASH leverages two key design innovations: (i) buffering bit updates in RAM and applying them in bulk to flash that helps to reduce random writes to flash, and (ii) a hierarchical bloom filter design consisting of component bloom filters, stored one per flash page, that helps to localize reads and writes on flash. We use two real-world data traces taken from representative bloom filter applications to drive and evaluate our design. BLOOMFLASH achieves bloom filter access times in the range of few tens of microseconds, thus allowing up to order of tens of thousands operations per sec.

Cloud data owners prefer to outsource documents in an encrypted form for the purpose of privacy preserving. Therefore it is essential to develop efficient and reliable ciphertext search techniques. One challenge is that the relationship between documents will be normally concealed in the process of encryption, which will lead to significant search accuracy performance degradation. Also the volume of data in data centers has experienced a dramatic growth. This will make it even more challenging to design ciphertext search schemes that can provide efficient and reliable online information retrieval on large volume of encrypted data. In this paper, a hierarchical clustering method is proposed to support more search semantics and also to meet the demand for fast ciphertext search within a big data environment. The proposed hierarchical approach clusters the documents based on the minimum relevance threshold, and then partitions the resulting clusters into sub-clusters until the constraint on the maximum size of cluster is reached. In the search phase, this approach can reach a linear computational complexity against an exponential size increase of document collection. In order to verify the authenticity of search results, a structure called minimum hash sub-tree is designed in this paper. Experiments have been conducted using the collection set built from the IEEE Xplore. The results show that with a sharp increase of documents in the dataset the search time of the proposed method increases linearly whereas the search time of the traditional method increases exponentially. Furthermore, the proposed method has an advantage over the traditional method in the rank privacy and relevance of retrieved documents.

Many applications require finding objects closest to a specified location that contains a set of keywords. For example, online yellow pages allow users to specify an address and a set of keywords. In return, the user obtains a list of businesses whose description contains these keywords, ordered by their distance from the specified address. The problems of nearest neighbor search on spatial data and keyword search on text data have been extensively studied separately. However, to the best of our knowledge there is no efficient method to answer spatial keyword queries, that is, queries that specify both a location and a set of keywords. In this work, we present an efficient method to answer top-k spatial keyword queries. To do so, we introduce an indexing structure called IR2-Tree (Information Retrieval R-Tree) which combines an R-Tree with superimposed text signatures. We present algorithms that construct and maintain an IR2-Tree, and use it to answer top-k spatial keyword queries. Our algorithms are experimentally compared to current methods and are shown to have superior performance and excellent scalability.

In this paper we exhibit several new classes of hash functions with certain desirable properties, and introduce two novel applications for hashing which make use of these functions. One class contains a small number of functions, yet is almost universal,. If the functions hash n-bit long names into m-bit indices, then specifying a member of the class requires only O((m + log, log,(n)) . log,(n)) bits as compared to O(n) bits for earlier techniques. For long names, this is about a factor of m larger than the lower bound of m + log, n -log, m bits. An application of this class is a provably secure authentication technique for sending messages over insecure lines. A second class of functions satisfies a much stronger property than universal,. We present the application of testing sets for equality. The authentication technique allows the receiver to be certain that a message is genuine. An “enemy”-even one with infinite computer resources-cannot forge or modify a message without detection. The set equality technique allows operations including “add member to set,” “delete member from set” and “test two sets for equality” to be performed in expected constant time and with less than a specified probability of error.

In-memory tree structured index search is a fundamental database operation. Modern processors provide tremendous computing power by integrating multiple cores, each with wide vector units. There has been much work to exploit modern processor architectures for database primitives like scan, sort, join and aggregation. However, unlike other primitives, tree search presents significant challenges due to irregular and unpredictable data accesses in tree traversal.
 In this paper, we present FAST, an extremely fast architecture sensitive layout of the index tree. FAST is a binary tree logically organized to optimize for architecture features like page size, cache line size, and SIMD width of the underlying hardware. FAST eliminates impact of memory latency, and exploits thread-level and datalevel parallelism on both CPUs and GPUs to achieve 50 million (CPU) and 85 million (GPU) queries per second, 5X (CPU) and 1.7X (GPU) faster than the best previously reported performance on the same architectures. FAST supports efficient bulk updates by rebuilding index trees in less than 0.1 seconds for datasets as large as 64Mkeys and naturally integrates compression techniques, overcoming the memory bandwidth bottleneck and achieving a 6X performance improvement over uncompressed index search for large keys on CPUs.

Geography and social relationships are inextricably intertwined; the people we interact with on a daily basis almost always live near us. As people spend more time online, data regarding these two dimensions -- geography and social relationships -- are becoming increasingly precise, allowing us to build reliable models to describe their interaction. These models have important implications in the design of location-based services, security intrusion detection, and social media supporting local communities.
 Using user-supplied address data and the network of associations between members of the Facebook social network, we can directly observe and measure the relationship between geography and friendship. Using these measurements, we introduce an algorithm that predicts the location of an individual from a sparse set of located users with performance that exceeds IP-based geolocation. This algorithm is efficient and scalable, and could be run on a network containing hundreds of millions of users.

Chain replication is a new approach to coordinating clusters of fail-stop storage servers. The approach is intended for supporting large-scale storage services that exhibit high throughput and availability without sacrificing strong consistency guarantees. Besides outlining the chain replication protocols themselves, simulation experiments explore the performance characteristics of a prototype implementation. Throughput, availability, and several objectplacement strategies (including schemes based on distributed hash table routing) are discussed.

This article discusses the challenges for Database Management in the Internet of Things. We provide scenarios to illustrate the new world that will be produced by the Internet of Things, where physical objects are fully integrated into the information highway. We discuss the different types of data that will be part of the Internet of Things. These include identification, positional, environmental, historical, and descriptive data. We consider the challenges brought by the need to manage vast quantities of data across heterogeneous systems. In particular, we consider the areas of querying, indexing, process modeling, transaction handling, and integration of heterogeneous systems. We refer to the earlier work that might provide solutions for these challenges. Finally we discuss a road map for the Internet of Things and respective technical priorities.

Many public knowledge bases are represented and stored as RDF graphs, where users can issue structured queries on such graphs using SPARQL. With massive queries over large and constantly growing RDF data, it is imperative that an RDF graph store should provide low latency and high throughput for concurrent query processing. However, prior systems still experience high perquery latency over large datasets and most prior designs have poor resource utilization such that each query is processed in sequence. We present Wukong, a distributed graph-based RDF store that leverages RDMA-based graph exploration to provide highly concurrent and low-latency queries over large data sets. Wukong is novel in three ways. First, Wukong provides an RDMA-friendly distributed key/value store that provides differentiated encoding and fine-grained partitioning of graph data to reduce RDMA transfers. Second, Wukong leverages full-history pruning to avoid the cost of expensive final join operations, based on the observation that the cost of one-sided RDMA operations is largely oblivious to the payload size to a certain extent. Third, countering conventional wisdom of preferring migration of execution over data, Wukong seamlessly combines data migration for low latency and execution distribution for high throughput by leveraging the low latency and high throughput of onesided RDMA operations, and proposes a worker-obliger model for efficient load balancing. Evaluation on a 6-node RDMA-capable cluster shows that Wukong significantly outperforms state-of-the-art systems like TriAD and Trinity.RDF for both latency and throughput, usually at the scale of orders of magnitude. Short for Sun Wukong, who is known as the Monkey King and is a main character in the Chinese classical novel “Journey to the West”. Since Wukong is known for his extremely fast speed (21,675 kilometers in one somersault) and the ability to fork himself to do massive multi-tasking, we term our system as Wukong. The source code and a brief instruction on how to install Wukong is available at http://ipads.se.sjtu.edu.cn/projects/wukong.

With the advancement of information and communication technology, data are being generated at an exponential rate via various instruments and collected at an unprecedented scale. Such large volume of data generated is referred to as big data, which now are revolutionizing all aspects of our life ranging from enterprises to individuals, from science communities to governments, as they exhibit great potentials to improve efficiency of enterprises and the quality of life. To obtain nontrivial patterns and derive valuable information from big data, a fundamental problem is how to properly place the collected data by different users to distributed clouds and to efficiently analyze the collected data to save user costs in data storage and processing, particularly the cost savings of users who share data. By doing so, it needs the close collaborations among the users, by sharing and utilizing the big data in distributed clouds due to the complexity and volume of big data. Since computing, storage and bandwidth resources in a distributed cloud usually are limited, and such resource provisioning typically is expensive, the collaborative users require to make use of the resources fairly. In this paper, we study a novel collaboration- and fairness-aware big data management problem in distributed cloud environments that aims to maximize the system throughout, while minimizing the operational cost of service providers to achieve the system throughput, subject to resource capacity and user fairness constraints. We first propose a novel optimization framework for the problem. We then devise a fast yet scalable approximation algorithm based on the built optimization framework. We also analyze the time complexity and approximation ratio of the proposed algorithm. We finally conduct experiments by simulations to evaluate the performance of the proposed algorithm. Experimental results demonstrate that the proposed algorithm is promising, and outperforms other heuristics.


Question: GHT: a geographic hashing technique for data-centric storage systems.

This paper introduces a novel routing protocol designed for ad hoc networks, which are temporary networks established by a group of wireless mobile devices without relying on pre-existing infrastructure or centralized administration. In these dynamic environments, mobile devices often require the assistance of other devices to forward packets to their intended destinations due to the limited transmission range of each device. The proposed protocol utilizes dynamic source routing, which is adept at swiftly adapting to changes in the network topology caused by frequent device movement. During periods of lower mobility, the protocol minimizes overhead, making it efficient and responsive to varying conditions. Through extensive simulations of packet-level data in mobile ad hoc networks, the protocol demonstrates robust performance across different scenarios, including varying host densities and movement rates. For most conditions, except the highest movement rates, the protocol's overhead is significantly low, reaching just 1% of the total data packets transmitted in a network of 24 mobile hosts at moderate movement rates. Moreover, the protocol ensures that the routes chosen are nearly optimal; in most cases, the average route length is within a factor of 1.01 of the optimal length, making it a highly effective solution for ad hoc network routing.

Modern large-scale heterogeneous computers incorporating GPUs offer impressive processing capabilities. It is desirable to fully utilize such systems for serving multiple users concurrently to visualize large data at interactive rates. However, as the disparity between data transfer speed and compute speed continues to increase in heterogeneous systems, data locality becomes crucial for performance. We present a new job scheduling design to support multi-user exploration of large data in a heterogeneous computing environment, achieving near optimal data locality and minimizing I/O overhead. The targeted application is a parallel visualization system which allows multiple users to render large volumetric data sets in both interactive mode and batch mode. We present a cost model to assess the performance of parallel volume rendering and quantify the efficiency of job scheduling. We have tested our job scheduling scheme on two heterogeneous systems with different configurations. The largest test volume data used in our study has over two billion grid points. The timing results demonstrate that our design effectively improves data locality for complex multi-user job scheduling problems, leading to better overall performance of the service.

The growing volume of Remote Sensing (RS) image archives demands for feature learning techniques and hashing functions which can: (1) accurately represent the semantics in the RS images; and (2) have quasi real-time performance during retrieval. This paper aims to address both challenges at the same time, by learning a semantic-based metric space for content based RS image retrieval while simultaneously producing binary hash codes for an efficient archive search. This double goal is achieved by training a deep network using a combination of different loss functions which, on the one hand, aim at clustering semantically similar samples (i.e., images), and, on the other hand, encourage the network to produce final activation values (i.e., descriptors) that can be easily binarized. Moreover, since RS annotated training images are too few to train a deep network from scratch, we propose to split the image representation problem in two different phases. In the first we use a general-purpose, pre-trained network to produce an intermediate representation, and in the second we train our hashing network using a relatively small set of training images. Experiments on two aerial benchmark archives show that the proposed method outperforms previous state-of-the-art hashing approaches by up to 5.4% using the same number of hash bits per image.

Heavy hitters are data items that occur at high frequency in a data set. They are among the most important items for an organization to summarize and understand during analytical processing. In data sets with sufficient skew, the number of heavy hitters can be relatively small. We take advantage of this small footprint to compute aggregate functions for the heavy hitters in fast cache memory in a single pass.
 We design cache-resident, shared-nothing structures that hold only the most frequent elements. Our algorithm works in three phases. It first samples and picks heavy hitter candidates. It then builds a hash table and computes the exact aggregates of these elements. Finally, a validation step identifies the true heavy hitters from among the candidates.
 We identify trade-offs between the hash table configuration and performance. Configurations consist of the probing algorithm and the table capacity that determines how many candidates can be aggregated. The probing algorithm can be perfect hashing, cuckoo hashing and bucketized hashing to explore trade-offs between size and speed.
 We optimize performance by the use of SIMD instructions, utilized in novel ways beyond single vectorized operations, to minimize cache accesses and the instruction footprint.

Reliability at massive scale is one of the biggest challenges we face at Amazon.com, one of the largest e-commerce operations in the world; even the slightest outage has significant financial consequences and impacts customer trust. The Amazon.com platform, which provides services for many web sites worldwide, is implemented on top of an infrastructure of tens of thousands of servers and network components located in many datacenters around the world. At this scale, small and large components fail continuously and the way persistent state is managed in the face of these failures drives the reliability and scalability of the software systems.
 This paper presents the design and implementation of Dynamo, a highly available key-value storage system that some of Amazon's core services use to provide an "always-on" experience. To achieve this level of availability, Dynamo sacrifices consistency under certain failure scenarios. It makes extensive use of object versioning and application-assisted conflict resolution in a manner that provides a novel interface for developers to use.

This work addresses a novel spatial keyword query called the m-closest keywords (mCK) query. Given a database of spatial objects, each tuple is associated with some descriptive information represented in the form of keywords. The mCK query aims to find the spatially closest tuples which match m user-specified keywords. Given a set of keywords from a document, mCK query can be very useful in geotagging the document by comparing the keywords to other geotagged documents in a database. To answer mCK queries efficiently, we introduce a new index called the bR*-tree, which is an extension of the R*-tree. Based on bR*-tree, we exploit a priori-based search strategies to effectively reduce the search space. We also propose two monotone constraints, namely the distance mutex and keyword mutex, as our a priori properties to facilitate effective pruning. Our performance study demonstrates that our search strategy is indeed efficient in reducing query response time and demonstrates remarkable scalability in terms of the number of query keywords which is essential for our main application of searching by document.

Clustering as one of the main branches of data mining, has gained an important place in the different applied fields. On the other hand, Peer-to-Peer (P2P) networks with features such as simplicity, low cost communication, and high availability resources, have gained a worldwide popularity in the present days. In P2P network, high volumes of data are distributed between dispersed data sources. In such distributed systems, sending data to a central site is not possible, due to processing, storage and handling costs. This issue makes the need for specific methods for distributed data clustering in P2P networks. This paper makes a technical review of distributed data clustering approaches in P2P network, to understand the research trends in this area. Keywords—Peer to Peer Networks, Distributed Data Mining, Distributed Data Clustering

In the emerging Internet of Things rich data on real-world objects and events will be generated in vast amounts and stored in widely distributed databases. In truly global and dynamic application scenarios, intermediate brokers are needed to find these data, even if the exact location and form of storage are initially unknown to the requester. Discovery Services are aimed to fill this gap: they respond to requests for data on specific objects with a list of corresponding data providers. In this paper, we frame functional requirements for Discovery Services, and perform an overview and analysis of five established approaches for implementing Discovery Services that are taken from literature and industrial practice. In order to compare their characteristics, we develop a quality framework based on literature review and an ISO standard for software quality.

In this paper we present an algorithm which automatically extracts geopositional coordinate index terms from text to support georeferenced document indexing and retrieval. Under this algorithm, words and phrases containing geographic place names or characteristics are extracted from a text document and used as input to database functions which use spatial reasoning to approximate statistically the geoposition being referenced in the text. We conclude with a discussion of preliminary results and future work.

Relational database systems have been the dominating technology to manage and analyze large data warehouses. Moreover, the ER model, the standard in database design has a close relationship with the relational model. Recently, there has been a surge of alternative technologies for large scale analytic processing, most of which are not based on the relational model. Out of these proposals, distributed file systems together with MapReduce have become strong competitors to relational database systems to analyze large data sets, exploiting parallel processing. Moreover, there is progress on using MapReduce to evaluate relational queries. With that motivation in mind, this panel will compare pros and cons of each technology for data warehousing and will identify research issues, considering practical aspects like ease of use, programming flexibility and cost; as well as technical aspects like data modeling, storage, hardware, scalability, query processing, fault tolerance and data mining.

The chunk-lookup disk bottleneck and the read amplification problems are two great challenges for deduplication-based storage systems and restrict the applicability of data deduplication for large-scale data volumes. Previous studies and our experimental evaluations have shown that the amount of redundant data shared among different types of applications is negligible. Based on the observations, we propose AA-Plus which effectively groups the hash index of the same application together and divides the whole hash index into different groups based on the application types. Moreover, it groups the data chunks of the same application together on the disks. The extensive trace-driven experiments conducted on our lightweight prototype implementation of AA-Plus show that compared with AA-Dedupe, AA-Plus significantly speeds up the write throughput by a factor of up to 6.9 and with an average of 3.1, and speeds up the read throughput by a factor of up to 3.3 and with an average of 1.9.


Question: GHT: A geographic hash table for storage in a data-centric approach.

This paper introduces the design, implementation, and evaluation of a location-support system called Cricket, specifically tailored for indoor environments to support mobile, location-based applications. Cricket facilitates location awareness for both mobile and stationary devices through a network of beacons strategically placed within the building. These beacons transmit information that listeners on the devices can hear and analyze to determine their physical location. The design of Cricket is guided by several key objectives, including the preservation of user privacy, decentralized governance, compatibility with diverse network infrastructures, and cost-effectiveness. Instead of actively tracking user movements, Cricket empowers devices to independently ascertain their location, leaving the decision on information sharing to the user. It operates without the need for centralized management or beacon coordination, and it serves devices irrespective of their network connectivity type. Each Cricket device is assembled using readily available components, keeping the cost under U.S. $10. The paper delves into the randomized beacon transmission algorithm, the innovative use of radio and ultrasonic signals to estimate distances, the listener algorithms designed to counteract multipath effects and interference, as well as practical strategies for beacon placement and configuration to enhance location accuracy. Our practical experience with Cricket reveals its potential to effortlessly enable location-dependent applications such as in-building navigation maps and device management, with minimal manual setup or effort.

We propose an algorithm for the hierarchical aggregation of observations in dissemination-based, distributed traffic information systems. Instead of carrying specific values (e.g., the number offree parking places in a given area), our aggregates contain a modified Flajolet-Martin sketch as a probabilistic approximation. The main advantage of this approach is that the aggregates are duplicate insensitive. This overcomes two central problems of existing aggregation schemes for VANET applications. First, when multiple aggregates of observations for the same area are available, it is possible to combine them into an aggregate containing all information from the original aggregates. This is fundamentally different from existing approaches where typically one of the aggregates is selected for further use while the rest is discarded. Second, any observation or aggregate can be included into higher level aggregates, regardless if it has already been previously - directly or indirectly - added. As a result of those characteristics the quality of the aggregates is high, while their construction is very flexible. We demonstrate these traits of our approach by a simulation study.

In this paper, we present our approach for geographic personalization of a content recommendation system. More specifically, our work focuses on recommending query topics to users. We do this by mining the search query logs to detect trending local topics. For a set of queries we compute their counts and what we call buzz scores, which is a metric for detecting trending behavior. We also compute the entropy of the geographic distribution of the queries as means of detecting their location affinity. We cluster the queries into trending topics and assign the topics to their corresponding location. Human editors then select a subset of these local topics and enter them into a recommendation system. In turn the recommendation system optimizes a pool of trending local and global topics by exploiting user feedback. We present some editorial evaluation of the technique and results of a live experiment. Inclusion of local topics in selected locations into the global pool of topics resulted in more than 6% relative increase in user engagement with the recommendation system compared to using the global topics exclusively.

New protocols for the data link and network layer are being proposed to address limitations of current protocols in terms of scalability, security, and manageability. High-speed routers and switches that implement these protocols traditionally perform packet processing using ASICs which offer high speed, low chip area, and low power. But with inflexible custom hardware, the deployment of new protocols could happen only through equipment upgrades. While newer routers use more flexible network processors for data plane processing, due to power and area constraints lookups in forwarding tables are done with custom lookup modules. Thus most of the proposed protocols can only be deployed with equipment upgrades. To speed up the deployment of new protocols, we propose a flexible lookup module, PLUG (Pipelined Lookup Grid). We can achieve generality without loosing efficiency because various custom lookup modules have the same fundamental features we retain: area dominated by memories, simple processing, and strict access patterns defined by the data structure. We implemented IPv4, Ethernet, Ethane, and SEATTLE in our dataflow-based programming model for the PLUG and mapped them to the PLUG hardware which consists of a grid of tiles. Throughput, area, power, and latency of PLUGs are close to those of specialized lookup modules.

Many applications work with graph-structured data. As graphs grow in size, indexing becomes essential to ensure sufficient query performance. We present the GRIPP index structure (GRaph Indexing based on Pre- and Postorder numbering) for answering reachability queries in graphs.
 GRIPP requires only linear time and space. Using GRIPP, we can answer reachability queries on graphs with 5 million nodes on average in less than 5 milliseconds, which is unrivaled by previous methods. We evaluate the performance and scalability of our approach on real and synthetic random and scale-free graphs and compare our approach to existing indexing schemes. GRIPP is implemented as stored procedure inside a relational database management system and can therefore very easily be integrated into existing graph-oriented applications.

Semantic hashing[1] seeks compact binary codes of data-points so that the Hamming distance between codewords correlates with semantic similarity. In this paper, we show that the problem of finding a best code for a given dataset is closely related to the problem of graph partitioning and can be shown to be NP hard. By relaxing the original problem, we obtain a spectral method whose solutions are simply a subset of thresholded eigenvectors of the graph Laplacian. By utilizing recent results on convergence of graph Laplacian eigenvectors to the Laplace-Beltrami eigenfunctions of manifolds, we show how to efficiently calculate the code of a novel datapoint. Taken together, both learning the code and applying it to a novel point are extremely simple. Our experiments show that our codes outperform the state-of-the art.

The chunk-lookup disk bottleneck and the read amplification problems are two great challenges for deduplication-based storage systems and restrict the applicability of data deduplication for large-scale data volumes. Previous studies and our experimental evaluations have shown that the amount of redundant data shared among different types of applications is negligible. Based on the observations, we propose AA-Plus which effectively groups the hash index of the same application together and divides the whole hash index into different groups based on the application types. Moreover, it groups the data chunks of the same application together on the disks. The extensive trace-driven experiments conducted on our lightweight prototype implementation of AA-Plus show that compared with AA-Dedupe, AA-Plus significantly speeds up the write throughput by a factor of up to 6.9 and with an average of 3.1, and speeds up the read throughput by a factor of up to 3.3 and with an average of 1.9.

The core design principle of Information-Centric Networking (ICN) is in the name based routing that enables users to ask for a data object by its name and makes the network deliver it to users from a nearby cache if available. Software-Defined Networking (SDN) lowers the cost and complexity of network management by decoupling architecture from infrastructure, which promises the continuous evolution of the network architecture in a flexible manner. The synergy between ICN supporting efficient data dissemination as the norm and SDN providing flexible management framework enables the combination to be a fully controllable framework for efficient data dissemination. In this paper, we propose a design of an ICN architecture within the framework of SDN.

With the advancement of information and communication technology, data are being generated at an exponential rate via various instruments and collected at an unprecedented scale. Such large volume of data generated is referred to as big data, which now are revolutionizing all aspects of our life ranging from enterprises to individuals, from science communities to governments, as they exhibit great potentials to improve efficiency of enterprises and the quality of life. To obtain nontrivial patterns and derive valuable information from big data, a fundamental problem is how to properly place the collected data by different users to distributed clouds and to efficiently analyze the collected data to save user costs in data storage and processing, particularly the cost savings of users who share data. By doing so, it needs the close collaborations among the users, by sharing and utilizing the big data in distributed clouds due to the complexity and volume of big data. Since computing, storage and bandwidth resources in a distributed cloud usually are limited, and such resource provisioning typically is expensive, the collaborative users require to make use of the resources fairly. In this paper, we study a novel collaboration- and fairness-aware big data management problem in distributed cloud environments that aims to maximize the system throughout, while minimizing the operational cost of service providers to achieve the system throughput, subject to resource capacity and user fairness constraints. We first propose a novel optimization framework for the problem. We then devise a fast yet scalable approximation algorithm based on the built optimization framework. We also analyze the time complexity and approximation ratio of the proposed algorithm. We finally conduct experiments by simulations to evaluate the performance of the proposed algorithm. Experimental results demonstrate that the proposed algorithm is promising, and outperforms other heuristics.

Data centers are widely used for big data analytics, which often involve data-parallel jobs, including query and web service. Meanwhile, cluster frameworks are rapidly developed for data-intensive applications in data center networks (DCNs). To promote the performance of these frameworks, many efforts have been paid to improve scheduling strategies and resource allocation algorithms. With the deployment of geo-distributed data centers and data-intensive applications, the optimization in DCNs regains pervasive attention in both industry and academia. Many solutions, such as the coflow-aware scheduling and speculative execution, have been proposed to meet various requirements. Therefore, we present a solid starting ground and comprehensive overview in this area to help readers quickly understand state-of-the-art technologies and research progress. We observe that algorithms in cluster frameworks are implemented with different guidelines and can be classified according to scheduling granularity, controller management, and prior-knowledge requirement. In addition, mechanisms for conquering crucial challenges in DCNs are discussed, including providing low latency and minimizing job completion time. Moreover, we analyze desirable properties of fault tolerance and scalability to illuminate the design principles of distributed systems. We hope that this paper will shed light on this promising land and serve as a guide for further researches.

With the increasing popularity of hand-held devices, location-based applications and services have access to accurate and real-time location information, raising serious privacy concerns for their users. The recently introduced notion of geo-indistinguishability tries to address this problem by adapting the well-known concept of differential privacy to the area of location-based systems. Although geo-indistinguishability presents various appealing aspects, it has the problem of treating space in a uniform way, imposing the addition of the same amount of noise everywhere on the map. In this paper we propose a novel elastic distinguishability metric that warps the geometrical distance, capturing the different degrees of density of each area. As a consequence, the obtained mechanism adapts the level of noise while achieving the same degree of privacy everywhere. We also show how such an elastic metric can easily incorporate the concept of a “geographic fence” that is commonly employed to protect the highly recurrent locations of a user, such as his home or work. We perform an extensive evaluation of our technique by building an elastic metric for Paris’ wide metropolitan area, using semantic information from the OpenStreetMap database. We compare the resulting mechanism against the Planar Laplace mechanism satisfying standard geo-indistinguishability, using two real-world datasets from the Gowalla and Brightkite location-based social networks. The results show that the elastic mechanism adapts well to the semantics of each area, adjusting the noise as we move outside the city center, hence offering better overall privacy. 1


Question: GHT: a geographic hash table for storage in data-centric systems.

This paper introduces RADAR, an innovative radio-frequency (RF) based framework designed for locating and tracking users within indoor environments. The widespread adoption of mobile computing devices and the expansion of local-area wireless networks have led to a heightened demand for location-aware systems and services. RADAR addresses this need by employing a network of strategically placed base stations that provide overlapping coverage in the target area. The system collects and analyzes signal strength data from these base stations, integrating empirical measurements with sophisticated signal propagation modeling to accurately determine user location. This capability paves the way for enhanced location-aware services and applications. The paper presents empirical findings that validate RADAR's effectiveness in achieving high precision in user location estimation.

**Question**: A Framework for Secure and Reliable Data Sharing in Cloud Computing

**Documents**: A Secure and Efficient Data Sharing Scheme for Cloud Storage
In this paper, we present a novel scheme for secure and efficient data sharing in cloud storage. The proposed scheme addresses the challenges of data privacy, integrity, and availability in a multi-user environment. It employs a combination of encryption and access control mechanisms to ensure that only authorized users can access and manipulate data. The scheme also includes mechanisms for efficient data retrieval and sharing among users. Experimental results demonstrate the effectiveness of the proposed scheme in terms of security, performance, and scalability.

**Similar Documents**: This paper unveils a groundbreaking approach to secure and efficient data sharing within cloud storage systems, particularly in scenarios involving multiple users. The proposed framework effectively tackles the critical issues of data privacy, integrity, and accessibility. It achieves this through the strategic use of encryption techniques and robust access control protocols, ensuring that data is accessible only to authenticated users and safeguarding against unauthorized access and manipulation. Additionally, the scheme incorporates advanced features for optimized data retrieval and sharing among users, facilitating collaborative work environments. Rigorous testing has confirmed the scheme's superiority in security, operational efficiency, and the ability to scale effectively to accommodate growing user bases.

With the existence of many large transaction databases, the huge amounts of data, the high scalability of distributed systems, and the easy partition and distribution of a centralized database, it is important to inuestzgate eficient methods for distributed mining of association rules. This study discloses some interesting relationships between locally large and globally large itemsets and proposes an interesting distributed association rule mining algorithm, FDM (Fast Distributed Mining of association rules), which generates a small number of candidate sets and substantially reduces the number of messages to be passed at mining association rules. Our performance study shows that FDM has a superior performance over the direct application of a typical sequential algorithm. Further performance enhancement leads to a few variations of the algorithm.

MapReduce is becoming ubiquitous in large-scale data analysis. Several recent works have shown that the performance of Hadoop MapReduce could be improved, for instance, by creating indexes in a non-invasive manner. However, they ignore the impact of the data layout used inside data blocks of Hadoop Distributed File System (HDFS). In this paper, we analyze different data layouts in detail in the context of MapReduce and argue that Row, Column, and PAX layouts can lead to poor system performance. We propose a new data layout, coined Trojan Layout, that internally organizes data blocks into attribute groups according to the workload in order to improve data access times. A salient feature of Trojan Layout is that it fully preserves the fault-tolerance properties of MapReduce. We implement our Trojan Layout idea in HDFS 0.20.3 and call the resulting system Trojan HDFS. We exploit the fact that HDFS stores multiple replicas of each data block on different computing nodes. Trojan HDFS automatically creates a different Trojan Layout per replica to better fit the workload. As a result, we are able to schedule incoming MapReduce jobs to data block replicas with the most suitable Trojan Layout. We evaluate our approach using three real-world workloads. We compare Trojan Layouts against Hadoop using Row and PAX layouts. The results demonstrate that Trojan Layout allows MapReduce jobs to read their input data up to 4.8 times faster than Row layout and up to 3.5 times faster than PAX layout.

The chunk-lookup disk bottleneck and the read amplification problems are two great challenges for deduplication-based storage systems and restrict the applicability of data deduplication for large-scale data volumes. Previous studies and our experimental evaluations have shown that the amount of redundant data shared among different types of applications is negligible. Based on the observations, we propose AA-Plus which effectively groups the hash index of the same application together and divides the whole hash index into different groups based on the application types. Moreover, it groups the data chunks of the same application together on the disks. The extensive trace-driven experiments conducted on our lightweight prototype implementation of AA-Plus show that compared with AA-Dedupe, AA-Plus significantly speeds up the write throughput by a factor of up to 6.9 and with an average of 3.1, and speeds up the read throughput by a factor of up to 3.3 and with an average of 1.9.

Database scale-out is commonly implemented by partitioning data across several database instances. This approach, however, has several restrictions. In particular, partitioned databases are inflexible in large-scale deployments and assume a partition-friendly workload in order to scale. In this paper, we analyze an alternative architecture design for distributed relational databases that overcomes the limitations of partitioned databases. The architecture is based on two fundamental principles: We decouple query processing and transaction management from data storage, and we share data across query processing nodes. The combination of these design choices provides scalability, elasticity, and operational flexibility without making any assumptions on the workload. As a drawback, sharing data among multiple database nodes causes synchronization overhead. To address this limitation, we introduce techniques for scalable transaction processing in shared-data environments. Specifically, we describe mechanisms for efficient data access, concurrency control, and data buffering. In combination with new hardware trends, the techniques enable performance characteristics that top state-of-the-art partitioned databases.

The rapid growing volumes of spatial data have brought significant challenges on developing high-performance spatial data processing techniques in parallel and distributed computing environments. Spatial joins are important data management techniques in gaining insights from large-scale geospatial data. While several distributed spatial join techniques based on spatial partitions have been implemented on top of existing Big Data systems, they are not capable of natively exploiting massively data parallel computing power provided by modern commodity Graphics Processing Units (GPUs). In this study, as an important component of our research initiative in developing high-performance spatial join techniques on GPUs, we have designed and implemented a polyline intersection based spatial join technique that is capable of exploiting massively data parallel computing power on GPUs. The proposed polyline intersection based spatial join technique is integrated into a customized lightweight distributed execution engine that natively supports spatial partitions. We empirically evaluate the performance of the proposed spatial join technique on both a standalone GPU-equipped workstation and Amazon EC2 GPU-accelerated clusters and demonstrate its high performance when comparing with the state-of-the-art.

Heavy hitters are data items that occur at high frequency in a data set. They are among the most important items for an organization to summarize and understand during analytical processing. In data sets with sufficient skew, the number of heavy hitters can be relatively small. We take advantage of this small footprint to compute aggregate functions for the heavy hitters in fast cache memory in a single pass.
 We design cache-resident, shared-nothing structures that hold only the most frequent elements. Our algorithm works in three phases. It first samples and picks heavy hitter candidates. It then builds a hash table and computes the exact aggregates of these elements. Finally, a validation step identifies the true heavy hitters from among the candidates.
 We identify trade-offs between the hash table configuration and performance. Configurations consist of the probing algorithm and the table capacity that determines how many candidates can be aggregated. The probing algorithm can be perfect hashing, cuckoo hashing and bucketized hashing to explore trade-offs between size and speed.
 We optimize performance by the use of SIMD instructions, utilized in novel ways beyond single vectorized operations, to minimize cache accesses and the instruction footprint.

We study an acceleration method for point-to-point shortest-path computations in large and sparse directed graphs with given nonnegative arc weights. The acceleration method is called the arc-flag approach and is based on Dijkstra's algorithm. In the arc-flag approach, we allow a preprocessing of the network data to generate additional information, which is then used to speedup shortest-path queries. In the preprocessing phase, the graph is divided into regions and information is gathered on whether an arc is on a shortest path into a given region. The arc-flag method combined with an appropriate partitioning and a bidirected search achieves an average speedup factor of more than 500 compared to the standard algorithm of Dijkstra on large networks (1 million nodes, 2.5 million arcs). This combination narrows down the search space of Dijkstra's algorithm to almost the size of the corresponding shortest path for long-distance shortest-path queries. We conduct an experimental study that evaluates which partitionings are best suited for the arc-flag method. In particular, we examine partitioning algorithms from computational geometry and a multiway arc separator partitioning. The evaluation was done on German road networks. The impact of different partitions on the speedup of the shortest path algorithm are compared. Furthermore, we present an extension of the speedup technique to multiple levels of partitions. With this multilevel variant, the same speedup factors can be achieved with smaller space requirements. It can, therefore, be seen as a compression of the precomputed data that preserves the correctness of the computed shortest paths.

The bloom filter is a probabilistic data structure that provides a compact representation of a set of elements. To keep false positive probabilities low, the size of the bloom filter must be dimensioned a priori to be linear in the maximum number of keys inserted, with the linearity constant ranging typically from one to few bytes. A bloom filter is most commonly used as an in memory data structure, hence its size is limited by the availability of RAM space on the machine. As datasets have grown over time to Internet scale, so have the RAM space requirements of bloom filters. If sufficient RAM space is not available, we advocate that flash memory may serve as a suitable medium for storing bloom filters, since it is about one-tenth the cost of RAM per GB while still providing access times orders of magnitude faster than hard disk. We present BLOOMFLASH, a bloom filter designed for flash memory based storage, that provides a new dimension of trade off with bloom filter access times to reduce RAM space usage (and hence system cost). The simple design of a single flat bloom filter on flash suffers from many performance bottlenecks, including in-place bit updates that are inefficient on flash and multiple reads and random writes spread out across many flash pages for a single lookup or insert operation. To mitigate these performance bottlenecks, BLOOMFLASH leverages two key design innovations: (i) buffering bit updates in RAM and applying them in bulk to flash that helps to reduce random writes to flash, and (ii) a hierarchical bloom filter design consisting of component bloom filters, stored one per flash page, that helps to localize reads and writes on flash. We use two real-world data traces taken from representative bloom filter applications to drive and evaluate our design. BLOOMFLASH achieves bloom filter access times in the range of few tens of microseconds, thus allowing up to order of tens of thousands operations per sec.

Carefully balancing load in distributed stream processing systems has a fundamental impact on execution latency and throughput. Load balancing is challenging because real-world workloads are skewed: some tuples in the stream are associated to keys which are significantly more frequent than others. Skew is remarkably more problematic in large deployments: having more workers implies fewer keys per worker, so it becomes harder to “average out” the cost of hot keys with cold keys. We propose a novel load balancing technique that uses a heavy hitter algorithm to efficiently identify the hottest keys in the stream. These hot keys are assigned to d ≥ 2 choices to ensure a balanced load, where d is tuned automatically to minimize the memory and computation cost of operator replication. The technique works online and does not require the use of routing tables. Our extensive evaluation shows that our technique can balance real-world workloads on large deployments, and improve throughput and latency by 150% and 60% respectively over the previous state-of-the-art when deployed on Apache Storm.

This paper proposes a comprehensive critical survey on the issues of warehousing and protecting big data, which are recognized as critical challenges of emerging big data research. Indeed, both are critical aspects to be considered in order to build truly, high-performance and highly-flexible big data management systems. We report on state-of-the-art approaches, methodologies and trends, and finally conclude by providing open problems and challenging research directions to be considered by future efforts.


Question: GHT: a geographic hash table for storage in data-centric systems.

In this seminal work by Alan Turing, titled "The Chemical Basis of Morphogenesis," published in 1952, Turing explores the theoretical underpinnings of how periodic patterns emerge in biological systems. Observing the intricate patterns found in nature, such as the stripes on a tiger, the spots on a leopard, or the spiral arrangement of leaves on a woodruff plant, it becomes evident that these patterns are not mere coincidences. Given the vast number of possible pattern variations, it is implausible that each specific design is genetically encoded. Turing's paper proposes a novel mechanism, involving the interaction of two morphogens that react and diffuse through the biological tissue. Under specific conditions concerning the diffusivity of these morphogens and the nature of their reactions, Turing's mechanism, often referred to as the Turing mechanism, can generate patterns in the concentration of these morphogens, which in turn define the observable structures. This mechanism offers a compelling explanation for the formation of periodic patterns in morphogenesis, providing a chemical basis for the development of complex biological structures.

We present Resilient Distributed Datasets (RDDs), a distributed memory abstraction that allows programmers to perform in-memory computations on large clusters while retaining the fault tolerance of data flow models like MapReduce. RDDs are motivated by two types of applications that current data flow systems handle inefficiently: iterative algorithms, which are common in graph applications and machine learning, and interactive data mining tools. In both cases, keeping data in memory can improve performance by an order of magnitude. To achieve fault tolerance efficiently, RDDs provide a highly restricted form of shared memory: they are read-only datasets that can only be constructed through bulk operations on other RDDs. However, we show that RDDs are expressive enough to capture a wide class of computations, including MapReduce and specialized programming models for iterative jobs such as Pregel. Our implementation of RDDs can outperform Hadoop by 20× for iterative jobs and can be used interactively to search a 1 TB dataset with latencies of 5–7 seconds.

Data-center applications running on distributed databases often suffer from unexpectedly high response time fluctuation which is caused by long tail latency. In this paper, we find that long tail latency of user writes is mainly created by the interference with garbage collection (GC) tasks running in various system layers. In order to address the tail latency problem, we propose a synchronized garbage collection technique, called SyncGC. By scheduling multiple GC instances to execute in sync with each other in an overlapped manner, SyncGC prevents user requests from being interfered with GC instances, thereby minimizing their negative impacts on tail latency. Our experimental results with Cassandra show that SyncGC reduces the 99.99th-percentile tail latency and the maximum latency by 35% and 37%, on average, respectively.

Heavy hitters are data items that occur at high frequency in a data set. They are among the most important items for an organization to summarize and understand during analytical processing. In data sets with sufficient skew, the number of heavy hitters can be relatively small. We take advantage of this small footprint to compute aggregate functions for the heavy hitters in fast cache memory in a single pass.
 We design cache-resident, shared-nothing structures that hold only the most frequent elements. Our algorithm works in three phases. It first samples and picks heavy hitter candidates. It then builds a hash table and computes the exact aggregates of these elements. Finally, a validation step identifies the true heavy hitters from among the candidates.
 We identify trade-offs between the hash table configuration and performance. Configurations consist of the probing algorithm and the table capacity that determines how many candidates can be aggregated. The probing algorithm can be perfect hashing, cuckoo hashing and bucketized hashing to explore trade-offs between size and speed.
 We optimize performance by the use of SIMD instructions, utilized in novel ways beyond single vectorized operations, to minimize cache accesses and the instruction footprint.

In most parallel DBSCAN algorithms, neighboring points are assigned to the same data partition for parallel processing to facilitate calculation of the density of the neighbors. This data partitioning scheme causes a few critical problems including load imbalance between data partitions, especially in a skewed data set. To remedy these problems, we propose a cell-based data partitioning scheme, pseudo random partitioning , that randomly distributes small cells rather than the points themselves. It achieves high load balance regardless of data skewness while retaining the data contiguity required for DBSCAN. In addition, we build and broadcast a highly compact summary of the entire data set, which we call a two-level cell dictionary , to supplement random partitions. Then, we develop a novel parallel DBSCAN algorithm, Random Partitioning-DBSCAN (shortly, RP-DBSCAN), that uses pseudo random partitioning together with a two-level cell dictionary. The algorithm simultaneously finds the local clusters to each data partition and then merges these local clusters to obtain global clustering. To validate the merit of our approach, we implement RP-DBSCAN on Spark and conduct extensive experiments using various real-world data sets on 12 Microsoft Azure machines (48 cores). In RP-DBSCAN, data partitioning and cluster merging are very light, and clustering on each split is not dragged out by a specific worker. Therefore, the performance results show that RP-DBSCAN significantly outperforms the state-of-the-art algorithms by up to 180 times.

In recent years, the increased need to house and process large volumes of data has prompted the need for distributed storage and querying systems. The growth of machine-readable RDF triples has prompted both industry and academia to develop new database systems, called “NoSQL,” with characteristics that differ from classical databases. Many of these systems compromise ACID properties for increased horizontal scalability and data availability. This thesis concerns the development and evaluation of a NoSQL triplestore. Triplestores are database management systems central to emerging technologies such as the Semantic Web and linked data. A triplestore comprises data storage using the RDF (resource description framework) data model and the execution of queries written in SPARQL. The triplestore developed here exploits an opensource stack comprising, Hadoop, HBase, and Hive. The evaluation spans several benchmarks, including the two most commonly used in triplestore evaluation, the Berlin SPARQL Benchmark, and the DBpedia benchmark, a query workload that operates an RDF representation of Wikipedia. Results reveal that the join algorithm used by the system plays a critical role in dictating query runtimes. Distributed graph databases must carefully optimize queries before generating MapReduce query plans as network traffic for large datasets can become prohibitive if the query is executed naively.

As data becomes dynamic, large, and distributed, there is increasing demand for what have become known as distributed stream algorithms. Since continuously collecting the data to a central server and processing it there is infeasible, a common approach is to define local conditions at the distributed nodes, such that—as long as they are maintained—some desirable global condition holds.
 Previous methods derived local conditions focusing on communication efficiency. While proving very useful for reducing the communication volume, these local conditions often suffer from heavy computational burden at the nodes. The computational complexity of the local conditions affects both the runtime and the energy consumption. These are especially critical for resource-limited devices like smartphones and sensor nodes. Such devices are becoming more ubiquitous due to the recent trend toward smart cities and the Internet of Things. To accommodate for high data rates and limited resources of these devices, it is crucial that the local conditions be quickly and efficiently evaluated.
 Here we propose a novel approach, designated CB (for Convex/Concave Bounds). CB defines local conditions using suitably chosen convex and concave functions. Lightweight and simple, these local conditions can be rapidly checked on the fly. CB’s superiority over the state-of-the-art is demonstrated in its reduced runtime and power consumption, by up to six orders of magnitude in some cases. As an added bonus, CB also reduced communication overhead in all the tested application scenarios.

Gossip algorithms have recently received significant attention, mainly because they constitute simple and robust algorithms for distributed information processing over networks. However for many topologies that are realistic for wireless ad-hoc and sensor networks (like grids and random geometric graphs), the standard nearest-neighbor gossip converges very slowly. A recently proposed algorithm called geographic gossip improves gossip efficiency by a p n/ log n factor for random geometric graphs, by exploiting geographic information of node locations. In this paper we prove that a variation of geographic gossip that averages along routed paths, improves efficiency by an additional p n/ log n factor and is order optimal for grids and random geometric graphs. Our analysis provides some general techniques and can be used to provide bounds on the performance of randomized message passing algorithms operating over various graph topologies.

Relational database systems have been the dominating technology to manage and analyze large data warehouses. Moreover, the ER model, the standard in database design has a close relationship with the relational model. Recently, there has been a surge of alternative technologies for large scale analytic processing, most of which are not based on the relational model. Out of these proposals, distributed file systems together with MapReduce have become strong competitors to relational database systems to analyze large data sets, exploiting parallel processing. Moreover, there is progress on using MapReduce to evaluate relational queries. With that motivation in mind, this panel will compare pros and cons of each technology for data warehousing and will identify research issues, considering practical aspects like ease of use, programming flexibility and cost; as well as technical aspects like data modeling, storage, hardware, scalability, query processing, fault tolerance and data mining.

Geospatial clustering is a very important research topic in the field of „Geospatial Knowledge Discovery.‟ Its main purpose is to group similar objects, from large geospatial datasets, into clusters, based on the objects‟ spatial and non-spatial attributes. This capability is very useful for better understanding the patterns and distributions of geographical phenomena. With the emerging availability of large amounts of geospatial data on the Internet, the demand for powerful methods of extracting meaningful information from large amounts of web-based geospatial data is thus also rapidly increasing. In this chapter, we present a geospatial clustering web service called GeoClustering. With this service, users are able to cluster online data sources easily and to then conveniently visualize the clustering results through a friendly map-centric user interface. Since it is web-based, GeoClustering offers a platform that can be accessed by anyone from anywhere at any time. To achieve better interoperability, GeoClustering offers an API and follows the OGC standard data format for spatial data exchange.

The enormous growth of data in a variety of applications has increased the need for high performance data mining based on distributed environments. However, standard data mining toolkits per se do not allow the usage of computing clusters. The success of MapReduce for analyzing large data has raised a general interest in applying this model to other, data intensive applications. Unfortunately current research has not lead to an integration of GUI based data mining toolkits with distributed file system based MapReduce systems. This paper defines novel principles for modeling and design of the user interface, the storage model and the computational model necessary for the integration of such systems. Additionally, it introduces a novel system architecture for interactive GUI based data mining of large data on clusters based on MapReduce that overcomes the limitations of data mining toolkits. As an empirical demonstration we show an implementation based on Weka and Hadoop.


Question: An investigation into data mining frameworks within the realm of cybersecurity.

This paper delves into the foundational concepts and techniques of data mining, specifically highlighting the pioneering work on association rule mining. The seminal contribution by Agrawal, Imielinski, and Swami [AIS93] introduced the field, which was further advanced by the Apriori algorithm for frequent itemset mining as discussed in Agrawal and Srikant [AS94b]. Independently, Mannila, Toivonen, and Verkamo [MTV94] developed a variant of the algorithm employing a comparable pruning strategy. These parallel efforts culminated in a collaborative publication by Agrawal, Mannila, Srikant, Toivonen, and Verkamo [AMS96], synthesizing the advancements in the field. The process of extracting association rules from frequent itemsets, a critical step in data mining, is detailed in Agrawal and Srikant [AS94a]. This foundational research has significantly influenced the development of data mining frameworks, particularly in the realm of cybersecurity, where the ability to uncover hidden patterns and relationships in data is paramount for threat detection and prevention.

Intrusion detection is a broad problem, and we need a greater range of tools than is currently available. In this article, we report a new approach. We have applied information retrieval techniques to index audit trails. These indexes can be extremely e cient at detecting attacks whose signature is an unusual combination of events, and they may consume only a very small additional amount of storage. This approach allows the intrusion detection community to adopt a wide range of techniques developed in applications ranging from library science to web search engines.

While the field of educational data mining (EDM) has generated many innovations for improving educational software and student learning, the mining of student data has recently come under a great deal of scrutiny. Many stakeholder groups, including public officials, media outlets, and parents, have voiced concern over the privacy of student data and their efforts have garnered national attention. The momentum behind and scrutiny of student privacy has made it increasingly difficult for EDM applications to transition from academia to industry. Based on experience as academic researchers transitioning into industry, we present three primary areas of concern related to student privacy in practice: policy, corporate social responsibility, and public opinion. Our discussion will describe the key challenges faced within these categories, strategies for overcoming them, and ways in which the academic EDM community can support the adoption of innovative technologies in large-scale production.

Cyber-physical Production Systems (CPPS) are one of the technical driving forces behind the transformation of industrial production towards "digital factory of the future" in the context of Industry 4.0. Security is a major concern for such systems as they become more intelligent, interconnected, and coupled with physical devices. For various security activities from security analysis to designing security controls and architecture, a systematic and structured view and presentation of security-related information is required. Based on the draft standard of Reference Architecture Model for Industry 4.0 (RAMI 4.0), we propose a practical approach to establish a security viewpoint in the CPPS reference architecture model. We investigate the feasibility of using an architecture modeling tool to implement the concept and leverage existing work on models of layered architecture. We demonstrate the applicability for security analysis in two example case studies.

Quite a large number of data mining techniques employed in financial fraud detection (FFD) are seen to be classification techniques. In this paper, we developed an algorithm to find the features of classification techniques (or method) that so much place it (classification techniques) in the heart of researchers in their various efforts in the study of insurance frauds detection. We also got to know the characteristics of insurance frauds data that made data mining classification techniques so much attracted to it (insurance data).

In recent years, Distributed Data Mining (DDM) has evolved in large space aiming at minimizing computation cost and memory overhead in processing huge geographically distributed data. There are two approaches of DDM -- homogeneous and heterogeneous classifier approach. This paper presents implementation of four homogeneous classifier techniques for DDM with different Electronic Health Records (EHRs) homogeneous datasets namely diabetes, hepatitis, hypothyroid and further analyzing results based on metric evaluation.

Mining grid data is an interesting research field which aims at analyzing grid systems with data mining techniques in order to efficiently discover new meaningful knowledge to enhance grid management. In this paper, we focus particularly on how extracted knowledge enables enhancing data replication and replica selection strategies which are important data management techniques commonly used in data grids. Indeed, relevant knowledge such as file access patterns, file correlations, user or job access behavior, prediction of future behavior or network performance, and so on, can be efficiently discovered. These findings are then used to enhance both data replication and replica selection strategies. Various works in this respect are then discussed along with their merits and demerits. In addition, we propose a new guideline to data mining application in the context of data replication and replica selection strategies. & 2015 Elsevier Ltd. All rights reserved.

What constitutes an enterprise architecture framework is a contested subject. The contents of present enterprise architecture frameworks thus differ substantially. This paper aims to alleviate the confusion regarding which framework contains what by proposing a meta framework for enterprise architecture frameworks. By using this meta framework, decision makers are able to express their requirements on what their enterprise architecture framework must contain and also to evaluate whether the existing frameworks meets these requirements. An example classification of common EA frameworks illustrates the approach.

The popularity of using Internet contains some risks of network attacks. Intrusion detection is one major research problem in network security, whose aim is to identify unusual access or attacks to secure internal networks. In literature, intrusion detection systems have been approached by various machine learning techniques. However, there is no a review paper to examine and understand the current status of using machine learning techniques to solve the intrusion detection problems. This chapter reviews 55 related studies in the period between 2000 and 2007 focusing on developing single, hybrid, and ensemble classifiers. Related studies are compared by their classifier design, datasets used, and other experimental setups. Current achievements and limitations in developing intrusion detection systems by machine learning are present and discussed. A number of future research directions are also provided. 2009 Elsevier Ltd. All rights reserved.

With the phenomenal growth of the Web, there is an everincreasing volume of data and information published in numerous Web pages. The research in Web mining aims to develop new techniques to effectively extract and mine useful knowledge or information from these Web pages [8]. Due to the heterogeneity and lack of structure of Web data, automated discovery of targeted or unexpected knowledge/information is a challenging task. It calls for novel methods that draw from a wide range of fields spanning data mining, machine learning, natural language processing, statistics, databases, and information retrieval. In the past few years, there was a rapid expansion of activities in the Web mining field, which consists of Web usage mining, Web structure mining, and Web content mining. Web usage mining refers to the discovery of user access patterns from Web usage logs. Web structure mining tries to discover useful knowledge from the structure of hyperlinks. Web content mining aims to extract/mine useful information or knowledge from Web page contents. For this special issue, we focus on Web content mining.

The healthcare industry is a vast field with a plethora of data about patients,added to the huge medical records every passing day. In terms of science, this industry is ’information rich’ yet ’knowledge poor’. However, data mining with its various analytical tools and techniques plays a major role in reducing the use of cumbersome tests used on patients to detect a disease. The aim of this paper is to employ and analyze different data mining techniques for the prediction of heart disease in a patient through extraction of interesting patterns from the dataset using vital parameters. This paper strives to bring out the methodology and implementation of these techniques-Artificial Neural Networks, Decision Tree and Naive Bayes and stress upon the results and conclusion induced on the basis of accuracy and time complexity. By far, the observations reveal that Artificial Neural Networks outperformed Naive Bayes and Decision Tree.


Question: An investigation into data mining frameworks within the realm of cyber security.

This paper presents the CRISP-DM 1.0 methodology, a comprehensive guide for data mining processes designed to be universally applicable across industries and technologies. The document includes an overview of the CRISP-DM approach, a detailed reference model, a user-friendly guide, and reporting standards, complemented by an appendix that provides additional resources and information. The material is owned exclusively by the CRISP-DM consortium partners, with all trademarks and service marks acknowledged by the consortium members.

The genesis of CRISP-DM dates back to late 1996, when data mining was in its nascent stage but poised for explosive growth. DaimlerChrysler, a pioneer in integrating data mining into business operations, SPSS (then ISL), a provider of data mining services since 1990 and the developer of the first commercial data mining workbench, Clementine, in 1994, and NCR, which aimed to offer value-added data mining services to its Teradata data warehouse clients, recognized the need for a standardized process. The question was whether the existing approaches were optimal, and whether every new adopter would have to learn through trial and error, as had been the case initially. Moreover, suppliers sought a way to assure potential customers of the maturity of data mining as a critical component of business processes.

The consortium envisioned a non-proprietary, freely accessible process model that could serve as a common framework for data mining practitioners. Thus, the CRISP-DM initiative was born, aiming to be industry-, tool-, and application-neutral. To ensure broad applicability and acceptance, the consortium sought input from a diverse range of stakeholders, including data warehouse vendors and management consultancies, by forming the CRISP-DM Special Interest Group (SIG). The SIG was launched with a call for interested parties to join a day-long workshop in Amsterdam, marking the beginning of a collaborative effort to develop and refine the CRISP-DM methodology.

The CRISP-DM framework addresses the critical issues of standardization and maturity in data mining, providing a structured approach that helps organizations navigate the complexities of data mining projects. By fostering a community of practitioners and stakeholders, CRISP-DM promotes best practices and standards, enabling businesses to confidently integrate data mining into their operations and decision-making processes.

What constitutes an enterprise architecture framework is a contested subject. The contents of present enterprise architecture frameworks thus differ substantially. This paper aims to alleviate the confusion regarding which framework contains what by proposing a meta framework for enterprise architecture frameworks. By using this meta framework, decision makers are able to express their requirements on what their enterprise architecture framework must contain and also to evaluate whether the existing frameworks meets these requirements. An example classification of common EA frameworks illustrates the approach.

This paper surveys knowledge management (KM) development using a literature review and classification of articles from 1995 to 2002 with keyword index in order to explore how KM technologies and applications have developed in this period. Based on the scope of 234 articles of knowledge management applications, this paper surveys and classifies KM technologies using the seven categories as: KM framework, knowledge-based systems, data mining, information and communication technology, artificial intelligence/expert systems, database technology, and modeling, together with their applications for different research and problem domains. Some discussion is presented, indicating future development for knowledge management technologies and applications as the followings: (1) KM technologies tend to develop towards expert orientation, and KM applications development is a problem-oriented domain. (2) Different social studies methodologies, such as statistical method, are suggested to implement in KM as another kind of technology. (3) Integration of qualitative and quantitative methods, and integration of KM technologies studies may broaden our horizon on this subject. (4) The ability to continually change and obtain new understanding is the power of KM technologies and will be the application of future works. q 2003 Elsevier Science Ltd. All rights reserved.

This paper discusses the relationship between data science and population-based algorithms, which include swarm intelligence and evolutionary algorithms. We reviewed two categories of literature, which include population-based algorithms solving data analysis problem and utilizing data analysis methods in population-based algorithms. With the exponential increment of data, the data science, or more specifically, the big data analytics has gained increasing attention among researchers. New and more efficient algorithms should be designed to handle this massive data problem. Based on the combination of population-based algorithms and data mining techniques, we understand better the insights of data analytics, and design more efficient algorithms to solve real-world big data analytics problems. Also, the weakness and strength of population-based algorithms could be analyzed via the data analytics along the optimization process, a crucial entity in population-based algorithms.

We survey promising directions for research in the area of formal methods and its applications, including fundamental questions about the combination of formal methods and machine learning, and applications to cyberphysical systems and education.

Intrusion detection plays a vital role in maintaining the stability of any network. The major requirements for any intrusion detection system are speed, accuracy and less memory. Though various intrusion detection methods are available, they excel at some points while lack in the others. This paper presents a comprehensive survey of the technologies that are used for detecting intrusions. It analyzes the pros and cons of each technology and the literature works that utilizes these technologies. Challenges faced by the current IDS and the requirements for IDS in the current network scenario are discussed in detail. A detailed study on the datasets that can be used for effective building of an IDS is discussed. The research framework is proposed and a discussion of the various technologies that can be used for improving the efficiency of the IDS is provided.

Nowadays, all kinds of information systems store detailed information in logs. Process mining has emerged as a way to analyze these systems based on these detailed logs. Unlike classical data mining, the focus of process mining is on processes. First, process mining allows us to extract a process model from an event log. Second, it allows us to detect discrepancies between a modeled process (as it was envisioned to be) and an event log (as it actually is). Third, it can enrich an existing model with knowledge derived from an event log. This paper presents our tool ProM, which is the world-leading tool in the area of process mining.

Threat analysis and mitigation, both essential for corporate security, are time consuming, complex and demand expert knowledge. We present an approach for simulating threats to corporate assets, taking the entire infrastructure into account. Using this approach effective countermeasures and their costs can be calculated quickly without expert knowledge and a subsequent security decisions will be based on objective criteria. The ontology used for the simulation is based on Landwehr’s [ALRL04] taxonomy of computer security and dependability.

— To classify data mining problems and algorithms we used two dimensions: data type and type of mining operations. One of the main issue that arise during the data mining process is treating data that contains temporal information. The area of temporal data mining has very much attention in the last decade because from the time related feature of the data, one can extract much significant information which can not be extracted by the general methods of data mining. Many interesting techniques of temporal data mining were proposed and shown to be useful in many applications. Since temporal data mining brings together techniques from different fields such as databases, statistics and machine learning the literature is scattered among many different sources. In this paper, we present a survey on techniques of temporal data mining.

Data Mining is the extraction of hidden predictive information from large databases. The process of discovering interesting, useful, nontrivial patterns from large spatial datasets is called spatial data mining. When time gets associated it becomes spatio temporal data mining. The study of spatio temporal data mining is of great concern for the study of mobile phone sensed data. Reality Mining is defined as the study of human social behavior based on mobile phone sensed data. It is based on the data collected by sensors in mobile phones, security cameras, RFID readers, etc. All allow for the measurement of human physical and social activity. In this paper Netviz, Gephi and Weka tools have been used to convert and analyze the Facebook. Further, analysis of a reality mining dataset is also presented.

Data mining techniques have been extensively applied in bioinformatics to analyze biomedical data. In this paper, we choose the Rapid-I¿s RapidMiner as our tool to analyze a Pima Indians Diabetes Data Set, which collects the information of patients with and without developing diabetes. The discussion follows the data mining process. The focus will be on the data preprocessing, including attribute identification and selection, outlier removal, data normalization and numerical discretization, visual data analysis, hidden relationships discovery, and a diabetes prediction model construction.