Question: Flower categorization using local and spatial visual information.

This paper presents a novel approach, Parameter-Sensitive Hashing (PSH), designed to address the challenges associated with parameter estimation in complex and high-dimensional problems, such as pose estimation. Traditional example-based methods, while effective for simpler systems or lower-dimensional inputs, struggle with the exponential increase in required examples and computational complexity for more intricate tasks. PSH introduces an innovative solution by learning a set of hashing functions that can efficiently index and retrieve relevant examples for a specific estimation task. Building upon the concept of locality-sensitive hashing, which can find approximate neighbors in time that is sublinear in the number of examples, PSH critically optimizes the choice of hash functions to maximize relevance to the estimation problem at hand. Through extensive experiments, the paper demonstrates the efficacy of PSH in rapidly and accurately estimating the articulated pose of human figures from a large database of example images. This breakthrough approach significantly reduces the computational burden and improves the efficiency of pose estimation tasks, showcasing the potential of PSH in handling complex parameter estimation problems. (Note: Portions of this research were conducted while G.S. and P.V. were affiliated with Mitsubishi Electric Research Labs in Cambridge, MA.)

Peripheral vision provides a less faithful representation of the visual input than foveal vision. Nonetheless, we can gain a lot of information about the world from our peripheral vision, for example in order to plan eye movements. The phenomenon of crowding shows that the reduction of information available in the periphery is not merely the result of reduced resolution. Crowding refers to visual phenomena in which identification of a target stimulus is significantly impaired by the presence of nearby stimuli, or flankers. What information is available in the periphery? We propose that the visual system locally represents peripheral stimuli by the joint statistics of responses of cells sensitive to different position, phase, orientation, and scale. This "textural" representation by summary statistics predicts the subjective "jumble" of features often associated with crowding. We show that the difficulty of performing an identification task within a single pooling region using this representation of the stimuli is correlated with peripheral identification performance under conditions of crowding. Furthermore, for a simple stimulus with no flankers, this representation can be adequate to specify the stimulus with some position invariance. This provides evidence that a unified neuronal mechanism may underlie peripheral vision, ordinary pattern recognition in central vision, and texture perception. A key component of our methodology involves creating visualizations of the information available in the summary statistics of a stimulus. We call these visualizations "mongrels" and show that they are highly useful in examining how the early visual system represents the visual input. Mongrels enable one to study the "equivalence classes" of our model, i.e., the sets of stimuli that map to the same representation according to the model.

Not withstanding its great success and wide adoption in Bag-of-visual Words representation, visual vocabulary created from single image local features is often shown to be ineffective largely due to three reasons. First, many detected local features are not stable enough, resulting in many noisy and non-descriptive visual words in images. Second, single visual word discards the rich spatial contextual information among the local features, which has been proven to be valuable for visual matching. Third, the distance metric commonly used for generating visual vocabulary does not take the semantic context into consideration, which renders them to be prone to noise. To address these three confrontations, we propose an effective visual vocabulary generation framework containing three novel contributions: 1) we propose an effective unsupervised local feature refinement strategy; 2) we consider local features in groups to model their spatial contexts; 3) we further learn a discriminant distance metric between local feature groups, which we call discriminant group distance. This group distance is further leveraged to induce visual vocabulary from groups of local features. We name it contextual visual vocabulary, which captures both the spatial and semantic contexts. We evaluate the proposed local feature refinement strategy and the contextual visual vocabulary in two large-scale image applications: large-scale near-duplicate image retrieval on a dataset containing 1.5 million images and image search re-ranking tasks. Our experimental results show that the contextual visual vocabulary shows significant improvement over the classic visual vocabulary. Moreover, it outperforms the state-of-the-art Bundled Feature in the terms of retrieval precision, memory consumption and efficiency.

In computer vision, the bag-of-visual words image representation has been shown to yield good results. Recent work has shown that modeling the spatial relationship between visual words further improves performance. Previous work extracts higher-order spatial features exhaustively. However, these spatial features are expensive to compute. We propose a novel method that simultaneously performs feature selection and feature extraction. Higher-order spatial features are progressively extracted based on selected lower order ones, thereby avoiding exhaustive computation. The method can be based on any additive feature selection algorithm such as boosting. Experimental results show that the method is computationally much more efficient than previous approaches, without sacrificing accuracy.

This paper describes a multimodal (image + text) learning approach for automatically identifying three graphical figure types commonly found in biomedical literature, namely, diagrams, statistical figures and flow charts. The goal is to improve retrieval of figures from biomedical journal articles. In this article, we describe a data fusion approach to combine information from both text and image sources, believed to contain complementary information. Text information about the image is extracted from the figure caption. The data fusion process includes a hybrid of evolutionary algorithm (EA) and Binary Particle Swarm Optimization (BPSO) called method applied to find an optimal subset of extracted image features. Chi-square statistic and information gain metric are used to select the optimal subset of extracted text features, which along with image features are input to Multi-Layer Perceptron Neural Network classifiers, whose outputs are characterized as fuzzy sets to determine the final classification result. Evaluation performed on 1707 figure images extracted from a test subset of Biome Central® journals extracted from U.S. National Library of Medicine's PubMed Central ® repository yielded classification accuracy as high as 96.1%.

A rational model of human categorization behavior is presented that assumes that categorization reflects the derivation of optimal estimates of the probability of unseen features of objects. A Bayesian analysis is performed of what optimal estimations would be if categories formed a disjoint partitioning of the object space and if features were independently displayed within a category. This Bayesian analysis is placed within an incremental categorization algorithm. The resulting rational model accounts for effects of central tendency of categories, effects of specific instances, learning of linearly nonseparable categories, effects of category labels, extraction of basic level categories, base-rate effects, probability matching in categorization, and trial-by-trial learning functions. Although the rational model considers just I level of categorization, it is shown how predictions can be enhanced by considering higher and lower levels. Considering prediction at the lower, individual level allows integration of this rational analysis of categorization with the earlier rational analysis of memory (Anderson & Milson, 1989).

In the past decade, SIFT descriptor has been witnessed as one of the most robust local invariant feature descriptors and widely used in various vision tasks. Most traditional image-classification systems depend on the gray-based SIFT descriptors, which only analyze the gray level variations of the images. Misclassification may happen since their color contents are ignored. In this article, we concentrate on improving the performance of existing image-classification algorithms by adding color information. To achieve this purpose, different kinds of colored SIFT descriptors are introduced and implemented. locality-constrained linear coding (LLC), a state-of-the-art sparse coding technology, is employed to construct the image-classification system for the evaluation. Moreover, we propose a simple $$\ell _2$$ ℓ 2 -norm regularized local distance to improve the traditional LLC method. The real experiments are carried out on several benchmarks. With the enhancements to color SIFT and $$\ell _2$$ ℓ 2 -norm regularization, the proposed image-classification system obtains approximately $$2\,\%$$ 2 % improvement of classification accuracy on the Caltech-101 dataset and approximately $$5\,\%$$ 5 % improvement of classification accuracy on the Caltech-256 dataset.

Indoor scene recognition is a multi-faceted and challenging problem due to the diverse intra-class variations and the confusing inter-class similarities that characterize such scenes. This paper presents a novel approach that exploits rich mid-level convolutional features to categorize indoor scenes. Traditional convolutional features retain the global spatial structure, which is a desirable property for general object recognition. We, however, argue that the structure-preserving property of the convolutional neural network activations is not of substantial help in the presence of large variations in scene layouts, e.g., in indoor scenes. We propose to transform the structured convolutional activations to another highly discriminative feature space. The representation in the transformed space not only incorporates the discriminative aspects of the target data set but also encodes the features in terms of the general object categories that are present in indoor scenes. To this end, we introduce a new large-scale data set of 1300 object categories that are commonly present in indoor scenes. Our proposed approach achieves a significant performance boost over the previous state-of-the-art approaches on five major scene classification data sets.

Selective visual attention is believed to be responsible for serializing visual information for recognizing one object at a time in a complex scene. But how can we attend to objects before they are recognized? In coherence theory of visual cognition, so-called proto-objects form volatile units of visual information that can be accessed by selective attention and subsequently validated as actual objects. We propose a biologically plausible model of forming and attending to proto-objects in natural scenes. We demonstrate that the suggested model can enable a model of object recognition in cortex to expand from recognizing individual objects in isolation to sequentially recognizing all objects in a more complex scene.

The fact that objects in the world appear in different ways depending on the scale of observation has important implications if one aims at describing them. It shows that the notion of scale is of utmost importance when processing unknown measurement data by automatic methods. In their seminal works, Witkin (1983) and Koenderink (1984) proposed to approach this problem by representing image structures at different scales in a so-called scale-space representation. Traditional scale-space theory building on this work, however, does not address the problem of how to select local appropriate scales for further analysis. This article proposes a systematic methodology for dealing with this problem. A framework is presented for generating hypotheses about interesting scale levels in image data, based on a general principle stating that local extrema over scales of different combinations of γ-normalized derivatives are likely candidates to correspond to interesting structures. Specifically, it is shown how this idea can be used as a major mechanism in algorithms for automatic scale selection, which adapt the local scales of processing to the local image structure. Support for the proposed approach is given in terms of a general theoretical investigation of the behaviour of the scale selection method under rescalings of the input pattern and by integration with different types of early visual modules, including experiments on real-world and synthetic data. Support is also given by a detailed analysis of how different types of feature detectors perform when integrated with a scale selection mechanism and then applied to characteristic model patterns. Specifically, it is described in detail how the proposed methodology applies to the problems of blob detection, junction detection, edge detection, ridge detection and local frequency estimation. In many computer vision applications, the poor performance of the low-level vision modules constitutes a major bottleneck. It is argued that the inclusion of mechanisms for automatic scale selection is essential if we are to construct vision systems to automatically analyse complex unknown environments.

• Lighting. When an object is in bright light, it looks brighter than when it’s in shadow, so a program can’t just look at image intensity values. • Within-class variation. Different instances of the same kind of object can look quite different to one another. For example, a green station wagon and a red convertible are both cars, so a program can’t simply compare a picture to one example. • Aspect. The same object can look very different when viewed at from different directions—pick up a book and compare its cover and its spine to see this effect. Again, this means that a program might need to have many examples of each type of object. • Deformation. Many objects can change their appearance significantly without their identity changing. For example, you can move your limbs around, change clothes, paint your face, or have your hair cut. You will look very different indeed, but you will still be a person.


Question: A method for indexing multi-dimensional data within cloud-based systems.

This paper explores the innovative design and implementation of Dynamo, Amazon's highly available and resilient key-value storage system, which is pivotal in supporting the company's global e-commerce operations and ensuring an "always-on" experience for users. Given the immense scale of Amazon's infrastructure, encompassing thousands of servers and network components distributed across various datacenters worldwide, maintaining reliability amidst continuous component failures poses a significant challenge. Dynamo's architecture is meticulously crafted to tackle these issues by prioritizing availability over consistency under certain failure scenarios. It achieves this through the strategic use of object versioning and an application-assisted conflict resolution mechanism, offering developers a unique and robust interface to manage persistent state effectively. This approach not only enhances system scalability but also significantly reduces the financial impact and customer trust erosion associated with outages, making Dynamo a cornerstone in Amazon's service reliability framework.

Disk-oriented approaches to online storage are becoming increasingly problematic: they do not scale gracefully to meet the needs of large-scale Web applications, and improvements in disk capacity have far outstripped improvements in access latency and bandwidth. This paper argues for a new approach to datacenter storage called RAMCloud, where information is kept entirely in DRAM and large-scale systems are created by aggregating the main memories of thousands of commodity servers. We believe that RAMClouds can provide durable and available storage with 100-1000x the throughput of disk-based systems and 100-1000x lower access latency. The combination of low latency and large scale will enable a new breed of dataintensive applications.

Dynamic Searchable Symmetric Encryption (DSSE) allows search/update operations over encrypted data via an encrypted index. However, DSSE has been shown to be vulnerable to statistical inference attacks, which can extract a significant amount of information from access patterns on encrypted index and files. While generic Oblivious Random Access Machine (ORAM) can hide access patterns, it has been shown to be extremely costly to be directly used in DSSE setting. By exploiting the distributed cloud infrastructure, we develop a series of Oblivious Distributed DSSE schemes called ODSE, which enable oblivious access on the encrypted index with a high security and improved efficiency over the use of generic ORAM. Specifically, ODSE schemes are 3×-57× faster than applying the state-of-the-art generic ORAMs on encrypted dictionary index in real network settings. One of the proposed ODSE schemes offers desirable security guarantees such as informationtheoretic security with robustness against malicious servers. These properties are achieved by exploiting some of the unique characteristics of searchable encryption and encrypted index, which permits us to harness the computation and communication efficiency of multi-server PIR and Write-Only ORAM simultaneously. We fully implemented ODSE and have conducted extensive experiments to assess the performance of our proposed schemes in a real cloud environment.

Cloud storage has rapidly become a cornerstone of many IT infrastructures, constituting a seamless solution for the backup, synchronization, and sharing of large amounts of data. Putting user data in the direct control of cloud service providers, however, raises security and privacy concerns related to the integrity of outsourced data, the accidental or intentional leakage of sensitive information, the profiling of user activities and so on. Furthermore, even if the cloud provider is trusted, users having access to outsourced files might be malicious and misbehave. These concerns are particularly serious in sensitive applications like personal health records and credit score systems. To tackle this problem, we present GORAM, a cryptographic system that protects the secrecy and integrity of outsourced data with respect to both an untrusted server and malicious clients, guarantees the anonymity and unlinkability of accesses to such data, and allows the data owner to share outsourced data with other clients, selectively granting them read and write permissions. GORAM is the first system to achieve such a wide range of security and privacy properties for outsourced storage. In the process of designing an efficient construction, we developed two new, generally applicable cryptographic schemes, namely, batched zero-knowledge proofs of shuffle and an accountability technique based on chameleon signatures, which we consider of independent interest. We implemented GORAM in Amazon Elastic Compute Cloud (EC2) and ran a performance evaluation demonstrating the scalability and efficiency of our construction.

Many applications work with graph-structured data. As graphs grow in size, indexing becomes essential to ensure sufficient query performance. We present the GRIPP index structure (GRaph Indexing based on Pre- and Postorder numbering) for answering reachability queries in graphs.
 GRIPP requires only linear time and space. Using GRIPP, we can answer reachability queries on graphs with 5 million nodes on average in less than 5 milliseconds, which is unrivaled by previous methods. We evaluate the performance and scalability of our approach on real and synthetic random and scale-free graphs and compare our approach to existing indexing schemes. GRIPP is implemented as stored procedure inside a relational database management system and can therefore very easily be integrated into existing graph-oriented applications.

Data mining techniques are becoming indispensable as the amount and complexity of available data is rapidly growing. Visual data mining techniques attempt to include a human observer in the loop and leverage human perception for knowledge extraction. This is commonly allowed by performing a dimensionality reduction into a visually easy-to-perceive 2D space, which might result in significant loss of important spatial and topological information. To address this issue, this paper presents the design and implementation of a unique 3D visual data mining framework - CAVE-SOM. The CAVE-SOM system couples the Self-Organizing Map (SOM) algorithm with the immersive Cave Automated Virtual Environment (CAVE). The main advantages of the CAVE-SOM system are: i) utilizing a 3D SOM to perform dimensionality reduction of large multi-dimensional datasets, ii) immersive visualization of the trained 3D SOM, iii) ability to explore and interact with the multi-dimensional data in an intuitive and natural way. The CAVE-SOM system uses multiple visualization modes to guide the visual data mining process, for instance the data histograms, U-matrix, connections, separations, uniqueness and the input space view. The implemented CAVE-SOM framework was validated on several benchmark problems and then successfully applied to analysis of wind-power generation data. The knowledge extracted using the CAVE-SOM system can be used for further informed decision making and machine learning.

New generation of e-commerce applications require data sch emas that are constantly evolving and sparsely populated. The co nventional horizontal row representation fails to meet these re quir ments. We represent objects in a vertical format storing an o bject as a set of tuples. Each tuple consists of an object identifier and attribute name-value pair. Schema evolution is now easy. Ho wever, writing queries against this format becomes cumberso me. We create a logical horizontal view of the vertical representa tion and transform queries on this view to the vertical table. We pres ent alternative implementations and performance results that s ow the effectiveness of the vertical representation for sparse da ta. We also identify additional facilities needed in database systems to support these applications well.

We propose to extend database systems by a Skyline operation. This operation $filters out a set of interesting points from a potentially large set of data points. A point is interesting if it is not dominated by any other point. For example, a hotel might be interesting for somebody traveling to Nassau if no other hotel is both cheaper and closer to the beach. We show how SQL can be extended to pose Skyline queries, present and evaluate alternative algorithms to implement the Skyline operation, and show how this operation can be combined with other database operations, e.g., join.

We consider extending decision support facilities toward large sophisticated networks, upon which multidimensional attributes are associated with network entities, thereby forming the so-called multidimensional networks. Data warehouses and OLAP (Online Analytical Processing) technology have proven to be effective tools for decision support on relational data. However, they are not well-equipped to handle the new yet important multidimensional networks. In this paper, we introduce Graph Cube, a new data warehousing model that supports OLAP queries effectively on large multidimensional networks. By taking account of both attribute aggregation and structure summarization of the networks, Graph Cube goes beyond the traditional data cube model involved solely with numeric value based group-by's, thus resulting in a more insightful and structure-enriched aggregate network within every possible multidimensional space. Besides traditional cuboid queries, a new class of OLAP queries, crossboid, is introduced that is uniquely useful in multidimensional networks and has not been studied before. We implement Graph Cube by combining special characteristics of multidimensional networks with the existing well-studied data cube techniques. We perform extensive experimental studies on a series of real world data sets and Graph Cube is shown to be a powerful and efficient tool for decision support on large multidimensional networks.

File correlation, which refers to a relationship among related files that can manifest in the form of their common access locality (temporal and/or spatial), has become an increasingly important consideration for performance enhancement in peta-scale storage systems. Previous studies on file correlations mainly concern with two aspects of files: file access sequence and semantic attribute. Based on mining with regard to these two aspects of file systems, various strategies have been proposed to optimize the overall system performance. Unfortunately, all of these studies consider either file access sequences or semantic attribute information separately and in isolation, thus unable to accurately and effectively mine file correlations, especially in large-scale distributed storage systems.
 This paper introduces a novel File Access coRrelation Mining and Evaluation Reference model (FARMER) for optimizing petascale file system performance that judiciously considers both file access sequences and semantic attributes simultaneously to evaluate the degree of file correlations by leveraging the Vector Space Model (VSM) technique adopted from the Information Retrieval field. We extract the file correlation knowledge from some typical file system traces using FARMER, and incorporate FARMER into a real large-scale object-based storage system as a case study to dynamically infer file correlations and evaluate the benefits and costs of a FARMER-enabled prefetching algorithm for the metadata servers under real file system workloads. Experimental results show that FARMER can mine and evaluate file correlations more accurately and effectively. More significantly, the FARMER-enabled prefetching algorithm is shown to reduce the metadata operations latency by approximately 24-35% when compared to a state-of-the-art metadata prefetching algorithm and a commonly used replacement policy.

We present LocationSpark, a spatial data processing system built on top of Apache Spark, a widely used distributed data processing system. LocationSpark offers a rich set of spatial query operators, e.g., range search, kNN, spatio-textual operation, spatial-join, and kNN-join. To achieve high performance, LocationSpark employs various spatial indexes for in-memory data, and guarantees that immutable spatial indexes have low overhead with fault tolerance. In addition, we build two new layers over Spark, namely a query scheduler and a query executor. The query scheduler is responsible for mitigating skew in spatial queries, while the query executor selects the best plan based on the indexes and the nature of the spatial queries. Furthermore, to avoid unnecessary network communication overhead when processing overlapped spatial data, We embed an efficient spatial Bloom filter into LocationSpark’s indexes. Finally, LocationSpark tracks frequently accessed spatial data, and dynamically flushes less frequently accessed data into disk. We evaluate our system on real workloads and demonstrate that it achieves an order of magnitude performance gain over a baseline framework.


Question: A method for indexing multi-dimensional data within cloud systems.

Introducing HadoopDB: A Fusion of MapReduce and DBMS for Enhanced Analytical Processing in Cloud Environments

The landscape of analytical data management is undergoing a transformative shift, with businesses increasingly transitioning from high-cost, proprietary hardware to cost-effective, commodity systems. This migration often takes place within shared-nothing MPP architectures, embedded in virtualized public or private cloud environments. The exponential growth in data volumes necessitates the concurrent processing capabilities of hundreds to thousands of machines to conduct in-depth analysis. The choice of technology for handling such vast datasets typically falls into two camps: parallel databases or MapReduce-based systems.

Supporters of parallel databases highlight their superior performance and efficiency, making them ideal for complex analytical tasks. Conversely, advocates of MapReduce systems emphasize their unparalleled scalability, fault tolerance, and adaptability in managing unstructured data. In light of these contrasting viewpoints, our research delves into the possibility of creating a hybrid solution that merges the best attributes of both technologies. The prototype developed in our study not only matches the performance and efficiency standards set by parallel databases but also inherits the scalability, fault tolerance, and flexibility associated with MapReduce systems.

Our hybrid approach, HadoopDB, represents a significant advancement in analytical workload management within cloud systems. By harnessing the strengths of both MapReduce and DBMS technologies, HadoopDB offers an optimized solution for multi-dimensional data indexing in cloud environments, ensuring efficient data processing while maintaining high levels of fault tolerance and scalability. This innovative blend of technologies is poised to redefine the standards for analytical data management in the era of big data and cloud computing.

We propose to extend database systems by a Skyline operation. This operation $filters out a set of interesting points from a potentially large set of data points. A point is interesting if it is not dominated by any other point. For example, a hotel might be interesting for somebody traveling to Nassau if no other hotel is both cheaper and closer to the beach. We show how SQL can be extended to pose Skyline queries, present and evaluate alternative algorithms to implement the Skyline operation, and show how this operation can be combined with other database operations, e.g., join.

Cloud computing is an emerging computing model in which resources of the computing communications are provided as services over the Internet. Privacy and security of cloud storage services are very important and become a challenge in cloud computing due to loss of control over data and its dependence on the cloud computing provider. While there is a huge amount of transferring data in cloud system, the risk of accessing data by attackers raises. Considering the problem of building a secure cloud storage service, current scheme is proposed which is based on combination of RSA and AES encryption methods to share the data among users in a secure cloud system. The proposed method allows providing difficulty for attackers as well as reducing the time of information transmission between user and cloud data storage.

Cloud computing promises a more cost effective enabling technology to outsource storage and computations. Existing approaches for secure outsourcing of data and arbitrary computations are either based on a single tamper-proof hardware, or based on recently proposed fully homomorphic encryption. The hardware based solutions are not scaleable, and fully homomorphic encryption is currently only of theoretical interest and very inefficient. In this paper we propose an architecture for secure outsourcing of data and arbitrary computations to an untrusted commodity cloud. In our approach, the user communicates with a trusted cloud (either a private cloud or built from multiple secure hardware modules) which encrypts and verifies the data stored and operations performed in the untrusted commodity cloud. We split the computations such that the trusted cloud is mostly used for security-critical operations in the less time-critical setup phase, whereas queries to the outsourced data are processed in parallel by the fast commodity cloud on encrypted data.

Cloud Computing has been envisioned as the next-generation architecture of IT Enterprise. It moves the application software and databases to the centralized large data centers, where the management of the data and services may not be fully trustworthy. This unique paradigm brings about many new security challenges, which have not been well understood. This work studies the problem of ensuring the integrity of data storage in Cloud Computing. In particular, we consider the task of allowing a third party auditor (TPA), on behalf of the cloud client, to verify the integrity of the dynamic data stored in the cloud. The introduction of TPA eliminates the involvement of client through the auditing of whether his data stored in the cloud is indeed intact, which can be important in achieving economies of scale for Cloud Computing. The support for data dynamics via the most general forms of data operation, such as block modification, insertion and deletion, is also a significant step toward practicality, since services in Cloud Computing are not limited to archive or backup data only. While prior works on ensuring remote data integrity often lacks the support of either public verifiability or dynamic data operations, this paper achieves both. We first identify the difficulties and potential security problems of direct extensions with fully dynamic data updates from prior works and then show how to construct an elegant verification scheme for seamless integration of these two salient features in our protocol design. In particular, to achieve efficient data dynamics, we improve the Proof of Retrievability model [1] by manipulating the classic Merkle Hash Tree (MHT) construction for block tag authentication. Extensive security and performance analysis show that the proposed scheme is highly efficient and provably secure.

While big data workflows haven been proposed recently as the next-generation data-centric workflow paradigm to process and analyze data of ever increasing in scale, complexity, and rate of acquisition, a scalable distributed data model is still missing that abstracts and automates data distribution, parallelism, and scalable processing. In the meanwhile, although NoSQL has emerged as a new category of data models, they are optimized for storing and querying of large datasets, not for ad-hoc data analysis where data placement and data movement are necessary for optimized workflow execution. In this paper, we propose a NoSQL data model that: 1) supports high-performance MapReduce-style workflows that automate data partitioning and data-parallelism execution. In contrast to the traditional MapReduce framework, our MapReduce-style workflows are fully composable with other workflows enabling dataflow applications with a richer structure, 2) automates virtual machine provisioning and deprovisioning on demand according to the sizes of input datasets, 3) enables a flexible framework for workflow executors that take advantage of the proposed NoSQL data model to improve the performance of workflow execution. Our case studies and experiments show the competitive advantages of our proposed data model. The proposed NoSQL data model is implemented in a new release of DATAVIEW, one of the most usable big data workflow systems in the community.

The ability to handle large scale graph data is crucial to an increasing number of applications. Much work has been dedicated to supporting basic graph operations such as subgraph matching, reachability, regular expression matching, etc. In many cases, graph indices are employed to speed up query processing. Typically, most indices require either super-linear indexing time or super-linear indexing space. Unfortunately, for very large graphs, super-linear approaches are almost always infeasible. In this paper, we study the problem of subgraph matching on billion-node graphs. We present a novel algorithm that supports efficient subgraph matching for graphs deployed on a distributed memory store. Instead of relying on super-linear indices, we use efficient graph exploration and massive parallel computing for query processing. Our experimental results demonstrate the feasibility of performing subgraph matching on web-scale graph data.

The Internet of Things (IoT) has provided a promising opportunity to build powerful industrial systems and applications by leveraging the growing ubiquity of Radio Frequency IDentification (RFID) and wireless sensors devices. Benefiting from RFID and sensor network technology, common physical objects can be connected, and are able to be monitored and managed by a single system. Such a network brings a series of challenges for data storage and processing in a cloud platform. IoT data can be generated quite rapidly, the volume of data can be huge and the types of data can be various. In order to address these potential problems, this paper proposes a data storage framework not only enabling efficient storing of massive IoT data, but also integrating both structured and unstructured data. This data storage framework is able to combine and extend multiple databases and Hadoop to store and manage diverse types of data collected by sensors and RFID readers. In addition, some components are developed to extend the Hadoop to realize a distributed file repository, which is able to process massive unstructured files efficiently. A prototype system based on the proposed framework is also developed to illustrate the framework's effectiveness.

In recent years, privacy preserving data mining has become an important problem because of the large amount of personal data which is tracked by many business applications. In many cases, users are unwilling to provide personal information unless the privacy of sensitive information is guaranteed. In this paper, we propose a new framework for privacy preserving data mining of multi-dimensional data. Previous work for privacy preserving data mining uses a perturbation approach which reconstructs data distributions in order to perform the mining. Such an approach treats each dimension independently and therefore ignores the correlations between the different dimensions. In addition, it requires the development of a new distribution based algorithm for each data mining problem, since it does not use the multi-dimensional records, but uses aggregate distributions of the data as input. This leads to a fundamental re-design of data mining algorithms. In this paper, we will develop a new and flexible approach for privacy preserving data mining which does not require new problem-specific algorithms, since it maps the original data set into a new anonymized data set. This anonymized data closely matches the characteristics of the original data including the correlations among the different dimensions. We present empirical results illustrating the effectiveness of the method.

The use of high-dimension features is unavoidable in many applications of image retrieval and techniques of dimension reductions are not always efficient. The space-filling curve reduces the number of dimensions to one while preserving the neighborhood relation. In this paper, Hilbert curve, the most neighborhood preserving space-filling curve, is used in shape retrieval. The retrieving is accelerated by ordering the shapes in 1-D dynamical data structure, which enables rapid insertions of new images without changing existing data. A proposal of fast mapping facilitate the computing of 1-D Hilbert indexes from high dimensional features.

In most parallel DBSCAN algorithms, neighboring points are assigned to the same data partition for parallel processing to facilitate calculation of the density of the neighbors. This data partitioning scheme causes a few critical problems including load imbalance between data partitions, especially in a skewed data set. To remedy these problems, we propose a cell-based data partitioning scheme, pseudo random partitioning , that randomly distributes small cells rather than the points themselves. It achieves high load balance regardless of data skewness while retaining the data contiguity required for DBSCAN. In addition, we build and broadcast a highly compact summary of the entire data set, which we call a two-level cell dictionary , to supplement random partitions. Then, we develop a novel parallel DBSCAN algorithm, Random Partitioning-DBSCAN (shortly, RP-DBSCAN), that uses pseudo random partitioning together with a two-level cell dictionary. The algorithm simultaneously finds the local clusters to each data partition and then merges these local clusters to obtain global clustering. To validate the merit of our approach, we implement RP-DBSCAN on Spark and conduct extensive experiments using various real-world data sets on 12 Microsoft Azure machines (48 cores). In RP-DBSCAN, data partitioning and cluster merging are very light, and clustering on each split is not dragged out by a specific worker. Therefore, the performance results show that RP-DBSCAN significantly outperforms the state-of-the-art algorithms by up to 180 times.


Question: A method for indexing multi-dimensional data within a cloud-based system.

This document discusses MapReduce, a pioneering framework designed for efficient data processing on extensive clusters, which has proven its versatility across a wide spectrum of practical applications. The core functionality of MapReduce is defined by the user through the specification of a map and a reduce function. The underlying runtime system, in turn, automates the process of distributing the computation across a multitude of machines, ensuring scalability and robustness by handling potential machine failures. Moreover, it optimizes inter-machine communication to maximize the utilization of network resources and disk I/O, thereby enhancing overall efficiency. The simplicity and ease of use of MapReduce have been widely recognized by developers, as evidenced by the creation of over ten thousand unique MapReduce programs at Google within a four-year timeframe. On a daily basis, Google's clusters execute an average of one hundred thousand MapReduce jobs, collectively processing a staggering volume of over twenty petabytes of data. This highlights the framework's capability to manage and process massive datasets in a cloud environment, making it a cornerstone technology for large-scale data processing tasks.

The Internet of Things (IoT) has provided a promising opportunity to build powerful industrial systems and applications by leveraging the growing ubiquity of Radio Frequency IDentification (RFID) and wireless sensors devices. Benefiting from RFID and sensor network technology, common physical objects can be connected, and are able to be monitored and managed by a single system. Such a network brings a series of challenges for data storage and processing in a cloud platform. IoT data can be generated quite rapidly, the volume of data can be huge and the types of data can be various. In order to address these potential problems, this paper proposes a data storage framework not only enabling efficient storing of massive IoT data, but also integrating both structured and unstructured data. This data storage framework is able to combine and extend multiple databases and Hadoop to store and manage diverse types of data collected by sensors and RFID readers. In addition, some components are developed to extend the Hadoop to realize a distributed file repository, which is able to process massive unstructured files efficiently. A prototype system based on the proposed framework is also developed to illustrate the framework's effectiveness.

SpatialHadoop is an extended MapReduce framework that supp orts global indexing that spatial partitions the data across machines providing orders of magnitude speedup, compared to traditi onal Hadoop. In this paper, we describe seven alternative partit ioning techniques and experimentally study their effect on the qua lity of the generated index and the performance of range and spatial join queries. We found that using a 1% sample is enough to produce high quality partitions. Also, we found that the total area o f partitions is a reasonable measure of the quality of indexes when r unning spatial join. This study will assist researchers in cho osing a good spatial partitioning technique in distributed enviro nments. 1. INDEXING IN SPATIALHADOOP SpatialHadoop [2, 3] provides a generic indexing algorithm which was used to implement grid, R-tree, and R+-tree based p artitioning. This paper extends our previous study by introduci ng four new partitioning techniques, Z-curve, Hilbert curve, Quad tree, and K-d tree, and experimentally evaluate all of the seven techn iques. The partitioning phase of the indexing algorithm runs in thr ee steps, where the first step is fixed and the last two steps are customiz ed for each partitioning technique. The first step computes number of desired partitionsn based on file size and HDFS block capacity which are both fixed for all partitioning techniques. The second st ep reads a random sample, with a sampling ratio ρ, from the input file and uses this sample to partition the space into cells such that number of sample points in each cell is at most ⌊k/n⌋, wherek is the sample size. The third step actually partitions the file by as signing each record to one or more cells. Boundary objects are handle d using either thedistribution or replication methods. Thedistribution method assigns an object to exactly one overlapping cell and the cell has to be expanded to enclose all contained records. The replication method avoids expanding cells by replicating each record to all overlapping cells but the query processor has to employ a duplicate avoidance technique to account for replicated records . ∗This work is supported in part by the National Science Founda tion, USA, under Grants IIS-0952977 and IIS-1218168 and the University of Minnesota Doctoral Disseration Fellowship. This work is licensed under the Creative Commons Attributio nNonCommercial-NoDerivs 3.0 Unported License. To view a cop y f this license, visit http://creativecommons.org/licenses/by-n c-nd/3.0/. Obtain permission prior to any use beyond those covered by the license. Contact copyright holder by emailing info@vldb.org. Articles fromthis volume were invited to present their results at the 41st Internatio l Conference on Very Large Data Bases, August 31st September 4th 2015, Koha la Coast, Hawaii. Proceedings of the VLDB Endowment, Vol. 8, No. 12 Copyright 2015 VLDB Endowment 2150-8097/15/08. 2. EXPERIMENTAL SETUP All experiments run on Amazon EC2 ‘m1.large’ instances which have a dual core processor, 7.5 GB RAM and 840 GB disk storage. We use Hadoop 1.2.1 running on Java 1.6 and CentOS 6. Each machine is configured to run three mappers and two reduce rs. Tables 1 and 2 summarize the datasets and configuration param eters used in our experiments, respectively. Default parame ters (in parentheses) are used unless otherwise mentioned. In the fo llowing part, we describe the partitioning techniques, the quer ies we run, and the performance metrics measured in this paper. 2.1 Partitioning Techniques This paper employsgrid and Quad tree as space partitioning techniques;STR, STR+, and K-d tree as data partitioning techniques; andZ-curve andHilbert curve as space filling curve (SFC) partitioning techniques. These techniques can also be grou ped, according to boundary object handling, into replication-based techniques (i.e., Grid, Quad, STR+, and K-d tree) and distributionbased techniques (i.e., STR, Z-Cruve, and Hilbert). Figure 1 illustrates these techniques, where sample points and partition bou daries are shown as dots and rectangles, respectively. 1. Uniform Grid: This technique does not require a random sample as it divides the input MBR using a uniform grid of ⌈√n⌉ × ⌈√n⌉ grid cells and employs the replication method to handle boundary objects. 2. Quad tree: This technique inserts all sample points into a quad tree [6] with node capacity of ⌊k/n⌋, wherek is the sample size. The boundaries of all leaf nodes are used as cell boundaries. W use the replication method to assign records to cells. 3. STR: This technique bulk loads the random sample into an Rtree using the STR algorithm [8] and the capacity of each node is set to⌊k/n⌋. The MBRs of leaf nodes are used as cell boundaries. Boundary objects are handled using the distribution method where it assigns a record r to the cell with maximal overlap. 4. STR+: This technique is similar to the STR technique but it uses the replication method to handle boundary objects. 5. K-d tree: This technique uses the K-d tree [1] partitioning method to partition the space into n cells. It starts with the input MBR as one cell and partitions it n − 1 times to producen cells. Records are assigned to cells using the replication method. 6. Z-curve: This technique sorts the sample points by their order on the Z-curve and partitions the curve into n splits, each containing roughly⌊k/n⌋ points. It uses the distribution method to assign a recordr to one cell by mapping the center point of its MBR to one of then splits. 7. Hilbert curve: This technique is exactly the same as the Zcurve technique but it uses Hilbert space filling curve which has better spatial properties. (a) Grid (b) Quad Tree (c) STR and STR+ (d) K-d Tree (e) Z-curve (f) Hilbert Curve Figure 1: Partitioning Techniques. Examples of nearly-empty partitions are starred. Name Size Records Average Record Size All Objects 88GB 250M 378 bytes Buildings 25GB 109M 234 bytes Roads 23GB 70M 337 bytes Lakes 8.6GB 9M 1KB Cities 1.4GB 170K 8.4KB

On-line analytical processing (OLAP) is a recent and important application of database systems. Typically, OLAP data is presented as a multidimensional \data cube." OLAP queries are complex and can take many hours or even days to run, if executed directly on the raw data. The most common method of reducing execution time is to precompute some of the queries into summary tables (subcubes of the data cube) and then to build indexes on these summary tables. In most commercial OLAP systems today, the summary tables that are to be precomputed are picked rst, followed by the selection of the appropriate indexes on them. A trial-and-error approach is used to divide the space available between the summary tables and the indexes. This two-step process can perform very poorly. Since both summary tables and indexes consume the same resource |space | their selection should be done together for the most e cient use of space. In this paper, we give algorithms that automate the selection of summary tables and indexes. In particular, we present a family of algorithms of increasing time complexities, and prove strong performance bounds for them. The algorithms with higher complexities have better performance bounds. However, the increase in the performance bound is diminishing, and we show that an algorithm of moderate complexity can perform fairly close to the optimal. This work was supported by NSF grant IRI{92{23405, ARO grant DAAH04{95{1{0192, and Air Force Contract F33615{93{ 1{1339. Present address of V. Harinarayan and A. Rajaraman: Junglee Corp., Palo Alto, CA.

Now a day's cloud computing is used in many areas like industry, military colleges etc to storing huge amount of data. We can retrieve data from cloud on request of user. To store data on cloud we have to face many issues. To provide the solution to these issues there are n number of ways. Cryptography and steganography techniques are more popular now a day's for data security. Use of a single algorithm is not effective for high level security to data in cloud computing. In this paper we have introduced new security mechanism using symmetric key cryptography algorithm and steganography. In this proposed system AES, blowfish, RC6 and BRA algorithms are used to provide block wise security to data. All algorithm key size is 128 bit. LSB steganography technique is introduced for key information security. Key information contains which part of file is encrypted using by which algorithm and key. File is splited into eight parts. Each and every part of file is encrypted using different algorithm. All parts of file are encrypted simultaneously with the help of multithreading technique. Data encryption Keys are inserted into cover image using LSB technique. Stego image is send to valid receiver using email. For file decryption purpose reverse process of encryption is applied.

Data access control is an effective way to ensure data security in the cloud. However, due to data outsourcing and untrusted cloud servers, the data access control becomes a challenging issue in cloud storage systems. Existing access control schemes are no longer applicable to cloud storage systems, because they either produce multiple encrypted copies of the same data or require a fully trusted cloud server. Ciphertext-policy attribute-based encryption (CP-ABE) is a promising technique for access control of encrypted data. However, due to the inefficiency of decryption and revocation, existing CP-ABE schemes cannot be directly applied to construct a data access control scheme for multiauthority cloud storage systems, where users may hold attributes from multiple authorities. In this paper, we propose data access control for multiauthority cloud storage (DAC-MACS), an effective and secure data access control scheme with efficient decryption and revocation. Specifically, we construct a new multiauthority CP-ABE scheme with efficient decryption, and also design an efficient attribute revocation method that can achieve both forward security and backward security. We further propose an extensive data access control scheme (EDAC-MACS), which is secure under weaker security assumptions.

Data Security is the most critical issues in a cloud computing environment. Authentication is a key technology for information security, which is a mechanism to establish proof of identities to get access of information in the system. Traditional password authentication does not provide enough security for information in cloud computing environment to the most modern means of attacks. In this paper, we propose a new multi-factor authentication framework for cloud computing. In this paper the features of various access control mechanisms are discussed and a novel framework of access control is proposed for cloud computing, which provides a multi -step and multifactor authentication of a user. The model proposed is well-organized and provably secure solution of access control for externally hosted applications.

With the emergence of cloud computing, the concept of information security has become a major issue. Indeed, the security of such a system is the greatest concern of computer scientists, providers of cloud and organizations who want to adopt and benefit from these services. Cloud computing providers must implement concepts ensuring network security, hardware, data storage and strategies of control and access to services. All these elements help to preserve data security and ensuring the availability of services associated with the Cloud, to better satisfy clients and acquire and build their trust. However, even if the data storage security in cloud servers is assured, reluctance remain when it comes to process the confidential data. Indeed, the fear that sensitive data is being used is a major obstacle in the adoption of cloud services by enterprises. To overcome this obstacle, the use of methods that can perform operations on encrypted data without knowing the secret key, seems to be an effective way to strengthen the confidentiality of information. In this paper we will examine the challenges facing Homomorphic Encryption methods to allow suppliers of cloud to perform operations on encrypted data, and provide the same results after treatment, as if they were performing calculations on raw data.

Now a day’s Peta-Bytes of data becomes the norm in industries. Handling, analyzing such big data is challenging task. Even frameworks like Hadoop (Open Source Implementation of MapReduce Paradigm) and NoSQL databases like Cassandra, HBase can be used to analyze and store such large data; heterogeneity of data is still an issue. Data centers usually have clusters formed using heterogeneous nodes. Ecosystem like Hadoop can be used to manage such types of cluster but it can not schedule jobs (Application) efficiently on this heterogeneous cluster when data is itself heterogeneous. Heterogeneity of data may be because of data format or because of the complexity. This paper is review of systems and algorithms for distributed management of huge size data, efficiency of these approaches.

Traditional information retrieval techniques that primarily rely on keyword-based linking of the query and document spaces face challenges such as the vocabulary mismatch problem where relevant documents to a given query might not be retrieved simply due to the use of different terminology for describing the same concepts. As such, semantic search techniques aim to address such limitations of keyword-based retrieval models by incorporating semantic information from standard knowledge bases such as Freebase and DBpedia. The literature has already shown that while the sole consideration of semantic information might not lead to improved retrieval performance over keyword-based search, their consideration enables the retrieval of a set of relevant documents that cannot be retrieved by keyword-based methods. As such, building indices that store and provide access to semantic information during the retrieval process is important. While the process for building and querying keyword-based indices is quite well understood, the incorporation of semantic information within search indices is still an open challenge. Existing work have proposed to build one unified index encompassing both textual and semantic information or to build separate yet integrated indices for each information type but they face limitations such as increased query process time. In this paper, we propose to use neural embeddings-based representations of term, semantic entity, semantic type and documents within the same embedding space to facilitate the development of a unified search index that would consist of these four information types. We perform experiments on standard and widely used document collections including Clueweb09-B and Robust04 to evaluate our proposed indexing strategy from both effectiveness and efficiency perspectives. Based on our experiments, we find that when neural embeddings are used to build inverted indices; hence relaxing the requirement to explicitly observe the posting list key in the indexed document: (a) retrieval efficiency will increase compared to a standard inverted index, hence reduces the index size and query processing time, and (b) while retrieval efficiency, which is the main objective of an efficient indexing mechanism improves using our proposed method, retrieval effectiveness also retains competitive performance compared to the baseline in terms of retrieving a reasonable number of relevant documents from the indexed corpus. Email addresses: fatemeh.lashkari@unb.ca (Fatemeh Lashkari), bagheri@ryerson.ca (Ebrahim Bagheri), ghorbani@unb.ca (Ali A. Ghorbani) Preprint submitted to Information Processing and Management September 11, 2018

We introduce a new compression scheme for high-dimensional vectors that approximates the vectors using sums of M codewords coming from M different codebooks. We show that the proposed scheme permits efficient distance and scalar product computations between compressed and uncompressed vectors. We further suggest vector encoding and codebook learning algorithms that can minimize the coding error within the proposed scheme. In the experiments, we demonstrate that the proposed compression can be used instead of or together with product quantization. Compared to product quantization and its optimized versions, the proposed compression approach leads to lower coding approximation errors, higher accuracy of approximate nearest neighbor search in the datasets of visual descriptors, and lower image classification error, whenever the classifiers are learned on or applied to compressed vectors.


Question: A method for indexing multi-dimensional data within a cloud computing environment.

This document introduces Xen, a sophisticated x86 virtual machine monitor designed to enable multiple commodity operating systems to coexist on standard hardware in a secure, resource-managed environment, without compromising performance or functionality. Unlike previous systems that either required specialized hardware, could not support mainstream operating systems, or prioritized compatibility over performance, Xen strikes a balance between these trade-offs. It offers resource isolation and performance guarantees, avoiding the risk of denial of service associated with best-effort provisioning.

Xen's design is centered around hosting up to 100 virtual machine instances concurrently on a contemporary server. The virtualization technique employed by Xen is notably efficient, allowing operating systems such as Linux and Windows XP to run simultaneously with minimal performance overhead—typically only a few percentage points compared to non-virtualized environments. This efficiency far surpasses that of rival commercial and open-source solutions, as demonstrated through a variety of microbenchmarks and comprehensive system tests.

In summary, Xen represents a significant advancement in virtualization technology, offering a secure, high-performance platform for hosting multiple operating systems on a single piece of hardware, thus maximizing resource utilization and flexibility.

This paper presents the design of a read-optimized relational DBMS that contrasts sharply with most current systems, which are write-optimized. Among the many differences in its design are: storage of data by column rather than by row, careful coding and packing of objects into storage including main memory during query processing, storing an overlapping collection of columnoriented projections, rather than the current fare of tables and indexes, a non-traditional implementation of transactions which includes high availability and snapshot isolation for read-only transactions, and the extensive use of bitmap indexes to complement B-tree structures. We present preliminary performance data on a subset of TPC-H and show that the system we are building, C-Store, is substantially faster than popular commercial products. Hence, the architecture looks very encouraging.

Cloud computing promises flexibility and high performance for users and high cost-efficiency for operators. Nevertheless, most cloud facilities operate at very low utilization, hurting both cost effectiveness and future scalability.
 We present Quasar, a cluster management system that increases resource utilization while providing consistently high application performance. Quasar employs three techniques. First, it does not rely on resource reservations, which lead to underutilization as users do not necessarily understand workload dynamics and physical resource requirements of complex codebases. Instead, users express performance constraints for each workload, letting Quasar determine the right amount of resources to meet these constraints at any point. Second, Quasar uses classification techniques to quickly and accurately determine the impact of the amount of resources (scale-out and scale-up), type of resources, and interference on performance for each workload and dataset. Third, it uses the classification results to jointly perform resource allocation and assignment, quickly exploring the large space of options for an efficient way to pack workloads on available resources. Quasar monitors workload performance and adjusts resource allocation and assignment when needed. We evaluate Quasar over a wide range of workload scenarios, including combinations of distributed analytics frameworks and low-latency, stateful services, both on a local cluster and a cluster of dedicated EC2 servers. At steady state, Quasar improves resource utilization by 47% in the 200-server EC2 cluster, while meeting performance constraints for workloads of all types.

Many important problems involve clustering large datasets. Although naive implementations of clustering are computationally expensive, there are established efficient techniques for clustering when the dataset has either (1) a limited number of clusters, (2) a low feature dimensionality, or (3) a small number of data points. However, there has been much less work on methods of efficiently clustering datasets that are large in all three ways at once—for example, having millions of data points that exist in many thousands of dimensions representing many thousands of clusters. We present a new technique for clustering these large, highdimensional datasets. The key idea involves using a cheap, approximate distance measure to efficiently divide the data into overlapping subsets we call canopies. Then clustering is performed by measuring exact distances only between points that occur in a common canopy. Using canopies, large clustering problems that were formerly impossible become practical. Under reasonable assumptions about the cheap distance metric, this reduction in computational cost comes without any loss in clustering accuracy. Canopies can be applied to many domains and used with a variety of clustering approaches, including Greedy Agglomerative Clustering, K-means and Expectation-Maximization. We present experimental results on grouping bibliographic citations from the reference sections of research papers. Here the canopy approach reduces computation time over a traditional clustering approach by more than an order of magnitude and decreases error in comparison to a previously used algorithm by 25%.

Approximate Nearest Neighbor (ANN) search in high dimensional space has become a fundamental paradigm in many applications. Recently, Locality Sensitive Hashing (LSH) and its variants are acknowledged as the most promising solutions to ANN search. However, state-of-the-art LSH approaches suffer from a drawback: accesses to candidate objects require a large number of random I/O operations. In order to guarantee the quality of returned results, sufficient objects should be verified, which would consume enormous I/O cost. To address this issue, we propose a novel method, called SortingKeys-LSH (SK-LSH), which reduces the number of page accesses through locally arranging candidate objects. We firstly define a new measure to evaluate the distance between the compound hash keys of two points. A linear order relationship on the set of compound hash keys is then created, and the corresponding data points can be sorted accordingly. Hence, data points that are close to each other according to the distance measure can be stored locally in an index file. During the ANN search, only a limited number of disk pages among few index files are necessary to be accessed for sufficient candidate generation and verification, which not only significantly reduces the response time but also improves the accuracy of the returned results. Our exhaustive empirical study over several real-world data sets demonstrates the superior efficiency and accuracy of SK-LSH for the ANN search, compared with state-of-the-art methods, including LSB, C2LSH and CK-Means.

Recently, MapReduce frameworks, e.g., Hadoop, have been used extensively in different applications that include tera-byte sorting, machine learning, and graph processing. With the huge volumes of spatial data coming from different sources, there is an increasing demand to exploit the efficiency of Hadoop, coupled with the flexibility of the MapReduce framework, in spatial data processing. However, Hadoop falls short in supporting spatial data efficiently as the core is unaware of spatial data properties. This paper describes SpatialHadoop; a full-edged MapReduce framework with native support for spatial data. SpatialHadoop is a comprehensive extension to Hadoop that injects spatial data awareness in each Hadoop layer, namely, the language, storage, MapReduce, and operations layers. In the language layer, SpatialHadoop adds a simple and ex- pressive high level language for spatial data types and operations. In the storage layer, SpatialHadoop adapts traditional spatial index structures, Grid, R-tree and R+-tree, to form a two-level spatial index. SpatialHadoop enriches the MapReduce layer by two new components, SpatialFileSplitter and SpatialRecordReader, for efficient and scalable spatial data processing. In the operations layer, SpatialHadoop is already equipped with a dozen of operations, including range query, kNN, and spatial join. The flexibility and open source nature of SpatialHadoop allows more spatial operations to be implemented efficiently using MapReduce. Extensive experiments on a real system prototype and real datasets show that SpatialHadoop achieves orders of magnitude better performance than Hadoop for spatial data processing.

In the big data era, duplicated copies of data may be generated by various smart devices. Although convergent encryption has been extensively adopted for secure deduplication, it is still very challenging to efficiently and reliably manage the convergent keys and support keyword search over encrypted data. This paper addresses this problem by presenting an efficient scheme for big data outsourcing with secure deduplication, named BDO-SD. Specifically, with the convergent encryption technique, data owners can outsource their data to the cloud server with data deduplication; with the idea of public key encryption and keywords search, a data user can query encrypted data while preserving its privacy. Security analysis demonstrates that our proposed BDO-SD scheme can achieve the data confidentiality, query privacy and key security. Simulation results show that compared with other schemes, our scheme can achieve better efficiency than the existing data deduplication schemes.

A broad class of algorithms for knowledge discovery in databases (KDD) relies heavily on similarity queries, i.e. range queries or nearest neighbor queries, in multidimensional feature spaces. Many KDD algorithms perform a similarity query for each point stored in the database. This approach causes serious performance degenerations if the considered data set does not fit into main memory. Usual cache strategies such as LRU fail because the locality of KDD algorithms is typically not high enough. In this paper, we propose to replace repeated similarity queries by the similarity join, a database primitive prevalent in multimedia database systems. We present a schema to transform query intensive KDD algorithms into a representation using the similarity join as a basic operation without affecting the correctness of the result of the considered algorithm. In order to perform a comprehensive experimental evaluation of our approach, we apply the proposed transformation to the clustering algorithm DBSCAN and to the hierarchical cluster structure analysis method OPTICS. Our technique allows the application of any similarity join algorithm, which may be based on index structures or not. In our experiments, we use a similarity join algorithm based on a variant of the X-tree. The experiments yield substantial performance improvements of our technique over the original algorithms. The traditional techniques are outperformed by factors of up to 33 for the X-tree and 54 for the R*-tree.

Now a day’s Peta-Bytes of data becomes the norm in industries. Handling, analyzing such big data is challenging task. Even frameworks like Hadoop (Open Source Implementation of MapReduce Paradigm) and NoSQL databases like Cassandra, HBase can be used to analyze and store such large data; heterogeneity of data is still an issue. Data centers usually have clusters formed using heterogeneous nodes. Ecosystem like Hadoop can be used to manage such types of cluster but it can not schedule jobs (Application) efficiently on this heterogeneous cluster when data is itself heterogeneous. Heterogeneity of data may be because of data format or because of the complexity. This paper is review of systems and algorithms for distributed management of huge size data, efficiency of these approaches.

This paper presents the design of an MPI-based parallel and distributed machine learning platform on large-scale HPC clusters. Researchers and practitioners can implement easily a class of parallelizable machine learning algorithms on the platform, or port quickly an existing non-parallel implementation of a parallelizable algorithm to the platform with only minor modifications. Complicated functions in parallel programming such as scheduling, caching and load balancing are handled automatically by the platform. The platform performance was evaluated in a series of stress tests by using a k-means clustering task on 7,500 hours of speech data (about 2.7 billion 52dimensional feature vectors). Good scalability is demonstrated on an HPC cluster with thousands of CPU cores.

Problem statement: Financial time series were usually large in size, unstructured and of high dimensionality. Since, the illustration of fin a cial time series shape was typically characterize d by a few number of important points. These important p oints moved in zigzag directions which could form technical patterns. However, these important p oints exhibited in different resolutions and diffic ult to determine. Approach: In this study, we proposed novel methods of financ i l time series indexing by considering their zigzag movement. The methods c onsist of two major algorithms: first, the identification of important points, namely the Zigz ag-Perceptually Important Points (ZIPs) identification method and next, the indexing method namely Zigzag based M-ary Tree (ZM-Tree) to structure and organize the important points. Results: The errors of the tree building and retrieving compared to the original time series increased when t important points increased. The dimensionality reduction using ZM-Tree based on tree pruning and n umber of retrieved points techniques performed better when the number of important points increase d. Conclusion: Our proposed techniques illustrated mostly acceptable performance in tree op rations and dimensionality reduction comparing to existing similar technique like Specialize Binar y Tree (SB-Tree).


Question: A method for indexing multi-dimensional data within a cloud computing environment.

This paper outlines the development of C-Store, a column-oriented Database Management System (DBMS) designed for read-optimized operations, marking a significant departure from the prevalent write-optimized systems. Its innovative design features include data storage by column instead of by row, meticulous object coding and packing into storage, including main memory during query processing, and the maintenance of an overlapping set of column-oriented projections, as opposed to the traditional tables and indexes. The system also introduces a non-standard transaction implementation, ensuring high availability and snapshot isolation for read-only transactions, along with the strategic use of bitmap indexes to supplement B-tree structures. Preliminary performance evaluations, conducted using a subset of TPC-H benchmarks, reveal that C-Store outperforms leading commercial products, suggesting a promising future for this architectural approach.

**Question**: The impact of social media on mental health in adolescents

**Documents**: The role of social media in adolescent mental health: A review
The use of social media among adolescents has grown rapidly in recent years, leading to concerns about its potential impact on mental health. This review examines the existing literature on the relationship between social media use and adolescent mental health. Studies have found mixed results, with some suggesting that social media use can lead to increased feelings of loneliness, anxiety, and depression, while others have found no significant association or even positive effects on mental health. Factors such as frequency of use, type of social media, and individual differences in personality and coping skills may play a role in moderating the relationship between social media and mental health. The review also discusses the potential mechanisms underlying the link between social media use and mental health, including social comparison, cyberbullying, and sleep disruption. Implications for future research and clinical practice are discussed.

**Similar Documents**: This comprehensive review explores the complex interplay between social media usage and the mental health of adolescents, a topic of growing concern given the exponential rise in social media engagement among this demographic. Research findings are contradictory, with some studies linking excessive social media use to heightened feelings of isolation, anxiety, and depression, while others report no significant correlation or even beneficial effects on mental well-being. The variability in outcomes appears to be influenced by factors such as the intensity of social media engagement, the specific platform in use, and individual characteristics like personality traits and coping mechanisms. The review delves into potential explanatory frameworks for the relationship between social media and mental health, including the negative effects of social comparison, the prevalence of cyberbullying, and the disruption of sleep patterns. The implications of these findings for future

Cloud computing is the one of the admired paradigms of current era, which facilitates the users with on demand services and pay as you use services. It has tremendous applications in almost every sphere such as education, gaming, social networking, transportation, medical, business, stock market, pattern matching, etc. Stock market is such an industry where lots of data is generated and benefits are reaped on the basis of accurate prediction. So prediction is a vital part of stock market. Therefore, an attempt is being made to predict the stock market based on the given data set of stock market along with some features; using the techniques available for predictive data mining. Machine learning is one of the upcoming trends of data mining; hence few machine learning algorithms have been used such as Decision tree, Linear model, Random forest and further their results have been compared using the classification evaluation parameters such as H, AUC, ROC, TPR, FPR, etc. Random forest have been consider as the most effective model as it yield the highest accuracy of 54.12% whereas decision tree and linear model gives the accuracy of 51.87% and 52.83% respectively.

This paper studies two design tasks faced by a geo-distributed cloud data market: which data to purchase data purchasing and where to place/replicate the data for delivery data placement. We show that the joint problem of data purchasing and data placement within a cloud data market can be viewed as a facility location problem and is thus NP-hard. However, we give a provably optimal algorithm for the case of a data market made up of a single data center and then generalize the structure from the single data center setting in order to develop a near-optimal, polynomial-time algorithm for a geo-distributed data market. The resulting design, $\mathsf {Datum}$ , decomposes the joint purchasing and placement problem into two subproblems, one for data purchasing and one for data placement, using a transformation of the underlying bandwidth costs. We show, via a case study, that $\mathsf {Datum}$ is near optimal within 1.6% in practical settings.

This paper proposes a comprehensive critical survey on the issues of warehousing and protecting big data, which are recognized as critical challenges of emerging big data research. Indeed, both are critical aspects to be considered in order to build truly, high-performance and highly-flexible big data management systems. We report on state-of-the-art approaches, methodologies and trends, and finally conclude by providing open problems and challenging research directions to be considered by future efforts.

Problem statement: Financial time series were usually large in size, unstructured and of high dimensionality. Since, the illustration of fin a cial time series shape was typically characterize d by a few number of important points. These important p oints moved in zigzag directions which could form technical patterns. However, these important p oints exhibited in different resolutions and diffic ult to determine. Approach: In this study, we proposed novel methods of financ i l time series indexing by considering their zigzag movement. The methods c onsist of two major algorithms: first, the identification of important points, namely the Zigz ag-Perceptually Important Points (ZIPs) identification method and next, the indexing method namely Zigzag based M-ary Tree (ZM-Tree) to structure and organize the important points. Results: The errors of the tree building and retrieving compared to the original time series increased when t important points increased. The dimensionality reduction using ZM-Tree based on tree pruning and n umber of retrieved points techniques performed better when the number of important points increase d. Conclusion: Our proposed techniques illustrated mostly acceptable performance in tree op rations and dimensionality reduction comparing to existing similar technique like Specialize Binar y Tree (SB-Tree).

Conjunctive Boolean queries are a key component of modern information retrieval systems, especially when Web-scale repositories are being searched. A conjunctive query q is equivalent to a |q|-way intersection over ordered sets of integers, where each set represents the documents containing one of the terms, and each integer in each set is an ordinal document identifier. As is the case with many computing applications, there is tension between the way in which the data is represented, and the ways in which it is to be manipulated. In particular, the sets representing index data for typical document collections are highly compressible, but are processed using random access techniques, meaning that methods for carrying out set intersections must be alert to issues to do with access patterns and data representation. Our purpose in this article is to explore these trade-offs, by investigating intersection techniques that make use of both uncompressed “integer” representations, as well as compressed arrangements. We also propose a simple hybrid method that provides both compact storage, and also faster intersection computations for conjunctive querying than is possible even with uncompressed representations.

Event log processing and analysis play a key role in applications ranging from security management, IT trouble shooting, to user behavior analysis. Recent years have seen a rapid growth in system scales and the corresponding rapid increase in the amount of log event data. At the same time, as logs are found to be a valuable information source, log analysis tasks have become more sophisticated demanding both interactive exploratory query processing and batch computation. Desirable query types include selection with time ranges and value filtering criteria, join within time windows, join between log data and reference tables, and various aggregation types. In such a situation, parallel solutions are necessary, but existing parallel and distributed solutions either support limited query types or perform only batch computations on logs. With a system called LogKV, this paper reports a first study of using Key-Value stores to support log processing and analysis, exploiting the scalability, reliability, and efficiency commonly found in Key-Value store systems. LogKV contains a number of unique techniques that are needed to handle log data in terms of event ingestion, load balancing, storage optimization, and query processing. Preliminary experimental results show that LogKV is a promising solution.

The Internet of Things (IoT) has provided a promising opportunity to build powerful industrial systems and applications by leveraging the growing ubiquity of Radio Frequency IDentification (RFID) and wireless sensors devices. Benefiting from RFID and sensor network technology, common physical objects can be connected, and are able to be monitored and managed by a single system. Such a network brings a series of challenges for data storage and processing in a cloud platform. IoT data can be generated quite rapidly, the volume of data can be huge and the types of data can be various. In order to address these potential problems, this paper proposes a data storage framework not only enabling efficient storing of massive IoT data, but also integrating both structured and unstructured data. This data storage framework is able to combine and extend multiple databases and Hadoop to store and manage diverse types of data collected by sensors and RFID readers. In addition, some components are developed to extend the Hadoop to realize a distributed file repository, which is able to process massive unstructured files efficiently. A prototype system based on the proposed framework is also developed to illustrate the framework's effectiveness.

Cloud computing provides convenient on-demand network access to a shared pool of configurable computing resources. The resources can be rapidly deployed with great efficiency and minimal management overhead. Cloud is an insecure computing platform from the view point of the cloud users, the system must design mechanisms that not only protect sensitive information by enabling computations with encrypted data, but also protect users from malicious behaviours by enabling the validation of the computation result. In this paper, we propose a new data encoding scheme called layered interleaving, designed for time-sensitive packet recovery in the presence of bursty loss. It is high-speed data recovery scheme with minimal loss probability and using a forward error correction scheme to handle bursty loss. The proposed approach is highly efficient in recovering the singleton losses almost immediately and from bursty data losses.

Cloud computing is a new paradigm to deliver services over the Internet. Data Security is the most critical issues in a cloud computing environment. Authentication is a key technology for information security, which is a mechanism to establish proof of identities to get access of information in the system. Traditional password authentication does not provide enough security for information in cloud computing environment to the most modern means of attacks. In this paper, we propose a new multi-factor authentication framework for cloud computing. The proposed framework provides a feasible and a most efficient mechanism which can closely integrate with the traditional authentication system.The proposed framework is verified by developing Cloud Access Management (CAM) system which authenticates the user based on multiple factors. Also using secret-splitting and encrypted value of arithmetic captcha is innovative factor for user authentication for cloud computing environment. Prototype model for cloud computing own cloud server is implemented using open sources technology. The proposed framework shows the close agreement with the standard criteria for security.

Indexing and query processing is an emerging research field in spatio temporal data. Most of the real-time applications such as location based services, fleet management, traffic prediction and radio frequency identification and sensor networks are based on spatiotemporal indexing and query processing. All the indexing and query processing applications is any one of the forms, such as spatio index access and supporting queries or spatio-temporal indexing method and support query or temporal dimension, while in spatial data it is considered as the second priority. In this paper, give the survey of the various uncertain indexing and query processing techniques. Most of the existing survey works on spatio-temporal are based on indexing methods and query processing, but presented separately. Both the indexing and querying are related, hence state of art of both the indexing and query processing techniques are considered together. This paper gives the details of spatio-temporal data classification, various types of indexing methods, query processing, application areas and research direction of spatio-temporal indexing and query