Question: A two-phase technique for clustering mixed categorical and numeric data.

This document introduces an innovative Similarity-Based Agglomerative Clustering (SBAC) algorithm, specifically designed to handle datasets with a mix of numeric and nominal features effectively. The algorithm employs a similarity metric, initially proposed by Goodall for biological taxonomy, which emphasizes the significance of uncommon feature value matches in the computation of similarity and makes no presumptions about the underlying distributions of feature values. This metric is pivotal in defining the similarity between object pairs. To construct a dendrogram, an agglomerative approach is utilized, and a straightforward distinctness heuristic is applied to derive a partition of the data. The SBAC algorithm's performance has been rigorously evaluated on both real-world and synthetic datasets, showcasing its efficacy in unsupervised learning tasks. Comparative analyses with alternative clustering methodologies highlight the superior performance of the SBAC approach in tackling mixed data types. Key terms include agglomerative clustering, conceptual clustering, feature weighting, interpretation, knowledge discovery, mixed numeric and nominal data, similarity measures, and aggregation.

This paper presents the results of an experimental study of some common document clustering techniques. In particular, we compare the two main approaches to document clustering, agglomerative hierarchical clustering and K-means. (For K-means we used a “standard” K-means algorithm and a variant of K-means, “bisecting” K-means.) Hierarchical clustering is often portrayed as the better quality clustering approach, but is limited because of its quadratic time complexity. In contrast, K-means and its variants have a time complexity which is linear in the number of documents, but are thought to produce inferior clusters. Sometimes K-means and agglomerative hierarchical approaches are combined so as to “get the best of both worlds.” However, our results indicate that the bisecting K-means technique is better than the standard K-means approach and as good or better than the hierarchical approaches that we tested for a variety of cluster evaluation metrics. We propose an explanation for these results that is based on an analysis of the specifics of the clustering algorithms and the nature of document data. 1 Background and Motivation Document clustering has been investigated for use in a number of different areas of text mining and information retrieval. Initially, document clustering was investigated for improving the precision or recall in information retrieval systems [Rij79, Kow97] and as an efficient way of finding the nearest neighbors of a document [BL85]. More recently, clustering has been proposed for use in browsing a collection of documents [CKPT92] or in organizing the results returned by a search engine in response to a user’s query [ZEMK97]. Document clustering has also been used to automatically generate hierarchical clusters of documents [KS97]. (The automatic generation of a taxonomy of Web documents like that provided by Yahoo!

Web mining is the sub category or application of data mining techniques to extract knowledge from Web data. With the advent of new advancements in technology the rapid use of new algorithms has been increased in the market. A data mining is one of the fast growing research field which is used in a wide areas of applications. The data mining consists of classification algorithms, association algorithms and searching algorithms. Different classification and clustering algorithm are used for the synthetic datasets. In this paper various techniques that are based on clustering and decision tree for synthesized data mining are discussed.

Missing values in datasets should be extracted from the datasets or should be estimated before they are used for classification, association rules or clustering in the preprocessing stage of data mining. In this study, we utilize a fuzzy c-means clustering hybrid approach that combines support vector regression and a genetic algorithm. In this method, the fuzzy clustering parameters, cluster size and weighting factor are optimized and missing values are estimated. The proposed novel hybrid method yields sufficient and sensible imputation performance results. The results are compared with those of fuzzy c-means genetic algorithm imputation, support vector regression genetic algorithm imputation and zero imputation. 2013 Elsevier Inc. All rights reserved.

A mixture model for preferences data, which adequately represents the composite nature of the elicitation mechanism in ranking processes, is proposed. Both probabilistic features of the mixture distribution and inferential and computational issues arising from the maximum likelihood parameters estimation are addressed. Moreover, empirical evidence from different data sets confirming the goodness of fit of the proposed model to many real preferences data is shown. © 2004 Elsevier B.V. All rights reserved.

Due to the technological progress over the last decades, today’s scientific and commercial applications are capable of generating, storing, and processing, massive amounts of data sets. This influences the type of data generated, which in turn means that with each data entry di erent aspects are combined and stored into one common database. Often the describing attributes are numeric; we name data with more than a handful attributes (dimensions) high-dimensional. Having to make use of these types of data archives provides new challenges to analysis techniques. The work of this thesis centers around the question of finding interesting patterns (meaningful information) in high-dimensional data sets. This task is highly challenging because of the so called curse of dimensionality, expressing that when dimensionality increases the data becomes sparse. This phenomena disturbs standard analysis techniques. Automatic techniques have to deal with the data complexity not only increasing their runtime, but also vitiating their computation functions (like distance functions). Moreover, exploring these data sets visually is hindered by the high number of dimensions that have to be displayed on the two dimensional screen space. This thesis is motivated by the idea that searching for interesting patterns in this kind of data can be done through a mixed approach of automation, visualization, and interaction. The amount of patterns a visualization contains can be measured by so called quality metrics. These automated functions can then filter the high number of highdimensional visualizations and present to the user a pre-filtered good subset for further investigation. We propose quality metrics for scatterplots and parallel coordinates focusing on di erent user tasks like identifying clusters and correlations. We also evaluate these measures with regard to (1) their ability to identify clusters in a variety of real and synthetic datasets; (2) their correlation with human perception of clusters in scatterplots. A thorough discussion of results follows reflecting the impact on directions for future research. As quality metrics were developed for a large number of di erent high-dimensional visualization techniques, we present our reflections on how these methods are related to each other and how the approach can be developed further. For this purpose, we provide an overview of approaches that use quality metrics in high-dimensional data visualization and propose a systematization based on a comprehensive literature review. In high-dimensional data, patterns exist often only in a subset of the dimensions. Subspace clustering techniques aim at finding these subspaces where clusters exist and which might otherwise be hidden if a traditional clustering algorithm is applied. While subspace clustering approaches tackle the sparsity problem in high-dimensional data well, designing e ective visualization to help analyzing the clustering result is not trivial. In addition to the cluster membership information, the relevant sets of dimensions and the overlaps of memberships and dimensions need to also be considered. Although, a number of techniques (for example, scatterplots, heat maps, dendrograms, hierarchical parallel coordinates) exist for visualizing traditional clustering results, little research has been done for visualizing subspace clustering results. Moreover, while extensive research has been carried out with regard to designing subspace clustering algorithms, surprisingly little attention has been paid to the developing of e ective visualization tools analyzing the viii clustering result. Appropriate visualization techniques will not only help in monitoring the clustering process but, with special mining techniques, they could also enable the domain expert to guide and even to steer the subspace clustering process to reveal the patterns of interest. To this goal, we envision a concept that combines subspace clustering algorithms and interactive scalable visual exploration techniques. This work includes the task of comparative visualization and feedback guided computation of alternative clusterings. Zusammenfassung Bedingt durch den technologischen Fortschritt der letzten Jahrzehnte sind heutige kommerzielle Applikationen in der Lage, riesige Datenmengen zu erzeugen, zu speichern und zu verarbeiten. Diese Entwicklung beeinflusst auch die Natur der erzeugten Daten, d.h. dass für jeden Dateneintrag unterschiedliche Aspekte in der gleichen Datenbank gespeichert werden. Oft sind die beschreibenden Attribute numerisch. Datensätze, die mehr als fünf solcher Attribute (Dimensionen) beinhalten, nenne ich hochdimensional. Der wertbringende Gebrauch solcher Datenarchive bringt neue Herausforderungen an Analysetechniken mit sich. Die vorliegende Dissertation bearbeitet die Fragestellung, wie interessante Muster (bedeutende Information) in hochdimensionalen Räumen gefunden werden können. Diese Aufgabenstellung ist durch das Problem des Fluches der Dimensionalität äußerst herausfordernd. Dieses Problem besagt, dass Daten im hochdimensionalen Raum spärlich vorkommen. Herkömmliche Analysetechniken werden dadurch beeinträchtigt. Automatische Methoden müssen die Datenkomplexität nicht nur ihre Laufzeit, sondern auch ihre Berechnungsfunktionen (z.B. Distanzfunktionen) betre end, einbeziehen. Außerdem wird die visuelle Exploration dieser Daten durch die Zweidimensionalität der Darstellungen beeinträchtigt. Diese Dissertation stützt sich auf das Konzept, dass die Suche nach interessanten Mustern in hochdimensionalen Datenmengen mit einem kombinierten Ansatz von automatischen, visuellen und interaktiven Methoden durchgeführt werden kann. Die Ausprägung der Muster einer Visualisierung kann durch sogenannte Qualitätsmaße gemessen werden. Durch diese automatischen Funktionen kann die große Menge an hochdimensionalen Visualisierungen eingegrenzt und dem Benutzer eine ausgewählte Menge zur weiteren Untersuchung zur Verfügung gestellt werden. Ich schlage Qualitätsmaße für Scatterplots und Parallele Koordinaten vor, die sich auf unterschiedliche Aufgaben, wie die Identifikation von Gruppen oder Korrelationen, konzentrieren. Zusätzlich werden diese Techniken bezüglich (1) ihrer Fähigkeit Cluster in unterschiedlichen realen und synthetischen Datensätzen und (2) ihrer Korrelation mit der menschlichen Wahrnehmung untersucht. Der ausführlichen Diskussion dieser Resultate folgen Überlegungen für die zukünftige Forschung. Da viele verschiedene Qualitätsmaße für eine Reihe weiterer hochdimensionaler Visualisierungen entwickelt wurden, werde ich Vorschläge für deren Vernetzung und Weiterentwicklung vorstellen. Hierfür wird eine Übersicht über die verschiedenen Ansätze erstellt, welcher eine Systematisierung zugrunde liegt, die aufgrund einer umfassenden Literaturauswertung zustande kam. Im hochdimensionalen Raum existieren manche Muster nur in verschiedenen Unterräumen des Datenraumes. Subspace Clustering Algorithmen wurden entwickelt, um Unterräume zu finden in denen Cluster existieren, die durch traditionelle Clustering Algorithmen nicht gefunden werden würden. Obwohl diese Algorithmen spärlich mit Daten besetzte, hochdimensionale Räume gut explorieren können, ist das Entwickeln von e ektiven Visualisierungstechniken, um diese Clusteringresultate zu analysieren, nicht trivial. Zusätzlich zu der Clusterzugehörigkeit von Elementen müssen die relevanten Attributmengen eines Clusters und die Objektund Dimensionsüberlappungen von Subspaceclusx tern dargestellt werden. Auch wenn eine Reihe von Techniken für die Visualisierung von traditionellen Clustering Resultaten existiert (z.B. Scatterplots, Heatmaps, Dendrogramme, hierarchische Parallele Koordinaten) gibt es nur wenige Ansätze, um das Resultat von Subspace Clustering Algorithmen zu visualisieren. Außerdem wurden bisher erstaunlich wenige Ansätze vorgestellt, die eine visuelle Analyse der Subspace Clustering Ergebnisse unterstützen können, obwohl im Bereich der Subspace Clustering Algorithmen viel Forschung betrieben wurde. Angemessene Visualisierungstechniken, die von speziellen Methoden zur Extraktion von Informationen unterstützt werden, würden nicht nur die Nachverfolgung der Clustering Ergebnisse ermöglichen, sondern auch Fachleuten dabei helfen, den Subspace Clustering Prozess so zu steuern, dass relevante Muster zum Vorschein kommen. Dieses Ziel vor Augen stelle ich ein Konzept vor, das Subspace Clustering Algorithmen mit interaktiven skalierbaren Visualisierungen kombiniert. Meine Ansätze widmen sich deshalb der Aufgabe der Visualisierung zum Vergleich von alternativen Clustergruppen, die durch Nutzerfeedback gesteuert werden.

To cluster increasingly massive data sets that are common today in data and text mining, we propose a parallel implementation of the k-means clustering algorithm based on the message passing model. The proposed algorithm exploits the inherent data-parallelism in the kmeans algorithm. We analytically show that the speedup and the scaleup of our algorithm approach the optimal as the number of data points increases. We implemented our algorithm on an IBM POWERparallel SP2 with a maximum of 16 nodes. On typical test data sets, we observe nearly linear relative speedups, for example, 15.62 on 16 nodes, and essentially linear scaleup in the size of the data set and in the number of clusters desired. For a 2 gigabyte test data set, our implementation drives the 16 node SP2 at more than 1.8 gigaflops.

Clustering methods are among the most commonly used tools for exploratory data analysis. However, using clustering to perform data analysis can be challenging for modern datasets that contain a large number of dimensions, are complex in nature, and lack a ground-truth labeling. Traditional tools, like summarization and plotting of clusters, are of limited benefit in a high-dimensional setting. On the other hand, while many clustering methods have been studied theoretically, such analysis often has limited instructive value for practical purposes due to unverifiable assumptions, oracle-dependent tuning parameters, or unquantified finite-sample effects.

The clustering techniques are the most important part of the data analysis and k-means is the oldest and popular clustering technique used. The paper discusses the traditional K-means algorithm with advantages and disadvantages of it. It also includes researched on enhanced k-means proposed by various authors and it also includes the techniques to improve traditional K-means for better accuracy and efficiency. There are two area of concern for improving K-means; 1) is to select initial centroids and 2) by assigning data points to nearest cluster by using equations for calculating mean and distance between two data points. The time complexity of the proposed K-means technique will be lesser that then the traditional one with increase in accuracy and efficiency. The main purpose of the article is to proposed techniques to enhance the techniques for deriving initial centroids and the assigning of the data points to its nearest clusters. The clustering technique proposed in this paper is enhancing the accuracy and time complexity but it still needs some further improvements and in future it is also viable to include efficient techniques for selecting value for initial clusters(k). Experimental results show that the improved method can effectively improve the speed of clustering and accuracy, reducing the computational complexity of the k-means. Introduction Clustering is a process of grouping data objects into disjointed clusters so that the data in the same cluster are similar, but data belonging to different cluster differ. A cluster is a collection of data object that are similar to one another are in same cluster and dissimilar to the objects are in other clusters [k-4]. At present the applications of computer technology in increasing rapidly which Unnati R. Raval et al, International Journal of Computer Science and Mobile Computing, Vol.5 Issue.5, May2016, pg. 191-203 © 2016, IJCSMC All Rights Reserved 192 created high volume and high dimensional data sets [10]. These data is stored digitally in electronic media, thus providing potential for the development of automatic data analysis, classification and data retrieval [10]. The clustering is important part of the data analysis which partitioned given dataset in to subset of similar data points in each subset and dissimilar to data from other clusters [1]. The clustering analysis is very useful with increasing in digital data to draw meaningful information or drawing interesting patters from the data sets hence it finds applications in many fields like bioinformatics, pattern recognition, image processing, data mining, marketing and economics etc [4]. There have been many clustering techniques proposed but K-means is one of the oldest and most popular clustering techniques. In this method the number of cluster (k) is predefined prior to analysis and then the selection of the initial centroids will be made randomly and it followed by iterative process of assigning each data point to its nearest centroid. This process will keep repeating until convergence criteria met. However, there are shortcomings of K-means, it is important to proposed techniques that enhance the final result of analysis. This article includes researched on papers [1,2,3,4,5,6] which made some very important improvements towards the accuracy and efficiency of the clustering technique. Basic K-means Algorithm : A centroid-based Clustering technique According to the basic K-mean clustering algorithm, clusters are fully dependent on the selection of the initial clusters centroids. K data elements are selected as initial centers; then distances of all data elements are calculated by Euclidean distance formula. Data elements having less distance to centroids are moved to the appropriate cluster. The process is continued until no more changes occur in clusters[k-1]. This partitioning clustering is most popular and fundamental technique [1]. It is vastly used clustering technique which requires user specified parameters like number of clusters k, cluster initialisation and cluster metric [2]. First it needs to define initial clusters which makes subsets (or groups) of nearest points (from centroid) inside the data set and these subsets (or groups) called clusters [1]. Secondly, it finds means value for each cluster and define new centroid to allocate data points to this new centroid and this iterative process will goes on until centroid [3] does not changes. The simplest algorithm for the traditional K-means [2] is as follows; Input: D = {d1, d2, d3,.......dn} // set of n numbers of data points K // The number of desire Clusters Output: A set of k clusters 1. Select k points as initial centroids. 2. Repeat 3. From K clusters by assigning each data point to its nearest centroid. 4. Recompute the centroid for each cluster until centroid does not change [2]. Unnati R. Raval et al, International Journal of Computer Science and Mobile Computing, Vol.5 Issue.5, May2016, pg. 191-203 © 2016, IJCSMC All Rights Reserved 193 Start

We present new deterministic methods that given two eigenspace models, each representing a set of n-dimensional observations will: (1) merge the models to yield a representation of the union of the sets; (2) split one model from another to represent the difference between the sets; as this is done, we accurately keep track of the mean. These methods are more efficient than computing new eigenspace models directly from the observations when the eigenmodels are dimensionally small compared to the total number of observations. Such methods are important because they provide a basis for novel techniques in machine learning, using a dynamic split-andmerge paradigm to optimally cluster observations. Here we present a theoretical derivation of the methods, empirical results relating to the efficiency and accuracy of the techniques, and three general applications, including the on-line construction of Gaussian mixture models.

Clustering is a process that aims to group the similar records in one cluster and dissimilar records in different clusters. K-means is one of the most popular and well-known clustering technique for its simplicity and light weight. However, the main drawback of K-means clustering technique is that it requires a user (data miner) to estimate the number of clusters in advance. Another limitation of K-means is that it has a tendency to get stuck at local optima. In order to overcome these limitations many evolutionary algorithm based clustering techniques have been proposed since the 1990s and applied to various fields. In this paper, we present an up-to-date review of some major evolutionary algorithm based clustering techniques for the last twenty (20) years (1995-2015). A total of 63 ranked (i.e. based on citation reports and JCR/CORE rank) evolutionary algorithm based clustering approaches are reviewed. Maximum of the techniques do not require any user to define the number of clusters in advance. We present the limitations and advantages of some evolutionary algorithm based clustering techniques. We also present a thorough discussion and future research directions of evolutionary algorithm based clustering techniques.


Question: A two-phase technique for grouping mixed categorical and numerical data.

Introducing Chameleon: A Dynamic Modeling Approach to Hierarchical Clustering
Clustering, a fundamental data mining technique, aims to organize data into groups based on maximizing intra-cluster similarity and minimizing inter-cluster similarity, thus elucidating the inherent structure of data distributions. This process is critical for various applications, such as segmenting customers by purchasing habits, categorizing web content, classifying genes and proteins by function, and identifying earthquake-prone areas using seismological data. However, traditional clustering algorithms often struggle with clusters of varying shapes, densities, and sizes due to their reliance on static models that may not accurately reflect the data's complexity.

Chameleon, a novel hierarchical clustering algorithm, addresses these limitations by incorporating dynamic modeling techniques. Unlike static models, Chameleon dynamically adapts to the characteristics of individual clusters as they are merged, providing a more nuanced approach to clustering. Existing algorithms often fail to consider either the interconnectivity or the proximity of items across clusters, leading to suboptimal clustering outcomes. For instance, algorithms that prioritize interconnectivity may erroneously merge clusters that are closer to another cluster in terms of distance, as illustrated in Figure 2.

Chameleon, on the other hand, integrates both interconnectivity and proximity into its clustering process, ensuring that clusters are merged based on a comprehensive understanding of their relationships. This dual consideration prevents the algorithm from merging clusters solely based on superficial criteria, such as proximity without regard for interconnectivity, or vice versa. By dynamically assessing these factors, Chameleon is better equipped to handle diverse cluster characteristics and achieve more accurate clustering results. This approach is particularly advantageous when dealing with complex datasets where clusters may exhibit a wide range of properties.

We propose a two-class classification model for grouping. Human segmented natural images are used as positive examples. Negative examples of grouping are constructed by randomly matching human segmentations and images. In a preprocessing stage an image is oversegmented into superpixels. We define a variety of features derived from the classical Gestalt cues, including contour, texture, brightness and good continuation. Information-theoretic analysis is applied to evaluate the power of these grouping cues. We train a linear classifier to combine these features. To demonstrate the power of the classification model, a simple algorithm is used to randomly search for good segmentations. Results are shown on a wide range of images.

Data mining technology has attracted significant interest as a means of identifying patterns and trends from large collections of data. It is however evident that the collection and analysis of data that include personal information may violate the privacy of the individuals to whom information refers. Privacy protection in data mining is then becoming a crucial issue that has captured the attention of many researchers. In this chapter, we first describe the concept of k-anonymity and illustrate different approaches for its enforcement. We then discuss how the privacy requirements characterized by k-anonymity can be violated in data mining and introduce possible approaches to ensure the satisfaction of k-anonymity in data mining.

Previous studies propose that associative classification has high classification accuracy and strong flexibility at handling unstructured data. However, it still suffers from the huge set of mined rules and sometimes biased classification or overfitting since the classification is based on only single high-confidence rule. In this study, we propose a new associative classification method, CMAR, i.e., Classification based on Multiple Association Rules. The method extends an efficient frequent pattern mining method, FP-growth, constructs a class distribution-associated FP-tree, and mines large database efficiently. Moreover, it applies a CR-tree structure to store and retrieve mined association rules efficiently, and prunes rules effectively based on confidence, correlation and database coverage. The classification is performed based on a weighted analysis using multiple strong association rules. Our extensive experiments on databases from UCI machine learning database repository show that CMAR is consistent, highly effective at classification of various kinds of databases and has better average classification accuracy in comparison with CBA and C4.5. Moreover, our performance study shows that the method is highly efficient and scalable in comparison with other reported associative classification methods.

Data mining technique has attracted attention in the information industry and society as a whole, because of the big amount of data and the imminent need to transform that data into useful information and knowledge. Recently conducted studies with successfully demarcated results using this technique, to estimate several parameters in a variety of domains. However, the effective use of data in some areas is still developing, as is the case of sports, which has shown moderate growth. In this context, the objective of this article is to present a systematic review of the literature about research involving sports data mining. As systematic searches were made out in five databases, resulting in 21 articles that answered a question that grounded this article.

Many different relative clustering validity criteria exist that are very useful in practice as quantitative measures for evaluating the quality of data partitions, and new criteria have still been proposed from time to time. These criteria are endowed with particular features that may make each of them able to outperform others in specific classes of problems. In addition, they may have completely different computational requirements. Then, it is a hard task for the user to choose a specific criterion when he or she faces such a variety of possibilities. For this reason, a relevant issue within the field of clustering analysis consists of comparing the performances of existing validity criteria and, eventually, that of a new criterion to be proposed. In spite of this, the comparison paradigm traditionally adopted in the literature is subject to some conceptual limitations. The present paper describes an alternative, possibly complementary methodology for comparing clustering validity criteria and uses it to make an extensive comparison of the performances of 40 criteria over a collection of 962,928 partitions derived from five well-known clustering algorithms and 1080 different data sets of a given class of interest. A detailed review of the relative criteria under investigation is also provided that includes an original comparative asymptotic analysis of their computational complexities. This work is intended to be a complement of the classic study reported in 1985 by Milligan and Cooper as well as a thorough extension of a preliminary paper by the authors themselves.  2010 Wiley Periodicals, Inc. Statistical Analysis and Data Mining 3:

Clustering is an important research topic in data mining that appears in a wide range of unsupervised classification applications. Partitional clustering algorithms such as the k-means algorithm are the most popular for clustering large datasets. The major problem with the k-means algorithm is that it is sensitive to the selection of the initial partitions and it may converge to local optima. In this paper, we present a hybrid two-phase GAI-PSO+k-means data clustering algorithm that performs fast data clustering and can avoid premature convergence to local optima. In the first phase we utilize the new genetically improved particle swarm optimization algorithm (GAI-PSO) which is a population-based heuristic search technique modeled on the hybrid of cultural and social rules derived from the analysis of the swarm intelligence (PSO) and the concepts of natural selection and evolution (GA). The GAI-PSO combines the standard velocity and position update rules of PSOs with the ideas of selection, mutation and crossover from GAs. The GAI-PSO algorithm searches the solution space to find the optimal initial cluster centroids for the next phase. The second phase is a local refining stage utilizing the k-means algorithm which can efficiently converge to the optimal solution. The proposed algorithm combines the ability of the globalized searching of the evolutionary algorithms and the fast convergence of the k-means algorithm and can avoid the drawback of both. The performance of the proposed algorithm is evaluated through several benchmark datasets. The experimental results show that the proposed algorithm is highly forceful and outperforms the previous approaches such as SA, ACO, PSO and k-means for the partitional clustering problem.

The ability to predict the students’ academic performance is very important in institution educational system. Recently some researchers have been proposed data mining techniques for higher education. In this paper, we compare two data mining techniques which are: Artificial Neural Network (ANN) and the combination of clustering and decision tree classification techniques for predicting and classifying students’ academic performance. The data set used in this research is the student data of Computer Science Department, Faculty of Science and Defence Technology, National Defence University of Malaysia (NDUM).

Discrete event simulation studies in a manufacturing context are a powerful instrument when modeling and evaluating processes of various industries. Usually simulation experts conduct simulation experiments for a predetermined system specification by manually varying parameters through educated assumptions and according to a prior defined goal. Moreover, simulation experts try to reduce complexity and number of simulation runs by excluding parameters that they consider as not influential regarding the simulation project scope. On the other hand, today's world of big data technology enables us to handle huge amounts of data. We therefore investigate the potential benefits of designing large scale experiments with a much broader coverage of possible system behavior. In this paper, we propose an approach for applying data mining methods on simulation data in combination with suitable visualization methods in order to uncover relationships in model behavior to discover knowledge that otherwise would have remained hidden. For a prototypical demonstration we used a clustering algorithm to divide large amounts of simulation output datasets into groups of similar performance values and depict those groups through visualizations to conduct a visual investigation process of the simulation data.

Diabetes is a major health problem in the United States. There is a long history of diabetic registries and databases with systematically collected patient information. We examine one such diabetic data warehouse, showing a method of applying data mining techniques, and some of the data issues, analysis problems, and results. The diabetic data warehouse is from a large integrated health care system in the New Orleans area with 30,383 diabetic patients. Methods for translating a complex relational database with time series and sequencing information to a flat file suitable for data mining are challenging. We discuss two variables in detail, a comorbidity index and the HgbA1c, a measure of glycemic control related to outcomes. We used the classification tree approach in Classification and Regression Trees (CART) with a binary target variable of HgbA1c >9.5 and 10 predictors: age, sex, emergency department visits, office visits, comorbidity index, dyslipidemia, hypertension, cardiovascular disease, retinopathy, end-stage renal disease. Unexpectedly, the most important variable associated with bad glycemic control is younger age, not the comorbiditity index or whether patients have related diseases. If we want to target diabetics with bad HgbA1c values, the odds of finding them is 3.2 times as high in those <65 years of age than those older. Data mining can discover novel associations that are useful to clinicians and administrators [corrected].

An enhanced technique for risk categorization is presented. This technique, PCA-ANN, provides an improved capability to discriminate high-risk software. The approach draws on the combined strengths of pattern recognition, multivariate statistics and neural networks. Principal component analysis is utilized to provide a means of normalizing and orthogonalizing the input data, thus eliminating the ill effects of multicollinearity. A neural network is used for risk determination/classification. A significant feature of this approach is a procedure, herein termed cross-normalization. This procedure provides the technique with capability to discriminate data sets that include disproportionately large numbers of high-risk software modules.


Question: A two-level compression technique targeting record-aware acceleration in big textual data analysis.

This paper introduces an efficient compression method tailored for natural language texts, offering several key advantages. Firstly, it enables highly efficient decompression of any segment of the text, making it particularly suitable for big textual data analysis. Secondly, it supports direct search for exact words and phrases on the compressed text itself, utilizing any sequential pattern-matching algorithm without the need for decompression. Thirdly, the method allows for word-based approximate and extended search functionalities to be executed efficiently, without requiring any decoding. The compression strategy employs a semistatic word-based model in conjunction with a Huffman code, optimized for byte-oriented coding rather than bit-oriented. This approach achieves a compression ratio of approximately 30% for typical English texts, outperforming popular compression tools like Compress (40%) and Gzip (35%). Compression times are comparable to Compress and roughly half that of Gzip, while decompression times are notably lower than Gzip and a third of Compress's decompression time. Three search algorithms are detailed, each offering flexibility in handling variations of basic word and phrase searches, including support for character sets, regular expressions, and approximate matching. These algorithms enable the exclusion of separators and stopwords during search operations without a significant increase in computational cost. Experimental results demonstrate that running these algorithms on compressed text is twice as fast as executing the best available software on uncompressed text when searching for simple words. For complex or approximate patterns, the algorithms are up to eight times faster than searching uncompressed text. The paper also explores the implications of this technique in the context of inverted files that reference logical blocks, advocating for the continuous compression of text, with decompression reserved solely for display purposes.

Visual analysis of high-volume time series data is ubiquitous in many industries, including finance, banking, and discrete manufacturing. Contemporary, RDBMS-based systems for visualization of high-volume time series data have difficulty to cope with the hard latency requirements and high ingestion rates of interactive visualizations. Existing solutions for lowering the volume of time series data disregard the semantics of visualizations and result in visualization errors. In this work, we introduce M4, an aggregation-based time series dimensionality reduction technique that provides errorfree visualizations at high data reduction rates. Focusing on line charts, as the predominant form of time series visualization, we explain in detail the drawbacks of existing data reduction techniques and how our approach outperforms state of the art, by respecting the process of line rasterization. We describe how to incorporate aggregation-based dimensionality reduction at the query level in a visualizationdriven query rewriting system. Our approach is generic and applicable to any visualization system that uses an RDBMS as data source. Using real world data sets from high tech manufacturing, stock markets, and sports analytics domains we demonstrate that our visualization-oriented data aggregation can reduce data volumes by up to two orders of magnitude, while preserving perfect visualizations.

Log-Structured Merge (LSM) Trees provide a tiered data storage and retrieval paradigm that is aractive for write-optimized data systems. Maintaining an ecient buer in memory and deferring updates past their initial write-time, the structure provides quick operations over hot data. Because each layer of the structure is logically separate from the others, the structure is also conducive to opportunistic and granular optimization. In this paper, we introduce the SkiplistBased LSM Tree (sLSM), a novel system in which the memory buer of the LSM is composed of a sequence of skiplists. We develop theoretical and experimental results that demonstrate that the breadth of tuning parameters inherent to the sLSM allows it broad exibility for excellent performance across a wide variety of workloads.

In this article, we first present the benefit of combing text analysis (topic models in particular) with interactive visualization. We then highlight examples from prior work on topic and time-oriented visual text analysis. Lastly, we present four challenges that warrant additional future research in the field of visual text analysis.

In this paper, we present a Text Summarisation tool, compendium, capable of generating the most common types of summaries. Regarding the input, singleand multi-document summaries can be produced; as the output, the summaries can be extractive or abstractiveoriented; and finally, concerning their purpose, the summaries can be generic, query-focused, or sentiment-based. The proposed architecture for compendium is divided in various stages, making a distinction between core and additional stages. The former constitute the backbone of the tool and are common for the generation of any type of summary, whereas the latter are used for enhancing the capabilities of the tool. The main contributions of compendium with respect to the state-of-the-art summarisation systems are that (i) it specifically deals with the problem of redundancy, by means of textual entailment; (ii) it combines statistical and cognitive-based techniques for determining relevant content; and (iii) it proposes an abstractive-oriented approach for facing the challenge of abstractive summarisation. The evaluation performed in different domains and textual genres, comprising traditional texts, as well as texts extracted from the Web 2.0, shows that compendium is very competitive and appropriate to be used as a tool for generating summaries.

Now a day’s Peta-Bytes of data becomes the norm in industries. Handling, analyzing such big data is challenging task. Even frameworks like Hadoop (Open Source Implementation of MapReduce Paradigm) and NoSQL databases like Cassandra, HBase can be used to analyze and store such large data; heterogeneity of data is still an issue. Data centers usually have clusters formed using heterogeneous nodes. Ecosystem like Hadoop can be used to manage such types of cluster but it can not schedule jobs (Application) efficiently on this heterogeneous cluster when data is itself heterogeneous. Heterogeneity of data may be because of data format or because of the complexity. This paper is review of systems and algorithms for distributed management of huge size data, efficiency of these approaches.

The Normalized Compression Distance (NCD) has been used in a number of domains to compare objects with varying feature types. This flexibility comes from the use of general purpose compression algorithms as the means of computing distances between byte sequences. Such flexibility makes NCD particularly attractive for cases where the right features to use are not obvious, such as malware classification. However, NCD can be computationally demanding, thereby restricting the scale at which it can be applied. We introduce an alternative metric also inspired by compression, the Lempel-Ziv Jaccard Distance (LZJD). We show that this new distance has desirable theoretical properties, as well as comparable or superior performance for malware classification, while being easy to implement and orders of magnitude faster in practice.

The amounts of currently produced data emphasize the importance of techniques for efficient data processing. Searching big data collections according to similarity of data well corresponds to human perception. This paper is focused on similarity search using the concept of sketches – a compact bit string representations of data objects compared by Hamming distance, which can be used for filtering big datasets. The object-to-sketch transformation is a form of the dimensionality reduction and thus there are two basic contradictory requirements: (1) The length of the sketches should be small for efficient manipulation, but (2) longer sketches retain more information about the data objects. First, we study various sketching methods for data modeled by metric space and we analyse their quality. Specifically, we study importance of several sketch properties for similarity search and we propose a high quality sketching technique. Further, we focus on the length of sketches by studying mutual influence of sketch properties such as correlation of their bits and the intrinsic dimensionality of a set of sketches. The outcome is an equation that allows us to estimate a suitable length of sketches for an arbitrary given dataset. Finally, we empirically verify proposed approach on two real-life datasets.

Resource Description Framework (RDF) represents a flexible and concise model for representing the metadata of resources on the web. Over the past years, with the increasing amount of RDF data, efficient and scalable RDF data management has become a fundamental challenge to achieve the Semantic Web vision. However, multiple approaches for RDF storage have been suggested, ranging from simple triple stores to more advanced techniques like vertical partitioning on the predicates or centralized approaches. Unfortunately, it is still a challenge to store a huge quantity of RDF quads due, in part, to the query processing for RDF data. This paper proposes a scalable solution for RDF data management that uses Apache Accumulo. We focus on introducing storage methods and indexing techniques that scale to billions of quads across multiple nodes, while providing fast and easy access to the data through conventional query mechanisms such as SPARQL. Our performance evaluation shows that in most cases our approach works well against large RDF datasets.

We present LocationSpark, a spatial data processing system built on top of Apache Spark, a widely used distributed data processing system. LocationSpark offers a rich set of spatial query operators, e.g., range search, kNN, spatio-textual operation, spatial-join, and kNN-join. To achieve high performance, LocationSpark employs various spatial indexes for in-memory data, and guarantees that immutable spatial indexes have low overhead with fault tolerance. In addition, we build two new layers over Spark, namely a query scheduler and a query executor. The query scheduler is responsible for mitigating skew in spatial queries, while the query executor selects the best plan based on the indexes and the nature of the spatial queries. Furthermore, to avoid unnecessary network communication overhead when processing overlapped spatial data, We embed an efficient spatial Bloom filter into LocationSpark’s indexes. Finally, LocationSpark tracks frequently accessed spatial data, and dynamically flushes less frequently accessed data into disk. We evaluate our system on real workloads and demonstrate that it achieves an order of magnitude performance gain over a baseline framework.

Next generation analytics will be all about graphs, though performance has been a fundamental challenge for large scale graph processing. In this paper, we present an industrial graph processing engine for exploring various large scale linked data, which exhibits superior performance due to the several innovations. This engine organizes a graph as a set of edge-sets, compatible with the traditional edge-centric sharding for graphs, but becomes more amenable for large scale processing. Each time only a portion of the sets are needed for computation and the data access patterns can be highly predictable for prefetch for many graph computing algorithms. Due to the sparsity of large scale graph structure, this engine differentiates logical edge-sets from the edge-sets physically stored on the disk, where multiple logical edge-sets can be organized into a same physical edge-set to increase the data locality. Besides, in contrast to existing solution, the data structures utilized for the physical edge-sets can vary from one to another. Such heterogeneous edge-set representation explores the best graph processing performance according to local data access patterns. We conduct experiments on a representative set of property graphs on multiple platforms, where the proposed system outperform the baseline systems consistently.


Question: A two-level compression technique targeting record-aware acceleration of big textual data analysis.

This paper presents an innovative method for monitoring the spread and evolution of short, unique phrases that maintain their integrity as they traverse online textual content. Our focus is on identifying a category of memes that exhibit extensive dissemination and dynamic variation over daily timescales, which aligns with the typical perception of news and events. By devising efficient algorithms to cluster textual variations of these memes, we are able to track their lifecycle and uncover their prevalence. Our primary area of investigation is the news cycle—the rhythmic patterns in news media that have been subject to qualitative interpretation but have not been quantitatively analyzed with precision. To achieve this, we monitored 1.6 million mainstream media sites and blogs for a three-month duration, analyzing a total of 90 million articles. Through our analysis, we discovered distinct and recurring temporal patterns in the news cycle. Notably, we observed an average delay of 2.5 hours between the peak attention a meme receives in mainstream media and its peak in blogs. Additionally, we found divergent behaviors surrounding the peak attention and a pulsating pattern in the transition of attention from news media to blogs. To further our understanding, we also developed and scrutinized a mathematical model that explains the temporal variations observed in our data.

Question: An Adaptive Multi-Objective Evolutionary Algorithm for Multi-Objective Optimization Problems

Documents: A Novel Multi-Objective Optimization Algorithm Based on Adaptive Reference Point Strategy
This paper proposes a novel multi-objective optimization algorithm, Adaptive Reference Point Strategy (ARPS), for solving complex multi-objective optimization problems. ARPS is designed to dynamically adjust the reference points based on the evolving population to guide the search towards the Pareto front. The algorithm is capable of handling problems with a large number of objectives and constraints, making it suitable for real-world applications. ARPS utilizes an adaptive mechanism to update the reference points, ensuring a balanced exploration and exploitation of the search space. This adaptive strategy allows ARPS to efficiently converge to the Pareto front while maintaining a good spread of solutions. To validate the effectiveness of ARPS, we compare it with several state-of-the-art multi-objective optimization algorithms on a set of benchmark problems. The experimental results demonstrate that ARPS outperforms the other algorithms in terms of solution quality and convergence rate. We also apply ARPS to a real-world multi-objective optimization problem in the field of environmental engineering, where it shows promising results in finding a set of non-dominated solutions that satisfy multiple conflicting objectives.

Similar Documents: This paper introduces an innovative multi-objective optimization algorithm, Adaptive Reference Point

In the big data era, duplicated copies of data may be generated by various smart devices. Although convergent encryption has been extensively adopted for secure deduplication, it is still very challenging to efficiently and reliably manage the convergent keys and support keyword search over encrypted data. This paper addresses this problem by presenting an efficient scheme for big data outsourcing with secure deduplication, named BDO-SD. Specifically, with the convergent encryption technique, data owners can outsource their data to the cloud server with data deduplication; with the idea of public key encryption and keywords search, a data user can query encrypted data while preserving its privacy. Security analysis demonstrates that our proposed BDO-SD scheme can achieve the data confidentiality, query privacy and key security. Simulation results show that compared with other schemes, our scheme can achieve better efficiency than the existing data deduplication schemes.

Text documents often contain valuable structured data that is hidden Yin regular English sentences. This data is best exploited infavailable as arelational table that we could use for answering precise queries or running data mining tasks.We explore a technique for extracting such tables from document collections that requires only a handful of training examples from users. These examples are used to generate extraction patterns, that in turn result in new tuples being extracted from the document collection.We build on this idea and present our Snowball system. Snowball introduces novel strategies for generating patterns and extracting tuples from plain-text documents.At each iteration of the extraction process, Snowball evaluates the quality of these patterns and tuples without human intervention,and keeps only the most reliable ones for the next iteration. In this paper we also develop a scalable evaluation methodology and metrics for our task, and present a thorough experimental evaluation of Snowball and comparable techniques over a collection of more than 300,000 newspaper documents.

Increasingly in the 21st century, our daily lives leave behind a detailed digital record: our shifting thoughts and opinions shared on Twitter, our social relationships, our purchasing habits, our information seeking, our photos and videos—even the movements of our bodies and cars. Naturally, for those interested in human behavior, this bounty of personal data is irresistible. Decision makers of all kinds, from company executives to government agencies to researchers and scientists, would like to base their decisions and actions on this data. In response, a new discipline of big data analytics is forming. Fundamentally, big data analytics is a workflow that distills terabytes of low-value data (e.g., every tweet) down to, in some cases, a single bit of high-value data (Should Company X acquire Company Y? Can we reject the null hypothesis?). The goal is to see the big picture from the minutia of our digital lives. It is no surprise today that big data is useful for HCI researchers and user interface design. As one example, A/B testing is a standard practice in the usability community to help determine relative differences in user performance using different interfaces. For many years, we have used strict laboratory conditions to evaluate interfaces, but more recently we have seen the ability to implement those tests quickly and on a large population by running controlled Interactions with Big Data Analytics

This tutorial gives an overview on state-of-the-art methods for the automatic construction of large knowledge bases and harnessing them for data and text analytics. It covers both big-data methods for building knowledge bases and knowledge bases being assets for big-data applications. The tutorial also points out challenges and research opportunities. 1. MOTIVATION AND SCOPE Comphrehensive machine-readable knowledge bases (KB’s) have been pursued since the seminal projects Cyc [19, 20] and WordNet [12]. In contrast to these manually created KB’s, great advances have recently been made on automating the building and curation of large KB’s [1, 16], using information extraction (IE) techniques and harnessing highquality Web sources like Wikipedia. Prominent endeavors of this kind include academic research projects such as DBpedia [3], KnowItAll [10], NELL [5] and YAGO [29], as well as industrial ones such as Freebase. These projects provide automatically constructed KB’s of facts about named entities, their semantic classes, and their mutual relationships. They contain millions of entities and billions of facts about them. Moreover, several KB’s are interlinked at the entity level, forming the backbone of the Web of Linked Data [14]. Such world knowledge in turn enables cognitive applications and knowledge-centric services like disambiguating natural-language text, entity linking, text summarization, deep question answering, and semantic search and analytics over entities and relations in Web and enterprise data (e.g., [2, 6, 8, 13]). Prominent examples of how KB’s can be harnessed include the Google Knowledge Graph [27] and the IBM Watson question answering system [17]. This tutorial presents state-of-the-art methods, recent advances, research opportunities, and open challenges along this avenue of knowledge harvesting and its applications. Particular emphasis is on the twofold role of KB’s for bigdata analytics: using scalable distributed algorithms for harvesting knowledge from Web and text sources, and leveraging entity-centric knowledge for deeper interpretation of and This work is licensed under the Creative Commons AttributionNonCommercial-NoDerivs 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/. Obtain permission prior to any use beyond those covered by the license. Contact copyright holder by emailing info@vldb.org. Articles from this volume were invited to present their results at the 40th International Conference on Very Large Data Bases, September 1st 5th 2014, Hangzhou, China. Proceedings of the VLDB Endowment, Vol. 7, No. 13 Copyright 2014 VLDB Endowment 2150-8097/14/08. better intelligence with big data. The following sections outline the structure of the tutorial. An extensive bibliography on this theme is given in [30]. 2. BUILDING KNOWLEDGE BASES Digital Knowledge: Today’s KB’s represent their data mostly in RDF-style SPO (subject-predicate-object) triples. We introduce this data model and the most salient KB projects, which include KnowItAll [10, 11], BabelNet [22], ConceptNet [28], DBpedia [3, 18], DeepDive [24], Freebase [4], ImageNet [7], NELL [5], Wikidata [31], WikiNet [21], WikiTaxonomy [26], and YAGO [29, 15]. We briefly discuss industrial projects like the Google Knowledge Graph and related work at Google [9, 13, 25], the EntityCube and Probase projects at Microsoft Research [23, 32], and IBM’s Watson project [17]. Harvesting Knowledge on Entities and Classes: Every entity in a KB (e.g., Steve Jobs) belongs to one or multiple classes (e.g., computer pioneer, entrepreneur). These classes are organized into a taxonomy, where more special classes are subsumed by more general classes (e.g., person). We discuss two families of methods to harvest such information: Wikipedia-based approaches that analyze the category system, and Web-based approaches that use techniques like set expansion. 3. HARVESTING FACTS AT WEB SCALE Harvesting Relational Facts: Relational facts express properties of and relationships between entities. There is a large spectrum of methods to extract such facts from Web documents. We give an overview on methods from pattern matching (e.g., regular expressions), computational linguistics (e.g., dependency parsing), statistical learning (e.g., factor graphs and MLN’s), and logical consistency reasoning (e.g., weighted MaxSat or ILP solvers). We also discuss to what extent these approaches scale to handle big data. Open Information Extraction: Alternatively to methods that operate on a pre-specified set of relations and entities, open information extraction harvests arbitrary SPO triples from natural language documents. It aggressively taps into noun phrases as entity candidates and verbal phrases as prototypic patterns for relations. We discuss recent methods that follow this direction. Some methods along these lines make clever use of big-data techniques like frequent sequence mining and map-reduce computation. Temporal and Multilingual Knowledge: Properly interpreting entities and facts in a KB often requires additional meta-information like entity names in different languages and the temporal scope of facts. We discuss tech-

Log-Structured Merge (LSM) Trees provide a tiered data storage and retrieval paradigm that is aractive for write-optimized data systems. Maintaining an ecient buer in memory and deferring updates past their initial write-time, the structure provides quick operations over hot data. Because each layer of the structure is logically separate from the others, the structure is also conducive to opportunistic and granular optimization. In this paper, we introduce the SkiplistBased LSM Tree (sLSM), a novel system in which the memory buer of the LSM is composed of a sequence of skiplists. We develop theoretical and experimental results that demonstrate that the breadth of tuning parameters inherent to the sLSM allows it broad exibility for excellent performance across a wide variety of workloads.

In this paper, we present a Text Summarisation tool, compendium, capable of generating the most common types of summaries. Regarding the input, singleand multi-document summaries can be produced; as the output, the summaries can be extractive or abstractiveoriented; and finally, concerning their purpose, the summaries can be generic, query-focused, or sentiment-based. The proposed architecture for compendium is divided in various stages, making a distinction between core and additional stages. The former constitute the backbone of the tool and are common for the generation of any type of summary, whereas the latter are used for enhancing the capabilities of the tool. The main contributions of compendium with respect to the state-of-the-art summarisation systems are that (i) it specifically deals with the problem of redundancy, by means of textual entailment; (ii) it combines statistical and cognitive-based techniques for determining relevant content; and (iii) it proposes an abstractive-oriented approach for facing the challenge of abstractive summarisation. The evaluation performed in different domains and textual genres, comprising traditional texts, as well as texts extracted from the Web 2.0, shows that compendium is very competitive and appropriate to be used as a tool for generating summaries.

Resource Description Framework (RDF) represents a flexible and concise model for representing the metadata of resources on the web. Over the past years, with the increasing amount of RDF data, efficient and scalable RDF data management has become a fundamental challenge to achieve the Semantic Web vision. However, multiple approaches for RDF storage have been suggested, ranging from simple triple stores to more advanced techniques like vertical partitioning on the predicates or centralized approaches. Unfortunately, it is still a challenge to store a huge quantity of RDF quads due, in part, to the query processing for RDF data. This paper proposes a scalable solution for RDF data management that uses Apache Accumulo. We focus on introducing storage methods and indexing techniques that scale to billions of quads across multiple nodes, while providing fast and easy access to the data through conventional query mechanisms such as SPARQL. Our performance evaluation shows that in most cases our approach works well against large RDF datasets.

A string-similarity measure quantifies the similarity between two text strings for approximate string matching or comparison. For example, the strings “Sam” and “Samuel” can be considered to be similar. Most existing work that computes the similarity of two strings only considers syntactic similarities, for example, number of common words or q-grams. While this is indeed an indicator of similarity, there are many important cases where syntactically-different strings can represent the same real-world object. For example, “Bill” is a short form of “William,” and “Database Management Systems” can be abbreviated as “DBMS.” Given a collection of predefined synonyms, the purpose of this article is to explore such existing knowledge to effectively evaluate the similarity between two strings and efficiently perform similarity searches and joins, thereby boosting the quality of approximate string matching.
 In particular, we first present an expansion-based framework to measure string similarities efficiently while considering synonyms. We then study efficient algorithms for similarity searches and joins by proposing two novel indexes, called SI-trees and QP-trees, which combine signature-filtering and length-filtering strategies. In order to improve the efficiency of our algorithms, we develop an estimator to estimate the size of candidates to enable an online selection of signature filters. This estimator provides strong low-error, high-confidence guarantees while requiring only logarithmic space and time costs, thus making our method attractive both in theory and in practice. Finally, the experimental results from a comprehensive study of the algorithms with three real datasets verify the effectiveness and efficiency of our approaches.

Human-quality text summarization systems are di cult to design, and even more di cult to evaluate, in part because documents can di er along several dimensions, such as length, writing style and lexical usage. Nevertheless, certain cues can often help suggest the selection of sentences for inclusion in a summary. This paper presents our analysis of news-article summaries generated by sentence selection. Sentences are ranked for potential inclusion in the summary using a weighted combination of statistical and linguistic features. The statistical features were adapted from standard IR methods. The potential linguistic ones were derived from an analysis of news-wire summaries. To evaluate these features we use a normalized version of precision-recall curves, with a baseline of random sentence selection, as well as analyze the properties of such a baseline. We illustrate our discussions with empirical results showing the importance of corpus-dependent baseline summarization standards, compression ratios and carefully crafted long queries.

While big data workflows haven been proposed recently as the next-generation data-centric workflow paradigm to process and analyze data of ever increasing in scale, complexity, and rate of acquisition, a scalable distributed data model is still missing that abstracts and automates data distribution, parallelism, and scalable processing. In the meanwhile, although NoSQL has emerged as a new category of data models, they are optimized for storing and querying of large datasets, not for ad-hoc data analysis where data placement and data movement are necessary for optimized workflow execution. In this paper, we propose a NoSQL data model that: 1) supports high-performance MapReduce-style workflows that automate data partitioning and data-parallelism execution. In contrast to the traditional MapReduce framework, our MapReduce-style workflows are fully composable with other workflows enabling dataflow applications with a richer structure, 2) automates virtual machine provisioning and deprovisioning on demand according to the sizes of input datasets, 3) enables a flexible framework for workflow executors that take advantage of the proposed NoSQL data model to improve the performance of workflow execution. Our case studies and experiments show the competitive advantages of our proposed data model. The proposed NoSQL data model is implemented in a new release of DATAVIEW, one of the most usable big data workflow systems in the community.


Question: A two-level compression technique targeting record-aware acceleration of big textual data analysis.

This paper presents Record-Aware Two-Level Compression (RaTLC), a novel technique designed to accelerate the analysis of big textual data. With the exponential growth of text-based data originating from diverse sources like social media, web content, and search queries, the need for efficient and scalable data analysis methods has become paramount. Traditional compression algorithms, while effective in reducing data size, often compromise the structural integrity of the data. RaTLC, however, addresses this issue by employing a dictionary-based approach that selectively compresses data without altering its original format and structure. This ensures that the compressed data can be directly utilized by data analysis platforms without the need for decompression. Through extensive testing with real-world datasets and benchmark MapReduce jobs on Hadoop, we validate the efficacy of RaTLC. Additionally, we offer a utility library for Hadoop, enabling existing MapReduce programs to operate seamlessly on compressed datasets with minimal adjustments. Our results indicate a significant reduction in data size, averaging around 30%, and up to a 32% enhancement in performance for I/O-intensive jobs on a Hadoop cluster. Notably, these improvements are achieved without additional overhead, serving as a complementary optimization to existing techniques.

In this paper, we describe a novel analytics system that enables query processing and predictive analytics over streams of aviation data. As part of an Internal Research and Development project, Boeing Research and Technology (BR&T) Advanced Air Traffic Management (AATM) built a system that makes predictions based upon descriptive patterns of archived aviation data. Boeing AATM has been receiving live Aircraft Situation Display to Industry (ASDI) data and archiving it for over two years. At the present time, there is not an easy mechanism to perform analytics on the data. The incoming ASDI data is large, compressed, and requires correlation with other flight data before it can be analyzed.
 The service exposes this data once it has been uncompressed, correlated, and stored in a data warehouse for further analysis using a variety of descriptive, predictive, and possibly prescriptive analytics tools. The service is being built partially in response to requests from Boeing Commercial Aviation (BCA) for analysis of capacity and flow in the US National Airspace System (NAS). The service utilizes a custom tool for correlating the raw ASDI feed, IBM Warehouse with DB2 for data management, WebSphere Message Broker for real-time message brokering, SPSS Modeler for statistical analysis, and Cognos BI for front-end business intelligence (BI) visualization. This paper describes a scalable service architecture, implementation and the value it adds to the aviation domain.

Efficient management of text data is a major concern of business organizations. In this direction, we propose a novel approach to extract structured knowledge from large corpora of unstructured business documents. This knowledge is represented in the form of object instances, which are common ways of organizing the available information about entities, and are modeled here using document templates. The approach itself is based on the observation that a significant fraction of these documents are created using the cut-copy-pastemethod, and thus, it is important to factor this observation into business document analysis projects. Correspondingly, our approach solves the problem of object instance extraction in two steps, namely similarity search and then extraction of object instances from the selected documents. Early qualitative results on a couple of carefully selected document corpora indicate the effective applicability of the approach for solving an important component of the efficient text management problem.

This paper presents VelocityOLAP (VOLAP), a distributed real-time OLAP system for high-velocity data. VOLAP makes use of dimension hierarchies, is highly scalable, exploits both multi-core and multi-processor parallelism, and can guarantee serializable execution of insert and query operations. In contrast to other high performance OLAP systems such as SAP HANA or IBM Netezza that rely on vertical scaling or special purpose hardware, VOLAP supports cost-efficient horizontal scaling on commodity hardware or modest cloud instances. Experiments on 20 Amazon EC2 nodes with TPC-DS data show that VOLAP is capable of bulk ingesting data at over 600 thousand items per second, and processing streams of interspersed insertions and aggregate queries at a rate of approximately 50 thousand insertions and 20 thousand aggregate queries per second with a database of 1 billion items. VOLAP is designed to support applications that perform large aggregate queries, and provides similar high performance for aggregations ranging from a few items to nearly the entire database.

Text mining is a new and exciting area of computer science that tries to solve the crisis of information overload by combining techniques from data mining, machine learning, natural language processing, information retrieval, and knowledge management. The Text Mining Handbook presents a comprehensive discussion of the latest techniques in text mining and link detection. In addition to providing an in-depth examination of core text mining and link detection algorithms and operations, the book examines advanced pre-processing techniques, knowledge representation considerations, and visualization approaches, ending with real-world applications.

A goodtext classifieris a classifierthat efficiently categorizeslarge setsof text documentsin a reasonabletime frameand with an acceptableaccuracy, and that provides classificationrules that are humanreadablefor possible fine-tuning. If the training of the classifier is also quick, this couldbecomein someapplicationdomainsa goodasset for the classifier . Many techniquesand algorithmsfor automatictext categorizationhavebeendevised.According to publishedliterature, somearemoreaccuratethanothers, andsomeprovidemore interpretableclassificationmodels thanothers. However, nonecancombineall thebeneficial propertiesenumer atedabove. In this paper, we presenta novel approach for automatictext categorizationthat borrowsfrom market basket analysistechniquesusingassociation rule miningin thedata-miningfield. We focuson two major problems:(1) findingthebesttermassociationrules in a textual databaseby generating and pruning; and (2) using the rules to build a text classifier . Our text categorization methodprovesto beefficientandeffective, andexperimentsonwell-knowncollectionsshowthattheclassifier performswell. In addition,trainingaswell asclassification arebothfastandthegeneratedrulesarehumanreadable.

In many applications involving continuous data streams, data arrival is bursty and data rate fluctuates over time. Systems that seek to give rapid or real-time query responses in such an environment must be prepared to deal gracefully with bursts in data arrival without compromising system performance. We discuss one strategy for processing bursty streams --- adaptive, load-aware scheduling of query operators to minimize resource consumption during times of peak load. We show that the choice of an operator scheduling strategy can have significant impact on the run-time system memory usage. We then present Chain scheduling, an operator scheduling strategy for data stream systems that is near-optimal in minimizing run-time memory usage for any collection of single-stream queries involving selections, projections, and foreign-key joins with stored relations. Chain scheduling also performs well for queries with sliding-window joins over multiple streams, and multiple queries of the above types. A thorough experimental evaluation is provided where we demonstrate the potential benefits of Chain scheduling, compare it with competing scheduling strategies, and validate our analytical conclusions.

Human-quality text summarization systems are di cult to design, and even more di cult to evaluate, in part because documents can di er along several dimensions, such as length, writing style and lexical usage. Nevertheless, certain cues can often help suggest the selection of sentences for inclusion in a summary. This paper presents our analysis of news-article summaries generated by sentence selection. Sentences are ranked for potential inclusion in the summary using a weighted combination of statistical and linguistic features. The statistical features were adapted from standard IR methods. The potential linguistic ones were derived from an analysis of news-wire summaries. To evaluate these features we use a normalized version of precision-recall curves, with a baseline of random sentence selection, as well as analyze the properties of such a baseline. We illustrate our discussions with empirical results showing the importance of corpus-dependent baseline summarization standards, compression ratios and carefully crafted long queries.

Compactions are a vital maintenance mechanism used by datastores based on the log-structured merge-tree to counter the continuous buildup of data files under update-intensive workloads. While compactions help keep read latencies in check over the long run, this comes at the cost of significantly degraded read performance over the course of the compaction itself. In this paper, we offer an in-depth analysis of compaction-related performance overheads and propose techniques for their mitigation. We offload large, expensive compactions to a dedicated compaction server to allow the datastore server to better utilize its resources towards serving the actual workload. Moreover, since the newly compacted data is already cached in the compaction server’s main memory, we fetch this data over the network directly into the datastore server’s local cache, thereby avoiding the performance penalty of reading it back from the filesystem. In fact, pre-fetching the compacted data from the remote cache prior to switching the workload over to it can eliminate local cache misses altogether. Therefore, we implement a smarter warmup algorithm that ensures that all incoming read requests are served from the datastore server’s local cache even as it is warming up. We have integrated our solution into HBase, and using the YCSB and TPC-C benchmarks, we show that our approach significantly mitigates compaction-related performance problems. We also demonstrate the scalability of our solution by distributing compactions across multiple compaction servers.

Text summarization is the process of automatically creating a shorter version of one or more text documents. This paper presents a qualitative and quantitative assessment of the 22 state-of-the-art extractive summarization systems using the CNN corpus, a dataset of 3,000 news articles.

Bloom filters are a probabilistic technique for large-scale set membership tests. They exhibit no false negative test results but are susceptible to false positive results. They are well-suited to both large sets and large numbers of membership tests. We implement the Bloom filters present in an accelerated version of BLAST, a genome biosequence alignment application, on NVIDIA GPUs and develop an analytic performance model that helps potential users of Bloom filters to quantify the inherent tradeoffs between throughput and false positive rates.


Question: A two-level compression technique targeting record-aware acceleration of big textual data analysis.

This paper introduces an unsupervised algorithm designed to distill meaningful insights from the vast corpus of online reviews, addressing the growing demand for effective text analysis tools in the digital age. Given the wide array of products and services that are subject to online reviews, supervised learning techniques can be limiting due to their reliance on pre-labeled data. Our proposed system adeptly identifies product aspects and assesses sentiment without human intervention. Its design is versatile, making it applicable across various domains and languages. A key feature is its consideration of how specific aspects can influence sentiment polarity, a factor often overlooked in previous studies. We showcase the efficacy of our method by comparing its performance in aspect extraction and sentiment analysis to more intricate semi-supervised approaches. Despite its simplicity, our unsupervised system yields comparable outcomes, demonstrating its potential as a powerful tool for textual data mining, particularly in scenarios where manual annotation and extensive domain knowledge are scarce.

This paper explores a simple and efficient baseline for text classification. Our experiments show that our fast text classifier fastText is often on par with deep learning classifiers in terms of accuracy, and many orders of magnitude faster for training and evaluation. We can train fastText on more than one billion words in less than ten minutes using a standard multicore CPU, and classify half a million sentences among 312K classes in less than a minute.

We study an acceleration method for point-to-point shortest-path computations in large and sparse directed graphs with given nonnegative arc weights. The acceleration method is called the arc-flag approach and is based on Dijkstra's algorithm. In the arc-flag approach, we allow a preprocessing of the network data to generate additional information, which is then used to speedup shortest-path queries. In the preprocessing phase, the graph is divided into regions and information is gathered on whether an arc is on a shortest path into a given region. The arc-flag method combined with an appropriate partitioning and a bidirected search achieves an average speedup factor of more than 500 compared to the standard algorithm of Dijkstra on large networks (1 million nodes, 2.5 million arcs). This combination narrows down the search space of Dijkstra's algorithm to almost the size of the corresponding shortest path for long-distance shortest-path queries. We conduct an experimental study that evaluates which partitionings are best suited for the arc-flag method. In particular, we examine partitioning algorithms from computational geometry and a multiway arc separator partitioning. The evaluation was done on German road networks. The impact of different partitions on the speedup of the shortest path algorithm are compared. Furthermore, we present an extension of the speedup technique to multiple levels of partitions. With this multilevel variant, the same speedup factors can be achieved with smaller space requirements. It can, therefore, be seen as a compression of the precomputed data that preserves the correctness of the computed shortest paths.

Hierarchical Pitman-Yor Process priors are compelling methods for learning language models, outperforming point-estimate based methods. However, these models remain unpopular due to computational and statistical inference issues, such as memory and time usage, as well as poor mixing of sampler. In this work we propose a novel framework which represents the HPYP model compactly using compressed suffix trees. Then, we develop an efficient approximate inference scheme in this framework that has a much lower memory footprint compared to full HPYP and is fast in the inference time. The experimental results illustrate that our model can be built on significantly larger datasets compared to previous HPYP models, while being several orders of magnitudes smaller, fast for training and inference, and outperforming the perplexity of the state-of-the-art Modified Kneser-Ney countbased LM smoothing by up to 15%.

We present a new indexing method for the approximate string matching problem. The method is based on a suffix array combined with a partitioning of the pattern. We analyze the resulting algorithm and show that the average retrieval time is , for some that depends on the error fraction tolerated and the alphabet size . It is shown that for approximately , where . The space required is four times the text size, which is quite moderate for this problem. We experimentally show that this index can outperform by far all the existing alternatives for indexed approximate searching. These are also the first experiments that compare the different existing schemes.

This paper describes novel aspects of the column store implemented in the MemSQL database engine and describes the design choices made to support real-time streaming workloads. Column stores have traditionally been restricted to data warehouse scenarios where low latency queries are a secondary goal, and where restricting data ingestion to be offline, batched, append-only, or some combination thereof is acceptable. In contrast, the MemSQL column store implementation treats low latency queries and ongoing writes as first class citizens, with a focus on avoiding interference between read, ingest, update, and storage optimization workloads through the use of fragmented snapshot transactions and optimistic storage reordering. This implementation broadens the range of serviceable column store workloads to include those with more stringent demands on query and data latency, such as those backing operational systems used by adtech, financial services, fraud detection and other real-time or data streaming applications.

A string similarity measure quantifies the similarity between two text strings for approximate string matching or comparison. For example, the strings "Sam" and "Samuel" can be considered similar. Most existing work that computes the similarity of two strings only considers syntactic similarities, e.g., number of common words or q-grams. While these are indeed indicators of similarity, there are many important cases where syntactically different strings can represent the same real-world object. For example, "Bill" is a short form of "William". Given a collection of predefined synonyms, the purpose of the paper is to explore such existing knowledge to evaluate string similarity measures more effectively and efficiently, thereby boosting the quality of string matching.
 In particular, we first present an expansion-based framework to measure string similarities efficiently while considering synonyms. Because using synonyms in similarity measures is, while expressive, computationally expensive (NP-hard), we propose an efficient algorithm, called selective-expansion, which guarantees the optimality in many real scenarios. We then study a novel indexing structure called SI-tree, which combines both signature and length filtering strategies, for efficient string similarity joins with synonyms. We develop an estimator to approximate the size of candidates to enable an online selection of signature filters to further improve the efficiency. This estimator provides strong low-error, high-confidence guarantees while requiring only logarithmic space and time costs, thus making our method attractive both in theory and in practice. Finally, the results from an empirical study of the algorithms verify the effectiveness and efficiency of our approach.

We present a framework of dynamic programming combinators that provides a high-level environment to describe the recursions typical of dynamic programming over sequence data in a style very similar to algebraic dynamic programming (ADP). Using a combination of type-level programming and stream fusion leads to a substantial increase in performance, without sacrificing much of the convenience and theoretical underpinnings of ADP.
 We draw examples from the field of computational biology, more specifically RNA secondary structure prediction, to demonstrate how to use these combinators and what differences exist between this library, ADP, and other approaches.
 The final version of the combinator library allows writing algorithms with performance close to hand-optimized C code.

The increasing growth of embedded networking applications has created a demand for high-performance logging systems capable of storing huge amounts of high-bandwidth, typically redundant data. An efficient way of maximizing the logger performance is doing a real-time compression of the logged stream. In this paper we present a flexible high-performance implementation of the LZSS compression algorithm capable of processing up to 50 MB/s on a Virtex-5 FPGA chip. We exploit the independently addressable dual-port block RAMs inside the FPGA chip to achieve an average performance of 2 clock cycles per byte. To make the compressed stream compatible with the ZLib library [1] we encode the LZSS algorithm output using a fixed Huffman table defined by the Deflate specification [2]. We also demonstrate how changing the amount of memory allocated to various internal tables impacts the performance and compression ratio. Finally, we provide a cycle-accurate estimation tool that allows finding a trade-off between FPGA resource utilization, compression ratio and performance for a specific data sample.

As applications within and outside the enterprise encounter increasing volumes of unstructured data, there has been renewed interest in the area of information extraction (IE) -- the discipline concerned with extracting structured information from unstructured text. Classical IE techniques developed by the NLP community were based on cascading grammars and regular expressions. However, due to the inherent limitations of grammarbased extraction, these techniques are unable to: (i) scale to large data sets, and (ii) support the expressivity requirements of complex information tasks. At the IBM Almaden Research Center, we are developing SystemT, an IE system that addresses these limitations by adopting an algebraic approach. By leveraging well-understood database concepts such as declarative queries and costbased optimization, SystemT enables scalable execution of complex information extraction tasks. In this paper, we motivate the SystemT approach to information extraction. We describe our extraction algebra and demonstrate the effectiveness of our optimization techniques in providing orders of magnitude reduction in the running time of complex extraction tasks.

Large corpora are ubiquitous in today’s world and memory quickly becomes the limiting factor in practical applications of the Vector Space Model (VSM). In this paper, we identify a gap in existing implementations of many of the popular algorithms, which is their scalability and ease of use. We describe a Natural Language Processing software framework which is based on the idea of document streaming, i.e. processing corpora document after document, in a memory independent fashion. Within this framework, we implement several popular algorithms for topical inference, including Latent Semantic Analysis and Latent Dirichlet Allocation, in a way that makes them completely independent of the training corpus size. Particular emphasis is placed on straightforward and intuitive framework design, so that modifications and extensions of the methods and/or their application by interested practitioners are effortless. We demonstrate the usefulness of our approach on a real-world scenario of computing document similarities within an existing digital library DML-CZ.