Question: Crossing the Limit of Single Transactions: Extracting Associations Among Multiple Transactions

This paper explores an innovative technique for discovering association rules in large databases with minimal I/O overhead, addressing a critical issue in database mining. Traditional algorithms for identifying association rules necessitate multiple scans of the database, making I/O operations a significant bottleneck for large-scale datasets. To mitigate this challenge, we propose an algorithm that significantly reduces database activity. The methodology involves selecting a random sample from the database, utilizing this sample to uncover all potential association rules that likely apply to the entire database, and subsequently validating these rules with the remaining data. Contrary to sampling-based approximation methods, our algorithm yields precise association rules, not estimates. While the approach is probabilistic, in the unlikely event that not all rules are captured during the initial sampling, any missing rules can be identified in a subsequent pass. Empirical results demonstrate the algorithm's efficiency in discovering association rules with just a single database pass, showcasing its potential for optimizing the mining of large databases.

— To classify data mining problems and algorithms we used two dimensions: data type and type of mining operations. One of the main issue that arise during the data mining process is treating data that contains temporal information. The area of temporal data mining has very much attention in the last decade because from the time related feature of the data, one can extract much significant information which can not be extracted by the general methods of data mining. Many interesting techniques of temporal data mining were proposed and shown to be useful in many applications. Since temporal data mining brings together techniques from different fields such as databases, statistics and machine learning the literature is scattered among many different sources. In this paper, we present a survey on techniques of temporal data mining.

We consider extending decision support facilities toward large sophisticated networks, upon which multidimensional attributes are associated with network entities, thereby forming the so-called multidimensional networks. Data warehouses and OLAP (Online Analytical Processing) technology have proven to be effective tools for decision support on relational data. However, they are not well-equipped to handle the new yet important multidimensional networks. In this paper, we introduce Graph Cube, a new data warehousing model that supports OLAP queries effectively on large multidimensional networks. By taking account of both attribute aggregation and structure summarization of the networks, Graph Cube goes beyond the traditional data cube model involved solely with numeric value based group-by's, thus resulting in a more insightful and structure-enriched aggregate network within every possible multidimensional space. Besides traditional cuboid queries, a new class of OLAP queries, crossboid, is introduced that is uniquely useful in multidimensional networks and has not been studied before. We implement Graph Cube by combining special characteristics of multidimensional networks with the existing well-studied data cube techniques. We perform extensive experimental studies on a series of real world data sets and Graph Cube is shown to be a powerful and efficient tool for decision support on large multidimensional networks.

The paper systematically compares two feature extraction algorithms to mine product features commented on in customer reviews. The first approach [17] identifies candidate features by applying a set of POS patterns and pruning the candidate set based on the log likelihood ratio test. The second approach [11] applies association rule mining for identifying frequent features and a heuristic based on the presence of sentiment terms for identifying infrequent features. We evaluate the performance of the algorithms on five product specific document collections regarding consumer electronic devices. We perform an analysis of errors and discuss advantages and limitations of the algorithms.

Given a collection of basic customer demographics (e.g., age and gender) andtheir behavioral data (e.g., item purchase histories), how can we predictsensitive demographics (e.g., income and occupation) that not every customermakes available?This demographics prediction problem is modeled as a classification task inwhich a customer's sensitive demographic y is predicted from his featurevector x. So far, two lines of work have tried to produce a"good" feature vector x from the customer's behavioraldata: (1) application-specific feature engineering using behavioral data and (2) representation learning (such as singular value decomposition or neuralembedding) on behavioral data. Although these approaches successfullyimprove the predictive performance, (1) designing a good feature requiresdomain experts to make a great effort and (2) features obtained fromrepresentation learning are hard to interpret. To overcome these problems, we present a Relational Infinite SupportVector Machine (R-iSVM), a mixture-of-experts model that can leveragebehavioral data. Instead of augmenting the feature vectors of customers, R-iSVM uses behavioral data to find out behaviorally similar customerclusters and constructs a local prediction model at each customer cluster. In doing so, R-iSVM successfully improves the predictive performance withoutrequiring application-specific feature designing and hard-to-interpretrepresentations. Experimental results on three real-world datasets demonstrate the predictiveperformance and interpretability of R-iSVM. Furthermore, R-iSVM can co-existwith previous demographics prediction methods to further improve theirpredictive performance.

Multiplicative weight-updating algorithms such as Winnow have been studied extensively in the COLT literature, but only recently have people started to use them in applications. In this paper, we apply a Winnow-based algorithm to a task in natural language: contextsensitive spelling correction. This is the task of xing spelling errors that happen to result in valid words, such as substituting to for too, casual for causal, and so on. Previous approaches to this problem have been statistics-based; we compare Winnow to one of the more successful such approaches, which uses Bayesian classi ers. We nd that: (1) When the standard (heavily-pruned) set of features is used to describe problem instances, Winnow performs comparably to the Bayesian method; (2) When the full (unpruned) set of features is used, Winnow is able to exploit the new features and convincingly outperform Bayes; and (3) When a test set is encountered that is dissimilar to the training set, Winnow is better than Bayes at adapting to the unfamiliar test set, using a strategy we will present for combining learning on the training set with unsupervised learning on the (noisy) test set. In Machine Learning: Proceedings of the 13th International Conference, Lorenza Saitta, ed., Morgan Kaufmann, San Francisco, CA, 1996, pages 182{190. This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonpro t educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Information Technology Center America; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Information Technology Center America. All rights reserved. Copyright c Mitsubishi Electric Information Technology Center America, 1996 201 Broadway, Cambridge, Massachusetts 02139 Dept. of Appl. Math. & CS, Weizmann Institute of Science, Rehovot 76100, Israel. 1. First printing, TR96-07, April 1996

NIMISHA PHILIP, SHERLY K.K Department of Computer Science & Engineering, Toc H Institute of Science & Technology email:nimishavellapally@gmail.com Department of Information Technology, Toc H Institute of Science & Technology, Ernakulam, Kerala, India shrly_shilu@yahoo.com Abstract — Globalization and increased use of the Internet for online shopping has resulted in a considerable proliferation of credit card transactions throughout the world.Higher acceptability and convenience of credit cards for purchases has not only given personal comfort to customers but also attracted a large number of attackers. As a result, credit card payment systems must be supported by efficient fraud detection capability for minimizing unwanted activities by adversaries. Most of the well known algorithms for fraud detection are based on supervised training Every cardholder has a certain shopping behavior, which establishes an activity profile for him. Existing FDS try to capture behavioral patterns as rules which are static .This becomes ineffective when cardholder develops new patterns of behavior Here, we propose a unsupervised method to dynamically profile behavior pattern of customer Then the incoming transactions are compared against the user profile to indicate the anomalies, based on which the corresponding warnings are outputted. A FP tree based pattern matching algorithm is used to evaluate how unusual the new transactions are.

We propose signature-accelerated transactional memory (SigTM), ahybrid TM system that reduces the overhead of software transactions. SigTM uses hardware signatures to track the read-set and write-set forpending transactions and perform conflict detection between concurrent threads. All other transactional functionality, including dataversioning, is implemented in software. Unlike previously proposed hybrid TM systems, SigTM requires no modifications to the hardware caches, which reduces hardware cost and simplifies support for nested transactions and multithreaded processor cores. SigTM is also the first hybrid TM system to provide strong isolation guarantees between transactional blocks and non-transactional accesses without additional read and write barriers in non-transactional code.
 Using a set of parallel programs that make frequent use of coarse-grain transactions, we show that SigTM accelerates software transactions by 30% to 280%. For certain workloads, SigTM can match the performance of a full-featured hardware TM system, while for workloads with large read-sets it can be up to two times slower. Overall, we show that SigTM combines the performance characteristics and strong isolation guarantees of hardware TM implementations with the low cost and flexibility of software TM systems.

Classical recommender systems provide users with a list of recommendations where each recommendation consists of a single item, e.g., a book or DVD. However, several applications can benefit from a system capable of recommending packages of items, in the form of sets. Sample applications include travel planning with a limited budget (price or time) and twitter users wanting to select worthwhile tweeters to follow given that they can deal with only a bounded number of tweets. In these contexts, there is a need for a system that can recommend top-k packages for the user to choose from.
 Motivated by these applications, we consider composite recommendations, where each recommendation comprises a set of items. Each item has both a value (rating) and a cost associated with it, and the user specifies a maximum total cost (budget) for any recommended set of items. Our composite recommender system has access to one or more component recommender system, focusing on different domains, as well as to information sources which can provide the cost associated with each item. Because the problem of generating the top recommendation (package) is NP-complete, we devise several approximation algorithms for generating top-k packages as recommendations. We analyze their efficiency as well as approximation quality. Finally, using two real and two synthetic data sets, we subject our algorithms to thorough experimentation and empirical analysis. Our findings attest to the efficiency and quality of our approximation algorithms for top-k packages compared to exact algorithms.

Market Basket Analysis is a popular data mining tool that can be used to search through data to find patterns of co-occurrence among objects. It is an algorithmic process that generates business rules and several metrics for each business rule such as support, confidence and lift that help researchers identify “interesting” patterns. Although useful, these popular metrics do not incorporate traditional significance testing. This paper describes how to easily add a well-known statistical significance test, the Pearson’s Chi Squared statistic, to the existing output generated by SAS® Enterprise Miner’s Association Node. The addition of this significance test enhances the ability of data analysts to make better decisions about which business rules are likely to be more useful.

We present DrTM, a fast in-memory transaction processing system that exploits advanced hardware features (i.e., RDMA and HTM) to improve latency and throughput by over one order of magnitude compared to state-of-the-art distributed transaction systems. The high performance of DrTM are enabled by mostly offloading concurrency control within a local machine into HTM and leveraging the strong consistency between RDMA and HTM to ensure serializability among concurrent transactions across machines. We further build an efficient hash table for DrTM by leveraging HTM and RDMA to simplify the design and notably improve the performance. We describe how DrTM supports common database features like read-only transactions and logging for durability. Evaluation using typical OLTP workloads including TPC-C and SmallBank show that DrTM scales well on a 6-node cluster and achieves over 5.52 and 138 million transactions per second for TPC-C and SmallBank Respectively. This number outperforms a state-of-the-art distributed transaction system (namely Calvin) by at least 17.9X for TPC-C.


Question: Crossing the Threshold of Transactions: Extracting Inter-Transaction Association Patterns

This paper delves into the novel challenge of extracting association rules from expansive relational tables that encompass both quantitative and categorical data. A typical association could be, for instance, "among individuals aged 50 to 60 who are married, a significant 10% own at least two vehicles." Our methodology for handling quantitative attributes involves a detailed partitioning of the data values, followed by a strategic amalgamation of contiguous partitions when required. We introduce metrics for assessing partial completeness, which provide insights into the potential information loss resulting from partitioning. However, this direct method can sometimes yield an excessive number of nearly identical rules. To address this, we employ a "greater-than-expected-value" measure of interest to single out the truly noteworthy rules from the output. We present an algorithm designed for mining quantitative association rules in this context. Lastly, we offer an analysis of the outcomes when applying this methodology to a genuine dataset, illustrating its practical utility and effectiveness.

The two areas of online transaction processing (OLTP) and online analytical processing (OLAP) present different challenges for database architectures. Currently, customers with high rates of mission-critical transactions have split their data into two separate systems, one database for OLTP and one so-called data warehouse for OLAP. While allowing for decent transaction rates, this separation has many disadvantages including data freshness issues due to the delay caused by only periodically initiating the Extract Transform Load-data staging and excessive resource consumption due to maintaining two separate information systems. We present an efficient hybrid system, called HyPer, that can handle both OLTP and OLAP simultaneously by using hardware-assisted replication mechanisms to maintain consistent snapshots of the transactional data. HyPer is a main-memory database system that guarantees the ACID properties of OLTP transactions and executes OLAP query sessions (multiple queries) on the same, arbitrarily current and consistent snapshot. The utilization of the processor-inherent support for virtual memory management (address translation, caching, copy on update) yields both at the same time: unprecedentedly high transaction rates as high as 100000 per second and very fast OLAP query response times on a single system executing both workloads in parallel. The performance analysis is based on a combined TPC-C and TPC-H benchmark.

This chapter examines the state of modern intrusion detection, with a particular emphasis on the emerging approach of data mining. The discussion paralleIs two important aspects of intrusion detection: general detection strategy (misuse detection versus anomaly detection) and data source (individual hosts versus network trafik). Misuse detection attempts to match known patterns of intrusion , while anomaly detection searches for deviations from normal behavior . Between the two approaches, only anomaly detection has the ability to detect unknown attacks. A particularly promising approach to anomaly detection combines association mining with other forms of machine learning such as classification. Moreover, the data source that an intrusion detection system employs significantly impacts the types of attacks it can detect. There is a tradeoff in the level of detailed information available verD. Barbará et al. (ed .), Applications of Data Mining in Computer Security © Kluwer Academic Publishers 2002 s

Previous studies propose that associative classification has high classification accuracy and strong flexibility at handling unstructured data. However, it still suffers from the huge set of mined rules and sometimes biased classification or overfitting since the classification is based on only single high-confidence rule. In this study, we propose a new associative classification method, CMAR, i.e., Classification based on Multiple Association Rules. The method extends an efficient frequent pattern mining method, FP-growth, constructs a class distribution-associated FP-tree, and mines large database efficiently. Moreover, it applies a CR-tree structure to store and retrieve mined association rules efficiently, and prunes rules effectively based on confidence, correlation and database coverage. The classification is performed based on a weighted analysis using multiple strong association rules. Our extensive experiments on databases from UCI machine learning database repository show that CMAR is consistent, highly effective at classification of various kinds of databases and has better average classification accuracy in comparison with CBA and C4.5. Moreover, our performance study shows that the method is highly efficient and scalable in comparison with other reported associative classification methods.

The main purpose of data mining and analytics is to find novel, potentially useful patterns that can be utilized in real-world applications to derive beneficial knowledge. For identifying and evaluating the usefulness of different kinds of patterns, many techniques/constraints have been proposed, such as support, confidence, sequence order, and utility parameters (e.g., weight, price, profit, quantity, etc.). In recent years, there has been an increasing demand for utility-oriented pattern mining (UPM). UPM is a vital task, with numerous high-impact applications, including cross-marketing, e-commerce, finance, medical, and biomedical applications. This survey aims to provide a general, comprehensive, and structured overview of the state-of-the-art methods of UPM. First, we introduce an in-depth understanding of UPM, including concepts, examples, and comparisons with related concepts. A taxonomy of the most common and state-of-the-art approaches for mining different kinds of high-utility patterns is presented, including Apriori-based, tree-based, projection-based, vertical-/horizontal-data-format-based, and other hybrid approaches. A comprehensive review of advanced topics of existing high-utility pattern mining techniques is offered, with a discussion of their pros and cons. Finally, we present several well-known open-source software packages for UPM. We conclude our survey with a discussion on open and practical challenges in this field.

In recent years, privacy preserving data mining has become an important problem because of the large amount of personal data which is tracked by many business applications. In many cases, users are unwilling to provide personal information unless the privacy of sensitive information is guaranteed. In this paper, we propose a new framework for privacy preserving data mining of multi-dimensional data. Previous work for privacy preserving data mining uses a perturbation approach which reconstructs data distributions in order to perform the mining. Such an approach treats each dimension independently and therefore ignores the correlations between the different dimensions. In addition, it requires the development of a new distribution based algorithm for each data mining problem, since it does not use the multi-dimensional records, but uses aggregate distributions of the data as input. This leads to a fundamental re-design of data mining algorithms. In this paper, we will develop a new and flexible approach for privacy preserving data mining which does not require new problem-specific algorithms, since it maps the original data set into a new anonymized data set. This anonymized data closely matches the characteristics of the original data including the correlations among the different dimensions. We present empirical results illustrating the effectiveness of the method.

In recent years, progress in hardware technology has resulted in the possibility of monitoring many events in real time. The volume of incoming data may be so large, that monitoring all individual data might be intractable. Revisiting any particular record can also be impossible in this environment. Therefore, many database schemes, such as aggregation, join, frequent pattern mining, and indexing, become more challenging in this context. This paper surveys the previous efforts to resolve these issues in processing data streams. The emphasis is on specifying and processing sliding window queries, which are supported in many stream processing engines. We also review the related work on stream query processing, including synopsis structures, plan sharing, operator scheduling, load shedding, and disorder control. Category: Ubiquitous computing

Local Process Models (LPMs) describe structured fragments of process behavior occurring in the context of less structured business processes. In contrast to traditional support-based LPM discovery, which aims to generate a collection of process models that describe highly frequent behavior, High-Utility Local Process Model (HU-LPM) discovery aims to generate a collection of process models that provide useful business insights by specifying a utility function. Mining LPMs is a computationally expensive task, because of the large search space of LPMs. In supportbased LPM mining, the search space is constrained by making use of the property that support is anti-monotonic. We show that in general, we cannot assume a provided utility function to be anti-monotonic, therefore, the search space of HU-LPMs cannot be reduced without loss. We propose four heuristic methods to speed up the mining of HU-LPMs while still being able to discover useful HU-LPMs. We demonstrate their applicability on three real-life data sets.

Amazon sales-rank gives a relative estimate of a product item’s popularity among other items in the same category. An early prediction of the Amazon sales-rank of a product would imply an early guess of its sales-popularity relative to the other products on Amazon, which is one of the largest e-commerce hub across the globe. Traditional methods suggest use of product review related features, e.g., volume of reviews, text content of the reviews etc. for the purpose of prediction. In contrast, we propose in this paper for the first time a network-assisted approach to construct suitable features for prediction. In particular, we build a co-purchase network treating the individual products as nodes, with edges in between if two products are bought with one another. The way a product is positioned in this network (e.g., its centrality, clustering coefficient etc.) turns out to be a strong indicator of its sales-rank. This network-assisted approach has two distinct advantages over the traditional baseline method based on review analysis – (i) it works even if the product has no reviews (relevant especially in the early stages of the product launch) and (ii) it is notably more discriminative in classifying a popular (i.e., low sales-rank) product from an unpopular (i.e., high sales-rank) one. Based on this observation, we build a supervised model to early classify a popular product from an unpopular one. We report our results on two different product categories (CDs and cell phones) and obtain remarkably better classification accuracy compared to the baseline scheme. When the top 100 (700) products based on sales-rank are labelled as popular and the bottom 100 (700) are labelled as unpopular, the classification accuracy of our method is 89.85% (82.1%) for CDs and 84.11% (84.8%) for cell phones compared to 46.37% (68.75%) and 83.17% (71.95%) respectively from the baseline method.

Online behavioural targeting is one of the most popular business strategies on the display advertising today. It is based primarily on analysing web user behavioural data with the usage of machine learning techniques with the aim to optimise web advertising. Being able to identify “unknown” and “first time seen” customers is of high importance in online advertising since a successful guess could identify “possible prospects” who would be more likely to purchase an advertisement’s product. By identifying prospective customers, online advertisers may be able to optimise campaign performance, maximise their revenue as well as deliver advertisements tailored to a variety of user interests. This work presents a hybrid approach benchmarking machine-learning algorithms and attribute preprocessing techniques in the context of behavioural targeting in process oriented environments. The performance of our suggested methodology is evaluated using the key performance metric in online advertising which is the predicted conversion rate. Our experimental results indicate that the presented process mining framework can significantly identify prospect customers in most cases. Our results seem promising, indicating that there is a need for further workflow research in online display

A case study on the theoretical and practical value of using process mining for the detection of fraudulent behavior in the procurement process Abstract This thesis presents the results of a six month research period on process mining and fraud detection. This thesis aimed to answer the research question as to how process mining can be utilized in fraud detection and what the benefits of using process mining for fraud detection are. Based on a literature study it provides a discussion of the theory and application of process mining and its various aspects and techniques. Using both a literature study and an interview with a domain expert, the concepts of fraud and fraud detection are discussed. These results are combined with an analysis of existing case studies on the application of process mining and fraud detection to construct an initial setup of two case studies, in which process mining is applied to detect possible fraudulent behavior in the procurement process. Based on the experiences and results of these case studies, the 1+5+1 methodology is presented as a first step towards operationalizing principles with advice on how process mining techniques can be used in practice when trying to detect fraud. This thesis presents three conclusions: (1) process mining is a valuable addition to fraud detection, (2) using the 1+5+1 concept it was possible to detect indicators of possibly fraudulent behavior (3) the practical use of process mining for fraud detection is diminished by the poor performance of the current tools. The techniques and tools that do not suffer from performance issues are an addition, rather than a replacement, to regular data analysis techniques by providing either new, quicker, or more easily obtainable insights into the process and possible fraudulent behavior. iii Occam's Razor: " One should not increase, beyond what is necessary, the number of entities required to explain anything " iv Contents


Question: A review of feature extraction techniques and duplicate detection methods in text mining.

This paper introduces CLOSET, a novel and efficient algorithm designed for the mining of frequent closed itemsets, an approach that significantly streamlines association mining by condensing the volume of derived rules while preserving their potency. Association mining typically yields a vast and potentially unwieldy set of frequent itemsets and rules. To address this, mining frequent closed itemsets has emerged as a viable solution, offering equivalent power to traditional association mining but with a markedly reduced rule set. CLOSET achieves this efficiency through the integration of three key innovations: (1) the utilization of a compressed frequent pattern tree (FP-tree) structure, which facilitates the mining of closed itemsets without the need for candidate generation, (2) the introduction of a single prefix path compression technique that enables the swift identification of frequent closed itemsets, and (3) the implementation of a partition-based projection mechanism for enhancing scalability in large-scale databases. Experimental results demonstrate CLOSET's superior performance, showcasing its efficiency, scalability, and speed, outpacing previously proposed methods in handling large databases.

A goodtext classifieris a classifierthat efficiently categorizeslarge setsof text documentsin a reasonabletime frameand with an acceptableaccuracy, and that provides classificationrules that are humanreadablefor possible fine-tuning. If the training of the classifier is also quick, this couldbecomein someapplicationdomainsa goodasset for the classifier . Many techniquesand algorithmsfor automatictext categorizationhavebeendevised.According to publishedliterature, somearemoreaccuratethanothers, andsomeprovidemore interpretableclassificationmodels thanothers. However, nonecancombineall thebeneficial propertiesenumer atedabove. In this paper, we presenta novel approach for automatictext categorizationthat borrowsfrom market basket analysistechniquesusingassociation rule miningin thedata-miningfield. We focuson two major problems:(1) findingthebesttermassociationrules in a textual databaseby generating and pruning; and (2) using the rules to build a text classifier . Our text categorization methodprovesto beefficientandeffective, andexperimentsonwell-knowncollectionsshowthattheclassifier performswell. In addition,trainingaswell asclassification arebothfastandthegeneratedrulesarehumanreadable.

In a phishing attack, an unsuspecting victim is lured, typically via an email, to a web site designed to steal sensitive information such as bank/credit card account numbers, login information for accounts, etc. Each year Internet users lose billions of dollars to this scourge. In this paper, we present a general semantic feature selection method for text problems based on the statistical t-test and WordNet, and we show its effectiveness on phishing email detection by designing classifiers that combine semantics and statistics in analyzing the text in the email. Our feature selection method is general and useful for other applications involving text-based analysis as well. Our email body-text-only classifier achieves more than 95% accuracy on detecting phishing emails with a false positive rate of 2.24%. Due to its use of semantics, our feature selection method is robust against adaptive attacks and avoids the problem of frequent retraining needed by machine learning classifiers.

A FUZZY LOGIC-BASED TEXT CLASSIFICATION METHOD FOR SOCIAL MEDIA

The efficacy of syntactic features for topic-independent authorship attribution is evaluated, taking a feature set of frequencies of words and punctuation marks as baseline. The features are ‘deep’ in the sense that they are derived by parsing the subject texts, in contrast to ‘shallow’ syntactic features for which a part-of-speech analysis is enough. The experiments are conducted on a corpus of novels written around the year 1900 by 20 different authors, and cover two tasks. In the first task, text samples are taken from books by one author, and the goal is to pair samples from the same book. In the second task, text samples are taken from several authors, but only one sample from each book, and the goal is to pair samples from the same author. In the first task, the baseline feature set outperformed the syntax-based feature set, but for the second task, the outcome was the opposite. This suggests that, compared to lexical features such as vocabulary and punctuation, syntactic features are more robust to changes in topic.

Outlier Detection is a Data Mining Application. Outlier contains noisy data which is researched in various domains. The various techniques are already being researched that is more generic. We surveyed on various techniques and applications of outlier detection that provides a novel approach that is more useful for the beginners. The proposed approach helps to clean data at university level in less time with great accuracy. This survey includes the existing outlier techniques and applications where the noisy data exists. Our paper defines critical review on various techniques used in different applications of outlier detection that are to be researched further and they gives a particular type of knowledge based data i.e. more useful in research activities. So where the Anomalies is present it will be detected through outlier detection techniques and monitored accordingly especially in educational Data Mining.

We present a novel approach to recognizing Textual Entailment. Structural features are constructed from abstract tree descriptions, which are automatically extracted from syntactic dependency trees. These features are then applied in a subsequence-kernel-based classifier to learn whether an entailment relation holds between two texts. Our method makes use of machine learning techniques using a limited data set, no external knowledge bases (e.g. WordNet), and no handcrafted inference rules. We achieve an accuracy of 74.5% for text pairs in the Information Extraction and Question Answering task, 63.6% for the RTE-2 test data, and 66.9% for the RET-3 test data.

We describe an open-source Java toolkit of methods for matching names and records. We summarize results obtained from using various string distance metrics on the task of matching entity names: these metrics include distance functions proposed by several different communities, including edit-distance metrics, fast heuristic string comparators, token-based distance metrics, and hybrid methods. We then describe an extension to the toolkit which allows records to be compared. We discuss some issues involved in performing a similar comparison for record-matching techniques, and finally present results for some baseline record-matching algorithms which are based on string comparisons between fields.

Entity matching that finds records referring to the same entity is an important operation in data cleaning and integration. Existing studies usually use a given similarity function to quantify the similarity of records, and focus on devising index structures and algorithms for efficient entity matching. However it is a big challenge to define “how similar is similar” for real applications, since it is rather hard to automatically select appropriate similarity functions. In this paper we attempt to address this problem. As there are a large number of similarity functions, and even worse thresholds may have infinite values, it is rather expensive to find appropriate similarity functions and thresholds. Fortunately, we have an observation that different similarity functions and thresholds have redundancy, and we have an opportunity to prune inappropriate similarity functions. To this end, we propose effective optimization techniques to eliminate such redundancy, and devise efficient algorithms to find the best similarity functions. The experimental results on both real and synthetic datasets show that our method achieves high accuracy and outperforms the baseline algorithms.

The popularity and utility of smartphones rely on their vibrant application markets; however, plagiarism threatens the long-term health of these markets. In this paper, we present a scalable approach to detecting similar Android apps based on semantic information. We implement our approach in a tool called AnDarwin and evaluate it on 265,359 apps collected from 17 markets including Google Play and numerous third-party markets. In contrast with earlier approaches, AnDarwin does not compare apps pairwise, thus greatly increasing its scalability. Additionally, AnDarwin does not rely on information other than the app code — such as the app’s market, signature, or description — thus greatly increasing its reliability. AnDarwin can automatically detect library code and remove it from the similarity analysis. We present two use cases for AnDarwin: finding similar apps by different developers (“clones”) and similar apps from the same developer (“rebranded”). In ten hours, AnDarwin detected at least 4,295 apps which have been the victims of cloning and 36,106 apps that are rebranded. By analyzing the clusters found by AnDarwin, we found 88 new variants of malware and identified 169 malicious apps based on differences in the requested permissions. In contrast to earlier approaches, AnDarwin can detect both full and partial app similarity. Additionally, AnDarwin can automatically detect similar code that is injected into many apps, which may indicate the spread of malware. Our evaluation demonstrates AnDarwin’s ability to accurately detect similar apps on a large scale.

Human-quality text summarization systems are di cult to design, and even more di cult to evaluate, in part because documents can di er along several dimensions, such as length, writing style and lexical usage. Nevertheless, certain cues can often help suggest the selection of sentences for inclusion in a summary. This paper presents our analysis of news-article summaries generated by sentence selection. Sentences are ranked for potential inclusion in the summary using a weighted combination of statistical and linguistic features. The statistical features were adapted from standard IR methods. The potential linguistic ones were derived from an analysis of news-wire summaries. To evaluate these features we use a normalized version of precision-recall curves, with a baseline of random sentence selection, as well as analyze the properties of such a baseline. We illustrate our discussions with empirical results showing the importance of corpus-dependent baseline summarization standards, compression ratios and carefully crafted long queries.


Question: Survey of feature extraction and duplicate detection methods in text mining.

This paper delves into the assessment and juxtaposition of methodologies for large-scale data analysis, with a focus on the MapReduce (MR) framework and parallel SQL database management systems (DBMS). Despite MR's recent popularity as a groundbreaking computational model for big data analysis [17], its core control flow has been a staple in parallel DBMS for over two decades. Here, we critically evaluate both approaches in terms of performance and development intricacy. To facilitate this comparison, we devise a benchmark encompassing a series of tasks executed on open-source MR and two parallel DBMS platforms. Each task is run under varying degrees of parallelism on a 100-node cluster to gauge system performance. Our findings highlight intriguing trade-offs. Despite the more time-consuming data loading and optimization processes in parallel DBMS compared to MR, the performance of DBMS significantly outshines MR. We explore the potential factors behind this pronounced performance gap and discuss architectural concepts that future systems should integrate from both MR and parallel DBMS frameworks.

We describe three semantic text similarity systems developed for the *SEM 2013 STS shared task and the results of the corresponding three runs. All of them shared a word similarity feature that combined LSA word similarity and WordNet knowledge. The first, which achieved the best mean score of the 89 submitted runs, used a simple term alignment algorithm augmented with penalty terms. The other two runs, ranked second and fourth, used support vector regression models to combine larger sets of features.

This paper consists of three parts: a preliminary typology of summaries in general; a description of the current and planned modules and performance of the SUMMARIST automated multilingual text summarization system being built sat ISI, and a discussion of three methods to evaluate summaries. 1. T H E N A T U R E O F S U M M A R I E S Early experimentation in the late 1950's and early 60's suggested that text summarization by computer was feasible though not s traightforward (Luhn, 59; Edmundson, 68). The methods developed then were fairly unsophisticated, relying primarily on surface level phenomena such as sentence position and word frequency counts, and focused on producing extracts (passages selected from the text, reproduced verbatim) rather than abstracts (interpreted portions of the text, newly generated). After a hiatus of some decades, the growing presence of large amounts of online text--in corpora and especially on the Web--renewed the interest in automated text summarization. During these intervening decades, progress in Natural Language Processing (NLP), coupled with great increases of computer memory and speed, made possible more sophisticated techniques, with very encouraging results. In the late 1990's, some relatively small research investments in the US (not more than 10 projects, including commercial efforts at Microsoft, Lexis-Nexis, Oracle, SRA, and TextWise, and university efforts at CMU, NMSU, UPenn, and USC/ISI) over three or four years have produced several systems that exhibit potential marketability, as well as several innovations that promise continued improvement. In addition, several recent workshops, a book collection, and several tutorials testify that automated text summarization has become a hot area. However, when one takes a moment to study the various systems and to consider what has really been achieved, one cannot help being struck by their underlying similarity, by the narrowness of their focus, and by the large numbers of unknown factors that surround the problem. For example, what precisely is a summary? No-one seems to know exactly. In our work, we use summary as the generic term and define it as follows: A summary is a text that is produced out of one or more (possibly multimedia) texts, that contains (some of) the same information of the original text(s), and that is no longer than half of the original text(s). To clarify the picture a little, we follow and extend (Sp~irck Jones, 97) by identifying the following aspects of variation. Any summary can be characterized by (at least) three major classes of characteristics: Invut: characteristics of the source text(s) Source size: single-document v s . multi-document: A single-document summary derives from a single input text (though the summarization process itself may employ information compiled earlier from other texts). A multi-document summary is one text that covers the content of more than one input text, and is usually used only when the input texts are thematically related. Specificity: domain-specific vs. general: When the input texts all pertain to a single domain, it may be appropr ia te to apply domain spec i f i c summarization techniques, focus on specific content, and output specific formats, compared to the general case. A domain-specific summary derives from input text(s) whose theme(s) pertain to a single restricted domain. As such, it can assume less term ambiguity, idiosyncratic word and grammar usage, specialized formatting, etc., and can reflect them in the summary.

Crime analysis has become a critical area for helping law enforcement agencies to protect civilians. As a result of a rapidly increasing population, crime rates have increased dramatically, and appropriate analysis has become a timeconsuming effort. Text mining is an effective tool that may help to solve this problem to classify crimes in effective manner. The proposed system aims to detect and classify crimes in Twitter posts that written in the Arabic language, one of the most widespread languages today. In this paper, classification techniques are used to detect crimes and identify their nature by different classification algorithms. The experiments evaluate different algorithms, such as SVM, DT, CNB, and KNN, in terms of accuracy and speed in the crime domain. Also, different features extraction techniques are evaluated, including rootbased stemming, light stemming, n-gram. The experiments revealed the superiority of n-gram over other techniques. Specifically, the results indicate the superiority of SVM with trigram over other classifiers, with a 91.55% accuracy. Keywords—Crimes; text mining; classification; features extraction techniques; arabic posts; twitter

Summarization is the way towards lessening the content of a text file to make it brief that holds all the critical purposes in the content of original text file. In the process of extractive summarization, one extracts only those sentences which are the most relevant sentences in the text document and that conveys the moral of the content. The extractive summarization techniques usually revolve around the idea of discovering most relevant and frequent keywords and then extract the sentences based on those keywords. Manual extraction or explanation of relevant keywords are a dreary procedure overflowing with errors including loads of manual exertion and time. In this paper, we proposed a hybrid approach to extract keyword automatically for multi-document text summarization in enewspaper articles. The performance of the proposed approach is compared with three additional keyword extraction techniques namely, term frequency-inverse document frequency (TF-IDF), term frequency-adaptive inverse document frequency (TF-AIDF), and a number of false alarm (NFA) for automatic keyword extraction and summarization in e-newspapers articles for better analysis. Finally, we showed that our proposed techniques had been outperformed over other techniques for automatic keyword extraction and summarization.

This paper deals with the problem of author identification. The common N-grams (CNG) method [6] is a language-independent profile-based approach with good results in many author identification experiments so far. A variation of this approach is presented based on new distance measures that are quite stable for large profile length values. Special emphasis is given to the degree upon which the effectiveness of the method is affected by the available training text samples per author. Experiments based on text samples on the same topic from the Reuters Corpus Volume 1 are presented using both balanced and imbalanced training corpora. The results show that CNG with the proposed distance measures is more accurate when only limited training text samples are available, at least for some of the candidate authors, a realistic condition in author identification problems.

New Event Detection is a challenging task that still offers scope for great improvement after years of effort. In this paper we show how performance on New Event Detection (NED) can be improved by the use of text classification techniques as well as by using named entities in a new way. We explore modifications to the document representation in a vector space-based NED system. We also show that addressing named entities preferentially is useful only in certain situations. A combination of all the above results in a multi-stage NED system that performs much better than baseline single-stage NED systems.

We propose a new unsupervised learning technique for extracting information from large text collections. We model documents as if they were generated by a two-stage stochastic process. Each author is represented by a probability distribution over topics, and each topic is represented as a probability distribution over words for that topic. The words in a multi-author paper are assumed to be the result of a mixture of each authors' topic mixture. The topic-word and author-topic distributions are learned from data in an unsupervised manner using a Markov chain Monte Carlo algorithm. We apply the methodology to a large corpus of 160,000 abstracts and 85,000 authors from the well-known CiteSeer digital library, and learn a model with 300 topics. We discuss in detail the interpretation of the results discovered by the system including specific topic and author models, ranking of authors by topic and topics by author, significant trends in the computer science literature between 1990 and 2002, parsing of abstracts by topics and authors and detection of unusual papers by specific authors. An online query interface to the model is also discussed that allows interactive exploration of author-topic models for corpora such as CiteSeer.

We present an evaluation framework for plagiarism detection.1 The framework provides performance measures that address the specifics of plagiarism detection, and the PAN-PC-10 corpus, which contains 64 558 artificial and 4 000 simulated plagiarism cases, the latter generated via Amazon’s Mechanical Turk. We discuss the construction principles behind the measures and the corpus, and we compare the quality of our corpus to existing corpora. Our analysis gives empirical evidence that the construction of tailored training corpora for plagiarism detection can be automated, and hence be done on a large scale.

This paper addresses the problem of extending an adaptive information filtering system to make decisions about the novelty and redundancy of relevant documents. It argues that relevance and redundance should each be modelled explicitly and separately. A set of five redundancy measures are proposed and evaluated in experiments with and without redundancy thresholds. The experimental results demonstrate that the cosine similarity metric and a redundancy measure based on a mixture of language models are both effective for identifying redundant documents.

With the advent of the Internet, large amount of digital text is generated everyday in the form of news articles, research publications, blogs, question answering forums and social media. It is important to develop techniques for extracting information automatically from these documents, as lot of important information is hidden within them. This extracted information can be used to improve access and management of knowledge hidden in large text corpora. Several applications such as Question Answering, Information Retrieval would benefit from this information. Entities like persons and organizations, form the most basic unit of the information. Occurrences of entities in a sentence are often linked through well-defined relations; e.g., occurrences of person and organization in a sentence may be linked through relations such as employed at. The task of Relation Extraction (RE) is to identify such relations automatically. In this paper, we survey several important supervised, semi-supervised and unsupervised RE techniques. We also cover the paradigms of Open Information Extraction (OIE) and Distant Supervision. Finally, we describe some of the recent trends in the RE techniques and possible future research directions. This survey would be useful for three kinds of readers i) Newcomers in the field who want to quickly learn about RE; ii) Researchers who want to know how the various RE techniques evolved over time and what are possible future research directions and iii) Practitioners who just need to know which RE technique works best in various settings.


Question: A review of feature extraction techniques and duplicate detection methods in text mining.

Starfish: An Adaptive and Self-Optimizing Framework for Big Data Processing

Hadoop, a leading framework for big data analytics, embodies the MAD (Magnetism, Agility, Depth) characteristics essential for handling vast and diverse datasets. Magnetism ensures that all data sources are integrated, Agility accommodates the rapid evolution of data, and Depth supports intricate analytical requirements. Starfish, a pioneering self-tuning system, enhances Hadoop's capabilities, making it MADDER (MAD, Data Lifecycle-aware, Elastic, Robust) and more autonomous.

Starfish introduces data lifecycle awareness to Hadoop, ensuring optimal performance at every stage of data's journey from creation to retirement. It dynamically adjusts resources and operational costs through elasticity, enabling efficient scaling. Furthermore, Starfish emphasizes robustness, guaranteeing system availability and predictability, crucial for mission-critical applications. Just-in-Time Job Optimization is a key feature, allowing Starfish to fine-tune job execution based on current data and resource conditions, maximizing efficiency and reducing latency.

Smart extraction of knowledge from social media has received the recent interest of the Biomedical and Health Informatics community for the simultaneous improvement of healthcare outcomes and lessen the expenses making use of consumergenerated reviews. Social media provides chances for patients and doctors to share their views and experiences without any obtrusion through online communities that might generate information, which is much beyond what is known by the domain experts. Nonetheless, for conventional public health surveillance systems, it is difficult to detect and then monitor the concerns related to health and the changes seen in attitudes of the public towards health-related problems. To solve this problem, several studies have shown the usage of information in social media for the discovery of biomedical and healthrelated information. Several disease-specific knowledge exchanges are now available on Face book and other portals of online social networking. These kind of new sources of information, support, and engagement have gone to become significant for patients who are suffering with the disease, and still the quality and the content of the knowledge contributed in these digital areas are not properly comprehended. The existing research methodologies are discussed with their merits and demerits, so that the further research works can be concentrated more. The experimental tests conducted were on all the research works in mat lab simulation environment and it is compared against each other to find the better approach under various performance measures such as Accuracy, Precision and Recall.

The Web has become an excellent source for gathering consumer opinions. There are now numerous Web sites containing such opinions, e.g., customer reviews of products, forums, discussion groups, and blogs. This paper focuses on online customer reviews of products. It makes two contributions. First, it proposes a novel framework for analyzing and comparing consumer opinions of competing products. A prototype system called Opinion Observer is also implemented. The system is such that with a single glance of its visualization, the user is able to clearly see the strengths and weaknesses of each product in the minds of consumers in terms of various product features. This comparison is useful to both potential customers and product manufacturers. For a potential customer, he/she can see a visual side-by-side and feature-by-feature comparison of consumer opinions on these products, which helps him/her to decide which product to buy. For a product manufacturer, the comparison enables it to easily gather marketing intelligence and product benchmarking information. Second, a new technique based on language pattern mining is proposed to extract product features from Pros and Cons in a particular type of reviews. Such features form the basis for the above comparison. Experimental results show that the technique is highly effective and outperform existing methods significantly.

Sifting through vast collections of unstructured or semistructured data beyond the reach of data mining tools, text mining tracks information sources, links isolated concepts in distant documents, maps relationships between activities, and helps answer questions.

In the era of information overload, text summarization can be defined as the process of extracting useful information from a large space of available content using traditional filtering methods. One of the major challenges in the domain of extraction based summarization is that a single statistical measure is not sufficient to produce efficient summaries which would be close to human-made ‘gold standard’, since each measure suffers from individual weaknesses. We deal with this problem by proposing a text summarization model that combines various statistical measures so that the pitfalls of an individual technique could be compensated by the strengths of others. Experimental results are presented to demonstrate the effectiveness of the proposed method using the TAC 2011 Multiling pilot dataset for English language and ROUGE summary evaluation tool.

Although there are a large number of thesauri for the biomedical domain many of them lack coverage in terms and their variant forms. Automatic thesaurus construction based on patterns was first suggested by Hearst [1], but it is still not clear how to automatically construct such patterns for different semantic relations and domains. In particular it is not certain which patterns are useful for capturing synonymy. The assumption of extant resources such as parsers is also a limiting factor for many languages, so it is desirable to find patterns that do not use syntactical analysis. Finally to give a more consistent and applicable result it is desirable to use these patterns to form synonym sets in a sound way. We present a method that automatically generates regular expression patterns by expanding seed patterns in a heuristic search and then develops a feature vector based on the occurrence of term pairs in each developed pattern. This allows for a binary classifications of term pairs as synonymous or non-synonymous. We then model this result as a probability graph to find synonym sets, which is equivalent to the well-studied problem of finding an optimal set cover. We achieved 73.2% precision and 29.7% recall by our method, out-performing hand-made resources such as MeSH and Wikipedia. We conclude that automatic methods can play a practical role in developing new thesauri or expanding on existing ones, and this can be done with only a small amount of training data and no need for resources such as parsers. We also concluded that the accuracy can be improved by grouping into synonym sets.

In recent times, the necessity of generating single document summary has gained popularity among the researchers due to its extensive applicability. The text summarization can be categorized with different approaches like: extractive and abstractive from single document or multi document, goal of text summarization (intent, focus and coverage), characteristic of text summarization (frequency-based, knowledge-based and discoursebased), level of processing (surface level, entities level and discourse level) and kind of information (lexicon, structure information and deep understanding). Recently, the efforts in research are transferred from single document summarization to multi document summarization Multi-document summarization considerably differs from single in issues related to compression, speed, redundancy and passage selection are critical in the formation of useful summaries. In this paper, we review the techniques that have done for multi document summarization. Next, we describe evaluation method In conclusion; we propose our future work for multi document

The set of features which the traditional feature selection algorithm of chi-square selected is not complete. This causes the low performance for the final text classification. Therefore, this paper proposes a method. The method utilizes word2vec to generate word vector to improve feature selection algorithm of the chi square. The algorithm applies the word vector generated by word2vec to the process of the traditional feature selection and uses these words to supplement the set of features as appropriate. Ultimately, the set of features obtained by this method has better discriminatory power. Because, the feature words with the better discriminatory power has the strong ability of distinguishing categories as its semantically similar words. On this base, multiple experiments have been carried out in this paper. The experimental results show that the performance of text classification can increase after extension of feature words.

Because time series data have a unique data structure, it is not easy to apply some existing data mining tools directly to the data. For example, in classification and clustering problems, each time point is often considered a variable and each time series is considered an observation. As the time dimension increases, the number of variables also increases, in proportion to the time dimension. Therefore, data mining tasks require some feature extraction techniques to summarize each time series in a form that has a significantly lower dimension. This paper describes various feature extraction methods for time series data that are implemented in SAS Enterprise MinerTM. INTRODUCTION Time series data mining has four major tasks: clustering, indexing, classification, and segmentation. Clustering finds groups of time series that have similar patterns. Indexing finds similar time series in order, given a query series. Classification assigns each time series to a known category by using a trained model. Segmentation partitions time series. Time series data can be considered multidimensional data; this means that there is one observation per time unit, and each time unit makes one dimension. Some common data mining techniques are used for the tasks. However, in the real world, each time series is usually a high-dimensional sequence, such as stock market data whose prices change over time or data collected by a medical device. When the time unit is seconds, the dimensions of data that accumulate in just one hour are 3,600. Moreover, the dimensions are highly correlated to one another. The number of variables in training data increases proportionally as the time dimension increases. Most existing data mining tools cannot be used efficiently on time series data without a dimension reduction. Therefore, a dimension reduction is required through feature extraction techniques that map each time series to a lower-dimensional space. The most common techniques of dimension reduction in time series are singular value decomposition (SVD), discrete Fourier transformation (DFT), discrete wavelet transformation (DWT), and line segment methods. Some classical time series analyses, such as seasonal, trend, seasonal decomposition, and correlation analyses, are also used for feature extraction. This paper shows how to use SAS Enterprise Miner to implement the feature extraction techniques. But it does not attempt to explain these time series techniques in detail and instead invites the interested reader to explore some of the literature on the subject: Keogh and Pazzani (2000a), Keogh et al. (2000), and Gavrilov et al. (2000) for time series dimensionality reduction and the TIMESERIES procedure documentation in SAS/ETS for the classical time series analyses. This paper is organized as follows. First, it explains feature extraction and dimension reduction techniques that use classical time series analysis, and then it describes feature extraction techniques that use some well-known mathematical functions for dimension reduction. Next, it demonstrates the performance of dimension reduction techniques by presenting some examples. The paper uses SAS Enterprise Miner 13.1 to demonstrate these techniques.

This paper presents an approach for knowledge discovery in texts extracted from the Web. Instead of analyzing words or attribute values, the approach is based on concepts, which are extracted from texts to be used as characteristics in the mining process. Statistical techniques are applied on concepts in order to find interesting patterns in concept distributions or associations. In this way, users can perform discovery in a high level, since concepts describe real world events, objects, thoughts, etc. For identifying concepts in texts, a categorization algorithm is used associated to a previous classification task for concept definitions. Two experiments are presented: one for political analysis and other for competitive intelligence. At the end, the approach is discussed, examining its problems and advantages in the Web context.

In recent years we have seen the emergence of a variety of scholarly datasets. Typically these capture ‘standard’ scholarly entities and their connections, such as authors, affiliations, venues, publications, citations, and others. However, as the repositories grow and the technology improves, researchers are adding new entities to these repositories to develop a richer model of the scholarly domain. In this paper, we introduce TechMiner, a new approach, which combines NLP, machine learning and semantic technologies, for mining technologies from research publications and generating an OWL ontology describing their relationships with other research entities. The resulting knowledge base can support a number of tasks, such as: richer semantic search, which can exploit the technology dimension to support better retrieval of publications; richer expert search; monitoring the emergence and impact of new technologies, both within and across scientific fields; studying the scholarly dynamics associated with the emergence of new technologies; and others. TechMiner was evaluated on a manually annotated gold standard and the results indicate that it significantly outperforms alternative NLP approaches and that its semantic features improve performance significantly with respect to both recall and precision.


Question: A review of feature extraction techniques and duplicate detection methods in text mining.

This paper explores efficient methodologies for extracting significant patterns, known as association rules, from large datasets in the field of data mining. Association rule mining is a crucial technique for discovering interesting relations between variables in extensive databases. The focus here is on fast algorithms designed to optimize the process of finding these rules. These algorithms are tailored to handle the computational challenges posed by massive datasets, ensuring that the extraction of association rules is both swift and accurate. Key strategies include optimizing search techniques, utilizing data structures that enhance retrieval speed, and implementing parallel and distributed processing to reduce computational time. The paper also discusses the importance of evaluating the efficiency and scalability of these algorithms, especially in the context of real-world applications where data volumes are continuously growing. By highlighting the latest advancements in fast algorithms for mining association rules, this work aims to guide researchers and practitioners in selecting the most suitable methods for their specific data mining tasks.

Temporal Text Mining (TTM) is concerned with discovering temporal patterns in text information collected over time. Since most text information bears some time stamps, TTM has many applications in multiple domains, such as summarizing events in news articles and revealing research trends in scientific literature. In this paper, we study a particular TTM task -- discovering and summarizing the evolutionary patterns of themes in a text stream. We define this new text mining problem and present general probabilistic methods for solving this problem through (1) discovering latent themes from text; (2) constructing an evolution graph of themes; and (3) analyzing life cycles of themes. Evaluation of the proposed methods on two different domains (i.e., news articles and literature) shows that the proposed methods can discover interesting evolutionary theme patterns effectively.

We approach the problem of academic literature search by considering an unpublished manuscript as a query to a search system. We use the text of previous literature as well as the citation graph that connects it to find relevant related material. We evaluate our technique with manual and automatic evaluation methods, and find an order of magnitude improvement in mean average precision as compared to a text similarity baseline.

This paper presents an approach for knowledge discovery in texts extracted from the Web. Instead of analyzing words or attribute values, the approach is based on concepts, which are extracted from texts to be used as characteristics in the mining process. Statistical techniques are applied on concepts in order to find interesting patterns in concept distributions or associations. In this way, users can perform discovery in a high level, since concepts describe real world events, objects, thoughts, etc. For identifying concepts in texts, a categorization algorithm is used associated to a previous classification task for concept definitions. Two experiments are presented: one for political analysis and other for competitive intelligence. At the end, the approach is discussed, examining its problems and advantages in the Web context.

The proliferation of Internet has not only generated huge volumes of unstructured information in the form of web documents, but a large amount of text is also generated in the form of emails, blogs, and feedbacks etc. The data generated from online communication acts as potential gold mines for discovering knowledge. Text analytics has matured and is being successfully employed to mine important information from unstructured text documents. Most of these techniques use Natural Language Processing techniques which assume that the underlying text is clean and correct. Statistical techniques, though not as accurate as linguistic mechanisms, are also employed for the purpose to overcome the dependence on clean text. The chief bottleneck for designing statistical mechanisms is however its dependence on appropriately annotated training data. None of these methodologies are suitable for mining information from online communication text data due to the fact that they are often noisy. These texts are informally written. They suffer from spelling mistakes, grammatical errors, improper punctuation and irrational capitalization. This paper focuses on opinion extraction from noisy text data. It is aimed at extracting and consolidating opinions of customers from blogs and feedbacks, at multiple levels of granularity. Ours is a hybrid approach, in which we initially employ a semi-supervised method to learn domain knowledge from a training repository which contains both noisy and clean text. Thereafter we employ localized linguistic techniques to extract opinion expressions from noisy text. We have developed a system based on this approach, which provides the user with a platform to analyze opinion expressions extracted from a repository.

Citations management is an important task in managing digital libraries. Citations provide valuable information e.g., used in evaluating an author's influences or scholarly quality (the impact factor of research journals). But although a reliable and effective autonomous citation management is essential, manual citation management can be extremely costly. Automatic citation mining on the other hand is a non-trivial task mainly due to non-conforming citation styles, spelling errors and the difficulty of reliably extracting text from PDF documents. In this paper we propose a novel rule-based autonomous citation mining technique, to address this important task. We define a set of common heuristics that together allow to improve the state of the art in automatic citation mining. Moreover, by first disambiguating citations based on venues, our technique significantly enhances the correct discovery of citations. Our experiments show that the proposed approach is indeed able to overcome limitations of current leading citation indexes such as ISI Web of Knowledge , Citeseer and Google Scholar.

Extractive summarization of text documents usually consists of ranking the document sentences and extracting the top-ranked sentences subject to the summary length constraints. In this paper, we explore the contribution of various supervised learning algorithms to the sentence ranking task. For this purpose, we introduce a novel sentence ranking methodology based on the similarity score between a candidate sentence and benchmark summaries. Our experiments are performed on three benchmark summarization corpora: DUC-2002, DUC2007 and MultiLing-2013. The popular linear regression model achieved the best results in all evaluated datasets. Additionally, the linear regression model, which included POS (Part-of-Speech)-based features, outperformed the one with statistical features only.

This paperpresentsthe architectureandthe multidocument summarizationtechniquesimplementedin the GISTEXTER system.The paperpresentsanalgorithmfor producing incrementalmulti-document summariesif extractiontemplatesof goodquality areavailable. An empirical methodof generatingad-hoctemplatesthat can be populated with information extractedfrom texts by automatically acquiredextractionpatternsis alsopresented . The resultsof GISTEXTER in the DUC-2001evaluations account for the advantagesof usingthetechniquespresentedin this paper .

Paper presents a survey of methods and approaches for keyword extraction task. In addition to the systematization of methods, the paper gathers a comprehensive review of existing research. Related work on keyword extraction is elaborated for supervised and unsupervised methods, with special emphasis on graphbased methods as well as Croatian keyword extraction. Selectivity-based keyword extraction method is proposed as a new unsupervised graph-based keyword extraction method which extracts nodes from a complex network as keyword candidates. The paper provides guidelines for future research and development of new graph-based approaches for keyword extraction.

Variable and feature selection have become the focus of much research in areas of application for which datasets with tens or hundreds of thousands of variabl es are available. These areas include text processing of internet documents, gene expression arr ay nalysis, and combinatorial chemistry. The objective of variable selection is three-fold: improvi ng the prediction performance of the predictors, providing faster and more cost-effective predict ors, and providing a better understanding of the underlying process that generated the data. The contrib utions of this special issue cover a wide range of aspects of such problems: providing a better definit ion of the objective function, feature construction, feature ranking, multivariate feature sele ction, efficient search methods, and feature validity assessment methods.

-A reliable method for extracting structural features from fingerprint images is presented. Viewing fingerprint images as a textured image, an orientation flow field is computed. The rest of the stages in the algorithm use the flow field to design adaptive filters for the input image. To accurately locate ridges, a waveform projection-based ridge segmentation algorithm is used. The ridge skeleton image is obtained and smoothed using morphological operators to detect the features. A large number of spurious features from the detected set of minutiae is deleted by a postprocessing stage. The performance of the proposed algorithm has been evaluated by computing a "goodness index" (GI) which compares the results of automatic extraction with manually extracted ground truth. The significance of the observed GI values is determined by comparing the index for a set of fingerprints against the GI values obtained under a baseline distribution. The detected features are observed to be reliable and accurate. Fingerprints Feature extraction Texture Flow orientation Minutiae Segmentation Skeleton


Question: Survey of feature extraction and duplicate detection techniques in text mining.

A Survey on Language-Agnostic Techniques for Document Summarization and Text Mining

This paper presents a comprehensive review of language-independent algorithms designed for single and multiple document summarization, focusing on their relevance to text mining. The study explores various similarity metrics, such as cosine similarity and string kernels, which are essential in the process of text summarization and feature extraction. These metrics enable the detection of duplicates and the extraction of meaningful features from text data, crucial for effective text mining operations.

The document provides an illustrative example of summarization in action, using news articles about Hurricane Gilbert. The algorithm processes multiple documents related to the hurricane, identifying key points such as the storm's movement, impact on the Dominican Republic, and reactions from local authorities. It highlights the Civil Defense's response, including alerts and preparations for high winds, heavy rains, and high seas. The summary also touches on the storm's strength, estimated population in affected areas, and the broader context of tropical storms in the region, including a mention of Hurricane Florence and its aftermath.

The paper emphasizes the importance of language-agnostic algorithms in text mining, which can process text data in any language, making them versatile tools for global information retrieval and analysis. These algorithms are particularly useful in environments where multilingual data is common, ensuring that valuable insights are not overlooked due to language barriers.

In conclusion, the survey underscores the role of similarity metrics and language-independent techniques in enhancing the efficiency and accuracy of text mining and summarization tasks. The case study on Hurricane Gilbert demonstrates the practical application of these algorithms in extracting relevant information from multiple sources, making it a valuable resource for researchers and practitioners in the field of text mining.

In today’s world redundancy is the most vital problem faced in almost all domains. Novelty detection is the identification of new or unknown data or signal that a machine learning system is not aware of during training. The problem becomes more intense when it comes to “Research Articles”. A method of identifying novelty at each sections of the article is highly required for determining the novel idea proposed in the research paper. Since research articles are semistructured, detecting novelty of information from them requires more accurate systems. Topic model provides a useful means to process them and provides a simple way to analyze them. This work compares the most predominantly used topic modelLatent Dirichlet Allocation with the hierarchical Pachinko Allocation Model. The results obtained are promising towards hierarchical Pachinko Allocation Model when used for document retrieval.

In this paper we have presented a brief current state of the Art for Arabic text representation and classification methods. First we describe some algorithms applied to classification on Arabic text. Secondly, we cite all major works when comparing classification algorithms applied on Arabic text, after this, we mention some authors who proposing new classification methods and finally we investigate the impact of preprocessing on Arabic TC.

The technology underlying text search engines has advanced dramatically in the past decade. The development of a family of new index representations has led to a wide range of innovations in index storage, index construction, and query evaluation. While some of these developments have been consolidated in textbooks, many specific techniques are not widely known or the textbook descriptions are out of date. In this tutorial, we introduce the key techniques in the area, describing both a core implementation and how the core can be enhanced through a range of extensions. We conclude with a comprehensive bibliography of text indexing literature.

The purpose of this research was to automatically extract catchphrases given a set of Legal documents. For this task, our focus was mainly on the Machine learning approaches: a comparative approach was used between the unsupervised and supervised approaches. The idea was to compare the different approaches to see which one of the two was comparatively better for automatic catchphrase extraction given a dataset of Legal documents. To perform this, two open source text mining software were used; one for the unsupervised approach while another one was used for the supervised approach. We then fine tuned some parameters for each tool before extracting catchphrases. The training dataset was used when fine tuning parameters in order to find optimal parameters that were then used for generating the final catchphrases. Different metrics were used to evaluate the results. We used the most common measures in Information Extraction which include Precision and Recall and the results from the two Machine learning approaches were compared. In general our results showed that the supervised approach performed far much better than the unsupervised approach.

In this paper, we present an implementation of a new method for text classification that achieves significantly faster results than most existing classifiers. We extract features that are computationally efficient and forgo more computationally expensive ones used by many other text classifiers, most notably n-grams (contiguous sequences of n words). We then analyze the feature vectors through a hybrid SVM technique that sequentially classifies them with a one-versus-all classifier and a sequence of binary classifiers. Our overall method finishes execution within seconds on large novels, with accuracy comparable to that of standard classifiers but with a significantly shorter runtime.

Crime analysis has become a critical area for helping law enforcement agencies to protect civilians. As a result of a rapidly increasing population, crime rates have increased dramatically, and appropriate analysis has become a timeconsuming effort. Text mining is an effective tool that may help to solve this problem to classify crimes in effective manner. The proposed system aims to detect and classify crimes in Twitter posts that written in the Arabic language, one of the most widespread languages today. In this paper, classification techniques are used to detect crimes and identify their nature by different classification algorithms. The experiments evaluate different algorithms, such as SVM, DT, CNB, and KNN, in terms of accuracy and speed in the crime domain. Also, different features extraction techniques are evaluated, including rootbased stemming, light stemming, n-gram. The experiments revealed the superiority of n-gram over other techniques. Specifically, the results indicate the superiority of SVM with trigram over other classifiers, with a 91.55% accuracy. Keywords—Crimes; text mining; classification; features extraction techniques; arabic posts; twitter

As the number of customer reviews grows very rapidly, it is essential to summarize useful opinions for buyers, sellers and producers. One key step of opinion mining is feature extraction. Most existing research focus on finding explicit features, only a few attempts have been made to extract implicit features. Nearly all existing research only concentrate on product features, few has paid attention to other features that relates to sellers, services and logistics. Therefore in this paper, we propose a novel co-occurrence association-based method, which aims to extract implicit features in customer reviews and provide more comprehensive and fine-grained mining results.

This paper describes our work which is based on discovering context for text document categorization. The document categorization approach is derived from a combination of a learning paradigm known as relation extraction and an technique known as context discovery. We demonstrate the effectiveness of our categorization approach using reuters 21578 dataset and synthetic real world data from sports domain. Our experimental results indicate that the learned context greatly improves the categorization performance as compared to traditional categorization approaches.

Argumentation mining aims to automatically identify structured argument data from unstructured natural language text. This challenging, multifaceted task is recently gaining a growing attention, especially due to its many potential applications. One particularly important aspect of argumentation mining is claim identification. Most of the current approaches are engineered to address specific domains. However, argumentative sentences are often characterized by common rhetorical structures, independently of the domain. We thus propose a method that exploits structured parsing information to detect claims without resorting to contextual information, and yet achieve a performance comparable to that of state-of-the-art methods that heavily rely on the context.

This research proposes an expanded semantic graph definition that serves as a basis for an expanded semantic graph representation and graph matching approach. This representation separates the content and context and adds a number of semantic structures that encapsulate inferred information. The expanded semantic graph approach facilitates finding additional matches, identifying and eliminating poor matches, and prioritizing matches based on how much new knowledge is provided. By focusing on information of interest, doing pre-processing, and reducing processing requirements, the approach is applicable to problems where related information of interest is sought across a massive body of free text documents. Key aspects of the approach include (1) expanding the nodes and edges through inference using DL-Safe rules, abductive hypotheses, and syntactic patterns, (2) separating semantic content into nodes and semantic context into edges, and (3) applying relatedness measures on a node, edge, and sub graph basis. Results from tests using a ground-truthed subset of a large dataset of law enforcement investigator emails illustrate the benefits of these approaches.