Question: Integrating concept hierarchies with statistical topic modeling techniques

Integrating Hierarchical Concepts and Probabilistic Topic Models for Enhanced Word Sense Disambiguation

This paper introduces an innovative probabilistic topic model, Latent Dirichlet Allocation with WordNet (LDAWN), which incorporates word sense as a latent variable. Our model is designed to simultaneously disambiguate a corpus and learn the thematic contexts in which each word operates, leveraging the WordNet hierarchy for this purpose. We propose a probabilistic posterior inference algorithm that facilitates the automatic identification of domains relevant to word senses, demonstrating that these learned domains significantly enhance the accuracy of word sense disambiguation (WSD) compared to conventional methods. Through a comparison with alternative contexts, we illustrate the superiority of our approach in improving WSD accuracy, showcasing the potential of integrating hierarchical concepts with statistical topic models for language understanding tasks.

We describe a method for the automatic acquisition of the hyponymy lexical relation from unrestricted text. Two goals motivate the approach: (i) avoidance of the need for pre-encoded knowledge and (ii) applicability across a wide range of text. We identify a set of lexicosyntactic patterns that are easily recognizable, that occur frequently and across text genre boundaries, and that indisputably indicate the lexical relation of interest. We describe a method for discovering these patterns and suggest that other lexical relations will also be acquirable in this way. A subset of the acquisition algorithm is implemented and the results are used to augment and critique the structure of a large hand-built thesaurus. Extensions and applications to areas such as information retrieval are suggested.

This paper explores the use of hierarchical structure for classifying a large, heterogeneous collection of web content. The hierarchical structure is initially used to train different second-level classifiers. In the hierarchical case, a model is learned to distinguish a second-level category from other categories within the same top level. In the flat non-hierarchical case, a model distinguishes a second-level category from all other second-level categories. Scoring rules can further take advantage of the hierarchy by considering only second-level categories that exceed a threshold at the top level.
We use support vector machine (SVM) classifiers, which have been shown to be efficient and effective for classification, but not previously explored in the context of hierarchical classification. We found small advantages in accuracy for hierarchical models over flat models. For the hierarchical approach, we found the same accuracy using a sequential Boolean decision rule and a multiplicative decision rule. Since the sequential approach is much more efficient, requiring only 14%-16% of the comparisons used in the other approaches, we find it to be a good choice for classifying text into large hierarchical structures.

In this paper, we present overview of our approach for clustering tweets. Due to short text of tweets, traditional text clustering mechanisms alone may not produce optimal results. We believe that there is an underlying theme/topic present in majority of tweets which is evident in growing usage of hashtag feature in the Twitter network. Clustering tweets based on these themes seems a more natural way for grouping. We propose to use Wikipedia topic taxonomy to discover the themes from the tweets and use the themes along with traditional word based similarity metric for clustering. We show some of our initial results to demonstrate the effectiveness of our approach.

We propose a generative model for text and other collections of discrete data that generalizes or improves on several previous models including naive Bayes/unigram, mixture of unigrams [6], and Hofmann's aspect model , also known as probabilistic latent semantic indexing (pLSI) [3]. In the context of text modeling, our model posits that each document is generated as a mixture of topics, where the continuous-valued mixture proportions are distributed as a latent Dirichlet random variable. Inference and learning are carried out efficiently via variational algorithms. We present empirical results on applications of this model to problems in text modeling, collaborative filtering, and text classification.

Given a topic of interest, a contrastive theme is a group of opposing pairs of viewpoints. We address the task of summarizing contrastive themes: given a set of opinionated documents, select meaningful sentences to represent contrastive themes present in those documents. Several factors make this a challenging problem: unknown numbers of topics, unknown relationships among topics, and the extraction of comparative sentences. Our approach has three core ingredients: contrastive theme modeling, diverse theme extraction, and contrastive theme summarization. Specifically, we present a hierarchical non-parametric model to describe hierarchical relations among topics; this model is used to infer threads of topics as themes from the nested Chinese restaurant process. We enhance the diversity of themes by using structured determinantal point processes for selecting a set of diverse themes with high quality. Finally, we pair contrastive themes and employ an iterative optimization algorithm to select sentences, explicitly considering contrast, relevance, and diversity. Experiments on three datasets demonstrate the effectiveness of our method.

The success of Semantic Web will heavily rely on the availability of formal ontologies to structure machine understanding data. However, there is still a lack of general methodologies for ontology automatic learning and population, i.e. the generation of domain ontologies from various kinds of resources by applying natural language processing and machine learning techniques In this paper, the authors present an ontology learning and population system that combines both statistical and semantic methodologies. Several experiments have been carried out, demonstrating the effectiveness of the proposed system. Keywords-Ontologies, Ontology Learning, Ontology Population, Latent Dirichlet Allocation

We propose a framework of word-vector adaptation, which makes vectors of visually similar concepts close to each other. Here, word vectors are real-valued vector representation of words, e.g., word2vec representation. Our basic idea is to assume that each concept has some hypernyms that are important to determine its visual features. For example, for a concept Swallow with hypernyms Bird, Animal and Entity, we believe Bird is the most important since birds have common visual features with their feathers etc. Adapted word vectors are obtained for each word by taking a weighted sum of a given original word vector and its hypernym word vectors. Our weight optimization makes vectors of visually similar concepts close to each other, by giving a large weight for such important hypernyms. We apply the adapted word vectors to zero-shot learning on the TRECVID 2014 semantic indexing dataset. We achieved 0.083 of Mean Average Precision, which is the best performance without using TRECVID training data to the best of our knowledge.

In this paper, we present an algorithm for extracting keywords representing the asserted main point in a document, without relying on external devices such as natural language processing tools or a document corpus. Our algorithm KeyGraph is based on the segmentation of a graph, representing the co-occurrence between terms in a document, into clusters. Each cluster corresponds to a concept on which author's idea is based, and top ranked terms by a statistic based on each term's relationship to these clusters are selected as keywords. This strategy comes from considering that a document is constructed like a building for expressing new ideas based on traditional concepts. The experimental results show that thus extracted terms match author's point quite accurately, even though KeyGraph does not use each term's average frequency in a corpus, i.e., KeyGraph is a contentsensitive, domain independent device of indexing.

We propose a hierarchical attention network for document classification. Our model has two distinctive characteristics: (i) it has a hierarchical structure that mirrors the hierarchical structure of documents; (ii) it has two levels of attention mechanisms applied at the wordand sentence-level, enabling it to attend differentially to more and less important content when constructing the document representation. Experiments conducted on six large scale text classification tasks demonstrate that the proposed architecture outperform previous methods by a substantial margin. Visualization of the attention layers illustrates that the model selects qualitatively informative words and sentences.

We argue that the quality of a summary can be evaluated based on how many concepts in the original document(s) that can be preserved after summarization. Here, a concept refers to an abstract or concrete entity or its action often expressed by diverse terms in text. Summary generation can thus be considered as an optimization problem of selecting a set of sentences with minimal answer loss. In this paper, we propose a document concept lattice that indexes the hierarchy of local topics tied to a set of frequent concepts and the corresponding sentences containing these topics. The local topics will specify the promising sub-spaces related to the selected concepts and sentences. Based on this lattice, the summary is an optimized selection of a set of distinct and salient local topics that lead to maximal coverage of concepts with the given number of sentences. Our summarizer based on the concept lattice has demonstrated competitive performance in Document Understanding Conference 2005 and 2006 evaluations as well as follow-on tests. 2007 Elsevier Ltd. All rights reserved.


Question: Integrating concept hierarchies with statistical topic modeling techniques

In this work, we introduce an innovative unsupervised learning method for uncovering insights from extensive text databases. Our approach is based on a probabilistic author-topic model, where documents are conceptualized as the outcome of a dual stochastic process. Each author is characterized by a distribution over topics, and each topic is defined by a distribution over words pertinent to that subject. The text of multi-author papers is considered as a blend of the individual topic mixtures of each author. We employ a Markov chain Monte Carlo algorithm to learn the topic-word and author-topic distributions from the data in an unsupervised manner. As a case study, we apply this methodology to a substantial corpus of 160,000 abstracts and 85,000 authors from the renowned CiteSeer digital library, training a model with 300 topics. We delve into the interpretation of the findings, including detailed analysis of specific topic and author models, ranking of authors by topic and topics by author, prominent trends in the computer science literature spanning 1990 to 2002, dissection of abstracts by topics and authors, and identification of atypical papers by certain authors. Additionally, we explore an interactive query interface that enables users to explore author-topic models for collections like CiteSeer, facilitating a deeper understanding of the underlying themes and connections within large text corpora.

We present the components of the ColumbiaGWU contribution to the 2016 TAC KBP BeSt Evaluation.

Many applications that process social data, such as tweets, must extract entities from tweets (e.g., “Obama” and “Hawaii” in “Obama went to Hawaii”), link them to entities in a knowledge base (e.g., Wikipedia), classify tweets into a set of predefined topics, and assign descriptive tags to tweets. Few solutions exist today to solve these problems for social data, and they are limited in important ways. Further, even though several industrial systems such as OpenCalais have been deployed to solve these problems for text data, little if any has been published about them, and it is unclear if any of the systems has been tailored for social media. In this paper we describe in depth an end-to-end industrial system that solves these problems for social data. The system has been developed and used heavily in the past three years, first at Kosmix, a startup, and later at WalmartLabs. We show how our system uses a Wikipedia-based global “real-time” knowledge base that is well suited for social data, how we interleave the tasks in a synergistic fashion, how we generate and use contexts and social signals to improve task accuracy, and how we scale the system to the entire Twitter firehose. We describe experiments that show that our system outperforms current approaches. Finally we describe applications of the system at Kosmix and WalmartLabs, and lessons learned.

This paper presents a new approach for measuring semantic similarity/distance between words and concepts. It combines a lexical taxonomy structure with corpus statistical information so that the semantic distance between nodes in the semantic space constructed by the taxonomy can be better quantified with the computational evidence derived from a distributional analysis of corpus data. Specifically, the proposed measure is a combined approach that inherits the edge-based approach of the edge counting scheme, which is then enhanced by the node-based approach of the information content calculation. When tested on a common data set of word pair similarity ratings, the proposed approach outperforms other computational models. It gives the highest correlation value (r = 0.828) with a benchmark based on human similarity judgements, whereas an upper bound (r = 0.885) is observed when human subjects replicate the same task.

In this paper we introduce the TTCS system, so named after Terms To Conceptual Spaces, that exploits a resource-driven approach relying on BabelNet, NASARI and ConceptNet. TTCS takes in input a term and its context of usage and produces as output a specific type of vector-based semantic representation, where conceptual information is encoded through the Conceptual Spaces (a geometric framework for common-sense knowledge representation and reasoning). The system has been evaluated in a twofold experimentation. In the first case we assessed the quality of the extracted common-sense conceptual information with respect to human judgments with an online questionnaire. In the second one we compared the performances of a conceptual categorization system that was run twice, once fed with extracted annotations and once with hand-crafted annotations. In both cases the results are encouraging and provide precious insights to make substantial improvements.

Hierarchical Pitman-Yor Process priors are compelling methods for learning language models, outperforming point-estimate based methods. However, these models remain unpopular due to computational and statistical inference issues, such as memory and time usage, as well as poor mixing of sampler. In this work we propose a novel framework which represents the HPYP model compactly using compressed suffix trees. Then, we develop an efficient approximate inference scheme in this framework that has a much lower memory footprint compared to full HPYP and is fast in the inference time. The experimental results illustrate that our model can be built on significantly larger datasets compared to previous HPYP models, while being several orders of magnitudes smaller, fast for training and inference, and outperforming the perplexity of the state-of-the-art Modified Kneser-Ney countbased LM smoothing by up to 15%.

We present a kernel-based algorithm for hierarchical text c lassification where the documents are allowed to belong to more than one category at a time. The clas sification model is a variant of the Maximum Margin Markov Network framework, where the classifi cation hierarchy is represented as a Markov tree equipped with an exponential family defined o n the edges. We present an efficient optimization algorithm based on incremental conditi onal gradient ascent in single-example subspaces spanned by the marginal dual variables. The optim ization is facilitated with a dynamic programming based algorithm that computes best update dire ctions in the feasible set. Experiments show that the algorithm can feasibly optimize t raining sets of thousands of examples and classification hierarchies consisting of hundreds of nodes. Training of the full hierarchical model is as efficient as training independent SVM-light clas sifiers for each node. The algorithm’s predictive accuracy was found to be competitive with other r ecently introduced hierarchical multicategory or multilabel classification learning algorithms .

Databases of text and text-annotated data constitute a significant fraction of the information available in electronic form. Searching and browsing are the typical ways that users locate items of interest in such databases. Interfaces that use multifaceted hierarchies represent a new powerful browsing paradigm which has been proven to be a successful complement to keyword searching. Thus far, multifaceted hierarchies have been created manually or semi-automatically, making it difficult to deploy multifaceted interfaces over a large number of databases. We present automatic and scalable methods for creation of multifaceted interfaces. Our methods are integrated with traditional relational databases and can scale well for large databases. Furthermore, we present methods for selecting the best portions of the generated hierarchies when the screen space is not sufficient for displaying all the hierarchy at once. We apply our technique to a range of large data sets, including annotated images, television programming schedules, and web pages. The results are promising and suggest directions for future research.

We describe a method for the automatic acquisition of the hyponymy lexical relation from unrestricted text. Two goals motivate the approach: (i) avoidance of the need for pre-encoded knowledge and (ii) applicability across a wide range of text. We identify a set of lexicosyntactic patterns that are easily recognizable, that occur frequently and across text genre boundaries, and that indisputably indicate the lexical relation of interest. We describe a method for discovering these patterns and suggest that other lexical relations will also be acquirable in this way. A subset of the acquisition algorithm is implemented and the results are used to augment and critique the structure of a large hand-built thesaurus. Extensions and applications to areas such as information retrieval are suggested.

In this paper, we describe latent subtopics discovered from Yelp restaurant reviews by running an online Latent Dirichlet Allocation (LDA) algorithm. The goal is to point out demand of customers from a large amount of reviews, with high dimensionality. These topics can provide meaningful insights to restaurants about what customers care about in order to increase their Yelp ratings, which directly affects their revenue. We used the open dataset from the Yelp Dataset Challenge with over 158,000 restaurant reviews. To find latent subtopics from reviews, we adopted Online LDA, a generative probabilistic model for collections of discrete data such as text corpora. We present the breakdown of hidden topics over all reviews, predict stars per hidden topics discovered, and extend our findings to that of temporal information regarding restaurants peak hours. Overall, we have found several interesting insights and a method which could definitely prove useful to restaurant owners.

In this paper, we present an algorithm for extracting keywords representing the asserted main point in a document, without relying on external devices such as natural language processing tools or a document corpus. Our algorithm KeyGraph is based on the segmentation of a graph, representing the co-occurrence between terms in a document, into clusters. Each cluster corresponds to a concept on which author's idea is based, and top ranked terms by a statistic based on each term's relationship to these clusters are selected as keywords. This strategy comes from considering that a document is constructed like a building for expressing new ideas based on traditional concepts. The experimental results show that thus extracted terms match author's point quite accurately, even though KeyGraph does not use each term's average frequency in a corpus, i.e., KeyGraph is a contentsensitive, domain independent device of indexing.


Question: Integrating concept hierarchies with statistical topic modeling techniques

This paper introduces a generative model for discrete data collections, such as text, that builds upon and enhances previous models like naive Bayes/unigram, mixture of unigrams, and Hofmann's aspect model, also known as probabilistic latent semantic indexing (pLSI). Our proposed model, referred to as Latent Dirichlet Allocation (LDA), assumes that each document is generated as a mixture of topics, with the mixture proportions being continuous values and distributed according to a latent Dirichlet random variable. This model offers a more nuanced approach to text modeling by incorporating topic distribution within documents. We employ variational algorithms for efficient inference and learning. The paper also includes empirical results demonstrating the model's application and effectiveness in various domains such as text modeling, collaborative filtering, and text classification. By integrating topic modeling with the latent Dirichlet distribution, LDA provides a robust framework for understanding the thematic structure of document collections.

This paper describes a novel Bayesian approach to unsupervised topic segmentation. Unsupervised systems for this task are driven by lexical cohesion: the tendency of wellformed segments to induce a compact and consistent lexical distribution. We show that lexical cohesion can be placed in a Bayesian context by modeling the words in each topic segment as draws from a multinomial language model associated with the segment; maximizing the observation likelihood in such a model yields a lexically-cohesive segmentation. This contrasts with previous approaches, which relied on hand-crafted cohesion metrics. The Bayesian framework provides a principled way to incorporate additional features such as cue phrases, a powerful indicator of discourse structure that has not been previously used in unsupervised segmentation systems. Our model yields consistent improvements over an array of state-of-the-art systems on both text and speech datasets. We also show that both an entropy-based analysis and a well-known previous technique can be derived as special cases of the Bayesian framework.1

Traditional graph based sentence ranking approaches modeled the documents as a text graph where vertices represent sentences and edges represent pairwise similarity relationships between two sentences. Such modeling cannot capture complex group relationships shared among multiple sentences which can be useful for sentence ranking. In this paper, we propose two different group relationships (sentence-topic relationship and document-topic relationship) shared among sentences, and construct a double-hypergraph integrating these relationships into a unified framework. Then, a double-hypergraph based sentence ranking algorithm is developed for query-focused multi-document summarization, in which Markov random walk is defined on each hypergraph and the mixture Markov chains are formed so as to perform transductive learning in the double-hypergraph. When evaluated on DUC datasets, performance of the proposed approach is remarkable.

Latent variable models can be enriched with a multi-dimensional structure to consider the many latent factors in a text corpus, such as topic, author perspective and sentiment. We introduce factorial LDA, a multi-dimensional model in which a document is influenced by K different factors, and each word token depends on a K-dimensional vector of latent variables. Our model incorporates structured word priors and learns a sparse product of factors. Experiments on research abstracts show that our model can learn latent factors such as research topic, scientific discipline, and focus (methods vs. applications). Our modeling improvements reduce test perplexity and improve human interpretability of the discovered factors.

We propose a new hierarchical Bayesian n-gram model of natural languages. Our model makes use of a generalization of the commonly used Dirichlet distributions called Pitman-Yor processes which produce power-law distributions more closely resembling those in natural languages. We show that an approximation to the hierarchical Pitman-Yor language model recovers the exact formulation of interpolated Kneser-Ney, one of the best smoothing methods forn-gram language models. Experiments verify that our model gives cross entropy results superior to interpolated Kneser-Ney and comparable to modified Kneser-Ney.

The success of Semantic Web will heavily rely on the availability of formal ontologies to structure machine understanding data. However, there is still a lack of general methodologies for ontology automatic learning and population, i.e. the generation of domain ontologies from various kinds of resources by applying natural language processing and machine learning techniques In this paper, the authors present an ontology learning and population system that combines both statistical and semantic methodologies. Several experiments have been carried out, demonstrating the effectiveness of the proposed system. Keywords-Ontologies, Ontology Learning, Ontology Population, Latent Dirichlet Allocation

Most of the existing multi-document summarization methods decompose the documents into sentences and work directly in the sentence space using a term-sentence matrix. However, the knowledge on the document side, i.e. the topics embedded in the documents, can help the context understanding and guide the sentence selection in the summarization procedure. In this paper, we propose a new Bayesian sentence-based topic model for summarization by making use of both the term-document and term-sentence associations. An efficient variational Bayesian algorithm is derived for model parameter estimation. Experimental results on benchmark data sets show the effectiveness of the proposed model for the multi-document summarization task.

Information Content (IC) is an important dimension of assessing the semantic similarity between two terms or word senses in word knowledge. The conventional method of obtaining IC of word senses is to combine knowledge of their hierarchical structure from an ontology like WordNet with actual usage in text as derived from a large corpus. In this paper, a new model of IC is presented, which relies on hierarchical structure alone. The model considers not only the hyponyms of each word sense but also its depth in the structure. The IC value is easier to calculate based on our model, and when used as the basis of a similarity approach it yields judgments that correlate more closely with human assessments than others, which using IC value obtained only considering the hyponyms and IC value got by employing corpus analysis.

Databases of text and text-annotated data constitute a significant fraction of the information available in electronic form. Searching and browsing are the typical ways that users locate items of interest in such databases. Interfaces that use multifaceted hierarchies represent a new powerful browsing paradigm which has been proven to be a successful complement to keyword searching. Thus far, multifaceted hierarchies have been created manually or semi-automatically, making it difficult to deploy multifaceted interfaces over a large number of databases. We present automatic and scalable methods for creation of multifaceted interfaces. Our methods are integrated with traditional relational databases and can scale well for large databases. Furthermore, we present methods for selecting the best portions of the generated hierarchies when the screen space is not sufficient for displaying all the hierarchy at once. We apply our technique to a range of large data sets, including annotated images, television programming schedules, and web pages. The results are promising and suggest directions for future research.

As the number of Internet users and the number of accessible Web pages grows, it is becoming increasingly difficult for users to find documents that are relevant to their particular needs. Users must either browse through a large hierarchy of concepts to find the information for which they are looking or submit a query to a publicly available search engine and wade through hundreds of results, most of them irrelevant. The core of the problem is that whether the user is browsing or searching, whether they are an eighth grade student or a Nobel prize winner, the identical information is selected and it is presented the same way. In this paper, we report on research that adapts information navigation based on a user profile structured as a weighted concept hierarchy. A user may create his or her own concept hierarchy and use them for browsing Web sites. Or, the user profile may be created from a reference ontology by ‘watching over the user’s shoulder’ while they browse. We show that these automatically created profiles reflect the user’s interests quite well and they are able to produce moderate improvements when applied to search results.

Today, the practice of returning entities from a knowledge base in response to search queries has become widespread. One of the distinctive characteristics of entities is that they are typed, i.e., assigned to some hierarchically organized type system (type taxonomy). The primary objective of this paper is to gain a better understanding of how entity type information can be utilized in entity retrieval. We perform this investigation in an idealized "oracle" setting, assuming that we know the distribution of target types of the relevant entities for a given query. We perform a thorough analysis of three main aspects: (i) the choice of type taxonomy, (ii) the representation of hierarchical type information, and (iii) the combination of type-based and term-based similarity in the retrieval model. Using a standard entity search test collection based on DBpedia, we find that type information proves most useful when using large type taxonomies that provide very specific types. We provide further insights on the extensional coverage of entities and on the utility of target types.


Question: Integrating concept hierarchies with statistical topic modeling techniques

This paper explores the methodology for uncovering scientific topics within a corpus of documents, a critical step in understanding document content. We focus on a generative model, initially proposed by Blei, Ng, and Jordan [Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003) J. Machine Learn. Res. 3, 993-1022], which posits that each document is produced through the selection of a distribution over topics, followed by the choice of each word from a topic determined by this distribution. To apply this model, we introduce a Markov chain Monte Carlo algorithm for performing inference. Using this algorithm, we analyze abstracts from the Proceedings of the National Academy of Sciences (PNAS), determining the number of topics through Bayesian model selection. Our analysis reveals that the identified topics accurately reflect the underlying structure of the data, aligning closely with the classifications provided by the articles' authors. We also discuss potential extensions of this work, such as identifying trending topics through temporal analysis and tagging abstracts to highlight semantic content, thereby enhancing the model's utility and applicability.

This paper presents a topic model that identifies interpretable low dimensional components in time-stamped data for capturing the evolution of trends. Unlike other models for time-stamped data, our proposal, the trend analysis model (TAM), focuses on the difference between temporal words and other words in each document to detect topic evolution over time. TAM introduces a latent trend class variable into each document and a latent switch variable into each token for handling these differences. The trend class has a probability distribution over temporal words, topics, and a continuous distribution over time, where each topic is responsible for generating words. The latter class uses a document specific probabilistic distribution to judge which variable each word comes from for generating words in each token. Accordingly, TAM can explain which topic co-occurrence pattern will appear at any given time, and represents documents of similar content and timestamp as sharing the same trend class. Therefore, TAM projects them on a latent space of trend dimensionality and allows us to predict the temporal evolution of words and topics in document collections. Experiments on various data sets show that the proposed model can capture interpretable low dimensionality sets of topics and timestamps, take advantage of previous models, and is useful as a generative model in the analysis of the evolution of trends.

We propose a Bayesian nonparametric topic model that rep- resents relationships between given labels and the corre- sponding words/phrases, from supervised articles. Unlike existing supervised topic models, our proposal, supervised N-gram topic model (SNT), focuses on both a number of topics and power-law distribution in the word frequencies to extract topic specific N-grams. To achieve this goal, SNT takes a Bayesian nonparametric approach to topic sampling, which generates word distribution jointly with the given variable in textual order, and then form each N-gram word as a hierarchy of Pitman-Yor process priors. Experiments on labeled text data show that SNT is useful as a generative model for discovering more phrases that complement human experts and domain specific knowledge than the existing al- ternatives. The results show that SNT can be applied to various tasks such as automatic annotation.

The paper describes the methodology used to develop a construction domain ontology, taking into account the wealth of existing semantic resources in the sector ranging from dictionaries to thesauri. Given the characteristics and settings of the construction industry, a modular, architecture-centric approach was adopted to structure and develop the ontology. The paper argues that taxonomies provide an ideal backbone for any ontology project. Therefore, a construction industry standard taxonomy was used to provide the seeds of the ontology, enriched and expanded with additional concepts extracted from large discipline-oriented document bases using information retrieval (IR) techniques.

A folksonomy refers to a collection of user-defined tags with which users describe contents published on the Web. With the flourish of Web 2.0, folksonomies have become an important mean to develop the Semantic Web. Because tags in folksonomies are authored freely, there is a need to understand the structure and semantics of these tags in various applications. In this paper, we propose a learning approach to create an ontology that captures the hierarchical semantic structure of folksonomies. Our experimental results on two different genres of real world data sets show that our method can effectively learn the ontology structure from the folksonomies.

An adaptive statistical language model is described, which successfully integrates long distance linguistic information with other knowledge sources. Most existing statistical language models exploit only the immediate history of a text. To extract information from further back in the document’s history, we propose and use trigger pairs as the basic information bearing elements. This allows the model to adapt its expectations to the topic of discourse. Next, statistical evidence from multiple sources must be combined. Traditionally, linear interpolation and its variants have been used, but these are shown here to be seriously deficient. Instead, we apply the principle of Maximum Entropy (ME). Each information source gives rise to a set of constraints, to be imposed on the combined estimate. The intersection of these constraints is the set of probability functions which are consistent with all the information sources. The function with the highest entropy within that set is the ME solution. Given consistent statistical evidence, a unique ME solution is guaranteed to exist, and an iterative algorithm exists which is guaranteed to converge to it. The ME framework is extremely general: any phenomenon that can be described in terms of statistics of the text can be readily incorporated. An adaptive language model based on the ME approach was trained on the Wall Street Journal corpus, and showed a 32–39% perplexity reduction over the baseline. When interfaced to SPHINX-II, Carnegie Mellon’s speech recognizer, it reduced its error rate by 10–14%. This thus illustrates the feasibility of incorporating many diverse knowledge sources in a single, unified statistical framework.  1996 Academic Press Limited

In this paper, we propose TopicRNN, a recurrent neural network (RNN)-based language model designed to directly capture the global semantic meaning relating words in a document via latent topics. Because of their sequential nature, RNNs are good at capturing the local structure of a word sequence – both semantic and syntactic – but might face difficulty remembering long-range dependencies. Intuitively, these long-range dependencies are of semantic nature. In contrast, latent topic models are able to capture the global semantic structure of a document but do not account for word ordering. The proposed TopicRNN model integrates the merits of RNNs and latent topic models: it captures local (syntactic) dependencies using an RNN and global (semantic) dependencies using latent topics. Unlike previous work on contextual RNN language modeling, our model is learned endto-end. Empirical results on word prediction show that TopicRNN outperforms existing contextual RNN baselines. In addition, TopicRNN can be used as an unsupervised feature extractor for documents. We do this for sentiment analysis on the IMDB movie review dataset and report an error rate of 6.28%. This is comparable to the state-of-the-art 5.91% resulting from a semi-supervised approach. Finally, TopicRNN also yields sensible topics, making it a useful alternative to document models such as latent Dirichlet allocation.

We introduce the Word Tree, a new visualization and information-retrieval technique aimed at text documents. A Word Tree is a graphical version of the traditional "keyword-in-context" method, and enables rapid querying and exploration of bodies of text. In this paper we describe the design of the technique, along with some of the technical issues that arise in its implementation. In addition, we discuss the results of several months of public deployment of word trees on Many Eyes, which provides a window onto the ways in which users obtain value from the visualization.

Today, the practice of returning entities from a knowledge base in response to search queries has become widespread. One of the distinctive characteristics of entities is that they are typed, i.e., assigned to some hierarchically organized type system (type taxonomy). The primary objective of this paper is to gain a better understanding of how entity type information can be utilized in entity retrieval. We perform this investigation in an idealized "oracle" setting, assuming that we know the distribution of target types of the relevant entities for a given query. We perform a thorough analysis of three main aspects: (i) the choice of type taxonomy, (ii) the representation of hierarchical type information, and (iii) the combination of type-based and term-based similarity in the retrieval model. Using a standard entity search test collection based on DBpedia, we find that type information proves most useful when using large type taxonomies that provide very specific types. We provide further insights on the extensional coverage of entities and on the utility of target types.

A family of probabilistic time series models is developed to analyze the time evolution of topics in large document collections. The approach is to use state space models on the natural parameters of the multinomial distributions that represent the topics. Variational approximations based on Kalman filters and nonparametric wavelet regression are developed to carry out approximate posterior inference over the latent topics. In addition to giving quantitative, predictive models of a sequential corpus, dynamic topic models provide a qualitative window into the contents of a large document collection. The models are demonstrated by analyzing the OCR'ed archives of the journal Science from 1880 through 2000.

Researchers and scientists increasingly find themselves in the position of having to quickly understand large amounts of technical material. Our goal is to effectively serve this need by using bibliometric text mining and summarization techniques to generate summaries of scientific literature. We show how we can use citations to produce automatically generated, readily consumable, technical extractive summaries. We first propose C-LexRank, a model for summarizing single scientific articles based on citations, which employs community detection and extracts salient information-rich sentences. Next, we further extend our experiments to summarize a set of papers, which cover the same scientific topic. We generate extractive summaries of a set of Question Answering (QA) and Dependency Parsing (DP) papers, their abstracts, and their citation sentences and show that citations have unique information amenable to creating a summary.


Question: Integrating concept hierarchies with statistical topic modeling techniques

Integrating Hierarchical Topics with Enhanced Pachinko Allocation
The Pachinko Allocation Model (PAM), a four-level structure introduced by Li and McCallum in 2006, captures topic correlations through a Directed Acyclic Graph (DAG). However, it falls short in depicting a nested hierarchy of topics, where some overarching word distributions encompass the vocabulary shared by more specialized topics. This paper introduces an advanced version of PAM, termed Hierarchical PAM, which explicitly models a hierarchical topic structure. This enhancement effectively merges the strengths of Hierarchical Latent Dirichlet Allocation (hLDA) for its hierarchical topic representation with PAM's capability to blend multiple leaves from the topic hierarchy. The outcomes of the experiments reveal enhancements in the likelihood of documents not included in the training set, as well as increased mutual information between topics automatically identified by the model and categories manually created by humans, such as journal classifications.

ANIMATE, HUMAN, etc. and encode type restrictions on nouns and adjectives and on the arguments of verbs. Subject codes use another set of primitives to classify senses of words by subject (ECONOMICS, ENGINEERING, etc.). Guthrie et al. (1991) demonstrate a typical use of this information: in addition to using the Lesk-based method of counting overlaps between definitions and contexts, they impose a correspondence of subject codes in an iterative process. No quantitative evaluation of this method is available, but Cowie et al. (1992) improve the method using simulated annealing and report results of 47% for sense distinctions and 72% for homographs. The use of LDOCE box codes, however, is problematic: the codes are not systematic (see, for example, Fontenelle, 1990); in later work, Braden-Harder (1993) showed that simply matching box or subject codes is not sufficient for disambiguation. For example, in I tipped the driver, the codes for several senses of the words in the sentence satisfy the necessary constraints (e.g. tip-money + human object or tip-tilt + movable solid object). In many ways, the supple7 Note that the assumptions underlying this method are very similar to Quillian’s: Thus one may think of a full concept analogically as consisting of all the information one would have if he looked up what will be called the “patriarch” word in a dictionary, then looked up every word in each of its definitions, then looked up every word found in each of these, and so on, continually branching outward[...] (Quillian, 1968, p. 238). However, Quillian’s network also keeps track of semantic relationships among the words encountered along the path between two words, which are encoded in his semantic network; the neural network avoids the overhead of creating the semantic network but loses this relational information. 13 mentary information in the LDOCE, and in particular the subject codes, are similar to those in a thesaurus, which, however, are more systematically structured. Inconsistencies in dictionaries, noted earlier, are not the only and perhaps not the major source of their limitations for WSD. While dictionaries provide detailed information at the lexical level, they lack pragmatic information that enters into sense determination (see, e.g., Hobbs, 1987). For example, the link between ash and tobacco, cigarette or tray in a network such as Quillian’s is very indirect, whereas in the Brown Corpus, the word ash co-occurs frequently with one of these words. It is therefore not surprising that corpora have become a primary source of information for WSD; this development is outlined below in section 2.3. 1.3.2 Thesauri. Thesauri provide information about relationships among words, most notably synonymy. Roget's International Thesaurus, which was put into machine-tractable form in the 1950's8 and has been used in a variety of applications including machine translation (Masterman, 1957), information retrieval (Sparck Jones, 1964, 1986), and content analysis (Sedelow and Sedelow, 1969; see also Sedelow and Sedelow, 1986, 1992), also supplies an explicit concept hierarchy consisting of up to eight increasingly refined levels. Typically, each occurrence of the same word under different categories of the thesaurus represent different senses of that word; i.e., the categories correspond roughly to word senses (Yarowsky, 1992). A set of words in the same category are semantically related. The earliest known use of Roget’s for WSD is the work of Masterman (1957), described above in section 2.1. Several years later, Patrick (1985) used Roget’s to discriminate among verb senses, by examining semantic clusters formed by “e-chains” derived from the thesaurus (Bryan, 1973, 1974; see also Sedelow and Sedelow, 1986). He uses “word-strong neighborhoods,” comprising word groups in low-level semicolon groups, which are the most closely related semantically in the thesaurus, and words connected to the group via chains. He is able to discriminate the correct sense of verbs such as inspire (to raise the spirits vs. to inhale, breathe in, sniff, etc.), question (to doubt vs. to ask a question) with “high reliability.” Bryan's earlier work had already demonstrated that homographs can be distinguished by applying a metric based on relationships defined by his chains (Bryan, 1973, 1974). Similar work is described in Sedelow and Mooney (1988). Yarowsky (1992) derives classes of words by starting with words in common categories in Roget's (4th ed.). A 100-word context of each word in the category is extracted from a corpus (the 1991 electronic text of Grolier's Ency8 The work of Masterman (1957) and Sparck Jones (1964) relied on a version of Roget’s that was hand-punched onto cards in the 1950’s; the Sedelow’s (1969) work relied on a machine readable version of the 3rd Edition. Roget’s is now widely available via anonymous ftp from various sites.

Traditional graph based sentence ranking approaches modeled the documents as a text graph where vertices represent sentences and edges represent pairwise similarity relationships between two sentences. Such modeling cannot capture complex group relationships shared among multiple sentences which can be useful for sentence ranking. In this paper, we propose two different group relationships (sentence-topic relationship and document-topic relationship) shared among sentences, and construct a double-hypergraph integrating these relationships into a unified framework. Then, a double-hypergraph based sentence ranking algorithm is developed for query-focused multi-document summarization, in which Markov random walk is defined on each hypergraph and the mixture Markov chains are formed so as to perform transductive learning in the double-hypergraph. When evaluated on DUC datasets, performance of the proposed approach is remarkable.

Latent variable models can be enriched with a multi-dimensional structure to consider the many latent factors in a text corpus, such as topic, author perspective and sentiment. We introduce factorial LDA, a multi-dimensional model in which a document is influenced by K different factors, and each word token depends on a K-dimensional vector of latent variables. Our model incorporates structured word priors and learns a sparse product of factors. Experiments on research abstracts show that our model can learn latent factors such as research topic, scientific discipline, and focus (methods vs. applications). Our modeling improvements reduce test perplexity and improve human interpretability of the discovered factors.

Query expansion is an important and commonly used technique for improving Web search results. Existing methods for query expansion have mostly relied on global or local analysis of document collection, click-through data, or simple ontologies such as WordNet. In this paper, we present the results of a systematic study of the methods leveraging the ConceptNet knowledge base, an emerging new Web resource, for query expansion. Specifically, we focus on the methods leveraging ConceptNet to improve the search results for poorly performing (or difficult) queries. Unlike other lexico-semantic resources, such as WordNet and Wikipedia, which have been extensively studied in the past, ConceptNet features a graph-based representation model of commonsense knowledge, in which the terms are conceptually related through rich relational ontology. Such representation structure enables complex, multi-step inferences between the concepts, which can be applied to query expansion. We first demonstrate through simulation experiments that expanding queries with the related concepts from ConceptNet has great potential for improving the search results for difficult queries. We then propose and study several supervised and unsupervised methods for selecting the concepts from ConceptNet for automatic query expansion. The experimental results on multiple data sets indicate that the proposed methods can effectively leverage ConceptNet to improve the retrieval performance of difficult queries both when used in isolation as well as in combination with pseudo-relevance feedback.

We introduce a type of Deep Boltzmann Machine (DBM) that is suitable for extracting distributed semantic representations from a large unstructured collection of documents. We overcome the apparent difficulty of training a DBM with judicious parameter tying. This enables an efficient pretraining algorithm and a state initialization scheme for fast inference. The model can be trained just as efficiently as a standard Restricted Boltzmann Machine. Our experiments show that the model assigns better log probability to unseen data than the Replicated Softmax model. Features extracted from our model outperform LDA, Replicated Softmax, and DocNADE models on document retrieval and document classification tasks.

Text classification to a hierarchical taxonomy of topics is a common and practical problem. Traditional approaches simply use bag-of-words and have achieved good results. However, when there are a lot of labels with different topical granularities, bagof-words representation may not be enough. Deep learning models have been proven to be effective to automatically learn different levels of representations for image data. It is interesting to study what is the best way to represent texts. In this paper, we propose a graph-CNN based deep learning model to first convert texts to graph-of-words, and then use graph convolution operations to convolve the word graph. Graph-ofwords representation of texts has the advantage of capturing non-consecutive and long-distance semantics. CNN models have the advantage of learning different level of semantics. To further leverage the hierarchy of labels, we regularize the deep architecture with the dependency among labels. Our results on both RCV1 and NYTimes datasets show that we can significantly improve large-scale hierarchical text classification over traditional hierarchical text classification and existing deep models.

We apply topic modelling to automatically induce word senses of a target word, and demonstrate that our word sense induction method can be used to automatically detect words with emergent novel senses, as well as token occurrences of those senses. We start by exploring the utility of standard topic models for word sense induction (WSI), with a pre-determined number of topics (=senses). We next demonstrate that a non-parametric formulation that learns an appropriate number of senses per word actually performs better at the WSI task. We go on to establish state-of-the-art results over two WSI datasets, and apply the proposed model to a novel sense detection task.

We argue that the quality of a summary can be evaluated based on how many concepts in the original document(s) that can be preserved after summarization. Here, a concept refers to an abstract or concrete entity or its action often expressed by diverse terms in text. Summary generation can thus be considered as an optimization problem of selecting a set of sentences with minimal answer loss. In this paper, we propose a document concept lattice that indexes the hierarchy of local topics tied to a set of frequent concepts and the corresponding sentences containing these topics. The local topics will specify the promising sub-spaces related to the selected concepts and sentences. Based on this lattice, the summary is an optimized selection of a set of distinct and salient local topics that lead to maximal coverage of concepts with the given number of sentences. Our summarizer based on the concept lattice has demonstrated competitive performance in Document Understanding Conference 2005 and 2006 evaluations as well as follow-on tests. 2007 Elsevier Ltd. All rights reserved.

A challenging problem in hierarchical classification is to leverage the hierarchical relations among classes for improving classification performance. An even greater challenge is to do so in a manner that is computationally feasible for large scale problems. This paper proposes a set of Bayesian methods to model hierarchical dependencies among class labels using multivariate logistic regression. Specifically, the parent-child relationships are modeled by placing a hierarchical prior over the children nodes centered around the parameters of their parents; thereby encouraging classes nearby in the hierarchy to share similar model parameters. We present variational algorithms for tractable posterior inference in these models, and provide a parallel implementation that can comfortably handle largescale problems with hundreds of thousands of dimensions and tens of thousands of classes. We run a comparative evaluation on multiple large-scale benchmark datasets that highlights the scalability of our approach and shows improved performance over the other state-of-the-art hierarchical methods.

Databases of text and text-annotated data constitute a significant fraction of the information available in electronic form. Searching and browsing are the typical ways that users locate items of interest in such databases. Faceted interfaces represent a new powerful paradigm that proved to be a successful complement to keyword searching. Thus far, the identification of the facets was either a manual procedure, or relied on apriori knowledge of the facets that can potentially appear in the underlying collection. In this paper, we present an unsupervised technique for automatic extraction of facets useful for browsing text databases. In particular, we observe, through a pilot study, that facet terms rarely appear in text documents, showing that we need external resources to identify useful facet terms. For this, we first identify important phrases in each document. Then, we expand each phrase with ";context"; phrases using external resources, such as WordNet and Wikipedia, causing facet terms to appear in the expanded database. Finally, we compare the term distributions in the original database and the expanded database to identify the terms that can be used to construct browsing facets. Our extensive user studies, using the Amazon Mechanical Turk service, show that our techniques produce facets with high precision and recall that are superior to existing approaches and help users locate interesting items faster.


Question: An intelligent anti-phishing strategy model for detecting phishing websites.

In the realm of combating phishing attacks, a significant cyber threat, the development of robust detection mechanisms is crucial. This paper presents an advanced solution, CANTINA+, designed to address the limitations of existing methods by leveraging a comprehensive set of features and machine learning algorithms. Traditionally, phishing detection has relied on human-curated URL blacklists or feature-based machine learning models. However, these approaches struggle with the detection of novel phishing sites due to their inherent limitations in feature richness and high false positive rates.

CANTINA+ is introduced as a multi-layered solution that combines the strength of feature-rich machine learning with filtering techniques to enhance detection accuracy while minimizing false positives. The framework includes eight innovative features, making it the most feature-dense approach in the literature. It utilizes the HTML Document Object Model (DOM), integrates search engine data, and harnesses third-party services to improve phishing site identification. Additionally, CANTINA+ incorporates two filters to reduce false positives and optimize performance. The first filter is a near-duplicate phishing detector, which employs hashing algorithms to identify highly similar phishing sites. The second filter is a login form classifier, which deems web pages without identifiable login forms as legitimate, streamlining the detection process.

Extensive evaluation of CANTINA+ was conducted using two distinct methods and a diverse dataset comprising 8118 phishing and 4883 legitimate web pages. In the randomized evaluation, CANTINA+ achieved a true positive rate (TP) of over 92% on unique testing phishing sites and over 99% on near-duplicate testing phishing sites, with a false positive rate (FP) of approximately 0.4% using 10% training data. In time-based evaluation, the framework maintained over 92% TP on unique testing phishing sites, over 99% TP on near-duplicate testing phishing sites, and a FP rate of about 1.4% under 20% training data with a two-week sliding window. Demonstrating a competitive edge in the anti-phishing domain, CANTINA+ achieved a FP rate of 0.4% and a TP rate of over 92%, solidifying its position as a potent solution against phishing threats.

Web applications are used on a large scale worldwide, which handles sensitive personal data of users. With web application that maintains data ranging from as simple as telephone number to as important as bank account information, security is a prime point of concern. With hackers aimed to breakthrough this security using various attacks, we are focusing on SQL injection attacks and XSS attacks. SQL injection attack is very common attack that manipulates the data passing through web application to the database servers through web servers in such a way that it alters or reveals database contents. While Cross Site Scripting (XSS) attacks focuses more on view of the web application and tries to trick users that leads to security breach. We are considering three tier web applications with static and dynamic behavior, for security. Static and dynamic mapping model is created to detect anomalies in the class of SQL Injection and XSS attacks.

Online reviews have become a valuable resource for decision making. However, its usefulness brings forth a curse ‒ deceptive opinion spam. In recent years, fake review detection has attracted significant attention. However, most review sites still do not publicly filter fake reviews. Yelp is an exception which has been filtering reviews over the past few years. However, Yelp’s algorithm is trade secret. In this work, we attempt to find out what Yelp might be doing by analyzing its filtered reviews. The results will be useful to other review hosting sites in their filtering effort. There are two main approaches to filtering: supervised and unsupervised learning. In terms of features used, there are also roughly two types: linguistic features and behavioral features. In this work, we will take a supervised approach as we can make use of Yelp’s filtered reviews for training. Existing approaches based on supervised learning are all based on pseudo fake reviews rather than fake reviews filtered by a commercial Web site. Recently, supervised learning using linguistic n-gram features has been shown to perform extremely well (attaining around 90% accuracy) in detecting crowdsourced fake reviews generated using Amazon Mechanical Turk (AMT). We put these existing research methods to the test and evaluate performance on the real-life Yelp data. To our surprise, the behavioral features perform very well, but the linguistic features are not as effective. To investigate, a novel information theoretic analysis is proposed to uncover the precise psycholinguistic difference between AMT reviews and Yelp reviews (crowdsourced vs. commercial fake reviews). We find something quite interesting. This analysis and experimental results allow us to postulate that Yelp’s filtering is reasonable and its filtering algorithm seems to be correlated with abnormal spamming behaviors.

In recent years, with rapid growth of social networking websites, users are very active in these platforms and large amount of data are aggregated. Among those social networking websites, Facebook is the most popular website that has most users. However, in Facebook, the abusing problem is a very critical issue, such as Hate Groups. Therefore, many researchers are devoting on how to detect potential hate groups, such as using the techniques of social networks analysis. However, we believe content is also a very important factors for hate groups detection. Thus, in this paper, we will propose an architecture to for hate groups detection which is based on the technique of Social Networks Analysis and Web Mining (Text Mining; Natural Language Processing). From the experiment result, it shows that content plays an critical role for hate groups detection and the performance is better than the system that just applying social networks analysis.

Network security is becoming an issue of paramount importance in the information technology era. Nowadays with the dramatic growth of communication and computer networks, security has become a critical subject for computer system. Intrusion detection is the art of detecting computer abuse and any attempt to break the networks. Intrusion detection system is an effective security tool that helps to prevent unauthorized access to network resources by analyzing the network traffic. Different algorithms, methods and applications are created and implemented to solve the problem of detecting the attacks in intrusion detection systems. Most methods detect attacks and categorize in two groups, normal or threat. One of the most promising areas of research in the area of Intrusion Detection deals with the applications of the Artificial Intelligence (AI) techniques. This proposed system presents a new approach of intrusion detection system based on artificial neural network. Multi Layer Percepron (MLP) architecture is used for Intrusion Detection System. The performance and evaluations are performed by using the set of benchmark data from a KDD (Knowledge discovery in Database) dataset. The proposed system detects the attacks and classifies them in six groups.

To combat the evolving Android malware attacks, in this paper, instead of only using Application Programming Interface (API) calls, we further analyze the different relationships between them and create higher-level semantics which require more efforts for attackers to evade the detection. We represent the Android applications (apps), related APIs, and their rich relationships as a structured heterogeneous information network (HIN). Then we use a meta-path based approach to characterize the semantic relatedness of apps and APIs. We use each meta-path to formulate a similarity measure over Android apps, and aggregate different similarities using multi-kernel learning to make predictions. Promising experimental results based on real sample collections from Comodo Cloud Security Center demonstrate that our developed system HinDroid outperforms other alternative Android malware detection techniques.

Web spamming refers to actions intended to mislead search engines into ranking some pages higher than they deserve. Recently, the amount of web spam has increased dramatically, leading to a degradation of search results. This paper presents a comprehensive taxonomy of current spamming techniques, which we believe can help in developing appropriate countermeasures.

We present WebSelF, a framework for web scraping which models the process of web scraping and decomposes it into four conceptually independent, reusable, and composable constituents. We have validated our framework through a full parameterized implementation that is flexible enough to capture previous work on web scraping. We conducted an experiment that evaluated several qualitatively different web scraping constituents (including previous work and combinations hereof) on about 11,000 HTML pages on daily versions of 17 web sites over a period of more than one year. Our framework solves three concrete problems with current web scraping and our experimental results indicate that composition of previous and our new techniques achieve a higher degree of accuracy, precision and specificity than existing tech-

Recently, malware has increasingly become a critical threat to embedded systems, while the conventional software solutions, such as antivirus and patches, have not been so successful in defending the ever-evolving and advanced malicious programs. In this paper, we propose a hardware-enhanced architecture, GuardOL, to perform online malware detection. GuardOL is a combined approach using processor and field-programmable gate array (FPGA). Our approach aims to capture the malicious behavior (i.e., high-level semantics) of malware. To this end, we first propose the frequency-centric model for feature construction using system call patterns of known malware and benign samples. We then develop a machine learning approach (using multilayer perceptron) in FPGA to train classifier using these features. At runtime, the trained classifier is used to classify the unknown samples as malware or benign, with early prediction. The experimental results show that our solution can achieve high classification accuracy, fast detection, low power consumption, and flexibility for easy functionality upgrade to adapt to new malware samples. One of the main advantages of our design is the support of early prediction-detecting 46% of malware within first 30% of their execution, while 97% of the samples at 100% of their execution, with <;3% false positives.

Naive users using a browser have no idea about the back-end of the page. The users might be tricked into giving away their credentials or downloading malicious data. Our aim is to create an extension for Chrome which will act as middleware between the users and the malicious websites, and mitigate the risk of users succumbing to such websites. Further, all harmful content cannot be exhaustively collected as even that is bound to continuous development. To counter this we are using machine learning-to train the tool and categorize the new content it sees every time into the particular categories so that corresponding action can be taken.

Nowadays it is very important to maintain a high level security to ensure safe and trusted communication of information between various organizations. But secured data communication over internet and any other network is always under threat of intrusions and misuses. So Intrusion Detection Systems have become a needful component in terms of computer and network security. There are various approaches being utilized in intrusion detections, but unfortunately any of the systems so far is not completely flawless. So, the quest of betterment continues. In this progression, here we present an Intrusion Detection System (IDS), by applying genetic algorithm (GA) to efficiently detect various types of network intrusions. Parameters and evolution processes for GA are discussed in details and implemented. This approach uses evolution theory to information evolution in order to filter the traffic data and thus reduce the complexity. To implement and measure the performance of our system we used the KDD99 benchmark dataset and obtained reasonable detection rate.


Question: An intelligent strategy model for detecting phishing websites using anti-phishing techniques.

This paper offers an extensive review of clustering algorithms, emphasizing their critical role in data analysis for comprehending diverse phenomena. Cluster analysis, as an exploratory technique requiring minimal prior knowledge, has evolved across various scientific communities, providing a rich array of methodologies. While this diversity presents a plethora of tools for researchers and practitioners, it simultaneously poses a challenge due to the overwhelming number of options available. We delve into clustering algorithms relevant to datasets encountered in statistics, computer science, and machine learning, showcasing their practical applications in standard benchmark datasets, the traveling salesman problem, and bioinformatics—a rapidly advancing field garnering significant interest. Additionally, we address related topics such as proximity measures and cluster validation, which are integral to the effective use of clustering techniques. Through this comprehensive survey, we aim to clarify the landscape of clustering algorithms, guiding readers in selecting the most appropriate methods for their specific data analysis needs.

In this paper (expanded from an invited talk at AISEC 2010), we discuss an emerging field of study: adversarial machine learning---the study of effective machine learning techniques against an adversarial opponent. In this paper, we: give a taxonomy for classifying attacks against online machine learning algorithms; discuss application-specific factors that limit an adversary's capabilities; introduce two models for modeling an adversary's capabilities; explore the limits of an adversary's knowledge about the algorithm, feature space, training, and input data; explore vulnerabilities in machine learning algorithms; discuss countermeasures against attacks; introduce the evasion challenge; and discuss privacy-preserving learning techniques.

The threat that attacks cyberspace is known as malware. In order to infect the technologic devices that are attacked, malware needs to evade the different antivirus systems. To avoid detection, an obfuscation technique must be applied so malware is updated and ready to be performed. No obstant, the technique implementation presents difficulties in terms of its required ability, evasion tests and infection functionality that turn outs to be a problem to keep malware updated. Therefore, a procedure is proposed that allows applying AVFUCKER or DSPLIT techniques. The purpose is to optimize the required technical means, reduce the antivirus analysis and malware functionality check times.

The popularity of using Internet contains some risks of network attacks. Intrusion detection is one major research problem in network security, whose aim is to identify unusual access or attacks to secure internal networks. In literature, intrusion detection systems have been approached by various machine learning techniques. However, there is no a review paper to examine and understand the current status of using machine learning techniques to solve the intrusion detection problems. This chapter reviews 55 related studies in the period between 2000 and 2007 focusing on developing single, hybrid, and ensemble classifiers. Related studies are compared by their classifier design, datasets used, and other experimental setups. Current achievements and limitations in developing intrusion detection systems by machine learning are present and discussed. A number of future research directions are also provided. 2009 Elsevier Ltd. All rights reserved.

We propose a novel trap-based architecture for detecting passive, “silent”, attackers who are eavesdropping on enterprise networks. Motivated by the increasing number of incidents where attackers sniff the local network for interesting information, such as credit card numbers, account credentials, and passwords, we introduce a methodology for building a trap-based network that is designed to maximize the realism of bait-laced traffic. Our proposal relies on a “record, modify, replay” paradigm that can be easily adapted to different networked environments. The primary contributions of our architecture are the ease of automatically injecting large amounts of believable bait, and the integration of different detection mechanisms in the back-end. We demonstrate our methodology in a prototype platform that uses our decoy injection API to dynamically create and dispense network traps on a subset of our campus wireless network. Our network traps consist of several types of monitored passwords, authentication cookies, credit cards, and documents containing beacons to alarm when opened. The efficacy of our decoys against a model attack program is also discussed, along with results obtained from experiments in the field. In addition, we present a user study that demonstrates the believability of our decoy traffic, and finally, we provide experimental results to show that our solution causes only negligible interference to ordinary users.

Botnets are networks formed with a number of machines infected by malware called bots. Detection of these malicious networks is becoming a major concern as they pose a serious threat to the network security. Most of the research on bot detection is based on particular botnet characteristics which fail to detect other types of botnets and bots. Furthermore, there are very few bot detection methods that considered real-life class-imbalanced dataset. A dataset is class-imbalanced if there are significantly more instances in one class than the other classes. In this paper, we develop three generic features to detect different types of bots regardless of their botnet characteristics. We develop five classification models based on those features to classify bots from a large, real-life, class-imbalanced network dataset. Results show that our methodology can detect bots more accurately than the existing methods. Experimental results also demonstrate that the developed methodology can successfully detect bots when the proportion of bots to normal activity is very small. We also provide a performance comparison of our methodology with a recent study on bot detection in a real-life, large, imbalanced dataset.

Criminal elements in today’s technology-driven society are using every means available at their disposal to launder the proceeds from their illegal activities. In response, international anti-money laundering (AML) efforts are being made. The events of September 11, 2001, highlighted the need for more sophisticated AML and anti-terrorist financing programs across the industry and nation. In the wake of this, regulators are focusing on the role that technology can play in compliance with laws and ultimately in law enforcement. Banks will have to employ or enhance AML tools and technology to satisfy rising regulatory expectations. While many AML solutions have been in place for some time within the banks, they are faced with the challenge of adapting to the ever-changing risks and methods related to money laundering. In order to provide support for AML decisions, we have formulated an AML conceptual model by following Simon [Simon, H. A. (1977). The new science of management decision. Englewood Cliffs, NJ: Prentice-Hall] decision-making process model. Based on this model, a novel and open multi-agent AML system prototype has been designed and developed. Intelligent agents with their properties of autonomy, reactivity, and proactivity are well suited for dynamic, ill-structured, and complex ML prevention controls. The advanced architecture is able to provide more adaptive, intelligent, and flexible solution for AML. This paper is the first attempt at intelligent agent financial application in the AML domain, with a decision-making/problem-solving process model, an innovative agent-based architecture, and a prototype system. 2007 Elsevier Ltd. All rights reserved.

As computers and information technologies become ubiquitous throughout society, the security of our networks and information technologies is a growing concern. As a result, many researchers have become interested in the security domain. Among them, there is growing interest in observing hacker communities for early detection of developing security threats and trends. Research in this area has often reported hackers openly sharing cybercriminal assets and knowledge with one another. In particular, the sharing of raw malware source code files has been documented in past work. Unfortunately, malware code documentation appears often times to be missing, incomplete, or written in a language foreign to researchers. Thus, analysis of such source files embedded within hacker communities has been limited. Here we utilize a subset of popular machine learning methodologies for the automated analysis of malware source code files. Specifically, we explore genetic algorithms to resolve questions related to feature selection within the context of malware analysis. Next, we utilize two common classification algorithms to test selected features for identification of malware attack vectors. Results suggest promising direction in utilizing such techniques to help with the automated analysis of malware source code.

In this work, we aim at building a bridge from poor behavioral data to an effective, quick-response, and robust behavior model for online identity theft detection. We concentrate on this issue in online social networks (OSNs) where users usually have composite behavioral records, consisting of multi-dimensional low-quality data, e.g., offline check-ins and online user generated content (UGC). As an insightful result, we find that there is a complementary effect among different dimensions of records for modeling users’ behavioral patterns. To deeply exploit such a complementary effect, we propose a joint model to capture both online and offline features of a user’s composite behavior. We evaluate the proposed joint model by comparing with some typical models on two real-world datasets: Foursquare and Yelp. In the widely-used setting of theft simulation (simulating thefts via behavioral replacement), the experimental results show that our model outperforms the existing ones, with the AUC values 0.956 in Foursquare and 0.947 in Yelp, respectively. Particularly, the recall (True Positive Rate) can reach up to 65.3% in Foursquare and 72.2% in Yelp with the corresponding disturbance rate (False Positive Rate) below 1%. It is worth mentioning that these performances can be achieved by examining only one composite behavior (visiting a place and posting a tip online simultaneously) per authentication, which guarantees the low response latency of our method. This study would give the cybersecurity community new insights into whether and how a real-time online identity authentication can be improved via modeling users’ composite behavioral patterns.

Many learning tasks such as spam filtering and credit card fraud detection face an active adversary that tries to avoid detection. For learning problems that deal with an active adversary, it is important to model the adversary's attack strategy and develop robust learning models to mitigate the attack. These are the two objectives of this paper. We consider two attack models: a free-range attack model that permits arbitrary data corruption and a restrained attack model that anticipates more realistic attacks that a reasonable adversary would devise under penalties. We then develop optimal SVM learning strategies against the two attack models. The learning algorithms minimize the hinge loss while assuming the adversary is modifying data to maximize the loss. Experiments are performed on both artificial and real data sets. We demonstrate that optimal solutions may be overly pessimistic when the actual attacks are much weaker than expected. More important, we demonstrate that it is possible to develop a much more resilient SVM learning model while making loose assumptions on the data corruption models. When derived under the restrained attack model, our optimal SVM learning strategy provides more robust overall performance under a wide range of attack parameters.

In addressing the growing problem of junk E-mail on the Internet, we examine methods for the automated construction of filters to eliminate such unwanted messages from a user’s mail stream. By casting this problem in a decision theoretic framework, we are able to make use of probabilistic learning methods in conjunction with a notion of differential misclassification cost to produce filters Which are especially appropriate for the nuances of this task. While this may appear, at first, to be a straight-forward text classification problem, we show that by considering domain-specific features of this problem in addition to the raw text of E-mail messages, we can produce much more accurate filters. Finally, we show the efficacy of such filters in a real world usage scenario, arguing that this technology is mature enough for deployment.


Question: An intelligent strategy model for the detection of phishing websites utilizing anti-phishing techniques.

This paper reviews and evaluates the effectiveness of various term-weighting strategies in the context of automatic text retrieval systems, emphasizing the impact of these strategies on retrieval performance over the past two decades. The accumulated evidence suggests that text indexing mechanisms that utilize single terms, appropriately weighted, yield superior retrieval outcomes compared to more complex text representations. The success of these systems is heavily reliant on the selection of effective term-weighting methodologies. This review consolidates the insights derived from extensive research on automatic term weighting and establishes benchmark models for single-term indexing, which can serve as a reference point for evaluating the performance of more intricate content analysis techniques. The concept of automatic text analysis was initially proposed by Luhn in the late 1950s [1], who posited that text retrieval systems could be developed by comparing content identifiers derived from both stored texts and user queries. These identifiers are typically represented by specific words extracted from the text, either automatically or manually by skilled indexers knowledgeable in the subject matter. In both scenarios, documents are characterized by term vectors of the form D = (t1, t2, ..., tp) (1) where each tk represents a term associated with a particular document D. Similarly, queries can be represented in vector format or as Boolean expressions. For instance, a query Q could be articulated as Q = (q1, q2, ..., qr) (2) This framework forms the foundational basis for understanding how term-weighting strategies influence the accuracy and efficiency of text retrieval systems. The subsequent sections delve into the details of these strategies and their implications for the development of more sophisticated content analysis algorithms.

Network security technology has become crucial in protecting government and industry computing infrastructure. Modern intrusion detection applications face complex requirements; they need to be reliable, extensible, easy to manage, and have low maintenance cost. In recent years, machine learning-based intrusion detection systems have demonstrated high accuracy, good generalization to novel types of intrusion, and robust behavior in a changing environment. This work aims to compare efficiency of machine learning methods in intrusion detection system, including artificial neural networks and support vector machine, with the hope of providing reference for establishing intrusion detection system in future. Compared with other related works in machine learning-based intrusion detectors, we propose to calculate the mean value via sampling different ratios of normal data for each measurement, which lead us to reach a better accuracy rate for observation data in real world. We compare the accuracy, detection rate, false alarm rate for 4 attack types. The extensive experimental results on the KDD-cup intrusion detection benchmark dataset demonstrate that the proposed approach produces higher performance than KDD Winner, especially for U2R and U2L type attacks.

The open nature of the World Wide Web makes evaluating webpage credibility challenging for users. In this paper, we aim to automatically assess web credibility by investigating various characteristics of webpages. Specifically, we first identify features from textual content, link structure, webpages design, as well as their social popularity learned from popular social media sites (e.g., Facebook, Twitter). A set of statistical analyses methods are applied to select the most informative features, which are then used to infer webpages credibility by employing supervised learning algorithms. Real dataset-based experiments under two application settings show that we attain an accuracy of 75% for classification, and an improvement of 53% for the mean absolute error (MAE), with respect to the random baseline approach, for regression.

Anti-money laundering (AML) refers to a set of financial and technological controls that aim to combat the entrance of dirty money into financial systems. A robust AML system must be able to automatically detect any unusual/anomalous financial transactions committed by a customer. The paper presents a hybrid anomaly detection approach that employs clustering to establish customers’ normal behaviors and uses statistical techniques to determine deviation of a particular transaction from the corresponding group behavior. The approach implements a variant of Euclidean Adaptive Resonance Theory, termed as TEART, to group customers in different clusters. The paper also suggests an anomaly index, named AICAF, for ranking transactions as anomalous. The approach has been tested on a real data set comprising of 8.2 million transactions and the results suggest that TEART scales well in terms of the partitions obtained when compared to the traditional K-means algorithm. The presented approach marks transactions having high AICAF values as suspicious.

In these days an increasing number of public and commercial services are used through the Internet, so that security of information becomes more important issue in the society information Intrusion Detection System (IDS) used against attacks for protected to the Computer networks. On another way, some data mining techniques also contribute to intrusion detection. Some data mining techniques used for intrusion detection can be classified into two classes: misuse intrusion detection and anomaly intrusion detection. Misuse always refers to known attacks and harmful activities that exploit the known sensitivity of the system. Anomaly generally means a generally activity that is able to indicate an intrusion. In this paper, comparison made between 23 related papers of using data mining techniques for intrusion detection. Our work provide an overview on data mining and soft computing techniques such as Artificial Neural Network (ANN), Support Vector Machine (SVM) and Multivariate Adaptive Regression Spline (MARS), etc. In this paper comparison shown between IDS data mining techniques and tuples used for intrusion detection. In those 23 related papers, 7 research papers use ANN and 4 ones use SVM, because of ANN and SVM are more reliable than other models and structures. In addition, 8 researches use the DARPA1998 tuples and 13 researches use the KDDCup1999, because the standard tuples are much more credible than others. There is no best intrusion detection model in present time. However, future research directions for intrusion detection should be explored in this paper. Keywords— intrusion detection, data mining, ANN

Online reviews have become a valuable resource for decision making. However, its usefulness brings forth a curse ‒ deceptive opinion spam. In recent years, fake review detection has attracted significant attention. However, most review sites still do not publicly filter fake reviews. Yelp is an exception which has been filtering reviews over the past few years. However, Yelp’s algorithm is trade secret. In this work, we attempt to find out what Yelp might be doing by analyzing its filtered reviews. The results will be useful to other review hosting sites in their filtering effort. There are two main approaches to filtering: supervised and unsupervised learning. In terms of features used, there are also roughly two types: linguistic features and behavioral features. In this work, we will take a supervised approach as we can make use of Yelp’s filtered reviews for training. Existing approaches based on supervised learning are all based on pseudo fake reviews rather than fake reviews filtered by a commercial Web site. Recently, supervised learning using linguistic n-gram features has been shown to perform extremely well (attaining around 90% accuracy) in detecting crowdsourced fake reviews generated using Amazon Mechanical Turk (AMT). We put these existing research methods to the test and evaluate performance on the real-life Yelp data. To our surprise, the behavioral features perform very well, but the linguistic features are not as effective. To investigate, a novel information theoretic analysis is proposed to uncover the precise psycholinguistic difference between AMT reviews and Yelp reviews (crowdsourced vs. commercial fake reviews). We find something quite interesting. This analysis and experimental results allow us to postulate that Yelp’s filtering is reasonable and its filtering algorithm seems to be correlated with abnormal spamming behaviors.

Many learning tasks such as spam filtering and credit card fraud detection face an active adversary that tries to avoid detection. For learning problems that deal with an active adversary, it is important to model the adversary's attack strategy and develop robust learning models to mitigate the attack. These are the two objectives of this paper. We consider two attack models: a free-range attack model that permits arbitrary data corruption and a restrained attack model that anticipates more realistic attacks that a reasonable adversary would devise under penalties. We then develop optimal SVM learning strategies against the two attack models. The learning algorithms minimize the hinge loss while assuming the adversary is modifying data to maximize the loss. Experiments are performed on both artificial and real data sets. We demonstrate that optimal solutions may be overly pessimistic when the actual attacks are much weaker than expected. More important, we demonstrate that it is possible to develop a much more resilient SVM learning model while making loose assumptions on the data corruption models. When derived under the restrained attack model, our optimal SVM learning strategy provides more robust overall performance under a wide range of attack parameters.

Social engineering is a type of attack that allows unauthorized access to a system to achieve specific objective. Commonly, the purpose is to obtain information for social engineers. Some successful social engineering attacks get victims’ information via human based retrieval approach, example technique terms as dumpster diving or shoulder surfing attack to get access to password. Alternatively, victims’ information also can be stolen using technical-based method such as from pop-up windows, email or web sites to get the password or other sensitive information. This research performed a preliminary analysis on social engineering attack taxonomy that emphasized on types of technical-based social engineering attack. Results from the analysis become a guideline in proposing a new generic taxonomy of Social Engineering Attack (SEA).

Online vendors employ collaborative filtering algorithms to provide recommendations to their customers so that they can increase their sales and profits. Although recommendation schemes are successful in e-commerce sites, they are vulnerable to shilling or profile injection attacks. On one hand, online shopping sites utilize collaborative filtering schemes to enhance their competitive edge over other companies. On the other hand, malicious users and/or competing vendors might decide to insert fake profiles into the user-item matrices in such a way so that they can affect the predicted ratings on behalf of their advantages. In the past decade, various studies have been conducted to scrutinize different shilling attacks strategies, profile injection attack types, shilling attack detection schemes, robust algorithms proposed to overcome such attacks, and evaluate them with respect to accuracy, cost/benefit, and overall performance. Due to their popularity and importance, we survey about shilling attacks in collaborative filtering algorithms. Giving an overall picture about various shilling attack types by introducing new classification attributes is imperative for further research. Explaining shilling attack detection schemes in detail and robust algorithms proposed so far might open a lead to develop new detection schemes and enhance such robust algorithms further, even propose new ones. Thus, we describe various attack types and introduce new dimensions for attack classification. Detailed description of the proposed detection and robust recommendation algorithms are given. Moreover, we briefly explain evaluation of the proposed schemes. We conclude the paper by discussing various open questions.

Modern malware families often rely on domain-generation algorithms (DGAs) to determine rendezvous points to their command-and-control server. Traditional defence strategies (such as blacklisting domains or IP addresses) are inadequate against such techniques due to the large and continuously changing list of domains produced by these algorithms. This paper demonstrates that a machine learning approach based on recurrent neural networks is able to detect domain names generated by DGAs with high precision. The neural models are estimated on a large training set of domains generated by various malwares. Experimental results show that this data-driven approach can detect malware-generated domain names with a F1 score of 0.971. To put it differently, the model can automatically detect 93 % of malware-generated domain names for a false positive rate of 1:100.

Firewalls are now an integral part of network security. An intelligent firewall that prevents unauthorized access to a system has been developed. Artificial intelligence applications are uniquely suited for the ever-changing, ever-evolving world of network security. Typical firewalls are only as good as the information provided by the Network Administrator. A new type of attack creates vulnerabilities, which a static firewall does not have the ability to avoid without human direction. An AI-managed firewall service, however, can protect a computer network from known and future threats. We report in this paper on research in progress concerning the integration of different security techniques. A main purpose of the project is to integrate a smart detection engine into a firewall. The smart detection engine will aim at not only detecting anomalous network traffic as in classical IDSs, but also detecting unusual structures data packets that suggest the presence of virus data. We will report in this paper on the concept of an intelligent firewall that contains a smart detection engine for potentially malicious data packets.