Question: A PRACTICAL APPROACH TO HYPERPARAMETER OPTIMIZATION FOR MACHINE LEARNING MODELS

Automating the Optimization Process: An Overview of Bayesian Optimization

The advent of Big Data has necessitated the creation of sophisticated systems that cater to vast user bases, employ intricate software architectures, and harness the power of extensive, diverse computing and storage infrastructures. These complex systems, such as recommendation engines, medical diagnostic tools, real-time gaming platforms, and speech recognition technologies, often encompass a multitude of adjustable parameters. Historically, these parameters have been manually set by different development teams, embedded within the software code. However, the collective optimization of these parameters can lead to substantial enhancements in system performance. Bayesian optimization has emerged as a potent technique for the simultaneous optimization of design choices, witnessing a surge in popularity in recent times. Its key advantage lies in the automation it offers, which not only boosts product quality but also elevates human efficiency. This paper serves as an introductory guide to Bayesian optimization, delving into its foundational principles, spotlighting its methodological nuances, and illustrating its applicability across a broad spectrum of domains.

In this paper, a novel and effective criterion based on the estimation of the signal-to-noise-ratio figure (SNRF) is proposed to optimize the number of hidden neurons in neural networks to avoid overfitting in the function approximation. SNRF can quantitatively measure the useful information left unlearned so that overfitting can be automatically detected from the training error only without use of a separate validation set. It is illustrated by optimizing the number of hidden neurons in a multi-layer perceptron (MLP) using benchmark datasets. The criterion can be further utilized in the optimization of other parameters of neural networks when overfitting needs to be considered.

We present a new family of subgradient methods that dynamica lly incorporate knowledge of the geometry of the data observed in earlier iterations to perfo rm more informative gradient-based learning. Metaphorically, the adaptation allows us to find n eedles in haystacks in the form of very predictive but rarely seen features. Our paradigm stems fro m recent advances in stochastic optimization and online learning which employ proximal funct ions to control the gradient steps of the algorithm. We describe and analyze an apparatus for adap tively modifying the proximal function, which significantly simplifies setting a learning rate nd results in regret guarantees that are provably as good as the best proximal function that can be cho sen in hindsight. We give several efficient algorithms for empirical risk minimization probl ems with common and important regularization functions and domain constraints. We experimen tally study our theoretical analysis and show that adaptive subgradient methods outperform state-o f-the-art, yet non-adaptive, subgradient algorithms.

Extreme Learning Machine (ELM) is an elegant technique for training Single-hidden Layer Feedforward Networks (SLFNs) with extremely fast speed that attracts significant interest recently. One potential weakness of ELM is the random generation of the input weights and hidden biases, which may deteriorate the classification accuracy. In this paper, we propose a new Memetic Algorithm (MA) based Extreme Learning Machine (M-ELM) for classification problems. M-ELM uses Memetic Algorithm which is a combination of population-based global optimization technique and individual-based local heuristic search method to find optimal network parameters for ELM. The optimized network parameters will enhance the classification accuracy and generalization performance of ELM. Experiments and comparisons on 22 benchmark data sets demonstrate that M-ELM is able to provide highly competitive results compared with other state-of-the-art varieties of ELM algorithms.

This paper puts forward two useful methods for self-adaptation of the mutation distribution - the concepts of derandomization and cumulation. Principle shortcomings of the concept of mutative strategy parameter control and two levels of derandomization are reviewed. Basic demands on the self-adaptation of arbitrary (normal) mutation distributions are developed. Applying arbitrary, normal mutation distributions is equiv-alent to applying a general, linear problem encoding. The underlying objective of mutative strategy parameter control is roughly to favor previously selected mutation steps in the future. If this objective is pursued rigor-ously, a completely derandomized self-adaptation scheme results, which adapts arbitrary normal mutation distributions. This scheme, called covariance matrix adaptation (CMA), meets the previously stated demands. It can still be considerably improved by cumulation - utilizing an evolution path rather than single search steps. Simulations on various test functions reveal local and global search properties of the evolution strategy with and without covariance matrix adaptation. Their performances are comparable only on perfectly scaled functions. On badly scaled, non-separable functions usually a speed up factor of several orders of magnitude is ob-served. On moderately mis-scaled functions a speed up factor of three to ten can be expected.

We have used information-theoretic ideas to derive a class of practical and nearly optimal schemes for adapting the size of a neural network. By removing unimportant weights from a network, several improvements can be expected: better generalization, fewer training examples required, and improved speed of learning and/or classification. The basic idea is to use second-derivative information to make a tradeoff between network complexity and training set error. Experiments confirm the usefulness of the methods on a real-world application.

In pattern recognition problems it has been noted that beyond a certain point the inclusion of additional parameters (that have been estimated) leads to higher probabilities of error. A simple problem has been formulated where the probability of error approaches zero as the dimensionality increases and all the parameters are known; on the other hand, the probability of error approaches one-half as the dimensionality increases and parameters are estimated.

Algorithms which construct classifiers from sample data -such as neural networks, radial basis functions, and decision trees -have attracted growing attention for their wide applicability. Researchers in the fields of Statistics, Artificial Intelligence, Machine Learning, Data Mining, and Pattern Recognition are continually introducing (or rediscovering) induction methods, and often publishing implementing code. It is natural for practitioners and potential users to wonder, "Which classification technique is best?", or more realistically, "What subset of methods tend to work well for a given type of dataset?". This book provides perhaps the best current answer to that question.

Skip-gram with negative sampling, a popular variant of Word2vec originally designed and tuned to create word embeddings for Natural Language Processing, has been used to create item embeddings with successful applications in recommendation. While these fields do not share the same type of data, neither evaluate on the same tasks, recommendation applications tend to use the same already tuned hyperparameters values, even if optimal hyperparameters values are often known to be data and task dependent. We thus investigate the marginal importance of each hyperparameter in a recommendation setting through large hyperparameter grid searches on various datasets. Results reveal that optimizing neglected hyperparameters, namely negative sampling distribution, number of epochs, subsampling parameter and window-size, significantly improves performance on a recommendation task, and can increase it by an order of magnitude. Importantly, we find that optimal hyper-parameters configurations for Natural Language Processing tasks and Recommendation tasks are noticeably different.

A f?umeworkfor learning parumeterized models oj‘optical flow from image sequences is presented. A class of motions is represented by U set of orthogonul basis $ow fields that are computed froni a training set using priiicipal component analysis. Many complex image motions can be represeiited by a linear combiriution of U small number of these basisjows. The learned motion models may be usedfilr optical flow estimation und for model-bused recognition. For optical flow estimation we describe a robust, multi-resolution scheme for directly computing the parameters of the learned flow models from image derivatives. As examples we consider learning motion discontinuities, non-rigid motion of human mouths, und urticulated human motion.

In this paper, we present an efficient fast learning algorithm for regression problems using meta-cognitive extreme learning machine(McELM). The proposed algorithm has two components, namely the cognitive component and meta-cognitive component. The cognitive component is an extreme learning machine (ELM) while the meta-cognitive component which controls the cognitive component employs a self-regulating learning mechanism to decide what to learn, when to learn and how to learn. The meta-cognitive component chooses suitable learning method based on the samples presented namely, delete sample, reserve sample and network update. The use of ELM improves the network speed and reduces computational cost. Unlike traditional ELM, the number of hidden layers is not fixed priori in McELM, instead, the network is built during the learning phase. This algorithm is evaluated on a set of benchmark regression and approximation problems and also on a real-world wind force and moment coefficient prediction problem. Performance results in this study highlight that McELM can achieve better results compared with conventional ELM, support vector regression (SVR).


Question: Practical methods for hyperparameter optimization in machine learning models.

This paper presents an innovative and robust automated machine learning (AutoML) framework that addresses the increasing demand for off-the-shelf machine learning systems that can be readily utilized by users without specialized expertise. For such systems to be effective, they must automatically select suitable algorithms, feature preprocessing techniques, and data preprocessing methods, as well as calibrate their respective hyperparameters. Building upon recent advancements in Bayesian optimization, we introduce a novel AutoML system that significantly enhances existing methodologies. This system, referred to as AUTO-SKLEARN, is based on the scikit-learn library and encompasses a structured hypothesis space comprising 15 classifiers, 14 feature preprocessing methods, 4 data preprocessing methods, and a total of 110 hyperparameters. AUTO-SKLEARN surpasses prior AutoML approaches by automatically leveraging past performance on similar datasets and constructing ensembles from models evaluated during the optimization phase. Our framework has achieved top performance in the initial phase of the ongoing ChaLearn AutoML challenge. Through a comprehensive evaluation on over 100 diverse datasets, we demonstrate that AUTO-SKLEARN significantly outperforms previous state-of-the-art AutoML solutions. Moreover, we provide insights into the performance gains attributed to each of our contributions and assess the efficacy of the individual components within AUTO-SKLEARN.

Statistical machine learning is concerned with learning models that describe observations. We train our models from data on tasks like machine translation or object recognition because we cannot explicitly write down programs to solve such problems. A statistical model is only useful when it generalizes to unseen data. Solomonoff114 has proved that one should choose the model that agrees with the observed data, while preferring the model that can be compressed the most, because such a choice guarantees the best possible generalization. The size of the best possible compression of the model is called the Kolmogorov complexity of the model. We define an algorithm as a function with small Kolmogorov complexity. This Ph.D. thesis outlines the problem of learning algorithms from data and shows several partial solutions to it. Our data model is mainly neural networks as they have proven to be successful in various domains like object recognition67,109,122, language modelling90, speech recognition48,39 and others. First, we examine empirical trainability limits for classical neural networks. Then, we extend them by providing interfaces, which provide a way to read memory, access the input, and postpone predictions. The model learns how to use them with reinforcement learning techniques like REINFORCE and Q-learning. Next, we ex-

Three Bayesian ideas are presented for supervised adaptive classifiers. First, it is argued that the output of a classifier should be obtained by marginalizing over the posterior distribution of the parameters; a simple approximation to this integral is proposed and demonstrated. This involves a "moderation" of the most probable classifier's outputs, and yields improved performance. Second, it is demonstrated that the Bayesian framework for model comparison described for regression models in MacKay (1992a,b) can also be applied to classification problems. This framework successfully chooses the magnitude of weight decay terms, and ranks solutions found using different numbers of hidden units. Third, an information-based data selection criterion is derived and demonstrated within this framework.

Magnetic Optimization Algorithm (MOA) has emerged as a promising optimization algorithm that is inspired by the principles of magnetic field theory. In this paper we improve the performance of the algorithm in two aspects. First an Opposition-Based Learning (OBL) approach is proposed for the algorithm which is applied to the movement operator of the algorithm. Second, by learning from the algorithm’s past experience, an adaptive parameter control strategy which dynamically sets the parameters of the algorithm during the optimization is proposed. To show the significance of the proposed parameter adaptation strategy, we compare the algorithm with two well-known parameter setting techniques on a number of benchmark problems. The results indicate that although the proposed algorithm with the adaptation strategy does not require to set the parameters of the algorithm prior to the optimization process, it outperforms MOA with other parameter setting strategies in most large-scale optimization problems. We also study the algorithm while employing the OBL by comparing it with the original version of MOA. Furthermore, the proposed algorithm is tested and compared with seven traditional population-based algorithms and eight state-of-theart optimization algorithms. The comparisons demonstrate that the proposed algorithm outperforms the traditional algorithms in most benchmark problems, and its results is comparative to those obtained by the state-of-the-art algorithms.

We consider the problem of automatic parameter selection in regularization-based radar image formation techniques. It has previously been shown that non-quadratic regularization produces feature-enhanced radar images; can yield superresolution; is robust to uncertain or limited data; and can generate enhanced images in non-conventional data collection scenarios such as sparse aperture imaging. However, this regularized imaging framework involves some hyper-parameters, whose choice is crucial because that directly affects the characteristics of the reconstruction. Hence there is interest in developing methods for automatic parameter choice. We investigate Stein’s unbiased risk estimator (SURE) and generalized cross-validation (GCV) for automatic selection of hyper-parameters in regularized radar imaging. We present experimental results based on the Air Force Research Laboratory (AFRL) “Backhoe Data Dome,” to demonstrate and discuss the effectiveness of these methods.

This work proposes a new meta-heuristic called Grey Wolf Optimizer (GWO) inspired by grey wolves (Canis lupus). The GWO algorithm mimics the leadership hierarchy and hunting mechanism of grey wolves in nature. Four types of grey wolves such as alpha, beta, delta, and omega are employed for simulating the leadership hierarchy. In addition, the three main steps of hunting, searching for prey, encircling prey, and attacking prey, are implemented. The algorithm is then benchmarked on 29 well-known test functions, and the results are verified by a comparative study with Particle Swarm Optimization (PSO), Gravitational Search Algorithm (GSA), Differential Evolution (DE), Evolutionary Programming (EP), and Evolution Strategy (ES). The results show that the GWO algorithm is able to provide very competitive results compared to these wellknown meta-heuristics. The paper also considers solving three classical engineering design problems (tension/compression spring, welded beam, and pressure vessel designs) and presents a real application of the proposed method in the field of optical engineering. The results of the classical engineering design problems and real application prove that the proposed algorithm is applicable to challenging problems with unknown search spaces.

A new heuristic approach for minimizing possibly nonlinear and non differentiable continuous space functions is presented. By means of an extensive testbed, which includes the De Jong functions, it will be demonstrated that the new method converges faster and with more certainty than Adaptive Simulated Annealing as well as the Annealed Nelder&Mead approach, both of which have a reputation for being very powerful. The new method requires few control variables, is robust, easy to use and lends itself very well to parallel computation. ________________________________________ 1)International Computer Science Institute, 1947 Center Street, Berkeley, CA 94704-1198, Suite 600, Fax: 510-643-7684. E-mail: storn@icsi.berkeley.edu. On leave from Siemens AG, ZFE T SN 2, OttoHahn-Ring 6, D-81739 Muenchen, Germany. Fax: 01149-636-44577, Email:rainer.storn@zfe.siemens.de. 2)836 Owl Circle, Vacaville, CA 95687, kprice@solano.community.net.

We consider Bayesian methods for multi-information source optimization (MISO), in which we seek to optimize an expensive-to-evaluate black-box objective function while also accessing cheaper but biased and noisy approximations (“information sources”). We present a novel algorithm that outperforms the state of the art for this problem by using a Gaussian process covariance kernel better suited to MISO than those used by previous approaches, and an acquisition function based on a one-step optimality analysis supported by efficient parallelization. We also provide a novel technique to guarantee the asymptotic quality of the solution provided by this algorithm. Experimental evaluations demonstrate that this algorithm consistently finds designs of higher value at less cost than previous approaches.

Optimizing task-related mathematical model is one of the most fundamental methodologies in statistic and learning areas. However, generally designed schematic iterations may hard to investigate complex data distributions in real-world applications. Recently, training deep propagations (i.e., networks) has gained promising performance in some particular tasks. Unfortunately, existing networks are often built in heuristic manners, thus lack of principled interpretations and solid theoretical supports. In this work, we provide a new paradigm, named Propagation and Optimization based Deep Model (PODM), to bridge the gaps between these different mechanisms (i.e., model optimization and deep propagation). On the one hand, we utilize PODM as a deeply trained solver for model optimization. Different from these existing network based iterations, which often lack theoretical investigations, we provide strict convergence analysis for PODM in the challenging nonconvex and nonsmooth scenarios. On the other hand, by relaxing the model constraints and performing end-to-end training, we also develop a PODM based strategy to integrate domain knowledge (formulated as models) and real data distributions (learned by networks), resulting in a generic ensemble framework for challenging real-world applications. Extensive experiments verify our theoretical results and demonstrate the superiority of PODM against these state-of-the-art approaches.

Motivated by a real-world problem, we study a novel budgeted optimization problem where the goal is to optimize an unknown function f(·) given a budget by requesting a sequence of samples from the function. In our setting, however, evaluating the function at precisely specified points is not practically possible due to prohibitive costs. Instead, we can only request constrained experiments. A constrained experiment, denoted by Q, specifies a subset of the input space for the experimenter to sample the function from. The outcome of Q includes a sampled experiment x, and its function output f(x). Importantly, as the constraints of Q become looser, the cost of fulfilling the request decreases, but the uncertainty about the location x increases. Our goal is to manage this trade-off by selecting a set of constrained experiments that best optimize f(·) within the budget. We study this problem in two different settings, the non-sequential (or batch) setting where a set of constrained experiments is selected at once, and the sequential setting where experiments are selected one at a time. We evaluate our proposed methods for both settings using synthetic and real functions. The experimental results demonstrate the efficacy of the proposed methods.

The support vector machine (SVM) is a popular classification technique. However, beginners who are not familiar with SVM often get unsatisfactory results since they miss some easy but significant steps. In this guide, we propose a simple procedure which usually gives reasonable results.


Question: Learning to rank non-factoid responses: selecting comments in web discussion forums.

This paper introduces a novel convolutional neural network architecture, the Dynamic Convolutional Neural Network (DCNN), designed for the semantic modeling of sentences. The DCNN employs Dynamic k-Max Pooling, a global pooling mechanism over linear sequences, to accurately represent sentences. A key feature of this network is its ability to process sentences of varying lengths and to induce a feature graph that effectively captures both short and long-range relationships within the text. Notably, the DCNN operates independently of a parse tree, making it a versatile tool for language understanding across different languages. The effectiveness of the DCNN is demonstrated through four experiments: binary and multi-class sentiment prediction on a small scale, six-way question classification, and Twitter sentiment prediction using distant supervision. The network performs exceptionally well in the first three experiments and achieves a significant error reduction of over 25% compared to the strongest baseline in the Twitter sentiment prediction task.

**Question**: The Use of Bayesian Networks for Anomaly Detection in Computer Networks

**Documents**: A Multilayer Bayesian Network Model for Network Anomaly Detection
In this paper, we propose a new multilayer Bayesian network model for network anomaly detection. The model is capable of learning the normal behavior of network traffic at different layers of the network protocol stack, and then detecting anomalies by comparing the current network traffic with the learned normal behavior. The model consists of multiple layers, each of which represents a different layer of the network protocol stack. Each layer contains a Bayesian network that models the dependencies among the variables at that layer. The variables in the Bayesian network represent the features of the network traffic at that layer. The model can be trained using unsupervised learning, where the normal behavior is learned from a set of normal network traffic data. During detection, the model computes the probability of the current network traffic given the learned normal behavior. If the probability is below a certain threshold, the current network traffic is considered anomalous. We evaluate the performance of the proposed model on several network datasets, and compare it with other existing anomaly detection methods. The results show that the proposed model outperforms the other methods in terms of detection accuracy and false alarm rate.

**Similar Documents**: This paper presents a novel multilayer Bayesian network model for detecting anomalies in computer networks. The proposed model is designed to learn the typical behavior of network traffic across various layers of the network protocol stack, enabling it to identify deviations from this norm. The model is structured in multiple layers, each corresponding to a specific layer of the protocol stack, with a Bayesian network within each

Given a question and a set of answer candidates, answer triggering determines whether the candidate set contains any correct answers. If yes, it then outputs a correct one. In contrast to existing pipeline methods which first consider individual candidate answers separately and then make a prediction based on a threshold, we propose an end-to-end deep neural network framework, which is trained by a novel group-level objective function that directly optimizes the answer triggering performance. Our objective function penalizes three potential types of error and allows training the framework in an end-to-end manner. Experimental results on the WIKIQA benchmark show that our framework outperforms the state of the arts by a 6.6% absolute gain under F1 measure1.

This thesis discusses the problems and the methods of finding relevant information in large collections of documents. The contribution of this thesis to this problem is to develop better content analysis methods which can be used to describe document content with

We propose a method for automatically labelling topics learned via LDA topic models. We generate our label candidate set from the top-ranking topic terms, titles of Wikipedia articles containing the top-ranking topic terms, and sub-phrases extracted from the Wikipedia article titles. We rank the label candidates using a combination of association measures and lexical features, optionally fed into a supervised ranking model. Our method is shown to perform strongly over four independent sets of topics, significantly better than a benchmark method.

This paper describes an approach and our experiences from the development, deployment and usability testing of a Natural Language Processing (NLP) and Information Retrieval system that supports the moderation of user comments on a large newspaper website. We highlight some of the differences between industry-oriented and academic research settings and their influence on the decisions made in the data collection and annotation processes, selection of document representation and machine learning methods. We report on classification results, where the problems to solve and the data to work with come from a commercial enterprise. In this context typical for NLP research, we discuss relevant industrial aspects. We believe that the challenges faced as well as the solutions proposed for addressing them can provide insights to others working in a similar setting. Data and experiment code related to this paper are available for download at https://ofai.github.io/million-post-corpus.

We introduce a new approach to the machine-assisted grading of short answer questions. We follow past work in automated grading by first training a similarity metric between student responses, but then go on to use this metric to group responses into clusters and subclusters. The resulting groupings allow teachers to grade multiple responses with a single action, provide rich feedback to groups of similar answers, and discover modalities of misunderstanding among students; we refer to this amplification of grader effort as “powergrading.” We develop the means to further reduce teacher effort by automatically performing actions when an answer key is available. We show results in terms of grading progress with a small “budget” of human actions, both from our method and an LDA-based approach, on a test corpus of 10 questions answered by 698 respondents.

Recently, several Web-scale knowledge harvesting systems have been built, each of which is competent at extracting information from certain types of data (e.g., unstructured text, structured tables on the web, etc.). In order to determine the response to a new query posed to such systems (e.g., is sugar a healthy food?), it is useful to integrate opinions from multiple systems. If a response is desired within a specific time budget (e.g., in less than 2 seconds), then maybe only a subset of these resources can be queried. In this paper, we address the problem of knowledge integration for on-demand time-budgeted query answering. We propose a new method, AskWorld, which learns a policy that chooses which queries to send to which resources, by accommodating varying budget constraints that are available only at query (test) time. Through extensive experiments on real world datasets, we demonstrate AskWorld’s capability in selecting most informative resources to query within test-time constraints, resulting in improved performance compared to competitive baselines.

Expert finding is an information retrieval task concerned wi th the search for the most knowledgeable people, in some topic, with basis on document s describing peoples activities. The task involves taking a user query as input and retur ning a list of people sorted by their level of expertise regarding the user query. This pape r introduces a novel approach for combining multiple estimators of expertise based on a mu ltisensor data fusion framework together with the Dempster-Shafer theory of evidence a nd Shannon’s entropy. More specifically, we defined three sensors which detect heteroge neous information derived from the textual contents, from the graph structure of the citati on patterns for the community of experts, and from profile information about the academic exp erts. Given the evidences collected, each sensor may define different candidates as ex perts and consequently do not agree in a final ranking decision. To deal with these conflicts , we applied the DempsterShafer theory of evidence combined with Shannon’s Entropy f rmula to fuse this information and come up with a more accurate and reliable final rankin g list. Experiments made over two datasets of academic publications from the Compute r Science domain attest for the adequacy of the proposed approach over the traditional s tate of the art approaches. We also made experiments against representative supervised s tat of the art algorithms. Results revealed that the proposed method achieved a similar p erformance when compared to these supervised techniques, confirming the capabilities o f the proposed framework. This work was supported by national funds through FCT Funda ção para a Ciência e a Tecnologia, under project PEst-OE/EEI/LA0021/2011 and supported by nationa l funds through FCT Fundação para a Ciência e a Tecnologia, under project PTDC/EIA-CCO/119722/2010

TopicShop is an interface that helps users evaluate and organize collections of web sites. The main interface components are site profiles, which contain information that helps users select high-quality items, and a work area, which offers thumbnail images, annotation, and lightweight grouping techniques to help users organize selected sites. The two components are linked to allow task integration. Previous work [2] demonstrated that subjects who used TopicShop were able to select significantly more highquality sites, in less time and with less effort. We report here on studies that confirm and extend these results. We also show that TopicShop subjects spent just half the time organizing sites, yet still created more groups and more annotations, and agreed more in how they grouped sites. Finally, TopicShop subjects tightly integrated the tasks of evaluating and organizing sites. INTRODUCTION In previous work [2], we motivated an important task for web users – gathering, evaluating, and organizing information resources for a given topic. Current web tools do not support this task well; specifically, they do not make it easy to evaluate collections of web sites to select the best ones or to organize sites for future reuse and sharing. Users have to browse and view sites one after another until they are satisfied they have a good set, or, more likely, they get tired and give up. Browsing a web site is an expensive operation, both in time and cognitive effort. And bookmarks, the most common form of keeping track of web sites, are a fairly primitive organizational technique. We designed and implemented the TopicShop system to provide comprehensive, integrated support for this task. TopicShop aids users in finding a set of relevant sites, in narrowing down the set into a smaller set of high quality sites, and in organizing sites for future use. TopicShop has evolved through a number of design iterations, driven by extensive user testing. We report here on lessons we learned from a pilot study, how these lessons re-shaped our understanding of the task and led to a significant re-design, and the results of a second, larger user study. RELATED WORK Our research program investigates the major information problems faced by users of the World Wide Web: • finding collections of items relevant to their interests; • identifying high-quality items within a collection; • finding items that contain a certain category of information, e.g., episode guides (for a television show) or song lyrics (for a musician); • organizing personalized subsets of items. We have addressed these problems by developing algorithms, implementing them in web crawling and analysis tools, creating interfaces to support users in exploring, comprehending, and organizing collections of web documents, and performing user studies [2, 3, 4, 15]. The work reported here focuses on understanding the user tasks of evaluating and organizing collections of web sites, as ill uminated by the design, evaluation, and re-design of interfaces to support these tasks. Other researchers have investigated these issues. Much recent work has been devoted to algorithms for adding meta-information to collections of web sites to enhance user comprehension, typically by analyzing the structure of links between sites. This approach builds on the intuition that when the author of one site chooses to link to another, this often implies both that the sites have similar content and that the author is endorsing the content of the linked-to site. Pirolli , Pitkow and colleagues [12, 13] experimented with link-based algorithms for clustering and categorizing web pages. Kleinberg’s HITS algorithm [8] defines authoritative and hub pages within a hypertext collection. Authorities and hubs are mutually dependent: a good authority is a page that is linked to by many hubs, and a good hub is one that links to many authorities. After evaluating items and selecting the interesting ones, users must organize the items for future use. Card, Robertson, and Mackinlay [5] introduced the concept of information workspaces to refer to environments in which information items can be stored and manipulated. A departure point for most such systems is the file manager popularized by the Apple Macintosh and then in Microsoft Windows. Such systems typically include a list view, which shows various properties of items, and an icon view, which lets users organize icons representing the items in a 2D space. Mander, Salomon, and Wong [10] enhanced the basic metaphor with the addition of “piles” . Users could create and manipulate piles of items. Interesting interaction techniques for displaying, browsing, and searching piles were designed and tested. Bookmarks are the most popular way to create personal information workspaces of web resources. Bookmarks consist of lists of URLs; typically the title of the web page is used as the label for the URL. Users may organize their bookmarks into a hierarchical category structure. Abrams, Baecker, and Chignell [1] carried out an extensive study of how several hundred web users used bookmarks. They observed a number of strategies for organizing bookmarks, including a flat ordered list, a single level of folders, and hierarchical folders. They also made four design recommendations to help users manage their bookmarks more effectively. First, bookmarks must be easy to organize, e.g., via automatic sorting techniques. Second, visualization techniques are necessary to provide comprehensive overviews of large sets of bookmarks. Third, rich representations of sites are required; many users noted that site titles are not accurate descriptors of site content. Finally, tools for managing bookmarks must be well i ntegrated with web browsers. Many researchers have created experimental information workspace interfaces, often designed expressly for web documents. Card, Robertson, and York [5] describe the WebBook, which uses a book metaphor to group a collection of related web pages for viewing and interaction, and the WebForager, an interface that lets users view and manage multiple WebBooks. Mackinlay, Rao, and Card [9] developed a novel user interface for accessing articles from a citation database. The central UI object is a “Butterfly” , which represents an article, its references, and its citers. The interface makes it easy for users to browse among related articles, group articles, and generate queries to retrieve articles that stand in a particular relationship to the current article. The Data Mountain of Robertson et al [14] represents documents as thumbnail images in a 3D virtual space. Users can move and group the images freely, with various interesting visual and audio cues used to help users arrange the documents. In a study comparing the use of Data Mountain to Internet Explorer Favorites, Data Mountain users retrieved items more quickly, with fewer incorrect or failed retrievals. Our research shares goals with much of the previous work. We focus on designing interfaces that make automatically extracted information about web sites readily accessible to users. We show that this increases users’ abilit y to select high-quality sites. Through ongoing user studies and redesign, we developed easy-to-use annotation and grouping techniques that let users organize items better and more quickly. Finally, we learned how users interleave work on various tasks and re-designed our interface to support such task interleaving. TOPICSHOPEXPLORER, VERSION 1 The TopicShop Explorer is implemented in C++ and runs on Microsoft Windows platforms. Version 1 was based directly on the Macintosh file manager / MS Windows Explorer metaphor (see [3] for detail of TopicShop Version 1 and the pilot study). Accordingly, users could view collections in either a details (Figure 1) or icons (Figure 2) view. The details view showed site profile information (see below) to help users evaluate sites, and the icons view let users organize sites spatially. Figure 1: TopicShop Explorer (version 1), details view. Each web site is represented by a small thumbnail image, the site title, and profile data including the links to/from other sites in the collection, and the number of pages, images, and audio files on the site. Users can sort by a property by clicking on the appropriate column. Figure 2: TopicShop Explorer (version 1), icons view. Each site is represented by a large thumbnail image and the site title. Users can organize sites by arranging them spatially, a technique especially useful in the early stages of exploration. The collections of sites and site profile data used in TopicShop are obtained by running a webcrawler/analyzer. The crawler takes a user-specified set of seed sites as input, and follows links from the seeds to construct a graph of the seed sites, pages contained on these sites, and, optionally, sites determined to be related based on their textual and hyperlink connections to the seeds. Site profiles are built by fetching a large number of pages from each site. Profiles contain content data, including the page title, an estimate of the page count, and a roster of audio files, movie files, and images; they also record links between sites in the collection. In addition, a thumbnail image of each site’s root page is constructed. The first goal of TopicShop is to help users evaluate and identify high quality sites. We sought to achieve this goal by providing site profile data and interface mechanisms for viewing and exploring the data. Making this data visible means that users no longer had to select sites to browse based solely on titles and (sometimes) brief textual annotations. (A chief complaint of subjects in the Abrams et al [1] study was that titles were inadequate descriptors of site content — and that was for sites that users already had browsed and decided to bookmark.) Instead, users may visit only sites that have been endorsed (linked to) by many other sites or sites that are rich in

A common task in many applications is to find persons who are knowledgeable about a given topic (i.e., expert finding). In this paper, we propose and develop a general probabilistic framework for studying expert finding problem and derive two families of generative models (candidate generation models and topic generation models) from the framework. These models subsume most existing language models proposed for expert finding. We further propose several techniques to improve the estimation of the proposed models, including incorporating topic expansion, using a mixture model to model candidate mentions in the supporting documents, and defining an email count-based prior in the topic generation model. Our experiments show that the proposed estimation strategies are all effective to improve retrieval accuracy.

Interest in online education is surging, as dramatized by the success of Khan Academy and recent Stanford online courses, but the technology for online education is in its infancy. Crowdsourcing mechanisms will likely be essential in order to reach the full potential of this medium. This paper sketches some of the challenges and directions we hope HCOMP researchers will ad-


Question: Learning to rank non-factoid responses: selecting comments in online forums.

This paper presents an innovative application of convolutional neural network (CNN) architectures to the task of matching natural language sentences. Semantic matching is a fundamental component in numerous natural language processing tasks, requiring algorithms that can effectively model the intricate structures of language elements and their interactions. To advance this area, we introduce CNN-based models specifically designed for matching pairs of sentences, drawing inspiration from convolutional strategies used in computer vision and speech recognition. Our models are characterized by their ability to represent the hierarchical nature of sentences through a layered composition and pooling mechanism, while also capturing a wide range of matching patterns at various levels of detail. A notable feature of these models is their versatility; they do not require any prior linguistic knowledge and can thus be readily applied to diverse matching tasks across different languages and domains. Comprehensive empirical evaluations on a spectrum of matching tasks have confirmed the effectiveness of our proposed models and their outperformance compared to existing competitor models.

For many internet users, searching for health advice online is the first step in seeking treatment. We present a Learning to Rank system that uses a novel set of syntactic and semantic features to improve consumer health search. Our approach was evaluated on the 2016 CLEF eHealth dataset, outperforming the best method by 26.6% in NDCG@10.

Written development communication (e.g. mailing lists, issue trackers) constitutes a precious source of information to build recommenders for software engineers, for example aimed at suggesting experts, or at redocumenting existing source code. In this paper we propose a novel, semi-supervised approach named DECA (Development Emails Content Analyzer) that uses Natural Language Parsing to classify the content of development emails according to their purpose (e.g. feature request, opinion asking, problem discovery, solution proposal, information giving etc), identifying email elements that can be used for specific tasks. A study based on data from Qt and Ubuntu, highlights a high precision (90%) and recall (70%) of DECA in classifying email content, outperforming traditional machine learning strategies. Moreover, we successfully used DECA for re-documenting source code of Eclipse and Lucene, improving the recall, while keeping high precision, of a previous approach based on ad-hoc heuristics.

Our society is increasingly digitalized. Every day, a tremendous amount of information is being created, shared, and digested through all kinds of cyber channels. Although people can easily acquire information from various sources (social media, news articles, etc.), the truthfulness of most received information remains unverified. In many real-life scenarios, false information has become the de facto cause that leads to detrimental decision makings, and techniques that can automatically filter false information are highly demanded. However, verifying whether a piece of information is trustworthy is difficult because: (1) selecting candidate snippets for fact checking is nontrivial; and (2) detecting supporting evidences, i.e. stances, suffers from the difficulty of measuring the similarity between claims and related evidences. We build ClaimVerif, a claim verification system that not only provides credibility assessment for any user-given query claim, but also rationales the assessment results with supporting evidences. ClaimVerif can automatically select the stances from millions of documents and employs two-step training to justify the opinions of the stances. Furthermore, combined with the credibility of stances sources, ClaimVerif degrades the score of stances from untrustworthy sources and alleviates the negative effects from rumor spreaders. Our empirical evaluations show that ClaimVerif achieves both high accuracy and efficiency in different claim verification tasks. It can be highly useful in practical applications by providing multi-dimension analysis for the suspicious statements, including the stances, opinions, source credibility and estimated judgements.

Recommender systems attempt to predict items in which a user might be interested, given some information about the user's and items' profiles. Most existing recommender systems use content-based or collaborative filtering methods or hybrid methods that combine both techniques (see the sidebar for more details). We created Informed Recommender to address the problem of using consumer opinion about products, expressed online in free-form text, to generate product recommendations. Informed recommender uses prioritized consumer product reviews to make recommendations. Using text-mining techniques, it maps each piece of each review comment automatically into an ontology.

Posting comments on the news is one of the most popular forms of user participation in online newspapers, and there is great potential for public discourse that is associated with this form of user communication. However, this potential arises only when several users participate in commenting and when their communication becomes interactive. Based on an adaption of Galtung and Ruge’s theory of newsworthiness, we hypothesized that a news article’s news factors affect both participation levels and interactivity in a news item’s comments section. The data from an online content analysis of political news are consistent with the hypotheses. This article explores the ways in which news factors affect participation levels and interactivity, and it discusses the theoretical, normative, and practical implications of those findings.

The accuracy of collaborative-filtering recommender systems largely depends on three factors: the quality of the rating prediction algorithm, and the quantity and quality of available ratings. While research in the field of recommender systems often concentrates on improving prediction algorithms, even the best algorithms will fail if they are fed poor-quality data during training, that is, garbage in, garbage out. Active learning aims to remedy this problem by focusing on obtaining better-quality data that more aptly reflects a user's preferences. However, traditional evaluation of active learning strategies has two major flaws, which have significant negative ramifications on accurately evaluating the system's performance (prediction error, precision, and quantity of elicited ratings). (1) Performance has been evaluated for each user independently (ignoring system-wide improvements). (2) Active learning strategies have been evaluated in isolation from unsolicited user ratings (natural acquisition).
 In this article we show that an elicited rating has effects across the system, so a typical user-centric evaluation which ignores any changes of rating prediction of other users also ignores these cumulative effects, which may be more influential on the performance of the system as a whole (system centric). We propose a new evaluation methodology and use it to evaluate some novel and state-of-the-art rating elicitation strategies. We found that the system-wide effectiveness of a rating elicitation strategy depends on the stage of the rating elicitation process, and on the evaluation measures (MAE, NDCG, and Precision). In particular, we show that using some common user-centric strategies may actually degrade the overall performance of a system. Finally, we show that the performance of many common active learning strategies changes significantly when evaluated concurrently with the natural acquisition of ratings in recommender systems.

The World Wide Web (WWW) has become a rapidly growing platform consisting of numerous sources which provide supporting or contradictory information about claims (e.g., “Chicken meat is healthy”). In order to decide whether a claim is true or false, one needs to analyze content of different sources of information on the Web, measure credibility of information sources, and aggregate all these information. This is a tedious process and the Web search engines address only part of the overall problem, viz., producing only a list of relevant sources. In this paper, we present ClaimEval, a novel and integrated approach which given a set of claims to validate, extracts a set of pro and con arguments from the Web information sources, and jointly estimates credibility of sources and correctness of claims. ClaimEval uses Probabilistic Soft Logic (PSL), resulting in a flexible and principled framework which makes it easy to state and incorporate different forms of prior-knowledge. Through extensive experiments on realworld datasets, we demonstrate ClaimEval’s capability in determining validity of a set of claims, resulting in improved accuracy compared to state-of-the-art baselines.

Asynchronous conversations are conversations where participants collaborate with each other at different times (e.g., email, blog, forum). The huge amount of textual data generated everyday in these conversations calls for automated methods of conversational text analysis. Topic segmentation and labeling is often considered a prerequisite for higher-level conversation analysis and has been shown to be useful in many NLP applications including summarization, information extraction and conversation visualization. Topic segmentation in asynchronous conversation refers to the task of clustering the sentences into a set of coherent topical clusters. In [6, 7], we presented unsupervised and supervised topic segmentation models for asynchronous conversations, which to our knowledge are the state-of-the-art. This study concerns topic labeling, that is, given a set of topical clusters in a conversation, the task is to assign appropriate topic labels to the clusters. For example, five topic labels in a Slashdot [2] blog conversation about releasing a new game called Daggerfall are game contents and size, game design, bugs/faults, other gaming options and performance/speed issues. Such topic labels can serve as a concise summary of the conversation and can also be used for indexing. Ideally, topic labels should be meaningful, semantically similar to the underlying topic, general (i.e., broad coverage of the topic) and discriminative (or exclusive) when there are multiple topics [10]. Traditionally, the top K terms in a multinomial topic model (e.g., LDA [3]) are used to represent a topic. However, as pointed out by [10], at the word-level, topic labels may become too general that it can impose cognitive difficulties on a user to interpret the meaning of the topic by associating the words together. On the other hand, if the labels are expressed at the sentence-level, they may become too specific to cover the whole theme of the topic. Based on these observations, [10] and other recent studies (e.g., [9]) advocate for phrase-level topic labels. This is also consistent with the monologue corpora built as part of the Topic Detection and Tracking (TDT) project [1] as well as with our own email and blog conversational corpora in which human annotators without specific instructions spontaneously generated labels at the phrase-level. Considering all these factors, we also treat phrase-level as the right level of granularity for a label in this work. Few prior studies have addressed the topic labeling problem in different settings [10, 9]. Common to their approaches is that they first mine topics in the form of topic-word distributions from the whole corpus using topic models like LDA. Then, they try to label the topics (i.e., topic-word distributions) with an appropriate label using the statistical association metrics (e.g., point-wise mutual information, t-test) computed from either the source corpus or an external knowledge base (e.g., Wikipedia). In contrast, our task is to label the topical clusters in a given conversation, where topics are closely related and distributional variations are subtle (e.g., game contents and size, game design). Therefore, corpus-based statistical association metrics are not reliable in our case. Also at the conversation-level, the topics are too specific to find their labels in an external source. To our knowledge, none has studied this problem before. Therefore, there is no standard corpus and no agreed-upon evaluation metrics available. Our contributions aim to remedy these problems. First, we present a blog and an email corpora annotated with topics. Second, we propose to generate topic labels using an extractive approach, that finds the most representative phrases from the text without relying on an external source. Since, graph-based key phrase ranking has proved to be the state-of-the-art (unsupervised) method [11], we adopt the same framework. We propose a novel biased random walk model that exploits the fact that the leading sentences in a topic often carry the most informative clues for its label. However, the phrases extracted only by considering the sentences in a topic may ignore the global aspect of the conversation. As another contribution, we propose to re-rank the phrases extracted from the whole conversation with respect to the individual topics and include the relevant ones. Experimental results show that our approach outperforms other ranking models including a general random walk model (i.e., TextRank) proposed by [11], a leadoriented model and a frequency-based model, and including the relevant conversation-level phrases improves the performance.

We describe our Alexa prize system (called ‘Alana’) which consists of an ensemble of bots, combining rule-based and machine learning systems, and using a contextual ranking mechanism to choose system responses. This paper reports on the version of the system developed and evaluated in the semi-finals of the competition (i.e. up to 15 August 2017), but not on subsequent enhancements. The ranker for this system was trained on real user feedback received during the competition, where we address the problem of how to train on the noisy and sparse feedback obtained during the competition. In order to avoid initial problems of inappropriate and boring utterances coming from big datasets such as Reddit and Twitter, we later focussed on ‘clean’ data sources such as news and facts. We report on experiments with different ranking functions and versions of our NewsBot. We find that a multiturn news strategy is beneficial, and that a ranker trained on the ratings feedback from users is also effective. Our system continuously improved using the data gathered over the course over the competition (1 July – 15 August) . Our final user score (averaged user rating over the whole semi-finals period) was 3.12, and we achieved 3.3 for the averaged user rating over the last week of the semi-finals (8-15 August 2017). We were also able to achieve long dialogues (average 10.7 turns) during the competition period. In subsequent weeks, after the end of the semi-final competition, we have achieved our highest scores of 3.52 (daily average, 18th October), 3.45 (weekly average on 23 and 24 October), and average dialogue lengths of 14.6 turns (1 October), and median dialogue length of 2.25 minutes (average for 7 days on 10th October).

This paper presents a bootstrapping process that learns linguistically rich extraction patterns for subjective (opinionated) expressions. High-precision classifiers label unannotated data to automatically create a large training set, which is then given to an extraction pattern learning algorithm. The learned patterns are then used to identify more subjective sentences. The bootstrapping process learns many subjective patterns and increases recall while maintaining high precision.


Question: Training a model to rank non-factoid responses: Selecting comments in online forums.

This paper explores the limitations of recurrent backpropagation in learning to store information over extended periods, primarily due to the issue of insufficient and decaying error backflow. Building upon Hochreiter's (1991) analysis of this challenge, we propose a novel method called Long Short-Term Memory (LSTM), an efficient gradient-based approach designed to overcome these limitations. LSTM employs a unique mechanism to maintain constant error flow through special units known as constant error carousels, enabling the bridging of time lags that exceed 1000 discrete-time steps. The system includes multiplicative gate units that learn when to allow or block the constant error flow, thereby controlling the access to information. LSTM's architecture is both spatially and temporally local, and its computational complexity per time step and weight is linear, denoted as O. 1. Our empirical evaluations, conducted using artificial data with a variety of representations including local, distributed, real-valued, and noisy patterns, highlight LSTM's superior performance. In comparison to real-time recurrent learning, backpropagation through time, recurrent cascade correlation, Elman networks, and neural sequence chunking, LSTM demonstrates a higher success rate and faster learning speed. Moreover, LSTM is the first recurrent network algorithm to effectively solve complex, artificial long-time-lag tasks that were previously unsolvable by other methods.

We investigate the problem of readeraware multi-document summarization (RA-MDS) and introduce a new dataset for this problem. To tackle RA-MDS, we extend a variational auto-encodes (VAEs) based MDS framework by jointly considering news documents and reader comments. To conduct evaluation for summarization performance, we prepare a new dataset. We describe the methods for data collection, aspect annotation, and summary writing as well as scrutinizing by experts. Experimental results show that reader comments can improve the summarization performance, which also demonstrates the usefulness of the proposed dataset. The annotated dataset for RA-MDS is available online1.

This paper investigates the effect of various types of linguistic features (lexical, syntactic and semantic) for training classifiers to detect threats of violence in a corpus of YouTube comments. Our results show that combinations of lexical features outperform the use of more complex syntactic and semantic features for this task.

Subjectivity taggingis distinguishing sentences used to present opinions and evaluations from sentences used to objectively present factual information. There are numerous applications for which subjectivity tagging is relevant, including information extraction and information retrieval. This paper identifies strong clues of subjectivity using the results of a method for clustering words according to distributional similarity (Lin 1998), seeded by a small amount of detailed manual annotation. These features are then further refined with the addition of lexical semantic features of adjectives, specifically polarityandgradability(Hatzivassiloglou & McKeown 1997), which can be automatically learned from corpora. In 10-fold cross validation experiments, features based on both similarity clusters and the lexical semantic features are shown to have higher precision than features based on each alone.

We publicly release a new large-scale dataset, called SearchQA, for machine comprehension, or question-answering. Unlike recently released datasets, such as DeepMind CNN/DailyMail and SQuAD, the proposed SearchQA was constructed to reflect a full pipeline of general question-answering. That is, we start not from an existing article and generate a question-answer pair, but start from an existing question-answer pair, crawled from J! Archive, and augment it with text snippets retrieved by Google. Following this approach, we built SearchQA, which consists of more than 140k question-answer pairs with each pair having 49.6 snippets on average. Each question-answer-context tuple of the SearchQA comes with additional meta-data such as the snippet’s URL, which we believe will be valuable resources for future research. We conduct human evaluation as well as test two baseline methods, one simple word selection and the other deep learning based, on the SearchQA. We show that there is a meaningful gap between the human and machine performances. This suggests that the proposed dataset could well serve as a benchmark for question-answering.

We consider the problem of combining ranking results from various sources. In the context of the Web, the main applications include building meta-search engines, combining ranking functions, selecting documents based on multiple criteria, and improving search precision through word associations. We develop a set of techniques for the rank aggregation problem and compare their performance to that of well-known methods. A primary goal of our work is to design rank aggregation techniques that can e ectively combat \spam," a serious problem in Web searches. Experiments show that our methods are simple, e cient, and e ective.

In this paper we present a detailed scheme for annotating expressions of opinions, beliefs, emotions, sentiment and speculation (private states) in the news and other discourse. We explore inter-annotator agreement for individual private state expressions, and show that these low-level annotations are useful for producing higher-level subjective sentence annotations.

In this work, we present a novel approach for evaluating job applicants in online recruitment systems, leveraging machine learning algorithms to solve the candidate ranking problem. An application of our approach is implemented in the form of a prototype system, whose functionality is showcased and evaluated in a real-world recruitment scenario. The proposed system extracts a set of objective criteria from the applicants’ LinkedIn profile, and infers their personality characteristics using linguistic analysis on their blog posts. Our system was found to perform consistently compared to human recruiters; thus, it can be trusted for the automation of applicant ranking and

Neural networks and word embeddings are powerful tools to capture latent factors. These tools can provide effective measures of similarities between users or items in the context of sparse data. We propose a novel approach that relies on neural networks and word embeddings to the problem of matching a learner looking for mentoring, and a tutor that is willing to provide this mentoring. Tutors and learners can issue multiple offers/requests on different topics. The approach matches over the whole array of topics specified by learners and tutors. Its performance for tutor-learner matching is compared with the state of the art. It yields similar results in terms of precision, but improves the recall.

Sarcasm is a sophisticated form of speech act widely used in online communities. Automatic recognition of sarcasm is, however, a novel task. Sarcasm recognition could contribute to the performance of review summarization and ranking systems. This paper presents SASI, a novel Semi-supervised Algorithm for Sarcasm Identification that recognizes sarcastic sentences in product reviews. SASI has two stages: semisupervised pattern acquisition, and sarcasm classification. We experimented on a data set of about 66000 Amazon reviews for various books and products. Using a gold standard in which each sentence was tagged by 3 annotators, we obtained precision of 77% and recall of 83.1% for identifying sarcastic sentences. We found some strong features that characterize sarcastic utterances. However, a combination of more subtle pattern-based features proved more promising in identifying the various facets of sarcasm. We also speculate on the motivation for using sarcasm in online communities and social networks.

We present a framework for question answering that can efficiently scale to longer documents while maintaining or even improving performance of state-ofthe-art models. While most successful approaches for reading comprehension rely on recurrent neural networks (RNNs), running them over long documents is prohibitively slow because it is difficult to parallelize over sequences. Inspired by how people first skim the document, identify relevant parts, and carefully read these parts to produce an answer, we combine a coarse, fast model for selecting relevant sentences and a more expensive RNN for producing the answer from those sentences. We treat sentence selection as a latent variable trained jointly from the answer only using reinforcement learning. Experiments demonstrate the state of the art performance on a challenging subset of the WIKIREADING dataset (Hewlett et al., 2016) and on a new dataset, while speeding up the model by 3.5x-6.7x.


Question: Training a model to rank non-factual responses: selecting comments in online forums.

This paper explores the effectiveness of Convolutional Neural Networks (CNNs) in sentence classification tasks when utilizing pre-trained word vectors as a foundation. Through a series of experiments, we demonstrate that a basic CNN model, with minimal adjustment of hyperparameters and static word vectors, can achieve remarkable accuracy on various benchmarks. Our results indicate that fine-tuning these vectors for specific tasks can lead to significant improvements in performance. Moreover, we introduce a slight architectural alteration to the CNN, enabling the simultaneous use of both task-specific and static word vectors. This enhanced CNN model surpasses existing state-of-the-art methods in 4 out of 7 tasks, encompassing sentiment analysis and question classification, showcasing its versatility and effectiveness in sentence-level classification.

**Question**: A Comparative Study of Feature Selection Methods for Intrusion Detection Systems

**Documents**: A Comprehensive Survey on Feature Selection in Intrusion Detection Systems
Intrusion Detection Systems (IDS) are essential tools for protecting networks and systems from unauthorized access and malicious activities. However, the performance of IDS can be significantly improved by reducing the dimensionality of the feature space through feature selection. This paper provides a comprehensive survey of feature selection methods applied in IDS, including filter, wrapper, and embedded methods. We discuss the advantages and disadvantages of each method and present a comparative analysis of their performance. Our survey reveals that wrapper methods generally outperform filter and embedded methods in terms of accuracy, but at the cost of higher computational complexity. We also highlight the importance of choosing the right feature selection method based on the specific requirements of the IDS and the characteristics of the dataset.

**Similar Documents**: This study offers an extensive review of feature selection techniques employed in Intrusion Detection Systems (IDS), which are critical for safeguarding networks and systems against unauthorized intrusions and malevolent activities. The paper delves into various feature selection methodologies, focusing on filter, wrapper, and embedded approaches, and evaluates their effectiveness in the context of IDS. It discusses the merits and demerits of each method, offering a detailed comparison of their respective performances. Our analysis indicates that while wrapper methods tend to excel in accuracy compared to filter and embedded methods, they do so with a higher computational cost. The paper underscores the significance of selecting the appropriate feature selection method, considering the specific demands of the IDS and the nature of the dataset being analyzed.

With social networks like Facebook, Twitter and Google+ attracting audiences of millions of users, they have been an important communication platform in daily life. This in turn attracts malicious users to the social networks as well, causing an increase in the incidence of low quality information. Low quality information such as spam and rumors is a nuisance to people and hinders them from consuming information that is pertinent to them or that they are looking for. Although individual social networks are capable of filtering a significant amount of low quality information they receive, they usually require large amounts of resources (e.g, personnel) and incur a delay before detecting new types of low quality information. Also the evolution of various low quality information posts lots of challenges to defensive techniques. My PhD thesis work focuses on the analysis and detection of low quality information in social networks. We introduce social spam analytics and detection framework SPADE across multiple social networks showing the efficiency and flexibility of cross-domain classification and associative classification. For evolutionary study of low quality information, we present the results on large-scale study on Web spam and email spam over a long period of time. Furthermore, we provide activity-based detection approaches to filter out low quality information in social networks: click traffic analysis of short URL spam, behavior analysis of URL spam and information diffusion analysis of rumor. Our framework and detection techniques show promising results in analyzing and detecting low quality information in social networks.

Inquiry is fundamental to communication, and machines cannot effectively collaborate with humans unless they can ask questions. In this thesis work, we explore how can we teach machines to ask clarification questions when faced with uncertainty, a goal of increasing importance in today’s automated society. We do a preliminary study using data from StackExchange, a plentiful online resource where people routinely ask clarifying questions to posts so that they can better offer assistance to the original poster. We build neural network models inspired by the idea of the expected value of perfect information: a good question is one whose expected answer is going to be most useful. To build generalizable systems, we propose two future research directions: a template-based model and a sequence-to-sequence based neural generative model.

Identifying incorrect content (i.e., semantic error) in text is a difficult task because of the ambiguous nature of written natural language and themany factors that canmake a statement semantically erroneous. Current methods identify semantic errors in a sentence by determining whether it contradicts the domain to which the sentence belongs. However, because thesemethods are constructedon expected logic contradictions, they cannot handle new or unexpected semantic errors. In this paper, we propose a new method for detecting semantic errors that is based on logic reasoning. Our proposed method converts text into logic clauses, which are later analyzed against a domain ontology by an automatic reasoner to determine its consistency. This approach can provide a complete analysis of the text, since it can analyze a single sentence or sets of multiple sentences. When there are multiple sentences to analyze, in order to avoid the high complexity of reasoning over a large set of logic clauses, we propose rules that reduce the set of sentences to analyze, based on the logic relationships between sentences. In our evaluation, we have found that our proposed method can identify a significant percentage of semantic errors and, in the case of multiple sentences, it does so without significant computational cost. We have also found that both the quality of the information extraction output and modeling elements B Dejing Dou dou@cs.uoregon.edu Fernando Gutierrez fernando@cs.uoregon.edu Nisansa de Silva nisansa@cs.uoregon.edu Stephen Fickas fickas@cs.uoregon.edu 1 Department of Computer and Information Science, 1202 University of Oregon, Eugene, OR 97403, USA of the ontology (i.e., property domain and range) affect the capability of detecting errors.

Identifying sentiments (the affective parts of opinions) is a challenging problem. We present a system that, given a topic, automatically finds the people who hold opinions about that topic and the sentiment of each opinion. The system contains a module for determining word sentiment and another for combining sentiments within a sentence. We experiment with various models of classifying and combining sentiment at word and sentence levels, with promising results.

Sarcasm is a form of speech act in which the speakers convey their message in an implicit way. The inherently ambiguous nature of sarcasm sometimes makes it hard even for humans to decide whether an utterance is sarcastic or not. Recognition of sarcasm can benefit many sentiment analysis NLP applications, such as review summarization, dialogue systems and review ranking systems. In this paper we experiment with semisupervised sarcasm identification on two very different data sets: a collection of 5.9 million tweets collected from Twitter, and a collection of 66000 product reviews from Amazon. Using the Mechanical Turk we created a gold standard sample in which each sentence was tagged by 3 annotators, obtaining F-scores of 0.78 on the product reviews dataset and 0.83 on the Twitter dataset. We discuss the differences between the datasets and how the algorithm uses them (e.g., for the Amazon dataset the algorithm makes use of structured information). We also discuss the utility of Twitter #sarcasm hashtags for the task.

We tackle Attitude Detection, which we define as the task of extracting the replier's attitude, i.e., a target-polarity pair, from a given one-round conversation. While previous studies considered Target Extraction and Polarity Classification separately, we regard them as subtasks of Attitude Detection. Our experimental results show that treating the two subtasks independently is not the optimal solution for Attitude Detection, as achieving high performance in each subtask is not sufficient for obtaining correct target-polarity pairs. Our jointly trained model AD-NET substantially outperforms the separately trained models by alleviating the target-polarity mismatch problem. Moreover, we proposed a method utilising the attitude detection model to improve retrieval-based chatbots by re-ranking the response candidates with attitude features. Human evaluation indicates that with attitude detection integrated, the new responses to the sampled queries from are statistically significantly more consistent, coherent, engaging and informative than the original ones obtained from a commercial chatbot.

Subjectivity taggingis distinguishing sentences used to present opinions and evaluations from sentences used to objectively present factual information. There are numerous applications for which subjectivity tagging is relevant, including information extraction and information retrieval. This paper identifies strong clues of subjectivity using the results of a method for clustering words according to distributional similarity (Lin 1998), seeded by a small amount of detailed manual annotation. These features are then further refined with the addition of lexical semantic features of adjectives, specifically polarityandgradability(Hatzivassiloglou & McKeown 1997), which can be automatically learned from corpora. In 10-fold cross validation experiments, features based on both similarity clusters and the lexical semantic features are shown to have higher precision than features based on each alone.

We consider the problem of combining ranking results from various sources. In the context of the Web, the main applications include building meta-search engines, combining ranking functions, selecting documents based on multiple criteria, and improving search precision through word associations. We develop a set of techniques for the rank aggregation problem and compare their performance to that of well-known methods. A primary goal of our work is to design rank aggregation techniques that can e ectively combat \spam," a serious problem in Web searches. Experiments show that our methods are simple, e cient, and e ective.

Relevance feedback has been shown to improve retrieval for a broad range of retrieval models. It is the most common way of adapting a retrieval model for a specific query. In this work, we expand this common way by focusing on an approach that enables us to do query-specific modification of a retrieval model for learning-to-rank problems. Our approach is based on using feedback documents in two ways: 1) to improve the retrieval model directly and 2) to identify a subset of training queries that are more predictive than others. Experiments with the Gov2 collection show that this approach can obtain statistically significant improvements over two baselines; learning-to-rank (SVM-rank) with no feedback and learning-to-rank with standard relevance feedback.

Our news are saturated with claims of “facts” made from data. Database research has in the past focused on how to answer queries, but has not devoted much attention to discerning more subtle qualities of the resulting claims, e.g., is a claim “cherry-picking”? This paper proposes a framework that models claims based on structured data as parameterized queries. A key insight is that we can learn a lot about a claim by perturbing its parameters and seeing how its conclusion changes. This framework lets us formulate practical fact-checking tasks—reverse-engineering (often intentionally) vague claims, and countering questionable claims—as computational problems. Along with the modeling framework, we develop an algorithmic framework that enables efficient instantiations of “meta” algorithms by supplying appropriate algorithmic building blocks. We present real-world examples and experiments that demonstrate the power of our model, efficiency of our algorithms, and usefulness of their results.