Question: An intelligent strategy model for the detection of phishing websites using anti-phishing techniques.

This paper explores a statistical methodology for the mechanized encoding and retrieval of literary information, addressing the complexities inherent in written communication. The process of conveying ideas through writing is inherently probabilistic, with authors opting for varying levels of specificity and utilizing diverse word combinations to transmit meaning effectively. These individual differences, coupled with the varied expression of similar concepts, pose significant challenges for automated literature searching. A statistical approach is proposed to tackle these issues, and the paper details the sequential steps involved in this system. These include conducting a statistical analysis of a corpus of documents within a specific domain, defining a set of core "notions" and the vocabulary that embodies them, compiling a thesaurus-like dictionary and index, and automating the document encoding process with the aid of this dictionary. Additionally, the encoding of topological notations, such as branched structures, is considered, alongside the storage of coded information. The establishment of a search pattern for identifying relevant data and the programming of suitable machines to execute these searches are also crucial components of the proposed system. This statistical approach aims to enhance the precision and efficiency of information retrieval in the literary domain.

**Question**: A Novel Approach for the Detection of Phishing Websites Using Machine Learning

**Documents**: An Advanced System for the Identification of Phishing Websites Utilizing Deep Learning Techniques
In the era of digital communication, the proliferation of phishing attacks has become a significant concern. This paper introduces an advanced system designed to combat this issue by employing deep learning techniques for the identification of phishing websites. The proposed system is based on a deep neural network architecture, specifically tailored to analyze and interpret the features of web pages. The system is trained on a large dataset of known phishing and legitimate websites, enabling it to learn the distinguishing characteristics of phishing attempts. Key features considered include URL structure, domain age, presence of HTTPS, and the use of suspicious keywords. The deep learning model is capable of detecting subtle patterns that are indicative of phishing activities, even when traditional methods fail. The system's performance is evaluated using various metrics, such as accuracy, precision, recall, and F1 score, and the results demonstrate its effectiveness in accurately distinguishing between phishing and legitimate websites. This advanced system represents a significant step forward in the ongoing battle against phishing, offering a robust solution for the protection of online users.

**Similar Documents**: This paper presents a cutting-edge solution for the detection of phishing websites, leveraging the power of deep learning algorithms. In the face of escalating phishing threats in the digital age, the proposed system utilizes a sophisticated deep neural network to accurately

Mobile malware has been growing in scale and complexity spurred by the unabated uptake of smartphones worldwide. Android is fast becoming the most popular mobile platform resulting in sharp increase in malware targeting the platform. Additionally, Android malware is evolving rapidly to evade detection by traditional signature-based scanning. Despite current detection measures in place, timely discovery of new malware is still a critical issue. This calls for novel approaches to mitigate the growing threat of zero-day Android malware. Hence, in this paper we develop and analyze proactive Machine Learning approaches based on Bayesian classification aimed at uncovering unknown Android malware via static analysis. The study, which is based on a large malware sample set of majority of the existing families, demonstrates detection capabilities with high accuracy. Empirical results and comparative analysis are presented offering useful insight towards development of effective static-analytic Bayesian classification based solutions for detecting unknown Android malware.

Firewalls are now an integral part of network security. An intelligent firewall that prevents unauthorized access to a system has been developed. Artificial intelligence applications are uniquely suited for the ever-changing, ever-evolving world of network security. Typical firewalls are only as good as the information provided by the Network Administrator. A new type of attack creates vulnerabilities, which a static firewall does not have the ability to avoid without human direction. An AI-managed firewall service, however, can protect a computer network from known and future threats. We report in this paper on research in progress concerning the integration of different security techniques. A main purpose of the project is to integrate a smart detection engine into a firewall. The smart detection engine will aim at not only detecting anomalous network traffic as in classical IDSs, but also detecting unusual structures data packets that suggest the presence of virus data. We will report in this paper on the concept of an intelligent firewall that contains a smart detection engine for potentially malicious data packets.

Due to the distributed nature of Denial-of-Service attacks, it is tremendously challenging to identify such malicious behavior using traditional intrusion detection systems in Wireless Sensor Networks (WSNs). In the current paper, a bio-inspired method is introduced, namely the cooperative-based fuzzy artificial immune system (Co-FAIS). It is a modular-based defense strategy derived from the danger theory of the human immune system. The agents synchronize and work with one another to calculate the abnormality of sensor behavior in terms of context antigen value (CAV) or attackers and update the fuzzy activation threshold for security response. In such a multi-node circumstance, the sniffer module adapts to the sink node to audit data by analyzing the packet components and sending the log file to the next layer. The fuzzy misuse detector module (FMDM) integrates with a danger detector module to identify the sources of danger signals. The infected sources are transmitted to the fuzzy Q-learning vaccination modules (FQVM) in order for particular, required action to enhance system abilities. The Cooperative Decision Making Modules (Co-DMM) incorporates danger detector module with the fuzzy Q-learning vaccination module to produce optimum defense strategies. To evaluate the performance of the proposed model, the Low Energy Adaptive Clustering Hierarchy (LEACH) was simulated using a network simulator. The model was subsequently compared against other existing soft computing methods, such as fuzzy logic controller (FLC), artificial immune system (AIS), and fuzzy Q-learning (FQL), in terms of detection accuracy, counter-defense, network lifetime and energy consumption, to demonstrate its efficiency and viability. The proposed method improves detection accuracy and successful defense rate performance against attacks compared to conventional empirical methods. & 2014 Elsevier Ltd. All rights reserved.

With over 50 billion downloads and more than 1.3 million apps in Google’s official market, Android has continued to gain popularity amongst smartphone users worldwide. At the same time there has been a rise in malware targeting the platform, with more recent strains employing highly sophisticated detection avoidance techniques. As traditional signature based methods become less potent in detecting unknown malware, alternatives are needed for timely zero-day discovery. Thus this paper proposes an approach that utilizes ensemble learning for Android malware detection. It combines advantages of static analysis with the efficiency and performance of ensemble machine learning to improve Android malware detection accuracy. The machine learning models are built using a large repository of malware samples and benign apps from a leading antivirus vendor. Experimental results and analysis presented shows that the proposed method which uses a large feature space to leverage the power of ensemble learning is capable of 97.3 % to 99% detection accuracy with very low false positive rates. Keywordsmobile security; Android; malware detection; ensemble learning; static analysis; machine learning; data mining; random forest

Intrusion detection is a new, retrofit approach for providing a sense of security in existing computers and data networks, while allowing them to operate in their current "open" mode. The goal of intrusion detection is to identify unauthorized use, misuse, and abuse of computer systems by both system insiders and external penetrators. The intrusion detection problem is becoming a challenging task due to the proliferation of heterogeneous computer networks since the increased connectivity of computer systems gives greater access to outsiders and makes it easier for intruders to avoid identification. Intrusion detection systems (IDSs) are based on the beliefs that an intruder's behavior will be noticeably different from that of a legitimate user and that many unauthorized actions are detectable. Typically, IDSs employ statistical anomaly and rulebased misuse models in order to detect intrusions. A number of prototype IDSs have been developed at several institutions, and some of them have also been deployed on an experimental basis in operational systems. In the present paper, several host-based and network-based IDSs are surveyed, and the characteristics of the corresponding systems are identified. The host-based systems employ the host operating system's audit trails as the main source of input to detect intrusive activity, while most of the network-based IDSs build their detection mechanism on monitored network traffic, and some employ host audit trails as well. An outline of a statistical anomaly detection algorithm employed in a typical IDS is also included.<<ETX>>

Social engineering is a type of attack that allows unauthorized access to a system to achieve specific objective. Commonly, the purpose is to obtain information for social engineers. Some successful social engineering attacks get victims’ information via human based retrieval approach, example technique terms as dumpster diving or shoulder surfing attack to get access to password. Alternatively, victims’ information also can be stolen using technical-based method such as from pop-up windows, email or web sites to get the password or other sensitive information. This research performed a preliminary analysis on social engineering attack taxonomy that emphasized on types of technical-based social engineering attack. Results from the analysis become a guideline in proposing a new generic taxonomy of Social Engineering Attack (SEA).

With the capability of infecting hundreds of thousands of hosts, worms represent a major threat to the Internet. The defense against Internet worms is largely an open problem. This paper investigates two important problems. Can a localized defense system detect new worms that were not seen before and moreover, capture the attack packets? How to identify polymorphic worms from the normal background traffic? We have two major contributions here. The first contribution is the design of a novel double-honeypot system, which is able to automatically detect new worms and isolate the attack traffic. The second contribution is the proposal of a new type of position-aware distribution signatures (PADS), which fit in the gap between the traditional signatures and the anomaly-based systems. We propose two algorithms based on expectation-maximization (EM) and Gibbs sampling for efficient computation of PADS from polymorphic worm samples. The new signature is capable of handling certain polymorphic worms. Our experiments show that the algorithms accurately separate new variants of the MSBlaster worm from the normal-traffic background.

Web spamming refers to actions intended to mislead search engines into ranking some pages higher than they deserve. Recently, the amount of web spam has increased dramatically, leading to a degradation of search results. This paper presents a comprehensive taxonomy of current spamming techniques, which we believe can help in developing appropriate countermeasures.

With the increase in mobile device use, there is a greater need for increasingly sophisticated malware detection algorithms. The research presented in this paper examines two types of features of Android applications, permission requests and system calls, as a way to detect malware. We are able to differentiate between benign and malicious apps by applying a machine learning algorithm. The model that is presented here achieved a classification accuracy of around 80% using permissions and 60% using system calls for a relatively small dataset. In the future, different machine learning algorithms will be examined to see if there is a more suitable algorithm. More features will also be taken into account and the training set will be expanded.

Most modern malware download attacks occur via the browser, typically due to social engineering and driveby downloads. In this paper, we study the “origin” of malware download attacks experienced by real network users, with the objective of improving malware download defenses. Specifically, we study the web paths followed by users who eventually fall victim to different types of malware downloads. To this end, we propose a novel incident investigation system, named WebWitness. Our system targets two main goals: 1) automatically trace back and label the sequence of events (e.g., visited web pages) preceding malware downloads, to highlight how users reach attack pages on the web; and 2) leverage these automatically labeled in-the-wild malware download paths to better understand current attack trends, and to develop more effective defenses. We deployed WebWitness on a large academic network for a period of ten months, where we collected and categorized thousands of live malicious download paths. An analysis of this labeled data allowed us to design a new defense against drive-by downloads that rely on injecting malicious content into (hacked) legitimate web pages. For example, we show that by leveraging the incident investigation information output by WebWitness we can decrease the infection rate for this type of drive-by downloads by almost six times, on average, compared to existing URL blacklisting approaches.


Question: An intelligent strategy model for detecting phishing websites as part of an anti-phishing approach.

An Enhanced Browser-Based Defense Mechanism Against Phishing Attacks
Phishing, a prevalent form of web-based identity theft, poses a serious threat by deceiving users into divulging sensitive personal information. In this paper, we explore the intricacies of phishing attacks and propose an advanced client-side defense strategy. Our solution, coined as PhishDefender, is a browser extension designed to scrutinize web pages and alert users to potential phishing attempts when data requests appear suspicious. Unlike existing approaches, PhishDefender employs sophisticated heuristics and machine learning algorithms to accurately distinguish between legitimate and fraudulent sites. The efficacy of our framework has been validated through extensive testing with real-world phishing websites, obtained from authoritative sources. However, given the evolving nature of phishing tactics, we anticipate that PhishDefender will be a critical tool in the ongoing battle against web-based identity theft.

Consumers’ purchase decisions are increasingly influenced by user-generated online reviews. Accordingly, there has been growing concern about the potential for posting deceptive opinion spam— fictitious reviews that have been deliberately written to sound authentic, to deceive the reader. In this paper, we explore generalized approaches for identifying online deceptive opinion spam based on a new gold standard dataset, which is comprised of data from three different domains (i.e. Hotel, Restaurant, Doctor), each of which contains three types of reviews, i.e. customer generated truthful reviews, Turker generated deceptive reviews and employee (domain-expert) generated deceptive reviews. Our approach tries to capture the general difference of language usage between deceptive and truthful reviews, which we hope will help customers when making purchase decisions and review portal operators, such as TripAdvisor or Yelp, investigate possible fraudulent activity on their sites.1

In this paper we discuss our research in developing general and systematic methods for intrusion detection. The key ideas are to use data mining techniques to discover consistent and useful patterns of system features that describe program and user behavior, and use the set of relevant system features to compute (inductively learned) classifiers that can recognize anomalies and known intrusions. Using experiments on the s ndmailsystem call data and the network tcpdumpdata, we demonstrate that we can construct concise and accurate classifiers to detect anomalies. We provide an overview on two general data mining algorithms that we have implemented: the association rules algorithm and the frequent episodes algorithm. These algorithms can be used to compute the intraand interaudit record patterns, which are essential in describing program or user behavior. The discovered patterns can guide the audit data gathering process and facilitate feature selection. To meet the challenges of both efficient learning (mining) and real-time detection, we propose an agent-based architecture for intrusion detection systems where the learning agents continuously compute and provide the updated (detection) models to the detection agents.

JavaScript is a browser scripting language that allows developers to create sophisticated client-side interfaces for web applications. However, JavaScript code is also used to carry out attacks against the user's browser and its extensions. These attacks usually result in the download of additional malware that takes complete control of the victim's platform, and are, therefore, called "drive-by downloads." Unfortunately, the dynamic nature of the JavaScript language and its tight integration with the browser make it difficult to detect and block malicious JavaScript code.
 This paper presents a novel approach to the detection and analysis of malicious JavaScript code. Our approach combines anomaly detection with emulation to automatically identify malicious JavaScript code and to support its analysis. We developed a system that uses a number of features and machine-learning techniques to establish the characteristics of normal JavaScript code. Then, during detection, the system is able to identify anomalous JavaScript code by emulating its behavior and comparing it to the established profiles. In addition to identifying malicious code, the system is able to support the analysis of obfuscated code and to generate detection signatures for signature-based systems. The system has been made publicly available and has been used by thousands of analysts.

Web spamming refers to actions intended to mislead search engines into ranking some pages higher than they deserve. Recently, the amount of web spam has increased dramatically, leading to a degradation of search results. This paper presents a comprehensive taxonomy of current spamming techniques, which we believe can help in developing appropriate countermeasures.

Consumers increasingly rate, review and research products online (Jansen, 2010; Litvin et al., 2008). Consequently, websites containing consumer reviews are becoming targets of opinion spam. While recent work has focused primarily on manually identifiable instances of opinion spam, in this work we study deceptive opinion spam—fictitious opinions that have been deliberately written to sound authentic. Integrating work from psychology and computational linguistics, we develop and compare three approaches to detecting deceptive opinion spam, and ultimately develop a classifier that is nearly 90% accurate on our gold-standard opinion spam dataset. Based on feature analysis of our learned models, we additionally make several theoretical contributions, including revealing a relationship between deceptive opinions and imaginative writing.

Cyber threat intelligence and security informatics play critical roles in the identification of the principal influencers and threats associated with criminal activity and extremism in web-based communities. This paper presents an effective, dynamic learning framework utilizing domain specific context information to design a novel classification model that can robustly identify malicious social media posts (e.g. enrollment propaganda for extremist groups) with expressions of extremism or criminal intent. Research towards automated identification of extreme online posts and their associated key suspects and threats faces numerous challenges, 1) Online data, particularly social media data, originated from numerous independent and heterogeneous sources are largely unstructured. 2) The tactics, techniques, and procedures (TTPs) of criminal activity and extremism are constantly evolving. 3) There are limited ground truth data to support the development of effective classification technologies. In this paper, we present a human-machine collaborative, semi-supervised learning system that can efficiently and effectively identify malicious social media posts in presence of these challenges. Our system and framework develops an initial classifier from limited annotated data and in an interactive manner evolves dynamically into a sophisticated model using shortlisted relevant samples, identified via a graph-based optimization method, solvable by maximum flow algorithm. This same method also may be used to refine the classifier as TTPs evolve. Under this framework, the classifier performance converges faster using roughly 1–2 orders of magnitude fewer annotated samples, as compared to fully supervised solutions, resulting in a reasonably acceptable accuracy of nearly 80%. We validate our framework using a large collection of English and non-English flagged words extracted from three web-based forums and manually verified by multiple independent annotators.

In this research, a technique for `Credit Card Fraud Detection' is developed. As fraudsters are increasing day by day. And fallacious transactions are done by the credit card and there are various types of fraud. So to solve this problem combination of technique is used like Genetic Algorithm, Behavior Based Technique and Hidden Markov Model. By this transaction is tested individually and whatever suits the best is further proceeded. And the foremost goal is to detect fraud by filtering the above techniques to get better result.

This paper presents a novel unsupervised approach to detect cyber attacks in Cyber-Physical Systems (CPS). We describe an unsupervised learning approach using a Recurrent Neural network which is a time series predictor as our model. We then use the Cumulative Sum method to identify anomalies in a replicate of a water treatment plant. The proposed method not only detects anomalies in the CPS but also identifies the sensor that was attacked. The experiments were performed on a complex dataset which is collected through a Secure Water Treatment Testbed (SWaT). Through the experiments, we show that the proposed technique is able to detect majority of the attacks designed by our research team with low false positive rates.

Ensuring the security of computers is a non-trivial task, with many techniques used by malicious users to compromise these systems. In recent years a new threat has emerged in the form of networks of hijacked zombie machines used to perform complex distributed attacks such as denial of service and to obtain sensitive data such as password information. These zombie machines are said to be infected with a dasiahotpsila - a malicious piece of software which is installed on a host machine and is controlled by a remote attacker, termed the dasiabotmaster of a botnetpsila. In this work, we use the biologically inspired dendritic cell algorithm (DCA) to detect the existence of a single hot on a compromised host machine. The DCA is an immune-inspired algorithm based on an abstract model of the behaviour of the dendritic cells of the human body. The basis of anomaly detection performed by the DCA is facilitated using the correlation of behavioural attributes such as keylogging and packet flooding behaviour. The results of the application of the DCA to the detection of a single hot show that the algorithm is a successful technique for the detection of such malicious software without responding to normally running programs.

Information security awareness web-based courses are commonly recommended in cyber security strategies to help build a security culture capable of addressing information systems breaches caused by user mistakes whose negligence or ignorance of policies may endanger information systems assets. A research gap exists on the impact of Information Security Awareness Web-Based Courses: these are failing in changing to a significant degree the behavior of participants regarding compliance and diligence, which translates into continuous vulnerabilities. The aim of this work is to contribute with a theoretical and empirical analysis on the potential strengths and weaknesses of Information Security Awareness Web-Based Courses and with two practical tools readily applicable for designers and reviewers of web-based or mediatized courses on information security awareness and education. The research design seeks to respond two research questions. The first on the formulation of a minimum set of criteria that could be applied to Information Security Awareness Web-Based Courses, to support their real impact on employee’s diligence and compliance, resulting in eleven criteria for courses’ assessment and a checklist. The second, about a controlled experiment to explore the actual impact of an existing course, in respect to diligence and compliance using phishing emails as educational tools, that reaffirms the theoretical assumptions arrived to earlier. The development of minimum criteria and their systematic implementation pursue behavioral change, emphasizes the importance of disciplinary integration in cyber security research, and advocates for the development of a solid security culture of diligence and compliance, capable of supporting the protection of organizations from information system threats. The results gathered in this study suggest that achieving positive results in the existing information security tests that follow security awareness courses does not necessarily imply that diligence or information security policies compliance are affected. These preliminary findings accumulate evidence on the importance of implementing the recommendations formulated in this work.


Question: Online learning strategies for adversaries influenced by memory: The cost of previous errors.

This paper investigates the issue of time series forecasting utilizing the ARMA (autoregressive moving average) model, with a focus on minimizing assumptions about the noise components. By employing techniques aimed at minimizing regret, we devise efficient online learning algorithms tailored to the prediction challenge, eschewing the common presumptions that the noise terms adhere to a Gaussian distribution, are identically distributed, or are independent. Additionally, we demonstrate that the performance of our proposed algorithm converges to that of the optimal ARMA model in hindsight, as the volume of data increases, showcasing its effectiveness in real-world applications.

**Question**: The Role of Social Media in Political Campaigns

**Documents**: Social Media and Political Mobilization: Evidence from a Field Experiment
This paper reports the results of a field experiment that examines the impact of social media on political mobilization. We randomly assigned 61,155 Facebook users to one of three groups: a treatment group that received information about voting, a treatment group that received information about voting and was encouraged to spread the information to their friends, and a control group that received no information. We find that the treatment groups were more likely to vote than the control group, and that the effect was largest for the group that received information about voting and was encouraged to spread it to their friends. We also find that the treatment groups were more likely to encourage their friends to vote, and that this effect was largest for the group that received information about voting and was encouraged to spread it to their friends.

**Similar Documents**: In this study, we present the findings from a randomized field experiment designed to assess the influence of social media on political engagement and mobilization. A total of 61,159 Facebook users were randomly allocated to one of three categories: a group that received voting information, a group that not only received voting information but was also prompted to share it with their social network, and a control group that did not receive any information. Our analysis reveals that the groups receiving voting information, particularly those encouraged to share it, exhibited a higher propensity to vote compared to the control group. Moreover, these groups were more inclined to motivate their friends to vote, with the most significant effect observed in the group that was both informed about voting and urged to disseminate this information. This research underscores the pivotal role social media can play in amplifying political participation and mobilization efforts.

Chapter 1 strongly advocates the stochastic back-propagation method to train neural networks. This is in fact an instance of a more general technique called stochastic gradient descent (SGD). This chapter provides background material, explains why SGD is a good learning algorithm when the training set is large, and provides useful recommendations.

Persuasion, defined as the act of exploiting an informational advantage in order to effect the decisions of others, is ubiquitous. Indeed, persuasive communication has been estimated to account for almost a third of all economic activity in the US. This paper examines persuasion through a computational lens, focusing on what is perhaps the most basic and fundamental model in this space: the celebrated Bayesian persuasion model of Kamenica and Gentzkow. Here there are two players, a sender and a receiver. The receiver must take one of a number of actions with a-priori unknown payoff, and the sender has access to additional information regarding the payoffs of the various actions for both players. The sender can commit to revealing a noisy signal regarding the realization of the payoffs of various actions, and would like to do so as to maximize her own payoff in expectation assuming that the receiver rationally acts to maximize his own payoff. When the payoffs of various actions follow a joint distribution (the common prior), the sender's problem is nontrivial, and its computational complexity depends on the representation of this prior. We examine the sender's optimization task in three of the most natural input models for this problem, and essentially pin down its computational complexity in each. When the payoff distributions of the different actions are i.i.d. and given explicitly, we exhibit a polynomial-time (exact) algorithmic solution, and a ``simple'' (1-1/e)-approximation algorithm. Our optimal scheme for the i.i.d. setting involves an analogy to auction theory, and makes use of Border's characterization of the space of reduced-forms for single-item auctions. When action payoffs are independent but non-identical with marginal distributions given explicitly, we show that it is #P-hard to compute the optimal expected sender utility. In doing so, we rule out a generalized Border's theorem, as defined by Gopalan et al, for this setting. Finally, we consider a general (possibly correlated) joint distribution of action payoffs presented by a black box sampling oracle, and exhibit a fully polynomial-time approximation scheme (FPTAS) with a bi-criteria guarantee. Our FPTAS is based on Monte-Carlo sampling, and its analysis relies on the principle of deferred decisions. Moreover, we show that this result is the best possible in the black-box model for information-theoretic reasons.

An approach to the online learning of Takagi-Sugeno (TS) type models is proposed in the paper. It is based on a novel learning algorithm that recursively updates TS model structure and parameters by combining supervised and unsupervised learning. The rule-base and parameters of the TS model continually evolve by adding new rules with more summarization power and by modifying existing rules and parameters. In this way, the rule-base structure is inherited and up-dated when new data become available. By applying this learning concept to the TS model we arrive at a new type adaptive model called the Evolving Takagi-Sugeno model (ETS). The adaptive nature of these evolving TS models in combination with the highly transparent and compact form of fuzzy rules makes them a promising candidate for online modeling and control of complex processes, competitive to neural networks. The approach has been tested on data from an air-conditioning installation serving a real building. The results illustrate the viability and efficiency of the approach. The proposed concept, however, has significantly wider implications in a number of fields, including adaptive nonlinear control, fault detection and diagnostics, performance analysis, forecasting, knowledge extraction, robotics, behavior modeling.

Multilayer connectionist models of memory based on the encoder model using the backpropagation learning rule are evaluated. The models are applied to standard recognition memory procedures in which items are studied sequentially and then tested for retention. Sequential learning in these models leads to 2 major problems. First, well-learned information is forgotten rapidly as new information is learned. Second, discrimination between studied items and new items either decreases or is nonmonotonic as a function of learning. To address these problems, manipulations of the network within the multilayer model and several variants of the multilayer model were examined, including a model with prelearned memory and a context model, but none solved the problems. The problems discussed provide limitations on connectionist models applied to human memory and in tasks where information to be learned is not all available during learning.

How do we learn to think better, and what can we do to promote such metacognitive learning? Here, we propose that cognitive growth proceeds through metacognitive reinforcement learning. We apply this theory to model how people learn how far to plan ahead and test its predictions about the speed of metacognitive learning in two experiments. In the first experiment, we find that our model can discern a reward structure that promotes metacognitive reinforcement learning from one that hinders it. In the second experiment, we show that our model can be used to design a feedback mechanism that enhances metacognitive reinforcement learning in an environment that hinders learning. Our results suggest that modeling metacognitive learning is a promising step towards promoting cognitive growth.

We address the question of when a network can be expected to generalize from m random training examples chosen from some arbitrary probability distribution, assuming that future test examples are drawn from the same distribution. Among our results are the following bounds on appropriate sample vs. network size. Assume 0 < ∊ 1/8. We show that if m O(W/∊ log N/∊) random examples can be loaded on a feedforward network of linear threshold functions with N nodes and W weights, so that at least a fraction 1 ∊/2 of the examples are correctly classified, then one has confidence approaching certainty that the network will correctly classify a fraction 1 ∊ of future test examples drawn from the same distribution. Conversely, for fully-connected feedforward nets with one hidden layer, any learning algorithm using fewer than (W/∊) random training examples will, for some distributions of examples consistent with an appropriate weight choice, fail at least some fixed fraction of the time to find a weight choice that will correctly classify more than a 1 ∊ fraction of the future test examples.

Recent studies have shown the vulnerability of reinforcement learning (RL) models in noisy settings. The sources of noises differ across scenarios. For instance, in practice, the observed reward channel is often subject to noise (e.g., when observed rewards are collected through sensors), and thus observed rewards may not be credible as a result. Also, in applications such as robotics, a deep reinforcement learning (DRL) algorithm can be manipulated to produce arbitrary errors. In this paper, we consider noisy RL problems where observed rewards by RL agents are generated with a reward confusion matrix. We call such observed rewards as perturbed rewards. We develop an unbiased reward estimator aided robust RL framework that enables RL agents to learn in noisy environments while observing only perturbed rewards. Our framework draws upon approaches for supervised learning with noisy data. The core ideas of our solution include estimating a reward confusion matrix and defining a set of unbiased surrogate rewards. We prove the convergence and sample complexity of our approach. Extensive experiments on different DRL platforms show that policies based on our estimated surrogate reward can achieve higher expected rewards, and converge faster than existing baselines. For instance, the state-of-the-art PPO algorithm is able to obtain 67.5% and 46.7% improvements in average on five Atari games, when the error rates are 10% and 30% respectively.

This paper presents a novel incremental algorithm that combines Q-learning, a well-known dynamic-programming based reinforcement learning method, with the TD(λ) return estimation process, which is typically used in actor-critic learning, another well-known dynamic-programming based reinforcement learning method. The parameter λ is used to distribute credit throughout sequences of actions, leading to faster learning and also helping to alleviate the non-Markovian effect of coarse state-space quatization. The resulting algorithm.Q(λ)-learning, thus combines some of the best features of the Q-learning and actor-critic learning paradigms. The behavior of this algorithm has been demonstrated through computer simulations.

In this paper we bring techniques from operations research to bear on the problem of choosing optimal actions in partially observable stochastic domains We begin by introducing the theory of Markov decision processes mdps and partially observable mdps pomdps We then outline a novel algorithm for solving pomdps o line and show how in some cases a nite memory controller can be extracted from the solution to a pomdp We conclude with a discussion of how our approach relates to previous work the complexity of nding exact solutions to pomdps and of some possibilities for nding approximate solutions

Boosting is a general method for improving the accuracy of any given learning algorithm. Focusing primarily on the AdaBoost algorithm, this chapter overviews some of the recent work on boosting including analyses of AdaBoost’s training error and generalization error; boosting’s connection to game theory and linear programming; the relationship between boosting and logistic regression; extensions of AdaBoost for multiclass classification problems; methods of incorporating human knowledge into boosting; and experimental and applied work using boosting.


Question: Online learning strategies for adversaries with memory: assessing the cost of previous errors.

This paper presents a series of algorithms designed for online convex optimization, a framework introduced by Zinkevich (ICML 2003) that encapsulates a broad range of decision-making scenarios, including Prediction from Expert Advice and Cover’s Universal Portfolios. In this setting, a decision-maker selects a sequence of points in Euclidean space from a fixed feasible set, encountering a sequence of (potentially unrelated) convex cost functions post-selection. Zinkevich demonstrated that a straightforward online gradient descent algorithm can achieve an additive regret of $O(\sqrt{T})$ for any sequence of T convex cost functions with bounded gradients, relative to the optimal single decision in hindsight. Here, we advance the field by providing algorithms that attain a regret of O(log (T)) for any sequence of strictly convex functions with bounded first and second derivatives. This generalizes the specific instances of Prediction from Expert Advice by Kivinen and Warmuth (EuroCOLT 1999) and Universal Portfolios by Cover (Math. Finance 1:1–19, 1991). Our paper introduces multiple algorithms achieving logarithmic regret, which are not only more comprehensive but also significantly more efficient to execute. The primary novel contributions involve the utilization of the Newton method for optimization, a fresh approach in this field, leading to an efficient algorithm. Our analysis uncovers an unexpected link between the intuitive follow-the-leader strategy and the Newton method. Moreover, we critically assess various algorithms, consolidating different prior methodologies including follow-the-leader, exponential weighting, Cover’s algorithm, and gradient descent.

LIBOL is an open-source library for large-scale online learning, which consists of a large family of efficient and scalable state-of-the-art online learning algorithms for large-scale online classification tasks. We have offered easy-to-use command-line tools and examples for users and developers, and also have made comprehensive documents available for both beginners and advanced users. LIBOL is not only a machine learning toolbox, but also a comprehensive experimental platform for conducting online learning research.

We provide a principled way of proving Õ(√T) high-probability guarantees for partial-information (bandit) problems over arbitrary convex decision sets. First, we prove a regret guarantee for the full-information problem in terms of “local” norms, both for entropy and self-concordant barrier regularization, unifying these methods. Given one of such algorithms as a black-box, we can convert a bandit problem into a full-information problem using a sampling scheme. The main result states that a high-probability Õ(√T) bound holds whenever the black-box, the sampling scheme, and the estimates of missing information satisfy a number of conditions, which are relatively easy to check. At the heart of the method is a construction of linear upper bounds on confidence intervals. As applications of the main result, we provide the first known efficient algorithm for the sphere with an Õ(√T) high-probability bound. We also derive the result for the n-simplex, improving the O(√nT log(nT)) bound of Auer et al [3] by replacing the log T term with log log T and closing the gap to the lower bound of Ω(√nT). While Õ(√T) high-probability bounds should hold for general decision sets through our main result, construction of linear upper bounds depends on the particular geometry of the set; we believe that the sphere example already exhibits the necessary ingredients. The guarantees we obtain hold for adaptive adversaries (unlike the in-expectation results of [1]) and the algorithms are efficient, given that the linear upper bounds on confidence can be computed.

We present a polynomial-time algorithm that, given as a input the description of a game with incomplete information and any number of players, produces a protocol for playing the game that leaks no partial information, provided the majority of the players is honest.
Our algorithm automatically solves all the multi-party protocol problems addressed in complexity-based cryptography during the last 10 years. It actually is a completeness theorem for the class of distributed protocols with honest majority. Such completeness theorem is optimal in the sense that, if the majority of the players is not honest, some protocol problems have no efficient solution [C].

Choosing optimal (or at least better) policies is an important problem in domains from medicine to education to finance and many others. One approach to this problem is through controlled experiments/trials but controlled experiments are expensive. Hence it is important to choose the best policies on the basis of observational data. This presents two difficult challenges: (i) missing counterfactuals, and (ii) selection bias. This paper presents theoretical bounds on estimation errors of counterfactuals from observational data by making connections to domain adaptation theory. It also presents a principled way of choosing optimal policies using domain adversarial neural networks. This illustrates the effectiveness of domain adversarial training together with various features of our algorithm on a semi-synthetic breast cancer dataset.

This paper investigates how adaptive software agents may utilize reinforcement learning algorithms such as Q-learning to make economic decisions such as setting prices in a competitive marketplace. For a single adaptive agent facing fixed-strategy opponents, ordinary Q-learning is guaranteed to find the optimal policy. However, for a population of agents each trying to adapt in the presence of other adaptive agents, the problem becomes non-stationary and history dependent, and it is not known whether any global convergence will be obtained, and if so, whether such solutions will be optimal. In this paper, we study simultaneous Q-learning by two competing seller agents in three moderately realistic economic models. This is the simplest case in which interesting multi-agent phenomena can occur, and the state space is small enough so that lookup tables can be used to represent the Q-functions. We find that, despite the lack of theoretical guarantees, simultaneous convergence to self-consistent optimal solutions is obtained in each model, at least for small values of the discount parameter. In some cases, exact or approximate convergence is also found even at large discount parameters. We show how the Q-derived policies increase profitability and damp out or eliminate cyclic price “wars” compared to simpler policies based on zero lookahead or short-term lookahead. In one of the models (the “Shopbot” model) where the sellers' profit functions are symmetric, we find that Q-learning can produce either symmetric or broken-symmetry policies, depending on the discount parameter and on initial conditions.

This paper describes an effort to model students' changing knowledge state during skill acquisition. Students in this research are learning to write short programs with the ACT Programming Tutor (APT). APT is constructed around a production rule cognitive model of programming knowledge, called theideal student model. This model allows the tutor to solve exercises along with the student and provide assistance as necessary. As the student works, the tutor also maintains an estimate of the probability that the student has learned each of the rules in the ideal model, in a process calledknowledge tracing. The tutor presents an individualized sequence of exercises to the student based on these probability estimates until the student has ‘mastered’ each rule. The programming tutor, cognitive model and learning and performance assumptions are described. A series of studies is reviewed that examine the empirical validity of knowledge tracing and has led to modifications in the process. Currently the model is quite successful in predicting test performance. Further modifications in the modeling process are discussed that may improve performance levels.

The robustness of neural networks to intended perturbations has recently attracted significant attention. In this paper, we propose a new method, learning with a strong adversary, that learns robust classifiers from supervised data by generating adversarial examples as an intermediate step. A new and simple way of finding adversarial examples is presented that is empirically stronger than existing approaches in terms of the accuracy reduction as a function of perturbation magnitude. Experimental results demonstrate that resulting learning method greatly improves the robustness of the classification models produced.

In practice, there are often explicit constraints on what representations or decisions are acceptable in an application of machine learning. For example it may be a legal requirement that a decision must not favour a particular group. Alternatively it can be that that representation of data must not have identifying information. We address these two related issues by learning flexible representations that minimize the capability of an adversarial critic. This adversary is trying to predict the relevant sensitive variable from the representation, and so minimizing the performance of the adversary ensures there is little or no information in the representation about the sensitive variable. We demonstrate this adversarial approach on two problems: making decisions free from discrimination and removing private information from images. We formulate the adversarial model as a minimax problem, and optimize that minimax objective using a stochastic gradient alternate min-max optimizer. We demonstrate the ability to provide discriminant free representations for standard test problems, and compare with previous state of the art methods for fairness, showing statistically significant improvement across most cases. The flexibility of this method is shown via a novel problem: removing annotations from images, from separate training examples of annotated and unannotated images, and with no a priori knowledge of the form of annotation provided to the model.

Long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997) can solve numerous tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset. Without resets, the state may grow indefinitely and eventually cause the network to break down. Our remedy is a novel, adaptive forget gate that enables an LSTM cell to learn to reset itself at appropriate times, thus releasing internal resources. We review illustrative benchmark problems on which standard LSTM outperforms other RNN algorithms. All algorithms (including LSTM) fail to solve continual versions of these problems. LSTM with forget gates, however, easily solves them, and in an elegant way.

Many learning algorithms generate complex models that are difficult for a human to interpret, debug, and extend. In this paper, we address this challenge by proposing a new learning paradigm called correctable learning, where the learning algorithm receives external feedback about which data examples are incorrectly learned. We define a set of metrics which measure the correctability of a learning algorithm. We then propose a simple and efficient correctable learning algorithm which learns local models for different regions of the data space. Given an incorrect example, our method samples data in the neighborhood of that example and learns a new, more correct local model over that region. Experiments over multiple classification and ranking datasets show that our correctable learning algorithm offers significant improvements over the state-of-the-art techniques.


Question: Online learning strategies for adversaries influenced by memory: The cost of previous errors.

This paper delves into the methodology of averaging expert predictions as a strategy for aggregating advice from multiple experts in a sequential decision-making context. The focus is on online learning scenarios where the algorithm must generate predictions based on the advice of these experts and then adjust its strategy based on the outcomes of these predictions. The algorithm maintains a weight for each expert, which is used to compute a prediction that is a weighted average of the experts' predictions. The update of these weights is guided by the observed outcomes, with a decrease in weight corresponding to the loss incurred by an expert's prediction. This approach is an adaptation of Vovk's Aggregating Algorithm, which employs exponential weight decay based on loss, but simplifies the prediction step by using a linear combination of expert predictions instead of a more complex nonlinear function. We demonstrate that for a broad range of loss functions, the simplified algorithm's cumulative loss does not exceed that of the best-performing expert by more than a factor of \(c \ln n\), where \(n\) is the number of experts and \(c\) is a constant that varies with the loss function. This bound matches the form of known bounds for the original Aggregating Algorithm, albeit with slightly less favorable constants. To provide a more refined analysis, we leverage the concept of relative entropy to derive tighter bounds and to provide a theoretical foundation for the weight update mechanism. This methodological adjustment retains the essence of the Aggregating Algorithm while simplifying its implementation and analysis, making it a viable alternative for online learning in adversarial environments with memory.

In an earlier paper [9], we introduced a new “boosting” algorithm called AdaBoost which, theoretically, can be used to significantly reduce the error of any learning algorithm that consistently generates classifiers whose performance is a little better than random guessing. We also introduced the related notion of a “pseudo-loss” which is a method for forcing a learning algorithm of multi-label concepts to concentrate on the labels that are hardest to discriminate. In this paper, we describe experiments we carried out to assess how well AdaBoost with and without pseudo-loss, performs on real learning problems. We performed two sets of experiments. The first set compared boosting to Breiman’s [1] “bagging” method when used to aggregate various classifiers (including decision trees and single attribute-value tests). We compared the performance of the two methods on a collection of machine-learning benchmarks. In the second set of experiments, we studied in more detail the performance of boosting using a nearest-neighbor classifier on an OCR problem.

Consider the pricing decision for a manager at large online retailer, such as Amazon.com, that sells millions of products. The pricing manager must decide on real-time prices for each of these product. Due to the large number of products, the manager must set retail prices without complete demand information. A manager can run price experiments to learn about demand and maximize long run profits. There are two aspects that make the online retail pricing different from traditional brick and mortar settings. First, due to the number of products the manager must be able to automate pricing. Second, an online retailer can make frequent price changes. Pricing differs from other areas of online marketing where experimentation is common, such as online advertising or website design, as firms do not randomize prices to different customers at the same time. In this paper we propose a dynamic price experimentation policy where the firm has incomplete demand information. For this general setting, we derive a pricing algorithm that balances earning profit immediately and learning for future profits. The proposed approach marries statistical machine learning and economic theory. In particular we combine multi-armed bandit (MAB) algorithms with partial identification of consumer demand into a unique pricing policy. Our automated policy solves this problem using a scalable distribution-free algorithm. We show that our method converges to the optimal price faster than standard machine learning MAB solutions to the problem. In a series of Monte Carlo simulations, we show that the proposed approach perform favorably compared to methods in computer science and revenue management. ∗Ross School of Business, University of Michigan. Contact: ericmsch@umich.edu †Rady School of Management, University of California, San Diego. Contact: kamisra@ucsd.edu ‡Department of Electrical Engineering and Computer Science, University of Michigan. Contact: jabernet@umich.edu

This paper describes the Q-routing algorithm for packet routing, in which a reinforcement learning module is embedded into each node of a switching network. Only local communication is used by each node to keep accurate statistics on which routing decisions lead to minimal delivery times. In simple experiments involving a 36-node, irregularly connected network, Q-routing proves superior to a nonadaptive algorithm based on precomputed shortest paths and is able to route e ciently even when critical aspects of the simulation, such as the network load, are allowed to vary dynamically. The paper concludes with a discussion of the tradeo between discovering shortcuts and maintaining stable policies.

Q-learning in single-agent environments is known to converge in the limit given sufficient exploration. The same algorithm has been applied, with some success, in multiagent environments, where traditional analysis techniques break down. Using established dynamical systems methods, we derive and study an idealization of Q-learning in 2-player 2-action repeated general-sum games. In particular, we address the discontinuous case of -greedy exploration and use it as a proxy for value-based algorithms to highlight a contrast with existing results in policy search. Analogously to previous results for gradient ascent algorithms, we provide a complete catalog of the convergence behavior of the -greedy Q-learning algorithm by introducing new subclasses of these games. We identify two subclasses of Prisoner’s Dilemma-like games where the application of Q-learning with -greedy exploration results in higher-than-Nash average payoffs for some initial conditions.

After the recent groundbreaking results of AlphaGo and AlphaZero, we have seen strong interests in deep reinforcement learning and artificial general intelligence (AGI) in game playing. However, deep learning is resource-intensive and the theory is not yet well developed. For small games, simple classical table-based Q-learning might still be the algorithm of choice. General Game Playing (GGP) provides a good testbed for reinforcement learning to research AGI. Q-learning is one of the canonical reinforcement learning methods, and has been used by (Banerjee & Stone, IJCAI 2007) in GGP. In this paper we implement Q-learning in GGP for three small-board games (Tic-Tac-Toe, Connect Four, Hex), to allow comparison to Banerjee et al.. We find that Q-learning converges to a high win rate in GGP. For the -greedy strategy, we propose a first enhancement, the dynamic algorithm. In addition, inspired by (Gelly & Silver, ICML 2007) we combine online search (Monte Carlo Search) to enhance offline learning, and propose QM-learning for GGP. Both enhancements improve the performance of classical Q-learning. In this work, GGP allows us to show, if augmented by appropriate enhancements, that classical table-based Q-learning can perform well in small games.

In pattern recognition problems it has been noted that beyond a certain point the inclusion of additional parameters (that have been estimated) leads to higher probabilities of error. A simple problem has been formulated where the probability of error approaches zero as the dimensionality increases and all the parameters are known; on the other hand, the probability of error approaches one-half as the dimensionality increases and parameters are estimated.

Online learning algorithms often have to operate in the presence of concept drifts. A recent study revealed that different diversity levels in an ensemble of learning machines are required in order to maintain high generalization on both old and new concepts. Inspired by this study and based on a further study of diversity with different strategies to deal with drifts, we propose a new online ensemble learning approach called Diversity for Dealing with Drifts (DDD). DDD maintains ensembles with different diversity levels and is able to attain better accuracy than other approaches. Furthermore, it is very robust, outperforming other drift handling approaches in terms of accuracy when there are false positive drift detections. In all the experimental comparisons we have carried out, DDD always performed at least as well as other drift handling approaches under various conditions, with very few exceptions.

Multilayer connectionist models of memory based on the encoder model using the backpropagation learning rule are evaluated. The models are applied to standard recognition memory procedures in which items are studied sequentially and then tested for retention. Sequential learning in these models leads to 2 major problems. First, well-learned information is forgotten rapidly as new information is learned. Second, discrimination between studied items and new items either decreases or is nonmonotonic as a function of learning. To address these problems, manipulations of the network within the multilayer model and several variants of the multilayer model were examined, including a model with prelearned memory and a context model, but none solved the problems. The problems discussed provide limitations on connectionist models applied to human memory and in tasks where information to be learned is not all available during learning.

We address the question of when a network can be expected to generalize from m random training examples chosen from some arbitrary probability distribution, assuming that future test examples are drawn from the same distribution. Among our results are the following bounds on appropriate sample vs. network size. Assume 0 < ∊ 1/8. We show that if m O(W/∊ log N/∊) random examples can be loaded on a feedforward network of linear threshold functions with N nodes and W weights, so that at least a fraction 1 ∊/2 of the examples are correctly classified, then one has confidence approaching certainty that the network will correctly classify a fraction 1 ∊ of future test examples drawn from the same distribution. Conversely, for fully-connected feedforward nets with one hidden layer, any learning algorithm using fewer than (W/∊) random training examples will, for some distributions of examples consistent with an appropriate weight choice, fail at least some fixed fraction of the time to find a weight choice that will correctly classify more than a 1 ∊ fraction of the future test examples.

Most recommender systems recommend a list of items. The user examines the list, from the first item to the last, and often chooses the first attractive item and does not examine the rest. This type of user behavior can be modeled by the cascade model. In this work, we study cascading bandits, an online learning variant of the cascade model where the goal is to recommend K most attractive items from a large set of L candidate items. We propose two algorithms for solving this problem, which are based on the idea of linear generalization. The key idea in our solutions is that we learn a predictor of the attraction probabilities of items from their features, as opposing to learning the attraction probability of each item independently as in the existing work. This results in practical learning algorithms whose regret does not depend on the number of items L. We bound the regret of one algorithm and comprehensively evaluate the other on a range of recommendation problems. The algorithm performs well and outperforms all baselines.


Question: Online learning strategies for adversaries with memory: assessing the cost of previous errors.

This document presents a novel framework for online convex programming, a method aimed at minimizing a convex function over a predefined convex set, F ⊆ R. In contrast to traditional convex programming, where the cost function is known beforehand, online convex programming requires decision-making under uncertainty. At each iteration of the process, a point in F must be selected prior to the revelation of the cost function for that particular step. This scenario aptly mirrors real-world optimization challenges found in various sectors, such as manufacturing, agriculture, and broader industrial applications, where the true value of outputs is only discerned post-production. A key contribution of this work is the introduction of an algorithm tailored for online convex programming. This algorithm is further contextualized within the realm of repeated games, demonstrating its versatility and adaptability. Notably, the proposed algorithm is revealed to be a generalized form of infinitesimal gradient ascent, a technique widely recognized for its effectiveness in optimization problems. The findings underscore the universal consistency of generalized infinitesimal gradient ascent (GIGA), thereby extending its applicability across a spectrum of optimization scenarios.

Structured learning with inexact inference is a fundamental problem. We propose variants of the structured perceptron algorithm under a general “violation-fixing” framework that guarantees convergence. This framework subsumes previous remedies including “early update” as special cases, and also explains why standard perceptron may fail with inexact search. We also propose new update methods within this framework which learn better models with dramatically reduced training times on state-of-the-art part-of-speech tagging and incremental parsing systems.

We propose a variant of temporal-di!erence learning that approximates average and di!erential costs of an irreducible aperiodic Markov chain. Approximations are comprised of linear combinations of "xed basis functions whose weights are incrementally updated during a single endless trajectory of the Markov chain. We present a proof of convergence (with probability 1) and a characterization of the limit of convergence. We also provide a bound on the resulting approximation error that exhibits an interesting dependence on the `mixing timea of the Markov chain. The results parallel previous work by the authors, involving approximations of discounted cost-to-go. ! 1999 Elsevier Science Ltd. All rights reserved.

Recent studies of probabilistic reasoning have postulated general-purpose inference algorithms that can be used to answer arbitrary queries. These algorithms are memoryless, in the sense that each query is processed independently, without reuse of earlier computation. We argue that the brain operates in the setting of amortized inference, where numerous related queries must be answered (e.g., recognizing a scene from multiple viewpoints); in this setting, memoryless algorithms can be computationally wasteful. We propose a simple form of flexible reuse, according to which shared inferences are cached and composed together to answer new queries. We present experimental evidence that humans exploit this form of reuse: the answer to a complex query can be systematically predicted from a person’s response to a simpler query if the simpler query was presented first and entails a sub-inference (i.e., a sub-component of the more complex query). People are also faster at answering a complex query when it is preceded by a sub-inference. Our results suggest that the astonishing efficiency of human probabilistic reasoning may be supported by interactions between inference and memory.

Poker is a game with incomplete and imperfect information. The ability to estimate opponent and interpret its actions makes a player as a world class player. Finding optimal game strategy is not enough to win poker game. As in real life as in online poker game, the most time of it consists of opponent analysis. This paper illustrates a development of poker agent using reinforcement learning with neural networks. 1 STAGE OF THE RESEARCH Poker is a game with incomplete and imperfect information. The ability to estimate opponent and interpret its actions makes a player as a world class player. Finding optimal game strategy is not enough to win poker game. As in real life as in online poker game, the most time of it consists of opponent analysis. Author is working on development of poker agent that would find optimal game strategy using reinforcement learning (RL) in combination with artificial neural network (ANN) for value function approximation. 2 OUTLINE OF OBJECTIVES This paper illustrates a development of poker agent using reinforcement learning with neural networks. Complete poker agent should have an ability to create optimal game strategy that makes decisions based on information:  Hand strength/potential estimation;  Table card estimation;  Opponent hand strength prediction;  Opponent classification (tight/loose passive/ aggressive);  Current state evaluation using neural network. AI Poker agent should be able to find an optimal strategy by itself (unsupervised learning) using information given above. It also should be able to adapt opponent play style and change its strategy during the game.

Model-based control strategies for robot manipulators can present numerous performance advantages when an accurate model of the system dynamics is available. In practice, obtaining such a model is a challenging task which involves modeling such physical processes as friction, which may not be well understood and difficult to model. Furthermore, uncertainties in the physical parameters of a system may be introduced from significant discrepancies between the manufacturer data and the actual system. Traditionally, adaptive and robust control strategies have been developed to deal with parametric uncertainty in the dynamic model, but often require knowledge of the structure of the dynamics. Recent approaches to model-based manipulator control involve data-driven learning of the inverse dynamics relationship, eliminating the need for any a-priori knowledge of the system model. Locally Weighted Projection Regression (LWPR) has been proposed for learning the inverse dynamics function of a manipulator. Due to its use of simple local, linear models, LWPR is suitable for online and incremental learning. Although global regression techniques such as Gaussian Process Regression (GPR) have been shown to outperform LWPR in terms of accuracy, due to its heavy computational requirements, GPR has been applied mainly to offline learning of inverse dynamics. More recent efforts in making GPR computationally tractable for real-time control have resulted in several approximations which operate on a select subset, or sparse representation of the entire training data set. Despite the significant advancements that have been made in the area of learning control, there has not been much work in recent years to evaluate these newer regression techniques against traditional model-based control strategies such as adaptive control. Hence, the first portion of this thesis provides a comparison between a fixed model-based control strategy, an adaptive controller and the LWPR-based learning controller. Simulations are carried out in order to evaluate the position and orientation tracking performance of each controller under varied end effector loading, velocities and inaccuracies in the known dynamic parameters. Both the adaptive controller and LWPR controller are shown to have comparable performance in the presence of parametric uncertainty. However, it is shown that the learning controller is unable to generalize well outside of the regions in which it has been trained. Hence, achieving good performance requires significant amounts of training in the anticipated region of operation. In addition to poor generalization performance, most learning controllers commence learning entirely from ‘scratch,’ making no use of any a-priori knowledge which may be available from the well-known rigid body dynamics (RBD) formulation. The second portion of this thesis develops two techniques for online, incremental learning algorithms which incorporate prior knowledge to improve generalization performance. First, prior knowledge

Machine learning algorithms, such as neural networks, create better predictive models when having access to larger datasets. In many domains, such as medicine and finance, each institute has only access to limited amounts of data, and creating larger datasets typically requires collaboration. However, there are privacy related constraints on these collaborations for legal, ethical, and competitive reasons. In this work, we present a feasible protocol for learning neural networks in a collaborative way while preserving the privacy of each record. This is achieved by combining Differential Privacy and Secure Multi-Party Computation with Machine Learning.

The back-propagation algorithm is widely used for learning in artificial neural networks. A challenge in machine learning is to create models that generalize to new data samples not seen in the training data. Recently, a common flaw in several machine learning algorithms was discovered: small perturbations added to the input data lead to consistent misclassification of data samples. Samples that easily mislead the model are called adversarial examples. Training a ”maxout” network on adversarial examples has shown to decrease this vulnerability, but also increase classification performance. This paper shows that adversarial training has a regularizing effect also in networks with logistic, hyperbolic tangent and rectified linear units. A simple extension to the back-propagation method is proposed, that adds an adversarial gradient to the training. The extension requires an additional forward and backward pass to calculate a modified input sample, or mini batch, used as input for standard back-propagation learning. The first experimental results on MNIST show that the ”adversarial back-propagation” method increases the resistance to adversarial examples and boosts the classification performance. The extension reduces the classification error on the permutation invariant MNIST from 1.60% to 0.95% in a logistic network, and from 1.40% to 0.78% in a network with rectified linear units. Based on these promising results, adversarial back-propagation is proposed as a stand-alone regularizing method that should be further investigated.

Most policy search algorithms require thousands of training episodes to find an effective policy, which is often infeasible with a physical robot. This survey article focuses on the extreme other end of the spectrum: how can a robot adapt with only a handful of trials (a dozen) and a few minutes? By analogy with the word “big-data”, we refer to this challenge as “micro-data reinforcement learning”. We show that a first strategy is to leverage prior knowledge on the policy structure (e.g., dynamic movement primitives), on the policy parameters (e.g., demonstrations), or on the dynamics (e.g., simulators). A second strategy is to create data-driven surrogate models of the expected reward (e.g., Bayesian optimization) or the dynamical model (e.g., model-based policy search), so that the policy optimizer queries the model instead of the real system. Overall, all successful micro-data algorithms combine these two strategies by varying the kind of model and prior knowledge. The current scientific challenges essentially revolve around scaling up to complex robots (e.g., humanoids), designing generic priors, and optimizing the computing time.

Coco (“cooperative/competitive”) values are a solution concept for two-player normalform games with transferable utility, when binding agreements and side payments between players are possible. In this paper, we show that coco values can also be defined for stochastic games and can be learned using a simple variant of Q-learning that is provably convergent. We provide a set of examples showing how the strategies learned by the Coco-Q algorithm relate to those learned by existing multiagent Q-learning algorithms.

Convex empirical risk minimization is a basic tool in machine learning and statistics. We provide new algorithms and matching lower bounds for differentially private convex empirical risk minimization assuming only that each data point's contribution to the loss function is Lipschitz and that the domain of optimization is bounded. We provide a separate set of algorithms and matching lower bounds for the setting in which the loss functions are known to also be strongly convex. Our algorithms run in polynomial time, and in some cases even match the optimal nonprivate running time (as measured by oracle complexity). We give separate algorithms (and lower bounds) for (ε, 0)and (ε, δ)-differential privacy; perhaps surprisingly, the techniques used for designing optimal algorithms in the two cases are completely different. Our lower bounds apply even to very simple, smooth function families, such as linear and quadratic functions. This implies that algorithms from previous work can be used to obtain optimal error rates, under the additional assumption that the contributions of each data point to the loss function is smooth. We show that simple approaches to smoothing arbitrary loss functions (in order to apply previous techniques) do not yield optimal error rates. In particular, optimal algorithms were not previously known for problems such as training support vector machines and the high-dimensional median.


Question: Online learning strategies for adversaries with memory: the cost of previous errors.

Advancing Skills Through Reading: A Cost-Effective Path to Knowledge

Acquire enhanced skills and insights in a time-efficient manner every day. It's not always feasible or cost-effective to travel extensively, seeking firsthand experiences and knowledge. Reading offers a viable alternative to enrich your understanding and expertise. Although direct experiences can be incredibly valuable, they often come at a significant financial cost. By immersing yourself in reading materials like "Prediction, Learning, and Games," you can maximize your learning potential while managing your budget effectively. This approach allows you to gain multiple benefits from a single investment of time and money, making it a smart strategy for personal and professional growth.

**Question**: A Comparative Study of Feature Selection Methods for Text Classification

**Documents**: A new feature selection method based on feature contribution for text classification
Feature selection is an important and challenging problem in text classification. In this paper, a new feature selection method based on feature contribution is proposed. First, a novel measure of feature contribution is defined, which takes into account the discriminative power of a feature as well as its relevance to the class. Then, a new feature selection algorithm is proposed, which selects features based on their contribution to the classification performance. The proposed method is evaluated on several benchmark datasets and compared with several state-of-the-art feature selection methods. Experimental results show that the proposed method outperforms the compared methods in terms of classification accuracy.

**Similar Documents**: An Innovative Feature Selection Technique for Enhanced Text Classification Accuracy

The process of feature selection in text classification is critical yet complex. This paper introduces an innovative feature selection technique that focuses on the concept of feature contribution. A new metric for assessing feature contribution is developed, which integrates both the discriminative capacity of a feature and its relevance to the class. Subsequently, a novel feature selection algorithm is proposed, which prioritizes features based on their contribution to the overall classification performance. The efficacy of this method is evaluated using various standard datasets and compared against leading feature selection techniques. The empirical findings confirm that the proposed method significantly surpasses existing methods in terms of classification accuracy, highlighting its potential for improving text classification outcomes.

Online portfolio selection has attracted increasing attention from data mining and machine learning communities in recent years. An important theory in financial markets is mean reversion, which plays a critical role in some state-of-the-art portfolio selection strategies. Although existing mean reversion strategies have been shown to achieve good empirical performance on certain datasets, they seldom carefully deal with noise and outliers in the data, leading to suboptimal portfolios, and consequently yielding poor performance in practice. In this paper, we propose to exploit the reversion phenomenon by using robust <inline-formula><tex-math notation="LaTeX"> $L_1$</tex-math><alternatives><inline-graphic xlink:type="simple" xlink:href="huang-ieq1-2563433.gif"/></alternatives> </inline-formula>-median estimators, and design a novel online portfolio selection strategy named “Robust Median Reversion” (RMR), which constructs optimal portfolios based on the improved reversion estimator. We examine the performance of the proposed algorithms on various real markets with extensive experiments. Empirical results show that RMR can overcome the drawbacks of existing mean reversion algorithms and achieve significantly better results. Finally, RMR runs in linear time, and thus is suitable for large-scale real-time algorithmic trading applications.

T rapid advance in information technology now makes it feasible for sellers to condition their price offers on consumers’ prior purchase behavior. In this paper we examine when it is profitable to engage in this form of price discrimination when consumers can adopt strategies to protect their privacy. Our baseline model involves rational consumers with constant valuations for the goods being sold and a monopoly merchant who can commit to a pricing policy. Applying results from the prior literature, we show that although it is feasible to price so as to distinguish high-value and low-value consumers, the merchant will never find it optimal to do so. We then consider various generalizations of this model, such as allowing the seller to offer enhanced services to previous customers, making the merchant unable to commit to a pricing policy, and allowing competition in the marketplace. In these cases we show that sellers will, in general, find it profitable to condition prices on purchase history.

This paper addresses the repeated acquisition of labels for data items when the labeling is imperfect. We examine the improvement (or lack thereof) in data quality via repeated labeling, and focus especially on the improvement of training labels for supervised induction of predictive models. With the outsourcing of small tasks becoming easier, for example via Amazon’s Mechanical Turk, it often is possible to obtain less-than-expert labeling at low cost. With low-cost labeling, preparing the unlabeled part of the data can become considerably more expensive than labeling. We present repeated-labeling strategies of increasing complexity, and show several main results. (i) Repeated-labeling can improve label quality and model quality, but not always. (ii) When labels are noisy, repeated labeling can be preferable to single labeling even in the traditional setting where labels are not particularly cheap. (iii) As soon as the cost of processing the unlabeled data is not free, even the simple strategy of labeling everything multiple times can give considerable advantage. (iv) Repeatedly labeling a carefully chosen set of points is generally preferable, and we present a set of robust techniques that combine different notions of uncertainty to select data points for which quality should be improved. The bottom line: the results show clearly that when labeling is not perfect, selective acquisition of multiple labels is a strategy that data miners should have in their repertoire; for certain label-quality/cost regimes, the benefit is substantial.

Learning to learn has emerged as an important direction for achieving artificial intelligence. Two of the primary barriers to its adoption are an inability to scale to larger problems and a limited ability to generalize to new tasks. We introduce a learned gradient descent optimizer that generalizes well to new tasks, and which has significantly reduced memory and computation overhead. We achieve this by introducing a novel hierarchical RNN architecture, with minimal perparameter overhead, augmented with additional architectural features that mirror the known structure of optimization tasks. We also develop a meta-training ensemble of small, diverse optimization tasks capturing common properties of loss landscapes. The optimizer learns to outperform RMSProp/ADAM on problems in this corpus. More importantly, it performs comparably or better when applied to small convolutional neural networks, despite seeing no neural networks in its meta-training set. Finally, it generalizes to train Inception V3 and ResNet V2 architectures on the ImageNet dataset for thousands of steps, optimization problems that are of a vastly different scale than those it was trained on. We release an open source implementation of the meta-training algorithm.

Implementing E cient, Portable Computations for Machine Learning

A long-lived autonomous agent should be able to respond online to novel instances of tasks from a familiar domain. Acting online requires ‘fast’ responses, in terms of rapid convergence, especially when the task instance has a short duration such as in applications involving interactions with humans. These requirements can be problematic for many established methods for learning to act. In domains where the agent knows that the task instance is drawn from a family of related tasks, albeit without access to the label of any given instance, it can choose to act through a process of policy reuse from a library in contrast to policy learning. In policy reuse, the agent has prior experience from the class of tasks in the form of a library of policies that were learnt from sample task instances during an offline training phase. We formalise the problem of policy reuse and present an algorithm for efficiently responding to a novel task instance by reusing a policy from this library of existing policies, where the choice is based on observed ‘signals’ which correlate to policy performance. We achieve this by posing the problem as a Bayesian choice problem with a corresponding notion of an optimal response, but the computation of that response is in many cases intractable. Therefore, to reduce the computation cost of the posterior, we follow a Bayesian optimisation approach and define a set of policy selection functions, which balance exploration in the policy library against exploitation of previously tried policies, together with a model of expected performance of the policy library on their corresponding task instances. We validate our method in several simulated domains of interactive, short-duration episodic tasks, showing rapid convergence in unknown task variations.

Chapter 1 strongly advocates the stochastic back-propagation method to train neural networks. This is in fact an instance of a more general technique called stochastic gradient descent (SGD). This chapter provides background material, explains why SGD is a good learning algorithm when the training set is large, and provides useful recommendations.

We propose a general multiparty computation protocol secure against an active adversary corrupting up to n−1 of the n players. The protocol may be used to compute securely arithmetic circuits over any finite field Fpk . Our protocol consists of a preprocessing phase that is both independent of the function to be computed and of the inputs, and a much more efficient online phase where the actual computation takes place. The online phase is unconditionally secure and has total computational (and communication) complexity linear in n, the number of players, where earlier work was quadratic in n. Moreover, the work done by each player is only a small constant factor larger than what one would need to compute the circuit in the clear. We show this is optimal for computation in large fields. In practice, for 3 players, a secure 64-bit multiplication can be done in 0.05 ms. Our preprocessing is based on a somewhat homomorphic cryptosystem. We extend a scheme by Brakerski et al., so that we can perform distributed decryption and handle many values in parallel in one ciphertext. The computational complexity of our preprocessing phase is dominated by the public-key operations, we need O(n/s) operations per secure multiplication where s is a parameter that increases with the security parameter of the cryptosystem. Earlier work in this model needed Ω(n) operations. In practice, the preprocessing prepares a secure 64-bit multiplication for 3 players in about 13 ms.

Classifying neural signals is a crucial step in the brain-computer interface (BCI). Although Deep Neural Network (DNN) has been shown to be surprisingly good at classification, DNN suffers from long training time and catastrophic forgetting. Catastrophic forgetting refers to a phenomenon in which a DNN tends to forget previously learned task when it learns a new task. Here we argue that the solution to this problem may be found in the human brain, specifically, by combining functions of the two regions: the striatum and the hippocampus, which is pivotal for reinforcement learning and memory recall relevant to the current context, respectively. The mechanism of these brain regions provides insights into resolving catastrophic forgetting and long training time of DNNs. Referring to the hippocampus-striatum network we discuss design principles of combining different types of DNNs for building a new BCI architecture, called “Meta BCI”.

We present a simple, agnostic active learning algorithm that works for any hypothesis class of bounded VC dimension, and any data distribution. Our algorithm extends a scheme of Cohn, Atlas, and Ladner [6] to the agnostic setting, by (1) reformulating it using a reduction to supervised learning and (2) showing how to apply generalization bounds even for the non-i.i.d. samples that result from selective sampling. We provide a general characterization of the label complexity of our algorithm. This quantity is never more than the usual PAC sample complexity of supervised learning, and is exponentially smaller for some hypothesis classes and distributions. We also demonstrate improvements experimentally.