Question: A state-conscious black-box web vulnerability scanner: Enemy of the State.

This paper unveils a groundbreaking methodology for navigating AJAX-enabled web applications by deducing alterations in user interface states. AJAX, heralded for enhancing the interactivity and responsiveness of web applications, simultaneously dismantles the conventional notion of a web "page," a cornerstone for conventional search engine crawlers. Our innovative approach entails a meticulous analysis of AJAX applications, focusing on the dynamic identification and reconstruction of user interface state transitions. We have devised a technique that dynamically constructs a state-flow graph, a comprehensive model that encapsulates the myriad navigation pathways and states intrinsic to AJAX applications. This reconstructed model serves as a blueprint for generating interconnected static pages, which can then be indexed by traditional search engines, effectively bridging the gap between AJAX sites and general search capabilities. Beyond this primary application, our crawling methodology holds promise for various other domains, including enhancements in accessibility, automated user interface testing, and exercising all interactive elements within AJAX applications on a state-by-state basis. To operationalize these concepts, we have developed an open-source tool, CRAWLJAX, which embodies the methodologies outlined in this paper. Our research is further substantiated by a practical case study, wherein we apply our approach to a selection of representative AJAX applications, and provide a detailed analysis of the outcomes, underscoring the efficacy and potential of our methodology.

Recent work on cache attacks has shown that CPU caches represent a powerful source of information leakage. However, existing attacks require manual identification of vulnerabilities, i.e., data accesses or instruction execution depending on secret information. In this paper, we present Cache Template Attacks. This generic attack technique allows us to profile and exploit cachebased information leakage of any program automatically, without prior knowledge of specific software versions or even specific system information. Cache Template Attacks can be executed online on a remote system without any prior offline computations or measurements. Cache Template Attacks consist of two phases. In the profiling phase, we determine dependencies between the processing of secret information, e.g., specific key inputs or private keys of cryptographic primitives, and specific cache accesses. In the exploitation phase, we derive the secret values based on observed cache accesses. We illustrate the power of the presented approach in several attacks, but also in a useful application for developers. Among the presented attacks is the application of Cache Template Attacks to infer keystrokes and—even more severe—the identification of specific keys on Linux and Windows user interfaces. More specifically, for lowercase only passwords, we can reduce the entropy per character from log2(26) = 4.7 to 1.4 bits on Linux systems. Furthermore, we perform an automated attack on the Ttable-based AES implementation of OpenSSL that is as efficient as state-of-the-art manual cache attacks.

We introduce a problem class which we term activity monitoring. Such problems involve monitoring the behavior of a large population of entities for interesting events requiring action. We present a framework within which each of the individual problems has a natural expression, as well as a methodology for evaluating performance of activity monitoring techniques. We show that two superficially different tasks, news story monitoring and intrusion detection, can be expressed naturally within the framework, and show that key differences in solution methods can be compared.

The steady evolution of the Web has paved the way for miscreants to take advantage of vulnerabilities to embed malicious content into web pages. Up on a visit, malicious web pages steal sensitive data, redirect victims to other malicious targets, or cease control of victim's system to mount future attacks. Approaches to detect malicious web pages have been reactively effective at special classes of attacks like drive-by-downloads. However, the prevalence and complexity of attacks by malicious web pages is still worrisome. The main challenges in this problem domain are (1) fine-grained capturing and characterization of attack payloads (2) evolution of web page artifacts and (3) exibility and scalability of detection techniques with a fast-changing threat landscape. To this end, we proposed a holistic approach that leverages static analysis, dynamic analysis, machine learning, and evolutionary searching and optimization to effectively analyze and detect malicious web pages. We do so by: introducing novel features to capture fine-grained snapshot of malicious web pages, holistic characterization of malicious web pages, and application of evolutionary techniques to fine-tune learning-based detection models pertinent to evolution of attack payloads. In this paper, we present key intuition and details of our approach, results obtained so far, and future work.

Applications for mobile platforms are being developed at a tremendous rate, but often without proper security implementation. Insecure mobile applications can cause serious information security and data privacy issues and can have severe repercussions on users and organizations alike.

Research and industrial experience reveal that code reviews as a part of software inspection might be the most cost-effective technique a team can use to reduce defects. Tools that automate code inspection mostly focus on the detection of a priori known defect patterns and security vulnerabilities. Automated detection of logical errors, due to a faulty implementation of applications’ functionality is a relatively uncharted territory. Automation can be based on profiling the intended behavior behind the source code. In this paper, we present a new code profiling method that combines an information flow analysis, the crosschecking of dynamic invariants with symbolic execution, and the use of fuzzy logic. Our goal is to detect logical errors and exploitable vulnerabilities. The theoretical underpinnings and the practical implementation of our approach are discussed. We test the APP_LogGIC tool that implements the proposed analysis on two realworld applications. The results show that profiling the intended program behavior is feasible in diverse applications. We discuss the heuristics used to overcome the problem of state space explosion and of the large data sets. Code metrics and test results are provided to demonstrate the effectiveness of the approach.

Nowadays, it is hard to find a popular Web site with a registration form that is not protected by an automated human proof test which displays a sequence of characters in an image, and requests the user to enter the sequence into an input field. This security mechanism is based on the Turing Test—one of the oldest concepts in Artificial Intelligence—and it is most often called Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA). This kind of test has been conceived to prevent the automated access to an important Web resource, for example, a Web mail service or a Social Network. There are currently hundreds of these tests, which are served millions of times a day, thus involving a huge amount of human work. On the other side, a number of these tests have been broken, that is, automated programs designed by researchers, hackers, and spammers have been able to automatically serve the correct answer. In this chapter, we present the history and the concept of CAPTCHAs, along with their applications and a wide review of their instantiations. We also discuss their evaluation, both from the user and the security perspectives, including usability, attacks, and countermeasures. We expect this chapter provides to the reader a good overview of this interesting field. CES IN COMPUTERS, VOL. 83 109 Copyright © 2011 Elsevier Inc. 65-2458/DOI: 10.1016/B978-0-12-385510-7.00003-5 All rights reserved. 110 J.M. GOMEZ HIDALGO AND G. ALVAREZ MARAÑON 1. I ntroduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 1 .1. T he Turing Test and the Origin of CAPTCHAs . . . . . . . . . . . . . . . 111 2. M otivation and Applications . . . . . . . . . . . . . . . . . . . . . . . . 116 2 .1. G eneral Description of CAPTCHAs . . . . . . . . . . . . . . . . . . . . . 116 2 .2. D esirable Properties of CAPTCHAs . . . . . . . . . . . . . . . . . . . . . 117 2 .3. I m plementation and Deployment . . . . . . . . . . . . . . . . . . . . . . 119 2 .4. A pplications and the Rise of the Robots . . . . . . . . . . . . . . . . . . . 121 3. T ypes of CAPTCHAs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 3 .1. O CR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 3 .2. I m age . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 3 .3. A udio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 3 .4. C ognitive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 4. E valuation of CAPTCHAs . . . . . . . . . . . . . . . . . . . . . . . . . 146 4 .1. E fficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 4 .2. A ccessibility Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 4 .3. P ractical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 5. S ecurity and Attacks on CAPTCHAs . . . . . . . . . . . . . . . . . . . . 156 5 .1. A ttacks on CAPTCHAs . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 5 .2. S ecurity Requirements on CAPTCHAs . . . . . . . . . . . . . . . . . . . 169 6. A lternatives to CAPTCHAs . . . . . . . . . . . . . . . . . . . . . . . . . 171 7. C onclusions and Future Trends . . . . . . . . . . . . . . . . . . . . . . . 173 R eferences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

Memory-based vulnerabilities are a major source of attack vectors. They allow attackers to gain unauthorized access to computers and their data. Previous research has made significant progress in detecting attacks. However, developers still need to locate and fix these vulnerabilities, a mostly manual and time-consuming process. They face a number of challenges. Particularly, the manifestation of an attack does not always coincide with the exploited vulnerabilities, and many attacks are hard to reproduce in the lab environment, leaving developers with limited information to locate them. In this paper, we propose Ravel, an architectural approach to pinpoint vulnerabilities from attacks. Ravel consists of an online attack detector and an offline vulnerability locator linked by a record & replay mechanism. Specifically, Ravel records the execution of a production system and simultaneously monitors it for attacks. If an attack is detected, the execution is replayed to reveal the targeted vulnerabilities by analyzing the program's memory access patterns under attack. We have built a prototype of Ravel based on the open-source FreeBSD operating system. The evaluation results in security and performance demonstrate that Ravel can effectively pinpoint various types of memory vulnerabilities and has low performance overhead.

Buffer overflow is one of the most common types of software security vulnerabilities. Although researchers have proposed various static and dynamic techniques for buffer overflow detection, buffer overflow attacks against both legacy and newly-deployed software systems are still quite prevalent. Compared with dynamic detection techniques, static techniques are more systematic and scalable. However, there are few studies on the effectiveness of state-of-the-art static buffer overflow detection techniques. In this paper, we perform an in-depth quantitative and qualitative study on static buffer overflow detection. More specifically, we obtain both the buggy and fixed versions of 100 buffer overflow bugs from 63 real-world projects totalling 28 MLoC (Millions of Lines of Code) based on the reports in Common Vulnerabilities and Exposures (CVE). Then, quantitatively, we apply Fortify, Checkmarx, and Splint to all the buggy versions to investigate their false negatives, and also apply them to all the fixed versions to investigate their false positives. We also qualitatively investigate the causes for the false-negatives and false-positives of studied techniques to guide the design and implementation of more advanced buffer overflow detection techniques. Finally, we also categorized the patterns of manual buffer overflow repair actions to guide automated repair techniques for buffer overflow. The experiment data is available at http://bo-study.github.io/Buffer-Overflow-Cases/.

Besides a large set of malware categories such as worms and Trojan horses, Advanced Persistent Threat (APT) is another more sophisticated attack entity emerging in the cyber threats environment. In this paper we propose a model of the APT detection problem as well as a methodology to implement it on a generic organization network. From our knowledge, the proposed method is the first to address the problem of modeling an APT and to provide a possible detection framework.

The continued growth and diversification of the Internet has been accompanied by an increasing prevalence of attacks and intrusions [40]. It can be argued, however, that a significant change in motivation for malicious activi ty has taken place over the past several years: from vandalism and recognition in th e hacker community, to attacks and intrusions for financial gain. This shift has bee n marked by a growing sophistication in the tools and methods used to conduct atta cks, thereby escalating the network security arms race. Our thesis is that the reactivemethods for network security that are predominant today are ultimately insufficient and that more proactivemethods are required. One such approach is to develop a foundational understanding of the mechanisms employed by malicious software (malware) which is often readi ly available in source form on the Internet. While it is well known that large IT secu rity companies maintain detailed databases of this information, these are not o penly available and we are not aware of any such open repository. In this paper we begin t he process of codifying the capabilities of malware by dissecting four widely-u sed Internet Relay Chat (IRC) botnet codebases. Each codebase is classified along se ven k y dimensions including botnet control mechanisms, host control mechani sms, propagation mechanisms, exploits, delivery mechanisms, obfuscation and de ception mechanisms. Our study reveals the complexity of botnet software, and we disc us es implications for defense strategies based on our analysis.


Question: A state-conscious black-box web vulnerability scanner: Enemy of the State.

This paper introduces an innovative framework for augmenting conventional type systems with flow-sensitive type qualifiers, enabling developers to annotate their code with these qualifiers for enhanced correctness verification. The framework's inference mechanism ensures the accuracy of these annotations without altering the underlying base types, ensuring efficiency through a constraint-based algorithm that seamlessly integrates flow-insensitive alias analysis, effect inference, and concepts from linear type systems to accommodate strong updates. To showcase the efficacy of flow-sensitive type qualifiers, the paper identifies a series of previously undetected locking anomalies within the Linux kernel, highlighting the framework's potential for uncovering subtle concurrency issues.

**Question**: A Comparative Study of Different Approaches to Solve the Traveling Salesman Problem

**Documents**: An Efficient Hybrid Algorithm for the Traveling Salesman Problem
In this paper, we propose a hybrid algorithm combining the strengths of the genetic algorithm and the local search algorithm for solving the traveling salesman problem (TSP). The genetic algorithm is used to generate the initial population and evolve the solution, while the local search algorithm is employed to refine the solutions by exploring the neighborhood of each solution to find better ones. The proposed hybrid algorithm is tested on a set of benchmark TSP instances, and the results show that it outperforms the genetic algorithm and the local search algorithm individually in terms of solution quality and convergence speed.

**Similar Documents**: This paper presents a novel hybrid algorithm that integrates the advantages of genetic algorithms and local search techniques to tackle the traveling salesman problem (TSP) effectively. The genetic algorithm component is utilized for the generation of the initial population and the evolution of solutions, while the local search component refines these solutions by exploring the vicinity of each solution to identify superior alternatives. The hybrid algorithm's performance is evaluated against a suite of standard TSP benchmarks, demonstrating its superiority over standalone genetic algorithms and local search algorithms in both solution quality and the rate of convergence.

Virtual machine introspection (VMI) describes the method of monitoring and analyzing the state of a virtual machine from the hypervisor level. This lends itself well to security applications, though the hardware virtualization support from Intel and AMD was not designed with VMI in mind. This results in many challenges for developers of hardware-supported VMI systems. This paper describes the design and implementation of our prototype framework, Nitro, for system call tracing and monitoring. Since Nitro is a purely VMI-based system, it remains isolated from attacks originating within the guest operating system and is not directly visible from within the guest. Nitro is extremely flexible as it supports all three system call mechanisms provided by the Intel x86 architecture and has been proven to work in Windows, Linux, 32-bit, and 64-bit environments. The high performance of our system allows for real-time capturing and dissemination of data without hindering usability. This is supported by extensive testing with various guest operating systems. In addition, Nitro is resistant to circumvention attempts due to a construction called hardware rooting. Finally, Nitro surpasses similar systems in both performance and functionality.

To address the growing concerns about the security of embedded systems, it is important to perform accurate analysis of firmware binaries, even when the source code or the hardware documentation are not available. However, research in this field is hindered by the lack of dedicated tools. For example, dynamic analysis is one of the main foundations of security analysis, e.g., through dynamic taint tracing or symbolic execution. Unlike static analysis, dynamic analysis relies on the ability to execute software in a controlled environment, often an instrumented emulator. However, emulating firmwares of embedded devices requires accurate models of all hardware components used by the system under analysis. Unfortunately, the lack of documentation and the large variety of hardware on the market make this approach infeasible in practice. In this paper we present Avatar, a framework that enables complex dynamic analysis of embedded devices by orchestrating the execution of an emulator together with the real hardware. We first introduce the basic mechanism to forward I/O accesses from the emulator to the embedded device, and then describe several techniques to improve the system’s performance by dynamically optimizing the distribution of code and data between the two environments. Finally, we evaluate our tool by applying it to three different security scenarios, including reverse engineering, vulnerability discovery and hardcoded backdoor detection. To show the flexibility of Avatar, we perform this analysis on three completely different devices: a GSM feature phone, a hard disk bootloader, and a wireless sensor node.

This paper sheds light on crucial weaknesses in the design of hidden services that allow us to break the anonymity of hidden service clients and operators passively. In particular, we show that the circuits, paths established through the Tor network, used to communicate with hidden services exhibit a very different behavior compared to a general circuit. We propose two attacks, under two slightly different threat models, that could identify a hidden service client or operator using these weaknesses. We found that we can identify the users’ involvement with hidden services with more than 98% true positive rate and less than 0.1% false positive rate with the first attack, and 99% true positive rate and 0.07% false positive rate with the second. We then revisit the threat model of previous website fingerprinting attacks, and show that previous results are directly applicable, with greater efficiency, in the realm of hidden services. Indeed, we show that we can correctly determine which of the 50 monitored pages the client is visiting with 88% true positive rate and false positive rate as low as 2.9%, and correctly deanonymize 50 monitored hidden service servers with true positive rate of 88% and false positive rate of 7.8% in an open world setting.

The continued growth and diversification of the Internet has been accompanied by an increasing prevalence of attacks and intrusions [40]. It can be argued, however, that a significant change in motivation for malicious activi ty has taken place over the past several years: from vandalism and recognition in th e hacker community, to attacks and intrusions for financial gain. This shift has bee n marked by a growing sophistication in the tools and methods used to conduct atta cks, thereby escalating the network security arms race. Our thesis is that the reactivemethods for network security that are predominant today are ultimately insufficient and that more proactivemethods are required. One such approach is to develop a foundational understanding of the mechanisms employed by malicious software (malware) which is often readi ly available in source form on the Internet. While it is well known that large IT secu rity companies maintain detailed databases of this information, these are not o penly available and we are not aware of any such open repository. In this paper we begin t he process of codifying the capabilities of malware by dissecting four widely-u sed Internet Relay Chat (IRC) botnet codebases. Each codebase is classified along se ven k y dimensions including botnet control mechanisms, host control mechani sms, propagation mechanisms, exploits, delivery mechanisms, obfuscation and de ception mechanisms. Our study reveals the complexity of botnet software, and we disc us es implications for defense strategies based on our analysis.

Malware is notoriously difficult to combat because it appears and spreads so quickly. In this article, we describe the design and implementation of CWSandbox, a malware analysis tool that fulfills our three design criteria of automation, effectiveness, and correctness for the Win32 family of operating systems

The constant increase of attacks against networks and their resources (as recently shown by the CodeRed worm) causes a necessity to protect these valuable assets. Firewalls are now a common installation to repel intrusion attempts in the first place. Intrusion detection systems (IDS), which try to detect malicious activities instead of preventing them, offer additional protection when the first defense perimeter has been penetrated. ID systems attempt to pin down attacks by comparing collected data to predefined signatures known to be malicious (signature based) or to a model of legal behavior (anomaly based).Anomaly based systems have the advantage of being able to detect previously unknown attacks but they suffer from the difficulty to build a solid model of acceptable behavior and the high number of alarms caused by unusual but authorized activities. We present an approach that utilizes application specific knowledge of the network services that should be protected. This information helps to extend current, simple network traffic models to form an application model that allows to detect malicious content hidden in single network packets. We describe the features of our proposed model and present experimental data that underlines the efficiency of our systems.

JavaScript malware-based attacks account for a large fraction of successful mass-scale exploitation happening today. Attackers like JavaScript-based attacks because they can be mounted against an unsuspecting user visiting a seemingly innocent web page. While several techniques for addressing these types of exploits have been proposed, in-browser adoption has been slow, in part because of the performance overhead these methods incur. In this paper, we propose ZOZZLE, a low-overhead solution for detecting and preventing JavaScript malware that is fast enough to be deployed in the browser. Our approach uses Bayesian classification of hierarchical features of the JavaScript abstract syntax tree to identify syntax elements that are highly predictive of malware. Our experimental evaluation shows that ZOZZLE is able to detect JavaScript malware through mostly static code analysis effectively. ZOZZLE has an extremely low false positive rate of 0.0003%, which is less than one in a quarter million. Despite this high accuracy, the ZOZZLE classifier is fast, with a throughput of over one megabyte of JavaScript code per second.

According to recent articles in popular technology websites, some mobile applications function in an insecure manner when presented with untrusted SSL certificates. These non-browser based applications seem to, in the absence of a standard way of alerting a user of an SSL error, accept any certificate presented to it. This paper intends to research these claims and show whether or not an invisible proxy based SSL attack can indeed steal user's credentials from mobile applications, and which types applications are most likely to be vulnerable to this attack vector. To ensure coverage of the most popular platforms, applications on both Android 4.2 and iOS 6 are tested. The results of our study showed that stealing credentials is indeed possible using invisible proxy man in the middle attacks.

T There are plenty of security software in market; each claiming the best, still we daily face problem of viruses and other malicious activities. If we know the basic working principal of such malware then we can very easily prevent most of them even without security software. Hackers and crackers are experts in psychology to manipulate people into giving them access or the information necessary to get access. This paper discusses the inner working of such attacks. Case study of Spyware is provided. In this case study, we got 100% success using social engineering techniques for deception on Linux operating system, which is considered as the most secure operating system. Few basic principal of defend, for the individual as well as for the organization, are discussed here, which will prevent most of such attack if followed.

The consideration of targeted security attacks is not yet common in the automotive domain where primarily safety requirements are considered. Hence this survey addresses the relatively new field of IT-security in automotive technology in order to motivate further research. With the emergence of automotive technologies like car-to-car (c2c) communication, the challenges increase. In order to show the necessity for such research we describe how in theory, a potential attack on the Electronic Throttle Control (ETC) could succeed, if no countermeasures are taken, with potentially drastic consequences on the safety of the vehicle, its occupants and its environment as a result. Also an attack on the power window system is shown using the implementation techniques already employed in the design of automotive components. Our goal is to show, that such exploited security vulnerabilities are likely to have serious consequences with respect to the vehicles safety. Therefore such attacks need to be considered and suitable countermeasures have to be identified. In order to achieve this, the connection of the two disciplines Safety and Security for automotive IT components needs to be discussed. We introduce two exemplary attacks, the Trojan Horse targeting at the Electronic Throttle Control System (ETC) and a replay attack on the electric window lift. The first attack is mainly discussed in theory, the second is elaborated in a practical simulation scenario. Finally we introduce a general investigation of potential ways of intrusion and present some first practical results from tests with current automotive hardware.


Question: A state-conscious black-box web vulnerability scanner: Enemy of the State.

This paper delves into the contrast between push and pull methodologies in AJAX, highlighting their distinct advantages and disadvantages within the context of real-time dynamic web data transmission. AJAX applications are crafted to offer users a highly interactive experience with minimal latency. To achieve this, they often need to disseminate real-time data, such as news updates, stock prices, and auction notifications, swiftly to users. However, the inherent limitations of the web's request/response model pose a challenge, as it does not support server-initiated data transmission. In response, AJAX applications typically resort to a pull mechanism, where the client periodically requests updates based on a set interval. An alternative approach is the push strategy, which allows the server to proactively transmit data when changes occur. Both methods come with their own set of trade-offs. This study examines the inherent constraints of browser-based applications and evaluates push-based solutions for AJAX. It also presents the findings of an empirical investigation contrasting the performance of push and pull strategies.

**Question**: Understanding the Impact of Online Reviews on Consumer Behavior

**Documents**: An Empirical Analysis of Online Product Reviews
Online product reviews have become an important source of information for consumers, influencing their purchase decisions. This paper presents an empirical analysis of online product reviews for a specific category of products, namely, books. We collected a dataset of book reviews from Amazon.com and analyzed the text of the reviews using natural language processing techniques. We found that the sentiment of the reviews, as measured by a sentiment analysis tool, has a significant impact on the sales rank of the books. We also found that the length of the reviews and the number of helpful votes received by the reviews are positively correlated with the sales rank. These findings suggest that online product reviews can significantly influence consumer behavior and sales.

**Similar Documents**: This study conducts a detailed investigation into the role of online product reviews in shaping consumer decisions, with a focus on the book category. Utilizing a comprehensive dataset sourced from Amazon.com, we apply advanced natural language processing methods to analyze the textual content of book reviews. Our results highlight the pivotal impact of review sentiment on the sales ranking of books. We discovered that positive sentiment, as quantified through sentiment analysis, is strongly correlated with higher sales rankings. Additionally, our analysis reveals that the length of reviews and the number of helpful votes they garner are also positively associated with sales ranking. These insights underscore the critical role that online product reviews play in guiding consumer behavior and influencing sales outcomes.

The importance of network security has grown tremendously and a collection of devices have been introduced, which can improve the security of a network. Network intrusion detection systems (NIDS) are among the most widely deployed such system; popular NIDS use a collection of signatures of known security threats and viruses, which are used to scan each packet's payload. Today, signatures are often specified as regular expressions; thus the core of the NIDS comprises of a regular expressions parser; such parsers are traditionally implemented as finite automata. Deterministic Finite Automata (DFA) are fast, therefore they are often desirable at high network link rates. DFA for the signatures, which are used in the current security devices, however require prohibitive amounts of memory, which limits their practical use.
 In this paper, we argue that the traditional DFA based NIDS has three main limitations: first they fail to exploit the fact that normal data streams rarely match any virus signature; second, DFAs are extremely inefficient in following multiple partially matching signatures and explodes in size, and third, finite automaton are incapable of efficiently keeping track of counts. We propose mechanisms to solve each of these drawbacks and demonstrate that our solutions can implement a NIDS much more securely and economically, and at the same time substantially improve the packet throughput.

A model of a real-time intrusion-detection expert system capable of detecting break-ins, penetrations, and other forms of computer abuse is described. The model is based on the hypothesis that security violations can be detected by monitoring a system's audit records for abnormal patterns of system usage. The model includes profiles for representing the behavior of subjects with respect to objects in terms of metrics and statistical models, and rules for acquiring knowledge about this behavior from audit records and for detecting anomalous behavior. The model is independent of any particular system, application environment, system vulnerability, or type of intrusion, thereby providing a framework for a general-purpose intrusion-detection expert system.

e emerging blockchain technology supports decentralized computing paradigm shi and is a rapidly approaching phenomenon. While blockchain is thought primarily as the basis of Bitcoin, its application has grown far beyond cryptocurrencies due to the introduction of smart contracts. Smart contracts are self-enforcing pieces of soware, which reside and run over a hosting blockchain. Using blockchain-based smart contracts for secure and transparent management to govern interactions (authentication, connection, and transaction) in Internet-enabled environments, mostly IoT, is a niche area of research and practice. However, writing trustworthy and safe smart contracts can be tremendously challenging because of the complicated semantics of underlying domain-specic languages and its testability. ere have been high-prole incidents that indicate blockchain smart contracts could contain various code-security vulnerabilities, instigating nancial harms. When it involves security of smart contracts, developers embracing the ability to write the contracts should be capable of testing their code, for diagnosing security vulnerabilities, before deploying them to the immutable environments on blockchains. However, there are only a handful of security testing tools for smart contracts. is implies that the existing research on automatic smart contracts security testing is not adequate and remains in a very stage of infancy. With a specic goal to more readily realize the application of blockchain smart contracts in security and privacy, we should rst understand their vulnerabilities before widespread implementation. Accordingly, the goal of this paper is to carry out a far-reaching experimental assessment of current static smart contracts security testing tools, for the most widely used blockchain, the Ethereum and its domain-specic programming language, Solidity, to provide the rst body of knowledge for creating more secure blockchainbased soware. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and the full citation on the rst page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

Attacks on host computers by malicious peripherals are a growing problem. Inexpensive and powerful peripherals, which attach to plug-and-play buses, have made such attacks easy to mount. Making matters worse, commodity operating systems lack systematic defenses, and users are often not aware of the scope of the problem. We present Cinch, a pragmatic response to this threat. Cinch uses virtualization to place the hardware in a logically separate, untrusted machine, and includes an interposition layer between the untrusted machine and the protected one. This layer accepts or rejects interaction with devices and enforces security policies that are easily configured and extended by users. We show that Cinch integrates with existing OSes, enforces policies that thwart real world attacks, and has low overhead.

To date, many attempts have been made to fingerprint users on the web. These fingerprints allow browsing sessions to be linked together and possibly even tied to a user’s identity. They can be used constructively by sites to supplement traditional means of user authentication such as passwords; and they can be used destructively to counter attempts to stay anonymous online. In this paper, we identify two new avenues for browser fingerprinting. The new fingerprints arise from the browser’s JavaScript execution characteristics, making them difficult to simulate or mitigate in practice. The first uses the innate performance signature of each browser’s JavaScript engine, allowing the detection of browser version, operating system and microarchitecture, even when traditional forms of system identification (such as the user-agent header) are modified or hidden. The second subverts the whitelist mechanism of the popular NoScript Firefox extension, which selectively enables web pages’ scripting privileges to increase privacy by allowing a site to determine if particular domains exist in a user’s NoScript whitelist. We have experimentally verified the effectiveness of our system fingerprinting technique using a 1,015-person study on Amazon’s Mechanical Turk platform.

Combinatorial Interaction Testing (CIT) is important because it tests the interactions between the many parameters that make up the configuration space of software systems. We apply this testing paradigm to a Python-based framework for rapid development of webbased applications called Django. In particular, we automatically create a CIT model for Django website configurations and run a state-of-the-art tool for CIT test suite generation to obtain sets of test configurations. Our automatic CIT-based approach is able to efficiently detect invalid configurations.

Recent work on cache attacks has shown that CPU caches represent a powerful source of information leakage. However, existing attacks require manual identification of vulnerabilities, i.e., data accesses or instruction execution depending on secret information. In this paper, we present Cache Template Attacks. This generic attack technique allows us to profile and exploit cachebased information leakage of any program automatically, without prior knowledge of specific software versions or even specific system information. Cache Template Attacks can be executed online on a remote system without any prior offline computations or measurements. Cache Template Attacks consist of two phases. In the profiling phase, we determine dependencies between the processing of secret information, e.g., specific key inputs or private keys of cryptographic primitives, and specific cache accesses. In the exploitation phase, we derive the secret values based on observed cache accesses. We illustrate the power of the presented approach in several attacks, but also in a useful application for developers. Among the presented attacks is the application of Cache Template Attacks to infer keystrokes and—even more severe—the identification of specific keys on Linux and Windows user interfaces. More specifically, for lowercase only passwords, we can reduce the entropy per character from log2(26) = 4.7 to 1.4 bits on Linux systems. Furthermore, we perform an automated attack on the Ttable-based AES implementation of OpenSSL that is as efficient as state-of-the-art manual cache attacks.

Intrusion-detection systems aim at detecting attacks against computer systems and networks, or against information systems in general, as it is difficult to provide provably secure information systems and maintain them in such a secure state for their entire lifetime and for every utilization. Sometimes, legacy or operational constraints do not even allow a fully secure information system to be realized at all. Therefore, the task of intrusion-detection systems is to monitor the usage of such systems and to detect the apparition of insecure states. They detect attempts and active misuse by legitimate users of the information systems or external parties to abuse their privileges or exploit security vulnerabilities. In this paper, we introduce a taxonomy of intrusion-detection systems that highlights the various aspects of this area. This taxonomy defines families of intrusion-detection systems according to their properties. It is illustrated by numerous examples from past and current projects. q 1999 Elsevier Science B.V. All rights reserved.

Networked systems still suffer from poor firewall configuration and monitoring. VisualFirewall seeks to aid in the configuration of firewalls and monitoring of networks by providing four simultaneous views that display varying levels of detail and time-scales as well as correctly visualizing firewall reactions to individual packets. The four implemented views, real-time traffic, visual signature, statistics, and IDS alarm, provide the levels of detail and temporality that system administrators need to properly monitor their systems in a passive or an active manner. We have visualized several attacks, and we feel that even individuals unfamiliar with networking concepts can quickly distinguish between benign and malignant traffic patterns with a minimal amount of introduction.

Businesses often interact with users via web-browsers and applications on mobile devices, and host services on cloud servers they may not own. Such highly-exposed environments employ white-box cryptography (WBC) for security protection. WBC operates on a security model far different from the traditional black-box model. The modern business world includes large commercial segments in which end-users are directly exposed to business application software hosted on web browsers, mobile phones, web-connected tablets, and an increasing number of other devices: the `internet of things' (IoT). Software applications and their communication activities now dominate much of the commercial world, and there have been countless hacks on such software, and on devices hosting it, with targets as diverse as mobile phones, web browser applications, vehicles, and even refrigerators! The business advantages of deploying computational power near the user encourage software migration to exposed network end-points, but this increasing exposure provides an ever growing attack surface. Here, we discuss goals and challenges of white-box cryptography and emerging approaches in a continual attempt to stay at least one step ahead of the attackers. We list some WBC techniques, both traditional and recent, indicating how they might be incorporated into a WBC AES implementation.


Question: Rapid learning of sparse Gaussian Markov Random Fields utilizing Cholesky factorization.

This document explores the concept of Probabilistic Graphical Models (PGMs), focusing on their foundational principles and practical techniques for implementation. PGMs represent a powerful framework for encoding probabilistic relationships among random variables, enabling efficient computation and inference in complex systems. A key component of PGMs is the representation of conditional dependencies through graphs, which allows for the compact and interpretable modeling of high-dimensional distributions. The document delves into the specifics of Bayesian networks and Markov random fields, two primary types of PGMs, and discusses their applications in various domains such as machine learning, computer vision, and bioinformatics. Techniques for learning PGMs from data, including parameter estimation and structure learning, are covered, with a particular emphasis on methods that leverage the sparsity of conditional dependencies. Additionally, the document highlights the importance of efficient inference algorithms, such as belief propagation and variational methods, which are essential for making predictions and decisions based on PGMs. Throughout the text, the role of Cholesky factorization in optimizing computations for Gaussian Markov random fields is underscored, showcasing its utility in achieving fast and accurate solutions in the context of sparse graphical models.

**Question**: An Efficient Algorithm for Solving the Economic Dispatch Problem with Valve-Point Effect

**Documents**: An efficient heuristic algorithm for the economic dispatch problem

**Similar Documents**: This paper presents an efficient heuristic algorithm designed to address the Economic Dispatch Problem (EDP) with a valve-point effect. The EDP, a crucial optimization task in power systems, involves determining the optimal allocation of generation among power plants to meet the load demand at the lowest possible cost, while adhering to operational constraints. The valve-point effect, a nonlinear phenomenon observed in thermal power plants, introduces additional complexity to the EDP by creating a series of small valleys in the cost curve, which traditional optimization methods often struggle to handle. The proposed heuristic algorithm, tailored to tackle this specific challenge, employs a multi-stage approach that combines the strengths of both global and local search strategies. Initially, a global search mechanism is applied to explore a wide range of potential solutions, ensuring a comprehensive coverage of the solution space. Subsequently, a local search technique is employed to refine the solutions, focusing on the vicinity of promising points to achieve higher precision. This two-step process not only enhances the algorithm's ability to locate near-optimal solutions but also accelerates the search process, making it particularly suitable for real-time applications in power systems. The algorithm's efficiency and effectiveness are demonstrated through extensive simulations conducted on standard test systems

This paper presents a Quadtree Convolutional Neural Network (QCNN) for efficiently learning from image datasets representing sparse data such as handwriting, pen strokes, freehand sketches, etc. Instead of storing the sparse sketches in regular dense tensors, our method decomposes and represents the image as a linear quadtree that is only refined in the non-empty portions of the image. The actual image data corresponding to non-zero pixels is stored in the finest nodes of the quadtree. Convolution and pooling operations are restricted to the sparse pixels, leading to better efficiency in computation time as well as memory usage. Specifically, the computational and memory costs in QCNN grow linearly in the number of non-zero pixels, as opposed to traditional CNNs where the costs are quadratic in the number of pixels. This enables QCNN to learn from sparse images much faster and process high resolution images without the memory constraints faced by traditional CNNs. We study QCNN on four sparse image datasets for sketch classification and simplification tasks. The results show that QCNN can obtain comparable accuracy with large reduction in computational and memory costs.

Many applications in speech, robotics, finance, and biology deal with sequential data, where ordering matters and recurrent structures are common. However, this structure cannot be easily captured by standard kernel functions. To model such structure, we propose expressive closed-form kernel functions for Gaussian processes. The resulting model, GP-LSTM, fully encapsulates the inductive biases of long short-term memory (LSTM) recurrent networks, while retaining the non-parametric probabilistic advantages of Gaussian processes. We learn the properties of the proposed kernels by optimizing the Gaussian process marginal likelihood using a new provably convergent semi-stochastic gradient procedure, and exploit the structure of these kernels for scalable training and prediction. This approach provides a practical representation for Bayesian LSTMs. We demonstrate state-of-the-art performance on several benchmarks, and thoroughly investigate a consequential autonomous driving application, where the predictive uncertainties provided by GP-LSTM are uniquely valuable.

Intelligent agents must be able to handle the complexity and uncertainty of the real world. Logical AI has focused mainly on the former, and statistical AI on the latter. Markov logic combines the two by attaching weights to first-order formulas and viewing them as templates for features of Markov networks. Inference algorithms for Markov logic draw on ideas from satisfiability, Markov chain Monte Carlo and knowledge-based model construction. Learning algorithms are based on the voted perceptron, pseudo-likelihood and inductive logic programming. Markov logic has been successfully applied to a wide variety of problems in natural language understanding, vision, computational biology, social networks and others, and is the basis of the open-source Alchemy system.

We present a method for training multi-label, massively multi-class image classification models, that is faster and more accurate than supervision via a sigmoid cross-entropy loss (logistic regression). Our method consists in embedding high-dimensional sparse labels onto a lower-dimensional dense sphere of unit-normed vectors, and treating the classification problem as a cosine proximity regression problem on this sphere. We test our method on a dataset of 300 million high-resolution images with 17,000 labels, where it yields considerably faster convergence, as well as a 7% higher mean average precision compared to logistic regression.

We propose a novel approximate inference algorithm that approximates a target distribution by amortising the dynamics of a user-selected MCMC sampler. The idea is to initialise MCMC using samples from an approximation network, apply the MCMC operator to improve these samples, and finally use the samples to update the approximation network thereby improving its quality. This provides a new generic framework for approximate inference, allowing us to deploy highly complex, or implicitly defined approximation families with intractable densities, including approximations produced by warping a source of randomness through a deep neural network. Experiments consider image modelling with deep generative models as a challenging test for the method. Deep models trained using amortised MCMC are shown to generate realistic looking samples as well as producing diverse imputations for images with regions of missing pixels.

The sparse matrix vector product (SpMV) is a key operation in engineering and scientific computing and, hence, it has been subjected to intense research for a long time. The irregular computations involved in SpMV make its optimization challenging. Therefore, enormous effort has been devoted to devise data formats to store the sparse matrix with the ultimate aim of maximizing the performance. Graphics Processing Units (GPUs) have recently emerged as platforms that yield outstanding acceleration factors. SpMV implementations for NVIDIA GPUs have already appeared on the scene. This work proposes and evaluates a new implementation of SpMV for NVIDIA GPUs based on a new format, ELLPACK-R, that allows storage of the sparse matrix in a regular manner. A comparative evaluation against a variety of storage formats previously proposed has been carried out based on a representative set of test matrices. The results show that, although the performance strongly depends on the specific pattern of the matrix, the implementation based on ELLPACK-R achieves higher overall performance. Moreover, a comparison with standard state-of-the-art superscalar processors reveals that significant speedup factors are achieved with GPUs. Copyright 2010 John Wiley & Sons, Ltd.

Deep convolutional neural networks (CNN) have achieved great success. On the other hand, modeling structural information has been proved critical in many vision problems. It is of great interest to integrate them effectively. In a classical neural network, there is no message passing between neurons in the same layer. In this paper, we propose a CRF-CNN framework which can simultaneously model structural information in both output and hidden feature layers in a probabilistic way, and it is applied to human pose estimation. A message passing scheme is proposed, so that in various layers each body joint receives messages from all the others in an efficient way. Such message passing can be implemented with convolution between features maps in the same layer, and it is also integrated with feedforward propagation in neural networks. Finally, a neural network implementation of endto-end learning CRF-CNN is provided. Its effectiveness is demonstrated through experiments on two benchmark datasets.

We study the satisfiability of randomly generated formulas formed by M clauses of exactly K literals over N Boolean variables. For a given value of N the problem is known to be most difficult when α = M/N is close to the experimental threshold αc separating the region where almost all formulas are SAT from the region where all formulas are UNSAT. Recent results from a statistical physics analysis suggest that the difficulty is related to the existence of a clustering phenomenon of the solutions when α is close to (but smaller than) αc. We introduce a new type of message passing algorithm which allows to find efficiently a satisfying assignment of the variables in this difficult region. This algorithm is iterative and composed of two main parts. The first is a message-passing procedure which generalizes the usual methods like Sum-Product or Belief Propagation: It passes messages that may be thought of as surveys over clusters of the ordinary messages. The second part uses the detailed probabilistic information obtained from the surveys in order to fix variables and simplify the problem. Eventually, the simplified problem that remains is solved by a conventional heuristic. © 2005 Wiley Periodicals, Inc. Random Struct. Alg., 27, 201–226, 2005

We propose a simple approach to combining first-order logic and probabilistic graphical models in a single representation. A Markov logic network (MLN) is a first-order knowledge base with a weight attached to each formula (or clause). Together with a set of constants representing objects in the domain, it specifies a ground Markov network containing one feature for each possible grounding of a first-order formula in the KB, with the corresponding weight. Inference in MLNs is performed by MCMC over the minimal subset of the ground network required for answering the query. Weights are efficiently learned from relational databases by iteratively optimizing a pseudo-likelihood measure. Optionally, additional clauses are learned using inductive logic programming techniques. Experiments with a real-world database and knowledge base in a university domain illustrate the promise of this approach.

In this paper we introduce deep Gaussian process (GP) models. Deep GPs are a deep belief network based on Gaussian process mappings. The data is modeled as the output of a multivariate GP. The inputs to that Gaussian process are then governed by another GP. A single layer model is equivalent to a standard GP or the GP latent variable model (GP-LVM). We perform inference in the model by approximate variational marginalization. This results in a strict lower bound on the marginal likelihood of the model which we use for model selection (number of layers and nodes per layer). Deep belief networks are typically applied to relatively large data sets using stochastic gradient descent for optimization. Our fully Bayesian treatment allows for the application of deep models even when data is scarce. Model selection by our variational bound shows that a five layer hierarchy is justified even when modelling a digit data set containing only 150 examples.


Question: Rapid learning of sparse Gaussian Markov Random Fields utilizing Cholesky factorization.

This paper introduces a novel approach for parameter estimation in linear models, known as the 'Lasso' (Least Absolute Shrinkage and Selection Operator). Unlike traditional methods, the Lasso minimizes the residual sum of squares under the constraint that the sum of the absolute values of the coefficients is less than a predetermined constant. This unique constraint leads to a model where some coefficients are exactly zero, thus facilitating the selection of important variables and enhancing model interpretability. Simulation studies highlight that the Lasso combines the best aspects of subset selection and ridge regression, offering models that are both interpretable and stable, akin to ridge regression, while also enabling variable selection, similar to subset selection. Interestingly, this methodology shares parallels with recent advancements in adaptive function estimation by Donoho and Johnstone. The Lasso technique is versatile and can be extended to a variety of statistical models, including generalized linear models and tree-based models, demonstrating its broad applicability in statistical analysis.

We propose a generative model for text and other collections of discrete data that generalizes or improves on several previous models including naive Bayes/unigram, mixture of unigrams [6], and Hofmann's aspect model , also known as probabilistic latent semantic indexing (pLSI) [3]. In the context of text modeling, our model posits that each document is generated as a mixture of topics, where the continuous-valued mixture proportions are distributed as a latent Dirichlet random variable. Inference and learning are carried out efficiently via variational algorithms. We present empirical results on applications of this model to problems in text modeling, collaborative filtering, and text classification.

The `1-regularized maximum likelihood estimation problem has recently become a topic of great interest within the machine learning, statistics, and optimization communities as a method for producing sparse inverse covariance estimators. In this paper, a proximal gradient method (G-ISTA) for performing `1-regularized covariance matrix estimation is presented. Although numerous algorithms have been proposed for solving this problem, this simple proximal gradient method is found to have attractive theoretical and numerical properties. G-ISTA has a linear rate of convergence, resulting in an O(log ε) iteration complexity to reach a tolerance of ε. This paper gives eigenvalue bounds for the G-ISTA iterates, providing a closed-form linear convergence rate. The rate is shown to be closely related to the condition number of the optimal point. Numerical convergence results and timing comparisons for the proposed method are presented. G-ISTA is shown to perform very well, especially when the optimal point is well-conditioned.

The objective of this work is to learn descriptors suitable for the sparse feature detectors used in viewpoint invariant matching. We make a number of novel contributions towards this goal. First, it is shown that learning the pooling regions for the descriptor can be formulated as a convex optimisation problem selecting the regions using sparsity. Second, it is shown that descriptor dimensionality reduction can also be formulated as a convex optimisation problem, using Mahalanobis matrix nuclear norm regularisation. Both formulations are based on discriminative large margin learning constraints. As the third contribution, we evaluate the performance of the compressed descriptors, obtained from the learnt real-valued descriptors by binarisation. Finally, we propose an extension of our learning formulations to a weakly supervised case, which allows us to learn the descriptors from unannotated image collections. It is demonstrated that the new learning methods improve over the state of the art in descriptor learning on the annotated local patches data set of Brown et al. and unannotated photo collections of Philbin et al. .

Graph matching plays a central role in solving correspondence problems in computer vision. Graph matching problems that incorporate pair-wise constraints can be cast as a quadratic assignment problem (QAP). Unfortunately, QAP is NP-hard and many algorithms have been proposed to solve different relaxations. This paper presents factorized graph matching (FGM), a novel framework for interpreting and optimizing graph matching problems. In this work we show that the affinity matrix can be factorized as a Kronecker product of smaller matrices. There are three main benefits of using this factorization in graph matching: (1) There is no need to compute the costly (in space and time) pair-wise affinity matrix; (2) The factorization provides a taxonomy for graph matching and reveals the connection among several methods; (3) Using the factorization we derive a new approximation of the original problem that improves state-of-the-art algorithms in graph matching. Experimental results in synthetic and real databases illustrate the benefits of FGM. The code is available at http://humansensing.cs.cmu.edu/fgm.

We introduce a new class of upper bounds on the log partition function of a Markov random field (MRF). This quantity plays an important role in various contexts, including approximating marginal distributions, parameter estimation, combinatorial enumeration, statistical decision theory, and large-deviations bounds. Our derivation is based on concepts from convex duality and information geometry: in particular, it exploits mixtures of distributions in the exponential domain, and the Legendre mapping between exponential and mean parameters. In the special case of convex combinations of tree-structured distributions, we obtain a family of variational problems, similar to the Bethe variational problem, but distinguished by the following desirable properties: i) they are convex, and have a unique global optimum; and ii) the optimum gives an upper bound on the log partition function. This optimum is defined by stationary conditions very similar to those defining fixed points of the sum-product algorithm, or more generally, any local optimum of the Bethe variational problem. As with sum-product fixed points, the elements of the optimizing argument can be used as approximations to the marginals of the original model. The analysis extends naturally to convex combinations of hypertree-structured distributions, thereby establishing links to Kikuchi approximations and variants.

We introduce the multi-prediction deep Boltzmann machine (MP-DBM). The MPDBM can be seen as a single probabilistic model trained to maximize a variational approximation to the generalized pseudolikelihood, or as a family of recurrent nets that share parameters and approximately solve different inference problems. Prior methods of training DBMs either do not perform well on classification tasks or require an initial learning pass that trains the DBM greedily, one layer at a time. The MP-DBM does not require greedy layerwise pretraining, and outperforms the standard DBM at classification, classification with missing inputs, and mean field prediction tasks.1

Determinantal point processes (DPPs) offer an elegant tool for encoding probabilities over subsets of a ground set. Discrete DPPs are parametrized by a positive semidefinite matrix (called the DPP kernel), and estimating this kernel is key to learning DPPs from observed data. We consider the task of learning the DPP kernel, and develop for it a surprisingly simple yet effective new algorithm. Our algorithm offers the following benefits over previous approaches: (a) it is much simpler; (b) it yields equally good and sometimes even better local maxima; and (c) it runs an order of magnitude faster on large problems. We present experimental results on both real and simulated data to illustrate the numerical performance of our technique.

This paper proposes an efficient sparse metric learning algorithm in high dimensional space via an l1-penalized log-determinant regularization. Compare to the most existing distance metric learning algorithms, the proposed algorithm exploits the sparsity nature underlying the intrinsic high dimensional feature space. This sparsity prior of learning distance metric serves to regularize the complexity of the distance model especially in the "less example number p and high dimension d" setting. Theoretically, by analogy to the covariance estimation problem, we find the proposed distance learning algorithm has a consistent result at rate O (√m2 log d)/n) to the target distance matrix with at most m nonzeros per row. Moreover, from the implementation perspective, this l1-penalized log-determinant formulation can be efficiently optimized in a block coordinate descent fashion which is much faster than the standard semi-definite programming which has been widely adopted in many other advanced distance learning algorithms. We compare this algorithm with other state-of-the-art ones on various datasets and competitive results are obtained.

A fundamental problem in computer vision is that of inferring the intrinsic, 3D structure of the world from flat, 2D images of that world. Traditional methods for recovering scene properties such as shape, reflectance, or illumination rely on multiple observations of the same scene to overconstrain the problem. Recovering these same properties from a single image seems almost impossible in comparison-there are an infinite number of shapes, paint, and lights that exactly reproduce a single image. However, certain explanations are more likely than others: surfaces tend to be smooth, paint tends to be uniform, and illumination tends to be natural. We therefore pose this problem as one of statistical inference, and define an optimization problem that searches for the most likely explanation of a single image. Our technique can be viewed as a superset of several classic computer vision problems (shape-from-shading, intrinsic images, color constancy, illumination estimation, etc) and outperforms all previous solutions to those constituent problems.

Relation learning is a fundamental problem in many vision tasks. Recently, high-order Boltzmann machine and its variants have shown their great potentials in learning various types of data relation in a range of tasks. But most of these models are learned in an unsupervised way, i.e., without using relation class labels, which are not very discriminative for some challenging tasks, e.g., face verification. In this paper, with the goal to perform supervised relation learning, we introduce relation class labels into conventional high-order multiplicative interactions with pairwise input samples, and propose a conditional high-order Boltzmann Machine (CHBM), which can learn to classify the data relation in a binary classification way. To be able to deal with more complex data relation, we develop two improved variants of CHBM: 1) latent CHBM, which jointly performs relation feature learning and classification, by using a set of latent variables to block the pathway from pairwise input samples to output relation labels and 2) gated CHBM, which untangles factors of variation in data relation, by exploiting a set of latent variables to multiplicatively gate the classification of CHBM. To reduce the large number of model parameters generated by the multiplicative interactions, we approximately factorize high-order parameter tensors into multiple matrices. Then, we develop efficient supervised learning algorithms, by first pretraining the models using joint likelihood to provide good parameter initialization, and then finetuning them using conditional likelihood to enhance the discriminant ability. We apply the proposed models to a series of tasks including invariant recognition, face verification, and action similarity labeling. Experimental results demonstrate that by exploiting supervised relation labels, our models can greatly improve the performance.


Question: Rapid learning of sparse Gaussian Markov Random Fields utilizing Cholesky factorization.

This document details the application of a proximal gradient method, referred to as G-ISTA, for the estimation of sparse inverse covariance matrices through `1-regularized maximum likelihood. This technique has garnered significant attention in the fields of machine learning, statistics, and optimization due to its effectiveness in generating sparse estimators. Despite the existence of various algorithms for tackling this problem, G-ISTA stands out due to its compelling theoretical and practical attributes. It boasts a linear convergence rate, which translates to an iteration complexity of O(log ε) to achieve a tolerance level of ε. The paper provides eigenvalue bounds for the G-ISTA iterations, offering a closed-form linear convergence rate that is closely linked to the condition number of the optimal solution. Experimental results and timing comparisons are presented, demonstrating G-ISTA's superior performance, particularly when dealing with well-conditioned optimal points. This method offers a promising alternative for sparse inverse covariance estimation, showcasing its efficiency and robustness in various scenarios.

Nonnegative matrix factorization (NMF) with group sparsity constraints is formulated as a probabilistic graphical model and, assuming some observed data have been generated by the model, a feasible variational Bayesian algorithm is derived for learning model parameters. When used in a supervised learning scenario, NMF is most often utilized as an unsupervised feature extractor followed by classification in the obtained feature subspace. Having mapped the class labels to a more general concept of groups which underlie sparsity of the coefficients, what the proposed group sparse NMF model allows is incorporating class label information to find low dimensional label-driven dictionaries which not only aim to represent the data faithfully, but are also suitable for class discrimination. Experiments performed in face recognition and facial expression recognition domains point to advantages of classification in such label-driven feature subspaces over classification in feature subspaces obtained in an unsupervised manner.

Kernel approximation via nonlinear random feature maps is w idely used in speeding up kernel machines. There are two main challenges for the conventional kernel ap proximation methods. First, before performing kernel approximation, a good kernel has to be chosen. Pickin g a good kernel is a very challenging problem in itself. Second, high-dimensional maps are often required i n order to achieve good performance. This leads to high computational cost in both generating the nonlinear ma ps, nd in the subsequent learning and prediction process. In this work, we propose to optimize the nonlinear m ps directly with respect to the classification objective in a data-dependent fashion. The proposed approa ch achieves kernel approximation and kernel learning in a joint framework. This leads to much more compact maps wit hout hurting the performance. As a by-product, the same framework can also be used to achieve more compact ke rnel maps to approximate a known kernel. We also introduce Circulant Nonlinear Maps, which uses a circu lant-structured projection matrix to speed up the nonlinear maps for high-dimensional data.

Many real-world applications require multi-label classification where multiple target labels are assigned to each instance. In multi-label classification, there exist the intrinsic correlations between the labels and features. These correlations are beneficial for multi-label classification task since they reflect the coexistence of the input and output spaces that can be exploited for prediction. Traditional classification methods have attempted to reveal these correlations in different ways. However, existing methods demand expensive computation complexity for finding such correlation structures. Furthermore, these approaches can not identify the suitable number of label-feature correlation patterns. In this paper, we propose a Bayesian nonparametric (BNP) framework for multi-label classification that can automatically learn and exploit the unknown number of multi-label correlation. We utilize the recent techniques in stochastic inference to derive the cheap (but efficient) posterior inference algorithm for the model. In addition, our model can naturally exploit the useful information from missing label samples. Furthermore, we extend the model to update parameters in an online fashion that highlights the flexibility of our model against the existing approaches. We compare our method with the state-of-the-art multi-label classification algorithms on real-world datasets using both complete and missing label settings. Our model achieves better classification accuracy while our running time is consistently much faster than the baselines in an order of magnitude.

The massive parallelism of graphics processing units (GPUs) offers tremendous performance in many high-performance computing applications. While dense linear algebra readily maps to such platforms, harnessing this potential for sparse matrix computations presents additional challenges. Given its role in iterative methods for solving sparse linear systems and eigenvalue problems, sparse matrix-vector multiplication (SpMV) is of singular importance in sparse linear algebra. In this paper we discuss data structures and algorithms for SpMV that are efficiently implemented on the CUDA platform for the fine-grained parallel architecture of the GPU. Given the memory-bound nature of SpMV, we emphasize memory bandwidth efficiency and compact storage formats. We consider a broad spectrum of sparse matrices, from those that are well-structured and regular to highly irregular matrices with large imbalances in the distribution of nonzeros per matrix row. We develop methods to exploit several common forms of matrix structure while offering alternatives which accommodate greater irregularity. On structured, grid-based matrices we achieve performance of 36 GFLOP/s in single precision and 16 GFLOP/s in double precision on a GeForce GTX 280 GPU. For unstructured finite-element matrices, we observe performance in excess of 15 GFLOP/s and 10 GFLOP/s in single and double precision respectively. These results compare favorably to prior state-of-the-art studies of SpMV methods on conventional multicore processors. Our double precision SpMV performance is generally two and a half times that of a Cell BE with 8 SPEs and more than ten times greater than that of a quad-core Intel Clovertown system.

We present an implementation of a probabilistic first-order logic called TensorLog, in which classes of logical queries are compiled into differentiable functions in a neuralnetwork infrastructure such as Tensorflow or Theano. This leads to a close integration of probabilistic logical reasoning with deep-learning infrastructure: in particular, it enables high-performance deep learning frameworks to be used for tuning the parameters of a probabilistic logic. Experimental results show that TensorLog scales to problems involving hundreds of thousands of knowledge-base triples and tens of thousands of examples.

Texture synthesis is important for many applications in computer graphics, vision, and image processing. However, it remains difficult to design an algorithm that is both efficient and capable of generating high quality results. In this paper, we present an efficient algorithm for realistic texture synthesis. The algorithm is easy to use and requires only a sample texture as input. It generates textures with perceived quality equal to or better than those produced by previous techniques, but runs two orders of magnitude faster. This permits us to apply texture synthesis to problems where it has traditionally been considered impractical. In particular, we have applied it to constrained synthesis for image editing and temporal texture generation. Our algorithm is derived from Markov Random Field texture models and generates textures through a deterministic searching process. We accelerate this synthesis process using tree-structured vector quantization.

We present a large-margin formulation and algorithm for structured output prediction that allows the use of latent variables. Our proposal covers a large range of application problems, with an optimization problem that can be solved efficiently using Concave-Convex Programming. The generality and performance of the approach is demonstrated through three applications including motiffinding, noun-phrase coreference resolution, and optimizing precision at k in information retrieval.

To accelerate the training of kernel machines, we propose to map the input data to a randomized low-dimensional feature space and then apply existing fast linear methods. Our randomized features are designed so that the inner products of the transformed data are approximately equal to those in the feature space of a user specified shift-invariant kernel. We explore two sets of random features, provide convergence bounds on their ability to approximate various radial basis kernels, and show that in large-scale classification and regression tasks linear machine learning algorithms that use these features outperform state-of-the-art large-scale kernel machines.

Gaussian processes allow for flexible specification of prior assumptions of unknown dynamics in state space models. We present a procedure for efficient Bayesian learning in Gaussian process state space models, where the representation is formed by projecting the problem onto a set of approximate eigenfunctions derived from the prior covariance structure. Learning under this family of models can be conducted using a carefully crafted particle MCMC algorithm. This scheme is computationally efficient and yet allows for a fully Bayesian treatment of the problem. Compared to conventional system identification tools or existing learning methods, we show competitive performance and reliable quantification of uncertainties in the model.

Gaussian processes (GPs) are a good choice for function approximation as they are flexible, robust to overfitting, and provide well-calibrated predictive uncertainty. Deep Gaussian processes (DGPs) are multi-layer generalizations of GPs, but inference in these models has proved challenging. Existing approaches to inference in DGP models assume approximate posteriors that force independence between the layers, and do not work well in practice. We present a doubly stochastic variational inference algorithm that does not force independence between layers. With our method of inference we demonstrate that a DGP model can be used effectively on data ranging in size from hundreds to a billion points. We provide strong empirical evidence that our inference scheme for DGPs works well in practice in both classification and regression.


Question: Rapid learning of sparse Gaussian Markov Random Fields utilizing Cholesky factorization.

In a pioneering effort to uncover conserved genetic modules across diverse species, a comprehensive gene-coexpression network has been constructed using a dataset of 3182 DNA microarrays from humans, flies, worms, and yeast. This study has revealed 22,163 coexpression relationships that have withstood the test of evolution, suggesting that these gene pairs confer a significant evolutionary advantage and are likely functionally interconnected. Among these relationships, numerous instances provide compelling evidence for the role of previously uncharacterized genes in fundamental biological processes, including cell cycle regulation, secretion mechanisms, and protein synthesis. To validate the predictive power of these coexpression links, experimental confirmation was sought for a subset, leading to the identification of cell proliferation functions for several genes. Through the assembly of these coexpression relationships into a network, the research has not only uncovered animal-specific components but also highlighted the interplay between newly evolved and ancient genetic modules, offering insights into the evolutionary landscape of gene function.

We propose a novel mixtures of Gaussian processes model in which the gating function is interconnected with a probabilistic logical model, in our case Markov logic networks. In this way, the resulting mixed graphical model, called Markov logic mixtures of Gaussian processes (MLxGP), solves joint Bayesian non-parametric regression and probabilistic relational inference tasks. In turn, MLxGP facilitates novel, interesting tasks such as regression based on logical constraints or drawing probabilistic logical conclusions about regression data, thus putting “machines reading regression data” in reach.

While Deep Neural Networks (DNNs) have achieved tremendous success for large vocabulary continuous speech recognition (LVCSR) tasks, training of these networks is slow. One reason is that DNNs are trained with a large number of training parameters (i.e., 10-50 million). Because networks are trained with a large number of output targets to achieve good performance, the majority of these parameters are in the final weight layer. In this paper, we propose a low-rank matrix factorization of the final weight layer. We apply this low-rank technique to DNNs for both acoustic modeling and language modeling. We show on three different LVCSR tasks ranging between 50-400 hrs, that a low-rank factorization reduces the number of parameters of the network by 30-50%. This results in roughly an equivalent reduction in training time, without a significant loss in final recognition accuracy, compared to a full-rank representation.

Relation learning is a fundamental problem in many vision tasks. Recently, high-order Boltzmann machine and its variants have shown their great potentials in learning various types of data relation in a range of tasks. But most of these models are learned in an unsupervised way, i.e., without using relation class labels, which are not very discriminative for some challenging tasks, e.g., face verification. In this paper, with the goal to perform supervised relation learning, we introduce relation class labels into conventional high-order multiplicative interactions with pairwise input samples, and propose a conditional high-order Boltzmann Machine (CHBM), which can learn to classify the data relation in a binary classification way. To be able to deal with more complex data relation, we develop two improved variants of CHBM: 1) latent CHBM, which jointly performs relation feature learning and classification, by using a set of latent variables to block the pathway from pairwise input samples to output relation labels and 2) gated CHBM, which untangles factors of variation in data relation, by exploiting a set of latent variables to multiplicatively gate the classification of CHBM. To reduce the large number of model parameters generated by the multiplicative interactions, we approximately factorize high-order parameter tensors into multiple matrices. Then, we develop efficient supervised learning algorithms, by first pretraining the models using joint likelihood to provide good parameter initialization, and then finetuning them using conditional likelihood to enhance the discriminant ability. We apply the proposed models to a series of tasks including invariant recognition, face verification, and action similarity labeling. Experimental results demonstrate that by exploiting supervised relation labels, our models can greatly improve the performance.

We describe how the pretraining algorithm for Deep Boltzmann Machines (DBMs) is related to the pretraining algorithm for Deep Belief Networks and we show that under certain conditions, the pretraining procedure improves the variational lower bound of a two-hidden-layer DBM. Based on this analysis, we develop a different method of pretraining DBMs that distributes the modelling work more evenly over the hidden layers. Our results on the MNIST and NORB datasets demonstrate that the new pretraining algorithm allows us to learn better generative models.

Mean-field variational methods are widely used for approximate posterior inference in many probabilistic models. In a typical application, mean-field methods approximately compute the posterior with a coordinate-ascent optimization algorithm. When the model is conditionally conjugate, the coordinate updates are easily derived and in closed form. However, many models of interest—like the correlated topic model and Bayesian logistic regression—are nonconjugate. In these models, mean-field methods cannot be directly applied and practitioners have had to develop variational algorithms on a case-by-case basis. In this paper, we develop two generic methods for nonconjugate models, Laplace variational inference and delta method variational inference. Our methods have several advantages: they allow for easily derived variational algorithms with a wide class of nonconjugate models; they extend and unify some of the existing algorithms that have been derived for specific models; and they work well on real-world data sets. We studied our methods on the correlated topic model, Bayesian logistic regression, and hierarchical Bayesian logistic regression.

Intelligent agents must be able to handle the complexity and uncertainty of the real world. Logical AI has focused mainly on the former, and statistical AI on the latter. Markov logic combines the two by attaching weights to first-order formulas and viewing them as templates for features of Markov networks. Inference algorithms for Markov logic draw on ideas from satisfiability, Markov chain Monte Carlo and knowledge-based model construction. Learning algorithms are based on the voted perceptron, pseudo-likelihood and inductive logic programming. Markov logic has been successfully applied to a wide variety of problems in natural language understanding, vision, computational biology, social networks and others, and is the basis of the open-source Alchemy system.

Convolutional networks are one of the most widely employed architectures in computer vision and machine learning. In order to leverage their ability to learn complex functions, large amounts of data are required for training. Training a large convolutional network to produce state-of-the-art results can take weeks, even when using modern GPUs. Producing labels using a trained network can also be costly when dealing with web-scale datasets. In this work, we present a simple algorithm which accelerates training and inference by a significant factor, and can yield improvements of over an order of magnitude compared to existing state-of-the-art implementations. This is done by computing convolutions as pointwise products in the Fourier domain while reusing the same transformed feature map many times. The algorithm is implemented on a GPU architecture and addresses a number of related challenges.

Efficiently learning mixture of Gaussians is a fundamental problem in statistics and learning theory. Given samples coming from a random one out of k Gaussian distributions in Rn, the learning problem asks to estimate the means and the covariance matrices of these Gaussians. This learning problem arises in many areas ranging from the natural sciences to the social sciences, and has also found many ma- chine learning applications. Unfortunately, learning mixture of Gaussians is an information theoretically hard problem: in order to learn the parameters up to a reasonable accuracy, the number of samples required is exponential in the number of Gaussian components in the worst case. In this work, we show that provided we are in high enough dimensions, the class of Gaussian mixtures is learnable in its most general form under a smoothed analysis framework, where the parameters are randomly perturbed from an adversarial starting point. In particular, given samples from a mixture of Gaussians with randomly perturbed parameters, when n ≥ Ω(k2), we give an algorithm that learns the parameters with polynomial running time and using polynomial number of samples.
 The central algorithmic ideas consist of new ways to de- compose the moment tensor of the Gaussian mixture by exploiting its structural properties. The symmetries of this tensor are derived from the combinatorial structure of higher order moments of Gaussian distributions (sometimes referred to as Isserlis' theorem or Wick's theorem). We also develop new tools for bounding smallest singular values of structured random matrices, which could be useful in other smoothed analysis settings.

Gaussian processes (GPs) are a good choice for function approximation as they are flexible, robust to overfitting, and provide well-calibrated predictive uncertainty. Deep Gaussian processes (DGPs) are multi-layer generalizations of GPs, but inference in these models has proved challenging. Existing approaches to inference in DGP models assume approximate posteriors that force independence between the layers, and do not work well in practice. We present a doubly stochastic variational inference algorithm that does not force independence between layers. With our method of inference we demonstrate that a DGP model can be used effectively on data ranging in size from hundreds to a billion points. We provide strong empirical evidence that our inference scheme for DGPs works well in practice in both classification and regression.

Learning good image priors is of utmost importance for the study of vision, computer vision and image processing applications. Learning priors and optimizing over whole images can lead to tremendous computational challenges. In contrast, when we work with small image patches, it is possible to learn priors and perform patch restoration very efficiently. This raises three questions - do priors that give high likelihood to the data also lead to good performance in restoration? Can we use such patch based priors to restore a full image? Can we learn better patch priors? In this work we answer these questions. We compare the likelihood of several patch models and show that priors that give high likelihood to data perform better in patch restoration. Motivated by this result, we propose a generic framework which allows for whole image restoration using any patch based prior for which a MAP (or approximate MAP) estimate can be calculated. We show how to derive an appropriate cost function, how to optimize it and how to use it to restore whole images. Finally, we present a generic, surprisingly simple Gaussian Mixture prior, learned from a set of natural images. When used with the proposed framework, this Gaussian Mixture Model outperforms all other generic prior methods for image denoising, deblurring and inpainting.