Question: A structured approach for social engineering attack methodologies.

**Similar Documents**: This comprehensive analysis delves into the intricate mechanisms of social engineering, exploring its various tactics, techniques, and procedures. Unlike conventional cybersecurity threats that target technical vulnerabilities, social engineering focuses on exploiting human behavior to gain unauthorized access or information. The paper examines the psychological underpinnings that make individuals susceptible to such attacks, including trust, compliance, and curiosity. It further discusses common social engineering strategies like phishing, baiting, and pretexting, and how they manipulate victims to divulge sensitive data or perform actions that compromise security. The document also highlights the importance of awareness and training in defending against social engineering attacks, emphasizing the role of education in strengthening the human element of security. Through detailed case studies and expert insights, this resource offers a thorough understanding of social engineering, equipping readers with the knowledge to identify and mitigate these sophisticated threats.

Although digital forensics investigations can vary drastically in their level of complexity, each investigative process must follow a rigorous path. Previously proposed frameworks are predominantly single-tier, higher order process models, focusing on the abstract, rather than more concrete principles of the investigation. We contend that these frameworks, although useful in explaining overarching concepts, require additional detail in order to be useful to investigators and researchers. We therefore propose a multi-tier, hierarchical framework to guide digital investigations. Our framework includes objectivesbased sub-phases that are applicable to various layers of abstraction.

The proposed security development lifecycle (SecDLC) model delivers a perpetual cycle of information security management and refinement. Using real-world examples, the authors show how SecDLC ensures the goals of preserving, monitoring, and improving security practices, policies, and standards in private and public sectors. The authors describe the four phases of SecDLC, comparing and contrasting them to existing security development models.

Phishing emails are semantic attacks that con people into divulging sensitive information using techniques to make the user believe that information is being requested by a legitimate source. In order to develop tools that will be effective in combating these schemes, we first must know how and why people fall for them. This study reports preliminary analysis of interviews with 20 non-expert computer users to reveal their strategies and understand their decisions when encountering possibly suspicious emails. One of the reasons that people may be vulnerable to phishing schemes is that awareness of the risks is not linked to perceived vulnerability or to useful strategies in identifying phishing emails. Rather, our data suggest that people can manage the risks that they are most familiar with, but don't appear to extrapolate to be wary of unfamiliar risks. We explore several strategies that people use, with varying degrees of success, in evaluating emails and in making sense of warnings offered by browsers attempting to help users navigate the web.

The importance of software security cannot be overstated. In the past, researchers have applied program analysis techniques to automatically detect security vulnerabilities and verify security properties. However, such techniques have limited success in reality because they require manually provided code-level security specifications. Manually writing and generating these code-level security specifications are tedious and error-prone. Additionally, they seldom exist in production software. In this paper, we propose a novel method and tool, called AutoISES, which Automatically Infers Security Specifications by statically analyzing source code, and then directly use these specifications to automatically detect security violations. Our experiments with the Linux kernel and Xen demonstrated the effectiveness of this approach – AutoISES automatically generated 84 security specifications and detected 8 vulnerabilities in the Linux kernel and Xen, 7 of which have already been confirmed by the corresponding developers.

It is becoming increasingly difficult to ignore the importance of using online social networks (OSNs) for various purposes such as marketing, education, entertainment, and business. However, OSNs also open the door for harmful activities and behaviors. Committing financial fraud and propagating malware and spam advertisements are very common criminal actions that people engage in by accessing uniform resource locators (URLs). It has been reported that advanced attackers tend to exploit human flaws rather than system flaws, thus, users are targeted in social media threats by hour. This research aims to understand the state of literature on detecting malicious URLs in OSNs, with a focus on two major aspects: URL and OSN objects. Although the literature presents these two aspects in a different context, this paper will mainly focus on their features in relation to malicious URL detection using classification methods. We firstly review three features of URLs: lexical features, hosts, and domains, then we discuss their limitations. We then introduce social spam analytics and detection models using combined features from both URLs and OSNs, particularly the use of user profiles and posts together with URL features, to enhance the detection of malicious behavior. This combination can help in understanding the interests of the user either explicitly, by stating choices in the profile, or implicitly, by analyzing the post behavior, as the spammers do not maintain a regular interest and tend to exploit events or top trend topics.

Information Security (InfoSec) research is far reaching and includes many approaches to deal with protecting and mitigating threats to the information assets and technical resources available within computer based systems. Although a predominant weakness in properly securing information assets is the individual user within an organization, much of the focus of extant security research is on technical issues. The purpose of this paper is to highlight future directions for Behavioral InfoSec research, which is a newer, growing area of research. The ensuing paper presents information about challenges currently faced and future directions that Behavioral InfoSec researchers should explore. These areas include separating insider deviant behavior from insider misbehavior, approaches to understanding hackers, improving information security compliance, cross-cultural Behavioral InfoSec research, and data collection and measurement issues in Behavioral InfoSec research. a 2012 Elsevier Ltd. All rights reserved.

Gamification is the use of game design elements in non-game context platforms such as services and marketing to motivate people to participate in planned activities to increase engagement and loyalty to achieve goals.
 Gamification has been applied to academic fields including software engineering in recent years. Some studies show that gamification can motivate engineers in Software Engineering (SE) if applied appropriately. However, most gamification implementations are lacking well-defined frameworks or models.
 This paper develops a software gamification model (SGM) with a well-defined framework that provides a robust process for implementing gamification for SE. This model also contains elements from applied psychology, Social Software Engineering (SSE), and the Capability Maturity Model (CMM). Having those elements ensures that the model has a solid foundation in gamification and SE disciplines. The model is further tailored for cross-cultural software development teams (CCSDT).
 CCSDT have been popular in recent years due to the fact that software development has become a global business. However, as more and more CCSDT are formed, many challenges and issues have been raised in those cross-cultural environments stemming from miscommunication, misunderstanding, cultural differences and conflicts. This paper uses the SGM to help resolve the issues.

This paper gives a new approach called Component based software engineering for designing an intrusion detection system. This paper also tells that before this component based software approach, other traditional approaches were used for designing intrusion detection system. But due to various drawbacks we proposed this component based approach for intrusion detection system. A comparison table is also illustrated in this paper on the basis of class diagram designed using UML language in a simulated environment. This comparison table will illustrate main difference between the traditional and component based approach and also shows that which parameters make this approach a better approach. Keywords—Component based software engineering, intrusion detection system, Classes etc.

A distributed detection method is proposed to detect single stage multi-point (SSMP) attacks on a Cyber Physical System (CPS). Such attacks aim at compromising two or more sensors or actuators at any one stage of a CPS and could totally compromise a controller and prevent it from detecting the attack. However, as demonstrated in this work, using the flow properties of water from one stage to the other, a neighboring controller was found effective in detecting such attacks. The method is based on physical invariants derived for each stage of the CPS from its design. The attack detection effectiveness of the method was evaluated experimentally against an operational water treatment testbed containing 42 sensors and actuators. Results from the experiments point to high effectiveness of the method in detecting a variety of SSMP attacks but also point to its limitations. Distributing the attack detection code among various controllers adds to the scalability of the proposed method.

In this paper, we study the quantification, practice, and implications of structural data (e.g., social data, mobility traces) De-Anonymization (DA). First, we address several open problems in structural data DA by quantifying perfect and (1-ε)-perfect structural data DA}, where ε is the error tolerated by a DA scheme. To the best of our knowledge, this is the first work on quantifying structural data DA under a general data model, which closes the gap between structural data DA practice and theory. Second, we conduct the first large-scale study on the de-anonymizability of 26 real world structural datasets, including Social Networks (SNs), Collaborations Networks, Communication Networks, Autonomous Systems, and Peer-to-Peer networks. We also quantitatively show the conditions for perfect and (1-ε)-perfect DA of the 26 datasets. Third, following our quantification, we design a practical and novel single-phase cold start Optimization based DA} (ODA) algorithm. Experimental analysis of ODA shows that about 77.7% - 83.3% of the users in Gowalla (.2M users and 1M edges) and 86.9% - 95.5% of the users in Google+ (4.7M users and 90.8M edges) are de-anonymizable in different scenarios, which implies optimization based DA is implementable and powerful in practice. Finally, we discuss the implications of our DA quantification and ODA and provide some general suggestions for future secure data publishing.


Question: A framework for understanding and defending against social engineering attacks.

The imperative for robust user education in safeguarding against social engineering breaches in secure computing environments is highlighted in this paper. Despite being entrusted with access to secure systems, individuals can inadvertently compromise security due to a lack of education, oversight, or succumbing to social pressures. The human element is often the most vulnerable in an otherwise fortified system, making them prime targets for social engineering tactics. These strategies, favored by hackers and cybercriminals, involve manipulating individuals into divulging sensitive information—such as login credentials—rather than exploiting technical vulnerabilities. Social engineering can manifest through both physical and psychological means, exploiting situations where victims feel secure, including workplaces, phone conversations, discarded documents, and online interactions. Psychological manipulation often creates a sense of urgency or authority, enabling the attacker to coax access details from unsuspecting employees.
The escalating importance of data privacy laws in the United States and globally, which impose stringent standards and penalties for non-compliance, underscores the need to assess the reliability of individuals and systems in protecting data. An audit simulating a social engineering attack serves as a critical tool for evaluating an organization's adherence to data privacy regulations. This process is crucial for entities tasked with safeguarding personal information.
The paper discusses the outcomes of an unannounced social engineering audit conducted within an organization concerned with data security. It delves into the interactions between the social engineer and audited personnel, the methodologies employed by the social engineer, and additional insights gleaned from the audit. The paper also explores potential measures to enhance user education and reduce vulnerabilities to social engineering through proactive training and awareness programs.

Lack of trust has been repeatedly identified as one of the most formidable barriers to people for engaging in e-commerce, involving transactions in which financial and personal information is submitted to merchants via the Internet. The future of e-commerce is tenuous without a general climate of online trust. Building consumer trust on the Internet presents a challenge for online merchants and is a research topic of increasing interest and importance. This paper provides an overview of the nature and concepts of trust from multi-disciplinary perspectives, and it reviews relevant studies that investigate the elements of online trust. Also, a framework of trust-inducing interface design features articulated from the existing literature is presented. The design features were classified into four dimensions, namely (1) graphic design, (2) structure design, (3) content design, and (4) social-cue design. By applying the design features identified within this framework to e-commerce web site interfaces, online merchants might then anticipate fostering optimal levels of trust in their customers. 2003 Elsevier Ltd. All rights reserved.

Phishing is an online identity theft, which aims to steal sensitive information such as username, password and online banking details from victims. To prevent this, phishing education needs to be considered. Game based education is becoming more and more popular. This paper introduces a mobile game prototype for the android platform based on a story, which simplifies and exaggerates real life. The elements of a game design framework for avoiding phishing attacks were used to address the game design issues and game design principles were used as a set of guidelines for structuring and presenting information. The overall mobile game design was aimed to enhance the user’s avoidance behaviour through motivation to protect themselves against phishing threats. The prototype mobile game design was presented on MIT App Inventor Emulator.

We identify three types of attack on the intellectual property contained in software, and three corresponding technical defenses. A potent defense against reverse engineering is obfuscation, a process that renders software unintelligible but still functional. A defense against software piracy is watermarking, a process that makes it possible to determine the origin of software. A defense against tampering is tamper-proofing, so that unauthorized modifications to software (for example to remove a watermark) will result in non-functional code. We briefly survey the available technology for each type of defense.

The proposed security development lifecycle (SecDLC) model delivers a perpetual cycle of information security management and refinement. Using real-world examples, the authors show how SecDLC ensures the goals of preserving, monitoring, and improving security practices, policies, and standards in private and public sectors. The authors describe the four phases of SecDLC, comparing and contrasting them to existing security development models.

We present SABER (Survivability Architecture: Block, Evade, React), a proposed survivability architecture that blocks, evades and reacts to a variety of attacks by using several security and survivability mechanisms in an automated and coordinated fashion. Contrary to the ad hoc manner in which contemporary survivable systems are built-using isolated, independent security mechanisms such as firewalls, intrusion detection systems and software sandboxes-SABER integrates several different technologies in an attempt to provide a unified framework for responding to the wide range of attacks malicious insiders and outsiders can launch.
 This coordinated multi-layer approach will be capable of defending against attacks targeted at various levels of the network stack, such as congestion-based DoS attacks, software-based DoS or code-injection attacks, and others. Our fundamental insight is that while multiple lines of defense are useful, most conventional, uncoordinated approaches fail to exploit the full range of available responses to incidents. By coordinating the response, the ability to survive successful security breaches increases substantially.
 We discuss the key components of SABER, how they will be integrated together, and how we can leverage on the promising results of the individual components to improve survivability in a variety of coordinated attack scenarios. SABER is currently in the prototyping stages, with several interesting open research topics.

In a phishing attack, an unsuspecting victim is lured, typically via an email, to a web site designed to steal sensitive information such as bank/credit card account numbers, login information for accounts, etc. Each year Internet users lose billions of dollars to this scourge. In this paper, we present a general semantic feature selection method for text problems based on the statistical t-test and WordNet, and we show its effectiveness on phishing email detection by designing classifiers that combine semantics and statistics in analyzing the text in the email. Our feature selection method is general and useful for other applications involving text-based analysis as well. Our email body-text-only classifier achieves more than 95% accuracy on detecting phishing emails with a false positive rate of 2.24%. Due to its use of semantics, our feature selection method is robust against adaptive attacks and avoids the problem of frequent retraining needed by machine learning classifiers.

This paper presents a graph-based approach to network vulnerability analysis. The method is flexible, allowing analysis of attacks from both outside and inside the network. It can analyze risks to a specific network asset, or examine the universe of possible consequences following a successful attack. The graph-based tool can identify the set of attack paths that have a high probability of success (or a low "effort" cost) for the attacker. The system could be used to test the effectiveness of making configuration changes, implementing an intrusion detection system, etc. The analysis system requires as input a database of common attacks, broken into atomic steps, specific network configuration and topology information, and an attacker profile. The attack information is "matched" with the network configuration information and an attacker profile to create a superset attack graph. Nodes identify a stage of attack, for example the class of machines the attacker has accessed and the user privilege level he or she has compromised. The arcs in the attack graph represent attacks or stages of attacks. By assigning probabilities of success on the arcs or costs representing level-of-effort for the attacker, various graph algorithms such as shortest-path algorithms can identify the attack paths with the highest probability of success.

Social networking sites such as Facebook, LinkedIn, and Xing have been reporting exponential growth rates and have millions of registered users. In this paper, we introduce a novel de-anonymization attack that exploits group membership information that is available on social networking sites. More precisely, we show that information about the group memberships of a user (i.e., the groups of a social network to which a user belongs) is sufficient to uniquely identify this person, or, at least, to significantly reduce the set of possible candidates. That is, rather than tracking a user's browser as with cookies, it is possible to track a person. To determine the group membership of a user, we leverage well-known web browser history stealing attacks. Thus, whenever a social network user visits a malicious website, this website can launch our de-anonymization attack and learn the identity of its visitors. The implications of our attack are manifold, since it requires a low effort and has the potential to affect millions of social networking users. We perform both a theoretical analysis and empirical measurements to demonstrate the feasibility of our attack against Xing, a medium-sized social network with more than eight million members that is mainly used for business relationships. Furthermore, we explored other, larger social networks and performed experiments that suggest that users of Facebook and LinkedIn are equally vulnerable.

During our digital social life, we share terabytes of information that can potentially reveal private facts and personality traits to unexpected strangers. Despite the research efforts aiming at providing efficient solutions for the anonymization of huge databases (including networked data), in online social networks the most powerful privacy protection is in the hands of the users. However, most users are not aware of the risks derived by the indiscriminate disclosure of their personal data. With the aim of fostering their awareness on private data leakage risk, some measures have been proposed that quantify the privacy risk of each user. However, these measures do not capture the objective risk of users since they assume that all user’s direct social connections are close (thus trustworthy) friends. Since this assumption is too strong, in this paper we propose an alternative approach: each user decides which friends are allowed to see each profile item/post and our privacy score is defined accordingly. We show that it can be easily computed with minimal user intervention by leveraging an active learning approach. Finally, we validate our measure on a set of real Facebook users.

Many large companies today face the risk of data breaches via malicious software, compromising their business. These types of attacks are usually executed using hacker assets. Researching hacker assets within underground communities can help identify the tools which may be used in a cyberattack, provide knowledge on how to implement and use such assets and assist in organizing tools in a manner conducive to ethical reuse and education. This study aims to understand the functions and characteristics of assets in hacker forums by applying classification and topic modeling techniques. This research contributes to hacker literature by gaining a deeper understanding of hacker assets in well-known forums and organizing them in a fashion conducive to educational reuse. Additionally, companies can apply our framework to forums of their choosing to extract their assets and appropriate functions.


Question: A framework for understanding and responding to social engineering attacks.

This paper aims to provide an extensive overview of the burgeoning field dedicated to the creation and application of ontologies. It highlights the significant challenge posed by the diversity in backgrounds, languages, tools, and methodologies, which hinders effective communication among individuals, organizations, and software systems. The paper argues that the establishment and execution of an explicit representation of a shared understanding—an ontology—in a specific domain can enhance communication, leading to increased reuse and sharing, improved interoperability, and more dependable software. To motivate the necessity for ontologies, the paper first clarifies their definition and functions. It then outlines a structured methodology for ontology development and evaluation, beginning with informal techniques that address issues such as scoping, ambiguity management, consensus building, and definition formulation. The paper subsequently explores the advantages of a more structured approach, revisiting the scoping phase and elaborating on the significance of formal languages and methods in ontology specification, implementation, and assessment. Lastly, the paper surveys the current state and practices in this evolving field, examining various case studies, software tools for ontology creation, critical research questions, and future prospects. Through this comprehensive exploration, the paper underscores the pivotal role of ontologies in facilitating more effective and efficient communication across diverse domains.

Social media has become an inevitable part of individuals personal and business lives. Its benefits come with various negative consequences. One major concern is the prevalence of detrimental online behavior on social media, such as online harassment and cyberbullying. In this study, we aim to address the computational challenges associated with harassment detection in social media by developing a machine learning framework with three distinguishing characteristics. (1) It uses minimal supervision in the form of expert-provided key phrases that are indicative of bullying or non-bullying. (2) It detects harassment with an ensemble of two learners that co-train one another; one learner examines the language content in the message, and the other learner considers the social structure. (3) It incorporates distributed word and graph-node representations by training nonlinear deep models. The model is trained by optimizing an objective function that balances a co-training loss with a weak-supervision loss. We evaluate the effectiveness of our approach using post-hoc, crowdsourced annotation of Twitter, Ask.fm, and Instagram data, finding that our deep ensembles outperform previous non-deep methods for weakly supervised harassment detection.

Attack models and attack graphs are efficient tools to describe and analyse attack scenarios aimed at computer networks. More precisely, attack graphs give all possible scenarios for an attacker to reach a certain goal, exploiting vulnerabilities of the targeted network. Nevertheless they give no information about the damages induced by these attacks, nor about the probability of exploitation of these scenarios. In this paper, we propose to combine attack graphs and CVSS framework, in order to add damage and exploitability probability information. Then, we define a notion of risk for each attack scenario, which is based on quantitative information added to attack graphs.

In this article we summarize the strengths and weaknesses of empirical research in software engineering. We argue that in order to improve the current situation we must create better studies and draw more credible interpretations from them. We finally present a roadmap for this improvement, which includes a general structure for software empirical studies and concrete steps for achieving these goals: designing better studies, collecting data more effectively, and involving others in our empirical enterprises.

Practical software security metrics and measurements are essential to the development of secure software [18]. In this paper, we propose to use a software system’s attack surface measurement as an indicator of the system’s security; the larger the attack surface, the more insecure the system. We formalize the notion of a system’s attack surface using an I/O automata model of the system [15] and define a quantitative measure of the attack surface in terms of three kinds of resources used in attacks on the system: methods, channels, and data. We demonstrate the feasibility of our approach by measuring the attack surfaces of two open source FTP daemons and two IMAP servers. Software developers can use our attack surface measurement method in the software development process and software consumers can use the method in their decision making process. This research was sponsored by the US Army Research Office (ARO) under contract no. DAAD190210389, SAP Labs, LLC under award no. 1010751, and the Software Engineering Institute. The views and conclusions contained in this document are those of the author and should not be interpreted as representing the official policies, either expressed or implied, of any sponsoring institution, the U.S. government or any other entity. Report Documentation Page Form Approved

Security breaches are largely caused by the vulnerable software. Since individuals and organizations mostly depend on softwares, it is important to produce in secured manner. The first step towards producing secured software is through gathering security requirements. This paper describes Software Security Requirements Gathering Instrument (SSRGI) that helps gather security requirements from the various stakeholders. This will guide the developers to gather security requirements along with the functional requirements and further incorporate security during other phases of software development. We subsequently present case studies that describe the integration of the SSRGI instrument with Software Requirements Specification (SRS) document as specified in standard IEEE 830-1998. Proposed SSRGI will support the software developers in gathering security requirements in detail during requirements gathering phase.

This paper gives an overview about the development of the field of Knowledge Engineering over the last 15 years. We discuss the paradigm shift from a transfer view to a modeling view and describe two approaches which considerably shaped research in Knowledge Engineering: Role-limiting Methods and Generic Tasks. To illustrate various concepts and methods which evolved in the last years we describe three modeling frameworks: CommonKADS, MIKE, and PROTÉGÉ-II. This description is supplemented by discussing some important methodological developments in more detail: specification languages for knowledge-based systems, problem-solving methods, and ontologies. We conclude with outlining the relationship of Knowledge Engineering to Software Engineering, Information Integration and Knowledge Management.

Rogue devices are an increasingly dangerous reality in the insider threat problem domain. Industry, government, and academia need to be aware of this problem and promote state-of-the-art detection methods.

Organizations develop technical and procedural measures to protect information systems. Relying only on technical based security solutions is not enough. Organizations must consider technical security solutions along with social, human, and organizational factors. The human element represents the employees (insiders) who use the information systems and other technology resources in their day-to-day operations. ISP awareness is essential to protect organizational information systems. This study adapts the Innovation Diffusion Theory to examine the antecedents of ISP awareness and its impact on the satisfaction with ISP and security practices. A sample of 236 employees in universities in the United States is collected to evaluate the research model. Results indicated that ISP quality, self-efficacy, and technology security awareness significantly impact ISP awareness. The current study presents significant contributions toward understanding the antecedents of ISP awareness and provides a starting point toward including satisfaction aspect in information security behavioral domain.

The automotive industry has successfully collaborated to release the ISO 26262 standard for developing safe software for cars. The standard describes in detail how to conduct hazard analysis and risk assessments to determine the necessary safety measures for each feature. However, the standard does not concern threat analysis for malicious attackers or how to select appropriate security countermeasures. We propose to apply ISO 27001 for this purpose and show how it can be applied together with ISO 26262. We show how ISO 26262 documentation can be re-used and enhanced to satisfy the analysis and documentation demands of the ISO 27001 standard. We illustrate our approach based on an electronic steering column lock system.

Online social networking communities such as Facebook and MySpace are extremely popular. These sites have changed how many people develop and maintain relationships through posting and sharing personal information. The amount and depth of these personal disclosures have raised concerns regarding online privacy. We expand upon previous research on users’ under-utilization of available privacy options by examining users’ current strategies for maintaining their privacy, and where those strategies fail, on the online social network site Facebook. Our results demonstrate the need for mechanisms that provide awareness of the privacy impact of users’ daily interactions.


Question: A biologically plausible deep learning methodology for neural networks in the brain.

This study introduces a novel mechanism that challenges the conventional wisdom regarding the backpropagation algorithm's applicability to the brain's learning processes. The brain's ability to process information through a layered structure of neurons, akin to deep learning architectures in machine learning, is a powerful feature. However, this complexity also poses a significant challenge in identifying which neurons are responsible for errors during learning. Traditionally, the backpropagation algorithm resolves this issue by calculating blame through a series of multiplications involving error signals and the weights of all synapses along a neuron's axon and beyond. This process, however, assumes a symmetrical and precise connectivity pattern that is unlikely to exist in the brain's biological network.

Our research shows that this stringent architectural requirement is not a necessity for effective error propagation. We propose a mechanism that simplifies the process by multiplying error signals with even random synaptic weights, rather than requiring a precise match. This method, surprisingly, can effectively transmit teaching signals through multiple layers of neurons, performing comparably to the traditional backpropagation algorithm on various tasks. Our findings not only reinvigorate the debate on how the brain might utilize error signals but also debunk long-standing assumptions about the algorithmic limitations on learning within biological systems. This breakthrough could potentially pave the way for a deeper understanding of the brain's learning mechanisms and inspire new, biologically plausible algorithms for artificial neural networks.

Research into the functioning of the human brain, particularly during the past decade, has greatly enhanced our understanding of cognitive behaviours which are fundamental to education: learning, memory, intelligence, emotion. Here, we argue the case that research findings from cognitive neuroscience hold implications for educational practice. In doing so we advance a bio-psycho-social position that welcomes multi-disciplinary perspectives on current educational challenges. We provide some examples of research implications which support conventional pedagogic wisdom, and others which are novel and perhaps counter-intuitive. As an example, we take a model of adaptive plasticity that relies on stimulus reinforcement and examine possible implications for pedagogy and curriculum depth. In doing so, we reject some popular but over-simplistic applications of neuroscience to education. In sum, the education profession could benefit from embracing rather than ignoring cognitive neuroscience. Moreover, educationists should be actively contributing to the research agenda of future brain research.

We consider deep neural networks, in which the output of each node is a quadratic function of its inputs. Similar to other deep architectures, these networks can compactly represent any function on a finite training set. The main goal of this paper is the derivation of an efficient layer-by-layer algorithm for training such networks, which we denote as the Basis Learner. The algorithm is a universal learner in the sense that the training error is guaranteed to decrease at every iteration, and can eventually reach zero under mild conditions. We present practical implementations of this algorithm, as well as preliminary experimental results. We also compare our deep architecture to other shallow architectures for learning polynomials, in particular kernel learning.

Restricted Boltzmann Machines (RBMs) have been demonstrated to perform efficiently on a variety of applications, such as dimensionality reduction and classification. Implementing RBMs on neuromorphic hardware has certain advantages, particularly from a concurrency and lowpower perspective. This paper outlines some of the requirements involved for neuromorphic adaptation of an RBM and attempts to address these issues with suitably targeted modifications for sampling and weight updates. Results show the feasibility of such alterations which will serve as a guide for future implementation of such algorithms in VLSI arrays of spiking neurons.

Several approaches to implementing symbol-like representations in neurally plausible models have been proposed. These approaches include binding through synchrony (Shastri & Ajjanagadde, ), "mesh" binding (van der Velde & de Kamps, ), and conjunctive binding (Smolensky, ). Recent theoretical work has suggested that most of these methods will not scale well, that is, that they cannot encode structured representations using any of the tens of thousands of terms in the adult lexicon without making implausible resource assumptions. Here, we empirically demonstrate that the biologically plausible structured representations employed in the Semantic Pointer Architecture (SPA) approach to modeling cognition (Eliasmith, ) do scale appropriately. Specifically, we construct a spiking neural network of about 2.5 million neurons that employs semantic pointers to successfully encode and decode the main lexical relations in WordNet, which has over 100,000 terms. In addition, we show that the same representations can be employed to construct recursively structured sentences consisting of arbitrary WordNet concepts, while preserving the original lexical structure. We argue that these results suggest that semantic pointers are uniquely well-suited to providing a biologically plausible account of the structured representations that underwrite human cognition.

A neural network architecture for the learning of recognition categories is derived. Real-time network dynamics are completely characterized through mathematical analysis and computer simulations. The architecture self-organizes and self-stabilizes its recognition codes in response to arbitrary orderings of arbitrarily many and arbitrarily complex binary input patterns. Top-down attentional and matching mechanisms are critical in self-stabilizing the code learning process. The architecture embodies a parallel search scheme which updates itself adaptively as the learning process unfolds. After learning self-stabilizes, the search process is automatically disengaged. Thereafter input patterns directly access their recognition codes without any search. Thus recognition time does not grow as a function of code complexity. A novel input pattern can directly access a category if it shares invariant properties with the set of familiar exemplars of that category. These invariant properties emerge in the form of learned critical feature patterns, or prototypes. The architecture possesses a context-sensitive self-scaling property which enables its emergent critical feature patterns to form. They detect and remember statistically predictive configurations of featural elements which are derived ~ ..from the set of all input patterns that are ever experienced. Four types of attentional

Online sequence learning from streaming data is one of the most challenging topics in machine learning. Neural network models represent promising candidates for sequence learning due to their ability to learn and recognize complex temporal patterns. In this paper, we present a comparative study of Hierarchical Temporal Memory (HTM), a neurally-inspired model, and other feedforward and recurrent artificial neural network models on both artificial and real-world sequence prediction algorithms. HTM and long-short term memory (LSTM) give the best prediction accuracy. HTM additionally demonstrates many other features that are desirable for real-world sequence learning, such as fast adaptation to changes in the data stream, robustness to sensor noise and fault tolerance. These features make HTM an ideal candidate for online sequence learning problems.

Chapter 1 strongly advocates the stochastic back-propagation method to train neural networks. This is in fact an instance of a more general technique called stochastic gradient descent (SGD). This chapter provides background material, explains why SGD is a good learning algorithm when the training set is large, and provides useful recommendations.

THE relationship between neuronal activity and psychophysical judgement has long been of interest to students of sensory processing. Previous analyses of this problem have compared the performance of human or animal observers in detection or discrimination tasks with the signals carried by individual neurons, but have been hampered because neuronal and perceptual data were not obtained at the same time and under the same conditions1–4. We have now measured the performance of monkeys and of visual cortical neurons while the animals performed a psychophysical task well matched to the properties of the neurons under study. Here we report that the reliability and sensitivity of most neurons on this task equalled or exceeded that of the monkeys. We therefore suggest that under our conditions, psychophysical judgements could be based on the activity of a relatively small number of neurons.

To better understand deep architectures and unsupervised learning, uncluttered by hardware details, we develop a general autoencoder framework for the comparative study of autoencoders, including Boolean autoencoders. We derive several results regarding autoencoders and autoencoder learning, including results on learning complexity, vertical and horizontal composition, and fundamental connections between critical points and clustering. Possible implications for the theory of deep architectures are discussed.

Discovering the structure inherent in a set of patterns is a fundamental aim of statistical inference or learning. One fruitful approach is to build a parameterized stochastic generative model, independent draws from which are likely to produce the patterns. For all but the simplest generative models, each pattern can be generated in exponentially many ways. It is thus intractable to adjust the parameters to maximize the probability of the observed patterns. We describe a way of finessing this combinatorial explosion by maximizing an easily computed lower bound on the probability of the observations. Our method can be viewed as a form of hierarchical self-supervised learning that may relate to the function of bottom-up and top-down cortical processing pathways.


Question: A biologically plausible algorithm for deep learning in neural networks.

Diving Deeper into Dopamine's Dual Role in Reward and Learning
Emerging neurophysiological findings indicate that specific neurons in various brain regions encode distinct signals related to past and anticipated rewards. Notably, dopamine neurons exhibit a rapid, phasic response that reflects the disparity between actual and anticipated rewards, playing a pivotal role in enhancing neural processing and facilitating learning of behavioral responses. This signal is notably separate from dopamine's sustained influence on a wide range of behavioral functions. Cells in the striatum, frontal cortex, and amygdala also process reward information, offering more nuanced insights for identifying, predicting rewards, and orchestrating goal-directed behaviors. These diverse reward signals serve complementary purposes, and the effective integration of these signals is crucial for the refinement of voluntary behaviors. Addictive psychostimulants, such as cocaine and amphetamines, may exploit this system by artificially amplifying the dopamine reward signal, thereby disrupting normal learning and reward processing mechanisms.

Training deep neural networks generally requires massive amounts of data and is very computation intensive. We show here that it may be possible to circumvent the expensive gradient descent procedure and derive the parameters of a neural network directly from properties of the training data. We show that, near convergence, the gradient descent equations for layers close to the input can be linearized and become stochastic equations with noise related to the covariance of data for each class. We derive the distribution of solutions to these equations and discover that it is related to a “supervised principal component analysis.” We implement these results on image datasets MNIST, CIFAR10 and CIFAR100 and find that, indeed, pretrained layers using our findings performs comparable or superior to neural networks of the same size and architecture trained with gradient descent. Moreover, our pretrained layers can often be calculated using a fraction of the training data, owing to the quick convergence of the covariance matrix. Thus, our findings indicate that we can cut the training time both by requiring only a fraction of the data used for gradient descent, and by eliminating layers in the costly backpropagation step of the training. Additionally, these findings partially elucidate the inner workings of deep neural networks and allow us to mathematically calculate optimal solutions for some stages of classification problems, thus significantly boosting our ability to solve such problems efficiently.

Intrabody nanonetworks for nervous system monitoring are envisioned as a key application of the Internet of Nano-Things (IoNT) paradigm, with the aim of developing radically new medical diagnosis and treatment techniques. Indeed, very recently, bionic devices have been implanted inside a living human brain as innovative treatment for drug-resistant epilepsy. In this context, this paper proposes a systems-theoretic communication model to capture the actual behavior of biological neurons. Specifically, biological neurons exhibit physical extension due to their projections called dendrites, which propagate the electrochemical stimulation received via synapses to the soma. Experimental evidences show that the dendrites exhibit two main features: 1) the compartmentalization at the level of the dendritic branches of the neuronal processes and 2) the location-dependent preference for different frequencies. Stemming from these experimental evidences, we propose to model the dendritic tree as a spatiotemporal filter bank, where each filter models the behavior in both space and time of a dendritic branch. Each filter is fully characterized along with the overall neuronal response. Furthermore, sufficient conditions on the incoming stimulus for inducing a null-neuronal response are derived. The conducted theoretical analysis shows that: 1) the neuronal information is encoded in the stimulus temporal pattern, i.e., it is possible to select the neuron to affect by changing the stimulus frequency content; in this sense, the communication among neurons is frequency-selective and 2) the spatial distribution of the dendrites affects the neuronal response; in this sense, the communication among neurons is spatial-selective. The theoretical analysis is validated through a real neuron morphology.

Cognitive theory has decomposed human mental abilities into cognitive (sub) systems, and cognitive neuroscience succeeded in disclosing a host of relationships between cognitive systems and specific structures of the human brain. However, an explanation of why specific functions are located in specific brain loci had still been missing, along with a neurobiological model that makes concrete the neuronal circuits that carry thoughts and meaning. Brain theory, in particular the Hebb-inspired neurocybernetic proposals by Braitenberg, now offers an avenue toward explaining brain–mind relationships and to spell out cognition in terms of neuron circuits in a neuromechanistic sense. Central to this endeavor is the theoretical construct of an elementary functional neuronal unit above the level of individual neurons and below that of whole brain areas and systems: the distributed neuronal assembly (DNA) or thought circuit (TC). It is shown that DNA/TC theory of cognition offers an integrated explanatory perspective on brain mechanisms of perception, action, language, attention, memory, decision and conceptual thought. We argue that DNAs carry all of these functions and that their inner structure (e.g., core and halo subcomponents), and their functional activation dynamics (e.g., ignition and reverberation processes) answer crucial localist questions, such as why memory and decisions draw on prefrontal areas although memory formation is normally driven by information in the senses and in the motor system. We suggest that the ability of building DNAs/TCs spread out over different cortical areas is the key mechanism for a range of specifically human sensorimotor, linguistic and conceptual capacities and that the cell assembly mechanism of overlap reduction is crucial for differentiating a vocabulary of actions, symbols and concepts.

Replicator neural networks self-organize by using their inputs as desired outputs; they internally form a compressed representation for the input data. A theorem shows that a class of replicator networks can, through the minimization of mean squared reconstruction error (for instance, by training on raw data examples), carry out optimal data compression for arbitrary data vector sources. Data manifolds, a new general model of data sources, are then introduced and a second theorem shows that, in a practically important limiting case, optimal-compression replicator networks operate by creating an essentially unique natural coordinate system for the manifold.

The hippocampus places us both in time and space. It does so over remarkably large spans: milliseconds to years, and centimeters to kilometers. This works for sensory representations, for memory, and for behavioral context. How does it fit in such wide ranges of time and space scales, and keep order among the many dimensions of stimulus context? A key organizing principle for a wide sweep of scales and stimulus dimensions is that of order in time, or sequences. Sequences of neuronal activity are ubiquitous in sensory processing, in motor control, in planning actions, and in memory. Against this strong evidence for the phenomenon, there are currently more models than definite experiments about how the brain generates ordered activity. The flip side of sequence generation is discrimination. Discrimination of sequences has been extensively studied at the behavioral, systems, and modeling level, but again physiological mechanisms are fewer. It is against this backdrop that I discuss two recent developments in neural sequence computation, that at face value share little beyond the label "neural." These are dendritic sequence discrimination, and deep learning. One derives from channel physiology and molecular signaling, the other from applied neural network theory - apparently extreme ends of the spectrum of neural circuit detail. I suggest that each of these topics has deep lessons about the possible mechanisms, scales, and capabilities of hippocampal sequence computation.

The performance of neural network classifiers is determined by a number of hyperparameters, including learning rate, batch size, and depth. A number of attempts have been made to explore these parameters in the literature, and at times, to develop methods for optimizing them. However, exploration of parameter spaces has often been limited. In this note, I report the results of large scale experiments exploring these different parameters and their interactions. 1 Datasets and Libraries All experiments reported here were carried out using the Torch library [1] and CUDA (some of the experiments have been reproduced on a smaller scale with other libraries). The dataset for all the experiments is MNIST [3, 2]. Characters were deskewed prior to all experiments. Deskewing significantly reduces error rates in nearest neighbor classifiers. Skew corresponds to a simple one-parameter family of linear transformations in feature space and causes decision regions to become highly anisotropic. Without deskewing, differences in performance between different architectures might primarily reduce to their ability to “learn deskewing”. With deskewing, MNIST character classification become more of an instance of a typical classification problem. Prior results on classifying deskewed MNIST data both with neural networks and with other methods are shown in the table below. 2 Logistic vs Softmax Outputs Multi-Layer Perceptrons (MLPs) used for classification usually attempt to approximate posterior probabilities and use those as their discriminant function. Two common approaches to this are the use of least square regression with logistic output units trained with a least square error measure (“logistic outputs”) 1 ar X iv :1 50 8. 02 78 8v 1 [ cs .N E ] 1 2 A ug 2 01 5 Method Test Error Preprocessing Reference Reduced Set SVM deg 5 polynomial 1 deskewing LeCun et al. 1998 SVM deg 4 polynomial 1.1 deskewing LeCun et al. 1998 K-nearest-neighbors, L3 1.22 deskewing, noise removal, blurring, 2 pixel shift Kenneth Wilder, U. Chicago K-nearest-neighbors, L3 1.33 deskewing, noise removal, blurring, 1 pixel shift Kenneth Wilder, U. Chicago 2-layer NN, 300 HU 1.6 deskewing LeCun et al. 1998 Table 1: Other previously reported results on the MNIST database. and a softmax output layer (“softmax outputs”). In the limit of infinite amounts of training data, both approaches converge to true posterior probability estimates. Softmax output layers have the property that they are guaranteed to produce a normalized posterior probability distribution across all classes, while least square regression with logistic output units generates independent probability estimates for each class membership without any guarantees that these probabilities sum up to one. Softmax is often preferred, although there is no obvious theoretical reason why it should yield better discriminant functions or lower classification error for finite training sets. In OCR and speech recognition, some practitioners have observed that logistic outputs yield better posterior probability estimates and better results when combined with probabilistic language models. In addition, when the sum of the posterior probability estimates derived from logistic outputs differs significantly from unity, that is a strong indication that the input lies outside the training set and should be rejected. Figure 1 shows a scatterplot of test vs training error for a large number of MLPs with one hidden layer at different learning rates, different number of hidden units, and different batch sizes. Such scatterplots show what error rates are achievable by the different architectures, hyperparameter choices, initializations, and order of sample presentations. The lowest points in the vertical direction indicate the lowest test set error achievable by the architecture in this set of experiments. The scatterplot shows that logistic outputs achieve test set error rates of about 1.0% vs 1.1% for softmax outputs. At the same time, logistic outputs never achieve zero percent training set error, while softmax outputs frequently do. In order to ascertain that the difference in test set error between the two

The predominant methodology in training deep learning advocates the use of stochastic gradient descent methods (SGDs). Despite its ease of implementation, SGDs are difficult to tune and parallelize. These problems make it challenging to develop, debug and scale up deep learning algorithms with SGDs. In this paper, we show that more sophisticated off-the-shelf optimization methods such as Limited memory BFGS (L-BFGS) and Conjugate gradient (CG) with line search can significantly simplify and speed up the process of pretraining deep algorithms. In our experiments, the difference between LBFGS/CG and SGDs are more pronounced if we consider algorithmic extensions (e.g., sparsity regularization) and hardware extensions (e.g., GPUs or computer clusters). Our experiments with distributed optimization support the use of L-BFGS with locally connected networks and convolutional neural networks. Using L-BFGS, our convolutional network model achieves 0.69% on the standard MNIST dataset. This is a state-of-theart result on MNIST among algorithms that do not use distortions or pretraining.

The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Until now neural networks have not been capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks that they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on a hand-written digit dataset and by learning several Atari 2600 games sequentially.

In this paper the task of training subsymbolic systems is considered as a combinatorial optimization problem and solved with the heuristic scheme of the reactive tabu search (RTS). An iterative optimization process based on a "modified local search" component is complemented with a meta-strategy to realize a discrete dynamical system that discourages limit cycles and the confinement of the search trajectory in a limited portion of the search space. The possible cycles are discouraged by prohibiting (i.e., making tabu) the execution of moves that reverse the ones applied in the most recent part of the search. The prohibition period is adapted in an automated way. The confinement is avoided and a proper exploration is obtained by activating a diversification strategy when too many configurations are repeated excessively often. The RTS method is applicable to nondifferentiable functions, is robust with respect to the random initialization, and effective in continuing the search after local minima. Three tests of the technique on feedforward and feedback systems are presented.

Human agents draw a variety of inferences effortlessly, spontaneously, and with remarkable efficiency — as though these inferences are a reflex response of their cognitive apparatus. Furthermore, these inferences are drawn with reference to a large body of background knowledge. This remarkable human ability seems paradoxical given the results about the complexity of reasoning reported by researchers in artificial intelligence. It also poses a challenge for cognitive science and computational neuroscience: How can a system of simple and slow neuron-like elements represent a large body of systematic knowledge and perform a range of inferences with such speed? We describe a computational model that is a step toward addressing the cognitive science challenge and resolving the artificial intelligence paradox. We show how a connectionist network can encode millions of facts and rules involving n-ary predicates and variables, and perform a class of inferences in a few hundred msec. Efficient reasoning requires the rapid representation and propagation of dynamic bindings. Our model achieves this by i) representing dynamic bindings as the synchronous firing of appropriate nodes, ii) rules as interconnection patterns that direct the propagation of rhythmic activity, iii) and long-term facts as temporal pattern-matching sub-networks. The model is consistent with recent neurophysiological findings which suggest that synchronous activity occurs in the brain and may play a representational role in neural information processing. The model also makes specific predictions about the nature of reflexive reasoning that are psychologically significant. It identifies constraints on the form of rules that may participate in such reasoning and relates the capacity of the working memory underlying reflexive reasoning to biological parameters such as the lowest frequency at which nodes can sustain synchronous oscillations and the coarseness of synchronization.


Question: A biologically plausible method for deep learning in neural networks within the brain.

This paper explores the application of Deep Big Multilayer Perceptrons (MLPs) in the realm of digit recognition, specifically targeting the competitive MNIST benchmark. Since its inception in 1998, the MNIST challenge has seen a series of record-breaking achievements, with the most recent significant progress occurring eight years ago, achieving an error rate of 0.4%. Utilizing a traditional online back-propagation algorithm for MLPs, we report an impressively low error rate of 0.35% for a single MLP and 0.31% for an ensemble of seven MLPs on the MNIST handwritten digit dataset. As of 2011, these results represent the state-of-the-art in the field. The key to this success lies in the use of multiple hidden layers, each with a large number of neurons, extensive augmentation of the training dataset to prevent overfitting, and the employment of graphics processing units (GPUs) to accelerate the training process. These findings highlight the enduring efficacy of classical MLP architectures when coupled with modern computational resources and advanced data augmentation techniques.

The relation between the structure of the nervous system and its function is more poorly understood than the relation between structure and function in any other organ system. We explore why bridging the structure-function divide is uniquely difficult in the brain. These difficulties also explain the thrust behind the enormous amount of innovation centered on microscopy in neuroscience. We highlight some recent progress and the challenges that remain.

In this paper, a new representation of neural tensor networks is presented. Recently, state-of-the-art neural tensor networks have been introduced to complete RDF knowledge bases. However, mathematical model representation of these networks is still a challenging problem, due to tensor parameters. To solve this problem, it is proposed that these networks can be represented as two-layer perceptron network. To complete the network topology, the traditional gradient based learning rule is then developed. It should be mentioned that for tensor networks there have been developed some learning rules which are complex in nature due to the complexity of the objective function used. Indeed, this paper is aimed to show that the tensor network can be viewed and represented by the two-layer feedforward neural network in its traditional form. The simulation results presented in the paper easily verify this claim.

In this paper we propose a network architecture that combines a rule-based approach with that of the neural network paradigm. Our primary motivation for this is to ensure that the knowledge embodied in the network is explicitly encoded in the form of understandable rules. This enables the network's decision to be understood, and provides an audit trail of how that decision was arrived at. We utilize an information theoretic approach to learning a model of the domain knowledge from examples. This model takes the form of a set of probabilistic conjunctive rules between discrete input evidence variables and output class variables. These rules are then mapped onto the weights and nodes of a feedforward neural network resulting in a directly specified architecture. The network acts as parallel Bayesian classifier, but more importantly, can also output posterior probability estimates of the class variables. Empirical tests on a number of data sets show that the rule-based classifier performs comparably with standard neural network classifiers, while possessing unique advantages in terms of knowledge representation and probability estimation.

Received Feb 10, 2017 Revised Apr 14, 2017 Accepted May 23, 2017 Deep learning is a technique of machine learning in artificial intelligence area. Deep learning in a refined "machine learning" algorithm that far surpasses a considerable lot of its forerunners in its capacities to perceive syllables and picture. Deep learning is as of now a greatly dynamic examination territory in machine learning and example acknowledgment society. It has increased colossal triumphs in an expansive zone of utilizations, for example, speech recognition, computer vision and natural language processing and numerous industry item. Neural network is used to implement the machine learning or to design intelligent machines. In this paper brief introduction to all machine learning paradigm and application area of deep machine learning and different types of neural networks with applications is discussed. Keyword:

Human agents draw a variety of inferences effortlessly, spontaneously, and with remarkable efficiency — as though these inferences are a reflex response of their cognitive apparatus. Furthermore, these inferences are drawn with reference to a large body of background knowledge. This remarkable human ability seems paradoxical given the results about the complexity of reasoning reported by researchers in artificial intelligence. It also poses a challenge for cognitive science and computational neuroscience: How can a system of simple and slow neuron-like elements represent a large body of systematic knowledge and perform a range of inferences with such speed? We describe a computational model that is a step toward addressing the cognitive science challenge and resolving the artificial intelligence paradox. We show how a connectionist network can encode millions of facts and rules involving n-ary predicates and variables, and perform a class of inferences in a few hundred msec. Efficient reasoning requires the rapid representation and propagation of dynamic bindings. Our model achieves this by i) representing dynamic bindings as the synchronous firing of appropriate nodes, ii) rules as interconnection patterns that direct the propagation of rhythmic activity, iii) and long-term facts as temporal pattern-matching sub-networks. The model is consistent with recent neurophysiological findings which suggest that synchronous activity occurs in the brain and may play a representational role in neural information processing. The model also makes specific predictions about the nature of reflexive reasoning that are psychologically significant. It identifies constraints on the form of rules that may participate in such reasoning and relates the capacity of the working memory underlying reflexive reasoning to biological parameters such as the lowest frequency at which nodes can sustain synchronous oscillations and the coarseness of synchronization.

To better understand deep architectures and unsupervised learning, uncluttered by hardware details, we develop a general autoencoder framework for the comparative study of autoencoders, including Boolean autoencoders. We derive several results regarding autoencoders and autoencoder learning, including results on learning complexity, vertical and horizontal composition, and fundamental connections between critical points and clustering. Possible implications for the theory of deep architectures are discussed.

Despite their great success, there is still no comprehensive theoretical understanding of learning with Deep Neural Networks (DNNs) or their inner organization. Previous work [Tishby and Zaslavsky (2015)] proposed to analyze DNNs in the Information Plane; i.e., the plane of the Mutual Information values that each layer preserves on the input and output variables. They suggested that the goal of the network is to optimize the Information Bottleneck (IB) tradeoff between compression and prediction, successively, for each layer. In this work we follow up on this idea and demonstrate the effectiveness of the InformationPlane visualization of DNNs. Our main results are: (i) most of the training epochs in standard DL are spent on compression of the input to efficient representation and not on fitting the training labels. (ii) The representation compression phase begins when the training errors becomes small and the Stochastic Gradient Decent (SGD) epochs change from a fast drift to smaller training error into a stochastic relaxation, or random diffusion, constrained by the training error value. (iii) The converged layers lie on or very close to the Information Bottleneck (IB) theoretical bound, and the maps from the input to any hidden layer and from this hidden layer to the output satisfy the IB self-consistent equations. This generalization through noise mechanism is unique to Deep Neural Networks and absent in one layer networks. (iv) The training time is dramatically reduced when adding more hidden layers. Thus the main advantage of the hidden layers is computational. This can be explained by the reduced relaxation time, as this it scales super-linearly (exponentially for simple diffusion) with the information compression from the previous layer. (v) As we expect critical slowing down of the stochastic relaxation near phase transitions on the IB curve, we expect the hidden layers to converge to such critical points.1

We consider the problem of learning from examples in layered linear feed-forward neural networks using optimization methods, such as back propagation, with respect to the usual quadratic error function E of the connection weights. Our main result is a complete description of the landscape attached to E in terms of principal component analysis. We show that E has a unique minimum corresponding to the projection onto the subspace generated by the first principal vectors of a covariance matrix associated with the training patterns. All the additional critical points of E are saddle points (corresponding to projections onto subspaces generated by higher order vectors). The auto-associative case is examined in detail. Extensions and implications for the learning algorithms are discussed.

The present phase of Machine Learning is characterized by supervised learning algorithms relying on large sets of labeled examples (n→∞). The next phase is likely to focus on algorithms capable of learning from very few labeled examples (n → 1), like humans seem able to do. We propose an approach to this problem and describe the underlying theory, based on the unsupervised, automatic learning of a “good” representation for supervised learning, characterized by small sample complexity (n). We consider the case of visual object recognition though the theory applies to other domains. The starting point is the conjecture, proved in specific cases, that image representations which are invariant to translations, scaling and other transformations can considerably reduce the sample complexity of learning. We prove that an invariant and unique (discriminative) signature can be computed for each image patch, I, in terms of empirical distributions of the dot-products between I and a set of templates stored during unsupervised learning. A module performing filtering and pooling, like the simple and complex cells described by Hubel and Wiesel, can compute such estimates. Hierarchical architectures consisting of this basic Hubel-Wiesel moduli inherit its properties of invariance, stability, and discriminability while capturing the compositional organization of the visual world in terms of wholes and parts. The theory extends existing deep learning convolutional architectures for image and speech recognition. It also suggests that the main computational goal of the ventral stream of visual cortex is to provide a hierarchical representation of new objects/images which is invariant to transformations, stable, and discriminative for recognition—and that this representation may be continuously learned in an unsupervised way during development and visual experience. This work was supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF 1231216. http://cbmm.mit.edu CBMM Paper March, 2014 1–24 ar X iv :1 31 1. 41 58 v5 [ cs .C V ] 1 1 M ar 2 01 4 Unsupervised learning of invariant representations with low sample complexity: the magic of sensory cortex or a new framework for machine learning? Fabio Anselmi ∗ †, Joel Z. Leibo ∗ , Lorenzo Rosasco ∗ † , Jim Mutch ∗ , Andrea Tacchetti ∗ † , and Tomaso Poggio ∗ † ∗Center for Brains, Minds and Machines, Massachusetts Institute of Technology, Cambridge, MA 02139, and †Istituto Italiano di Tecnologia, Genova, 16163 The present phase of Machine Learning is characterized by supervised learning algorithms relying on large sets of labeled examples (n→∞). The next phase is likely to focus on algorithms capable of learning from very few labeled examples (n → 1), like humans seem able to do. We propose an approach to this problem and describe the underlying theory, based on the unsupervised, automatic learning of a “good” representation for supervised learning, characterized by small sample complexity (n). We consider the case of visual object recognition though the theory applies to other domains. The starting point is the conjecture, proved in specific cases, that image representations which are invariant to translations, scaling and other transformations can considerably reduce the sample complexity of learning. We prove that an invariant and unique (discriminative) signature can be computed for each image patch, I, in terms of empirical distributions of the dot-products between I and a set of templates stored during unsupervised learning. A module performing filtering and pooling, like the simple and complex cells described by Hubel and Wiesel, can compute such estimates. Hierarchical architectures consisting of this basic Hubel-Wiesel moduli inherit its properties of invariance, stability, and discriminability while capturing the compositional organization of the visual world in terms of wholes and parts. The theory extends existing deep learning convolutional architectures for image and speech recognition. It also suggests that the main computational goal of the ventral stream of visual cortex is to provide a hierarchical representation of new objects/images which is invariant to transformations, stable, and discriminative for recognition—and that this representation may be continuously learned in an unsupervised way during development and visual experience.1 Invariance | Hierarchy | Convolutional networks | Visual cortex It is known that Hubel and Wiesel’s original proposal [1] for visual area V1—of a module consisting of complex cells (C-units) combining the outputs of sets of simple cells (S-units) with identical orientation preferences but differing retinal positions—can be used to construct translationinvariant detectors. This is the insight underlying many networks for visual recognition, including HMAX [2] and convolutional neural nets [3, 4]. We show here how the original idea can be expanded into a comprehensive theory of visual recognition relevant for computer vision and possibly for visual cortex. The first step in the theory is the conjecture that a representation of images and image patches, with a feature vector that is invariant to a broad range of transformations—such as translation, scale, expression of a face, pose of a body, and viewpoint—makes it possible to recognize objects from only a few labeled examples, as humans do. The second step is proving that hierarchical architectures of Hubel-Wiesel (‘HW’) modules (indicated by ∧ in Fig. 1) can provide such invariant representations while maintaining discriminative information about the original image. Each ∧ -module provides a feature vector, which we call a signature, for the part of the visual field that is inside its “receptive field”; the signature is invariant to (R) affine transformations within the receptive field. The hierarchical architecture, since it computes a set of signatures for l=4

Statistical process control (SPC) is a method for improving the quality o f products. Control charting plays a most important role in SPC. SPC control charts arc used for monitoring and detecting unnatural process behaviour. Unnatural patterns in control charts indicate unnatural causes for variations. Control chart pattern recognition is therefore important in SPC. Past research shows that although certain types o f charts, such as the CUSUM chart, might have powerful detection ability, they lack robustness and do not function automatically. In recent years, neural network techniques have been applied to automatic pattern recognition. Spiking Neural Networks (SNNs) belong to the third generation o f artificial neural networks, with spiking neurons as processing elements. In SNNs, time is an important feature for information representation and processing. This thesis proposes the application o f SNN techniques to control chart pattern recognition. It is designed to present an analysis o f the existing learning algorithms o f SNN for pattern recognition and to explain how and why spiking neurons have more computational power in comparison to the previous generation o f neural networks. This thesis focuses on the architecture and the learning procedure o f the network. Four new learning algorithms are presented with their specific architecture: Spiking Learning Vector Quantisation (S-LVQ), Enhanced-Spiking Learning Vector Quantisation (NS-LVQ), S-LVQ with Bees and NS-LVQ with Bees. The latter two algorithms employ a new intelligent swarm-based optimisation called the Bees Algorithm to optimise the LVQ pattern recognition networks. Overall, the aim o f the research is to develop a simple architecture for the proposed network as well as to develop a network that is efficient for application to control chart pattern recognition. Experiments show that the proposed architecture and the learning procedure give high pattern recognition accuracies.