Question: Bat algorithm and cuckoo search: an instructional guide.

This paper delves into the application of Lévy flights, a form of non-local search mechanism, integrated with simulated annealing for solving complex, non-convex stochastic optimization problems. Unlike traditional search methods that are often confined to local movements, Lévy flights, characterized by their large, infrequent jumps, enable a more expansive exploration of the search space. This non-local search feature is particularly advantageous in scenarios where the potential landscape is unknown and the global minimum is the target. The temperature parameter in the simulated annealing process, which governs the acceptance of worse solutions, is adjusted to decrease polynomially, facilitating a rapid convergence to the ground state. This combination of Lévy flights and simulated annealing not only accelerates the search for the global minimum but also enhances the robustness of the optimization process against getting trapped in local optima.

Evolutionary computation (EC) is a kind of optimization methodology inspired by the mechanisms of biological evolution and behaviors of living organisms. In the literature, the terminology evolutionary algorithms is frequently treated the same as EC. This article focuses on making a survey of researches based on using ML techniques to enhance EC algorithms. In the framework of an ML-technique enhanced-EC algorithm (MLEC), the main idea is that the EC algorithm has stored ample data about the search space, problem features, and population information during the iterative search process, thus the ML technique is helpful in analyzing these data for enhancing the search performance. The paper presents a survey of five categories: ML for population initialization, ML for fitness evaluation and selection, ML for population reproduction and variation, ML for algorithm adaptation, and ML for local search.

Swarm intelligence is one of the most promising area for the researchers in the field of numerical optimization. Researchers have developed many algorithms by simulating the swarming behavior of various creatures like ants, honey bees, fish, birds and the findings are very motivating. In this paper, a new approach for numerical optimization is proposed by modeling the foraging behavior of spider monkeys. Spider monkeys have been categorized as fission–fusion social structure based animals. The animals which follow fission– fusion social systems, split themselves from large to smaller groups and vice-versa based on the scarcity or availability of food. The proposed swarm intelligence approach is named as Spider Monkey Optimization (SMO) algorithm and can broadly be classified as an algorithm inspired by intelligent foraging behavior of fission–fusion social structure based animals.

This paper presents a method to employ particle swarms optim izers in a cooperative configuration. This is achieved by splitting the input vector into several sub-vectors, each w hich is optimized cooperatively in its own swarm. The applic ation of this technique to neural network training is investigate d, with promising results.

This paper puts forward two useful methods for self-adaptation of the mutation distribution - the concepts of derandomization and cumulation. Principle shortcomings of the concept of mutative strategy parameter control and two levels of derandomization are reviewed. Basic demands on the self-adaptation of arbitrary (normal) mutation distributions are developed. Applying arbitrary, normal mutation distributions is equiv-alent to applying a general, linear problem encoding. The underlying objective of mutative strategy parameter control is roughly to favor previously selected mutation steps in the future. If this objective is pursued rigor-ously, a completely derandomized self-adaptation scheme results, which adapts arbitrary normal mutation distributions. This scheme, called covariance matrix adaptation (CMA), meets the previously stated demands. It can still be considerably improved by cumulation - utilizing an evolution path rather than single search steps. Simulations on various test functions reveal local and global search properties of the evolution strategy with and without covariance matrix adaptation. Their performances are comparable only on perfectly scaled functions. On badly scaled, non-separable functions usually a speed up factor of several orders of magnitude is ob-served. On moderately mis-scaled functions a speed up factor of three to ten can be expected.

Many software engineering problems have been addressed with search algorithms. Search algorithms usually depend on several parameters (e.g., population size and crossover rate in genetic algorithms), and the choice of these parameters can have an impact on the performance of the algorithm. It has been formally proven in the No Free Lunch theorem that it is impossible to tune a search algorithm such that it will have optimal settings for all possible problems. So, how to properly set the parameters of a search algorithm for a given software engineering problem? In this paper, we carry out the largest empirical analysis so far on parameter tuning in search-based software engineering. More than one million experiments were carried out and statistically analyzed in the context of test data generation for object-oriented software using the EvoSuite tool. Results show that tuning does indeed have impact on the performance of a search algorithm. But, at least in the context of test data generation, it does not seem easy to find good settings that significantly outperform the “default” values suggested in the literature. This has very practical value for both researchers (e.g., when different techniques are compared) and practitioners. Using “default” values is a reasonable and justified choice, whereas parameter tuning is a long and expensive process that might or might not pay off in the end.

Editor: Peter Anderson Rochester Institute of Technology 1 Lomb Memorial Dr. Rochester, NY 14623 pga@cs.rit.edu Each technique receives similar coverage, including a description of its new and prerequisite vocabulary, its model, the bug assumptlons that underlie its effectiveness, typical applications, examples (almost all based on standard IRS tax forms), caveats and limitations, automation and tools for the technique, a self-evaluation quiz, and exercises. Common threads in the presentation show the role

A new heuristic approach for minimizing possibly nonlinear and non differentiable continuous space functions is presented. By means of an extensive testbed, which includes the De Jong functions, it will be demonstrated that the new method converges faster and with more certainty than Adaptive Simulated Annealing as well as the Annealed Nelder&Mead approach, both of which have a reputation for being very powerful. The new method requires few control variables, is robust, easy to use and lends itself very well to parallel computation. ________________________________________ 1)International Computer Science Institute, 1947 Center Street, Berkeley, CA 94704-1198, Suite 600, Fax: 510-643-7684. E-mail: storn@icsi.berkeley.edu. On leave from Siemens AG, ZFE T SN 2, OttoHahn-Ring 6, D-81739 Muenchen, Germany. Fax: 01149-636-44577, Email:rainer.storn@zfe.siemens.de. 2)836 Owl Circle, Vacaville, CA 95687, kprice@solano.community.net.


Question: Bat algorithm and cuckoo search: an instructional guide.

This document introduces a novel meta-heuristic algorithm, known as Cuckoo Search (CS), designed to address optimization challenges. Inspired by the unique brood parasitism observed in certain cuckoo species, coupled with the Lévy flight patterns exhibited by various birds and fruit flies, CS presents a fresh perspective on solving complex optimization problems. To evaluate the efficacy of this algorithm, we conduct a series of tests using benchmark functions and subsequently compare its performance against established methods such as genetic algorithms and particle swarm optimization. Our analysis not only validates the robustness of CS but also highlights its competitive edge in optimization tasks. The paper concludes with an exploration of the implications of these findings and proposes avenues for future research to further enhance and expand the application of Cuckoo Search in various domains.

We explore data-driven methods for gaining insight into the dynamics of a two population genetic algorithm (GA), which has been effective in tests on constrained optimization problems. We track and compare one population of feasible solutions and another population of infeasible solutions. Feasible solutions are selected and bred to improve their objective function values. Infeasible solutions are selected and bred to reduce their constraint violations. Interbreeding between populations is completely indirect, that is, only through their offspring that happen to migrate to the other population. We introduce an empirical measure of distance, and apply it between individuals and between population centroids to monitor the progress of evolution. We find that the centroids of the two populations approach each other and stabilize. This is a valuable characterization of convergence. We find the infeasible population influences, and sometimes dominates, the genetic material of the optimum solution. Since the infeasible population is not evaluated by the objective function, it is free to explore boundary regions, where the optimum is likely to be found. Roughly speaking, the No Free Lunch theorems for optimization show that all blackbox algorithms (such as Genetic Algorithms) have the same average performance over the set of all problems. As such, our algorithm would, on average, be no better than random search or any other blackbox search method. However, we provide two general theorems that give conditions that render null the No Free Lunch results for the constrained optimization problem class we study. The approach taken here thereby escapes the No Free Lunch implications, per se.

Many real-world problems involve two types of problem difficulty: i) multiple, conflicting objectives and ii) a highly complex search space. On the one hand, instead of a single optimal solution competing goals give rise to a set of compromise solutions, generally denoted as Pareto-optimal. In the absence of preference information, none of the corresponding trade-offs can be said to be better than the others. On the other hand, the search space can be too large and too complex to be solved by exact methods. Thus, efficient optimization strategies are required that are able to deal with both difficulties. Evolutionary algorithms possess several characteristics that are desirable for this kind of problem and make them preferable to classical optimization methods. In fact, various evolutionary approaches to multiobjective optimization have been proposed since 1985, capable of searching for multiple Paretooptimal solutions concurrently in a single simulation run. However, in spite of this variety, there is a lack of extensive comparative studies in the literature. Therefore, it has remained open up to now: • whether some techniques are in general superior to others, • which algorithms are suited to which kind of problem, and • what the specific advantages and drawbacks of certain methods are. The subject of this work is the comparison and the improvement of existing multiobjective evolutionary algorithms and their application to system design problems in computer engineering. In detail, the major contributions are: • An experimental methodology to compare multiobjective optimizers is developed. In particular, quantitative measures to assess the quality of trade-off fronts are introduced and a set of general test problems is defined, which are i) easy to formulate, ii) represent essential aspects of real-world problems, and iii) test for different types of problem difficulty. • On the basis of this methodology, an extensive comparison of numerous evolutionary techniques is performed in which further aspects such as the influence of elitism and the population size are also investigated. • A novel approach to multiobjective optimization, the strength Pareto evolutionary algorithm, is proposed. It combines both established and new techniques in a unique manner. • Two complex multicriteria applications are addressed using evolutionary algorithms: i) the automatic synthesis of heterogeneous hardware/systems and ii) the multidimensional exploration of software implementations for digital signal processors. Zusammenfassung Viele praktische Optimierungsprobleme sind durch zwei Eigenschaften charakterisiert: a) mehrere, teilweise im Konflikt stehende Zielfunktionen sind involviert, und b) der Suchraum ist hochgradig komplex. Einerseits f ̈ uhren widersprüchliche Optimierungskriterien dazu, dass es statt eines klar definierten Optimums eine Menge von Kompromissl ̈ o ungen, allgemein als Pareto-optimal bezeichnet, gibt. Insofern keine Gewichtung der Kriterien vorliegt, m ̈ ussen die entsprechenden Alternativen als gleichwertig betrachtet werden. Andererseits kann der Suchraum eine bestimmte Gr ̈ osse und Komplexit ̈ at überschreiten, so dass exakte Optimierungsverfahren nicht mehr anwendbar sind. Erforderlich sind demnach effiziente Suchstrategien, die beiden Aspekten gerecht werden. Evolutionäre Algorithmen sind aufgrund mehrerer Merkmale f ̈ ur diese Art von Problem besonders geeignet; vor allem im Vergleich zu klassischen Methoden weisen sie gewisse Vorteile auf. Doch obwohl seit 1985 verschiedenste evolutionäre Ansätze entwickelt wurden, die mehrere Pareto-optimale L ̈ osungen in einem einzigen Simulationslauf generieren k ̈ onnen, mangelt es in der Literatur an umfassenden Vergleichsstudien. Folglich blieb bislang ungekl ̈ art, • ob bestimmte Techniken anderen Methoden generell  ̈ ub rlegen sind, • welche Algorithmen f ̈ ur welche Art von Problem geeignet sind und • wo die spezifischen Vorund Nachteile einzelner Verfahren liegen. Die vorliegende Arbeit hat zum Gegenstand, bestehende evolution ̈ are Mehrzieloptimierungsverfahren zu vergleichen, zu verbessern und auf Entwurfsprobleme im Bereich der Technischen Informatik anzuwenden. Im Einzelnen werden folgende Themen behandelt: • Eine Methodik zum experimentellen Vergleich von Mehrzieloptimierungsverfahren wird entwickelt. Unter anderem werden quantitative Qualit ̈ atsmasse für Mengen von Kompromissl ̈ osungen eingef ̈ uhrt und mehrere Testfunktionen definiert, die a) eine einfache Problembeschreibung besitzen, b) wesentliche Merkmale realer Optimierungsprobleme repr ̈ asentieren und c) erlauben, verschiedene Einflussfaktoren separat zu  ̈ uberprüfen. • Auf der Basis dieser Methodik wird ein umfangreicher Vergleich diverser evolutionärer Techniken durchgef ̈ uhrt, wobei auch weitere Aspekte wie die Auswirkungen von Elitism und der Populationsgr ̈ osse auf den Optimierungsprozess untersucht werden. • Ein neues Verfahren, der Strength-Pareto-Evolutionary-Algorithm, wird vorgestellt. Es kombiniert auf spezielle Art und Weise bew ̈ ahrte und neue Konzepte miteinander. • Zwei komplexe Mehrzielprobleme werden auf der Basis evolution ̈ arer Methoden untersucht: a) die automatische Synthese von heterogenen Hardware/Software-Systemen und b) die mehrdimensionale Exploration von Softwareimplementierungen f ̈ ur digitale Signalverarbeitungsprozessoren. I would like to thank Prof. Dr. Lothar Thiele for the valuable discussions concerning this research, Prof. Dr. Kalyanmoy Deb for his willingness to be the co-examiner of my thesis,

If you are looking for Genetic Algorithms Data Structures Evolution Programs in pdf file you can find it here. This is the best place for you where you can find the genetic algorithms data structures evolution programs document. Sign up for free to get the download links. There are 3 sources of download links that you can download and save it in your desktop.

Gradient descent optimization algorithms, while increasingly popular, are often used as black-box optimizers, as practical explanations of their strengths and weaknesses are hard to come by. This article aims to provide the reader with intuitions with regard to the behaviour of different algorithms that will allow her to put them to use. In the course of this overview, we look at different variants of gradient descent, summarize challenges, introduce the most common optimization algorithms, review architectures in a parallel and distributed setting, and investigate additional strategies for optimizing gradient descent.

Multi-objectiveevolutionaryalgorithmswhichusenon-dominatedsorting andsharinghave beenmainly criticizedfor their (i) computational complexity (where is thenumberof objectivesand is thepopulationsize), (ii) non-elitismapproach, and(iii) theneedfor specifyingasharingparameter . In this paper , we suggesta non-dominatedsortingbasedmulti-objective evolutionaryalgorithm(wecalledit theNon-dominatedSortingGA-II or NSGA-II) which alleviatesall theabove threedifficulties.Specifically, a fastnon-dominatedsorting approachwith computationalcomplexity is presented. Second,a selectionoperatoris presentedwhich createsa mating pool by combiningthe parentandchild populationsandselectingthe best(with respectto fitnessand spread) solutions.Simulationresultson five difficult testproblemsshow that theproposedNSGA-II is ableto find muchbetterspreadof solutionsin all problemscomparedto PAES—anotherelitist multi-objective EA which paysspecial attentiontowardscreatinga diversePareto-optimalfront. Becauseof NSGA-II’s low computational requirements, elitist approach, andparameter -lesssharingapproach,NSGA-II shouldfind increasingapplicationsin theyearsto come.

Much of the testing of optimization software is inadequate because the number of test functmns is small or the starting points are close to the solution. In addition, there has been too much emphasm on measurmg the efficmncy of the software and not enough on testing reliability and robustness. To address this need, we have produced a relatwely large but easy-to-use collection of test functions and designed gmdelines for testing the reliability and robustness of unconstrained optimization software.

A new global optimization algorithm for functions of many continuous variables is presented, derived from the basic Simulated annealing method. Our main contribution lies in dealing with high-dimensionality minimization problems, which are often difficult to solve by all known minimization methods with or without gradient. In this article we take a special interest in the variables discretization issue. We also develop and implement several complementary stopping criteria. The original Metropolis iterative random search, which takes place in a Euclidean space Rn, is replaced by another similar exploration, performed within a succession of Euclidean spaces Rp, with p <<n. This Enhanced Simulated Annealing (ESA) algorithm was validated first on classical highly multimodal functions of 2 to 100 variables. We obtained significant reductions in the number of function evaluations compared to six other global optimization algorithms, selected according to previously published computational results for the same set of test functions. In most cases, ESA was able to closely approximate known global optima. The reduced ESA computational cost helped us to refine further the obtained global results, through the use of some local search. We have used this new minimizing procedure to solve complex circuit design problems, for which the objective function evaluation can be exceedingly costly.

The objective of image segmentation is to extract meaningful objects. A meaningful segmentation selects the proper threshold values to optimize a criterion using entropy. The conventional multilevel thresholding methods are efficient for bi-level thresholding. However, they are computationally expensive when extended to multilevel thresholding since they exhaustively search the optimal thresholds to optimize the objective functions. To overcome this problem, two successful swarm-intelligence-based global optimization algorithms, cuckoo search (CS) algorithm and wind driven optimization (WDO) for multilevel thresholding using Kapur’s entropy has been employed. For this purpose, best solution as fitness function is achieved through CS and WDO algorithm using Kapur’s entropy for optimal multilevel thresholding. A new approach of CS andWDO algorithm is used for selection of optimal threshold value. This algorithm is used to obtain the best solution or best fitness value from the initial random threshold values, and to evaluate the quality of a solution, correlation function is used. Experimental results have been examined on standard set of satellite images using various numbers of thresholds. The results based on Kapur’s entropy reveal that CS, ELR-CS and WDO method can be accurately and efficiently used in multilevel thresholding problem. 2013 Elsevier Ltd. All rights reserved.

A combination of distributed computation, positive feedback and constructive greedy heuristic is proposed as a new approach to stochastic optimization and problem solving. Positive feedback accounts for rapid discovery of very good solutions, distributed computation avoids premature convergence, and greedy heuristic helps the procedure to find acceptable solutions in the early stages of the search process. An application of the proposed methodology to the classical travelling salesman problem shows that the system can rapidly provide very good, if not optimal, solutions. We report on many simulation results and discuss the working of the algorithm. Some hints about how this approach can be applied to a variety of optimization problems are also given.


Question: Bat algorithm and cuckoo search: an instructional guide.

This paper presents an innovative metaheuristic optimization technique, known as the Cuckoo Search (CS) algorithm, designed to tackle structural optimization challenges. The CS algorithm, augmented with Lévy flights, is initially tested on a well-established benchmark problem involving nonlinear constraints to ensure its accuracy and reliability. To further validate its effectiveness in structural engineering optimization, the CS algorithm is applied to a series of 13 design problems previously documented in the literature. The outcomes of the CS algorithm are then meticulously compared against a range of established optimization techniques, highlighting its cutting-edge status in the field. Impressively, the solutions derived from the CS algorithm consistently outperform those obtained through other contemporary methods. The paper concludes with an in-depth exploration of the distinctive search mechanisms employed by the CS algorithm and their potential implications for future research endeavors in optimization.

Swarm Intelligence is the part of Artificial Intel ligence based on study of actions of individuals in various decentralized sys tems. The Bee Colony Optimization (BCO) metaheuristic has been introduce fairly recently as a new direction in the field of Swarm Intelligence. Artif icial bees represent agents, which collaboratively solve complex combinatorial optimiz ation problem. The chapter presents a classification and analysis of the resul ts achieved using Bee Colony Optimization (BCO) to model complex engineering and management processes. The primary goal of this chapter is to acquaint readers with the basic principles of Bee Colony Optimization, as well as to indicate potenti al BCO applications in engineering and management.

In this paper, we introduce a new parameter, called inertia weight, into the original particle swarm optimizer. Simulations have been done to illustrate the signilicant and effective impact of this new parameter on the particle swarm optimizer.

This paper introduces a novel parameter automation strategy for the particle swarm algorithm and two further extensions to improve its performance after a predefined number of generations. Initially, to efficiently control the local search and convergence to the global optimum solution, time-varying acceleration coefficients (TVAC) are introduced in addition to the time-varying inertia weight factor in particle swarm optimization (PSO). From the basis of TVAC, two new strategies are discussed to improve the performance of the PSO. First, the concept of "mutation" is introduced to the particle swarm optimization along with TVAC (MPSO-TVAC), by adding a small perturbation to a randomly selected modulus of the velocity vector of a random particle by predefined probability. Second, we introduce a novel particle swarm concept "self-organizing hierarchical particle swarm optimizer with TVAC (HPSO-TVAC)". Under this method, only the "social" part and the "cognitive" part of the particle swarm strategy are considered to estimate the new velocity of each particle and particles are reinitialized whenever they are stagnated in the search space. In addition, to overcome the difficulties of selecting an appropriate mutation step size for different problems, a time-varying mutation step size was introduced. Further, for most of the benchmarks, mutation probability is found to be insensitive to the performance of MPSO-TVAC method. On the other hand, the effect of reinitialization velocity on the performance of HPSO-TVAC method is also observed. Time-varying reinitialization step size is found to be an efficient parameter optimization strategy for HPSO-TVAC method. The HPSO-TVAC strategy outperformed all the methods considered in this investigation for most of the functions. Furthermore, it has also been observed that both the MPSO and HPSO strategies perform poorly when the acceleration coefficients are fixed at two.

This paper aims at incorporating immune operators in genetic algorithms as an advanced method for solving the problem of test data generation. The new proposed hybrid algorithm is called immune genetic algorithm (IGA). A full description of this algorithm is presented before investigating its application in the context of software test data generation using some benchmark programs. Moreover, the algorithm is compared with other evolutionary algorithms.

The task of optimizing a complex system presents at least two levels of problems for the system designer. First, a class of optimization algorithms must be chosen that is suitable for application to the system. Second, various parameters of the optimization algorithm need to be tuned for efficiency. A class of adaptive search procedures called genetic algorithms (GA) has been used to optimize a wide variety of complex systems. GA's are applied to the second level task of identifying efficient GA's for a set of numerical optimization problems. The results are validated on an image registration problem. GA's are shown to be effective for both levels of the systems optimization problem.

The genetic algorithm includes some parameters that should be adjusted, so as to get reliable results. Choosing a representation of the problem addressed, an initial population, a method of selection, a crossover operator, mutation operator, the probabilities of crossover and mutation, and the insertion method creates a variant of genetic algorithms. Our work is part of the answer to this perspective to find a solution for this combinatorial problem. What are the best parameters to select for a genetic algorithm that creates a variety efficient to solve the Traveling Salesman Problem (TSP)? In this paper, we present a comparative analysis of different mutation operators, surrounded by a dilated discussion that justifying the relevance of genetic operators chosen to solving the TSP problem.

Geographic Information Systems have gradually acquired greater relevance as tools to support decision-making processes, and during the last decades they have been used in conjunction with Multi-Criteria Decision Analysis techniques (GIS-MCDA) to solve real-world spatial problems. GIS-MCDA can be generally divided in two main approaches: Multi-Attribute and MultiObjective techniques. Until now most of the applications of GIS-MCDA have been focused only on using the multi-attribute approach, and less than 10% of the research has been related to a specific type of multi-objective technique: the use of heuristic/meta-heuristic algorithms. The present study explores how different heuristic/meta-heuristic methods perform on solving a spatial multi-objective optimisation problem. To achieve this, four algorithms representing different types of heuristics methods were implemented, and applied to solve the same spatial multi-objective optimisation problem related with an evacuation planning situation. The implemented algorithms were Standard Particle Swarm Optimisation (SPSO), Non-dominated Sorting Genetic Algorithm II (NSGA-II), Archived Multi-Objective Simulated Annealing (AMOSA) and Multi-Objective Grey Wolf Optimiser (MOGWO). The results show that the four algorithms were effective on solving the given problem, although in general AMOSA and MOGWO had a higher performance for the evaluated aspects (number of solutions, effectiveness of the optimisation, diversity, execution time and repeatability). However, the differences in the results obtained from each algorithm were not clear enough to state that one type of heuristic is superior than others. Since AMOSA and MOGWO are the most recent algorithms among the implemented ones, they include several improvements achieved by the latest research, and their superior performance could be connected to these improvements more than to the specific type of algorithms they belong to. Further research is suggested to explore the suitability of these methods for many-objectives spatial problems, to consider the temporal variability and dynamism of real-world situations, to create a standard set of algorithms to be used for benchmarking, and to integrate them with the currently available GIS-MCDA tools. Despite this, from the performed research it is possible to conclude that heuristics methods are reliable techniques for solving spatial problems with multiple and conflictive objectives, and future research and practical implementations in this field can strengthen the capacities of GIS as a multi-criteria decisionmaking support tool.

This paper reports a Fast Local Search (FLS) algorithm which helps to improve the efficiency of hill climbing and a Guided Local Search (GLS) Algorithm which is developed to help local search to escape local optima and distribute search effort. To illustrate how these algorithms work, this paper describes their application to British Telecom’s workforce scheduling problem, which is a hard real life problem. The effectiveness of FLS and GLS are demonstrated by the fact that they both out-perform all the methods applied to this problem so far, which include simulated annealing, genetic algorithms and constraint logic programming.


Question: Bat algorithm and cuckoo search: an instructional guide.

This paper provides an overview of the latest developments in continuous nonlinear multiobjective optimization (MOO) techniques, consolidating and linking a range of diverse terminologies and methodologies. The survey categorizes these methods into three primary groups: those requiring a priori preference articulation, those allowing a posteriori preference articulation, and those that do not require any preference articulation. Additionally, the paper examines genetic algorithms in detail. Critical insights are offered from three perspectives: the merits and drawbacks of individual methods, the comparative strengths of different method classes, and the overall landscape of MOO. A summary of the key features of the most impactful methods is presented. The conclusions highlight less commonly acknowledged concepts and their relevance to engineering problems. It is evident that no single method holds universal supremacy; the choice of a specific approach is contingent upon the nature of information available in the problem, the user's preferences, the desired solution characteristics, and the accessibility of software tools.

This paper looks into efficient implementation of one dimensional low pass Finite Impulse Response filters using certain commonly used and state-of-the-art optimization techniques. Methods like Parks-McClellan (PM) equiripple design, Quantum-behaved Particle Swarm Optimization (QPSO) and Cuckoo Search Algorithm (CSA) with Levy Flight are employed and overall performance is further improved by hybridization and adaptive step size update. Various performance metrics are analyzed with a focus on increasing convergence speed to reach global optima faster. It is seen that the improved search methods used in this work, i.e., Simulated Annealing based Weighted Mean Best QPSO (SAWQPSO) and Adaptive CSA (ACSA) effect significant reductions in convergence time with ACSA proving to be the faster one. The designed filter is used in the receiver stage of a Frequency Modulated Radio Transmission model using a Quadrature-Phase Shift Keyed (QPSK) Modulator and Demodulator. Its efficiency is validated by obtaining near perfect correlation between the message and recovered signals.

The objective of image segmentation is to extract meaningful objects. A meaningful segmentation selects the proper threshold values to optimize a criterion using entropy. The conventional multilevel thresholding methods are efficient for bi-level thresholding. However, they are computationally expensive when extended to multilevel thresholding since they exhaustively search the optimal thresholds to optimize the objective functions. To overcome this problem, two successful swarm-intelligence-based global optimization algorithms, cuckoo search (CS) algorithm and wind driven optimization (WDO) for multilevel thresholding using Kapur’s entropy has been employed. For this purpose, best solution as fitness function is achieved through CS and WDO algorithm using Kapur’s entropy for optimal multilevel thresholding. A new approach of CS andWDO algorithm is used for selection of optimal threshold value. This algorithm is used to obtain the best solution or best fitness value from the initial random threshold values, and to evaluate the quality of a solution, correlation function is used. Experimental results have been examined on standard set of satellite images using various numbers of thresholds. The results based on Kapur’s entropy reveal that CS, ELR-CS and WDO method can be accurately and efficiently used in multilevel thresholding problem. 2013 Elsevier Ltd. All rights reserved.

This paper demonstrates the computational advantages of a multiobjective framework that can overcome the generic and domain-related challenges in optical system design and optimisation. Non-dominated sorting genetic algorithms-II (Deb, 2003) is employed in this study. The optical systems studied in this paper are Cooke triplets, Petzval lens systems and achromatic doublets. We report the results of four studies. In the first study, we optimise the optical systems using computationally efficient image quality objective functions. Our approach uses only two paraxial rays to estimate the objective functions and thus improves the computational efficiency. This timesaving measure can partially compensate for the typically enormous number of fitness function evaluations required in evolutionary algorithms. The reduction in reliability due to the computations from a single ray pair is compensated by the availability of multiple objective functions that help us to navigate to the optima. In the second study, hybridisation of evolutionary and gradient-based approaches and scaling techniques are employed to speed up convergence and enforce the constraints. The third study shows how recent developments in optical system design research can be better integrated in a multi-objective framework. The fourth study optimises an achromatic doublet with suitable constraints applied to the thicknesses and image distance.

Particle Swarm Optimization (PSO) has been shown to perform very well on a wide range of optimization problems. One of the drawbacks to PSO is that the base algorithm assumes continuous variables. In this paper, we present a version of PSO that is able to optimize over discrete variables. This new PSO algorithm, which we call Integer and Categorical PSO (ICPSO), incorporates ideas from Estimation of Distribution Algorithms (EDAs) in that particles represent probability distributions rather than solution values, and the PSO update modifies the probability distributions. In this paper, we describe our new algorithm and compare its performance against other discrete PSO algorithms. In our experiments, we demonstrate that our algorithm outperforms comparable methods on both discrete benchmark functions and NK landscapes, a mathematical framework that generates tunable fitness landscapes for evaluating EAs.

Direct search methods are best known as unconstrained optimization techniques that do not explicitly use derivatives. Direct search methods were formally proposed and widely applied in the 1960s but fell out of favor with the mathematical optimization community by the early 1970s because they lacked coherent mathematical analysis. Nonetheless, users remained loyal to these methods, most of which were easy to program, some of which were reliable. In the past fifteen years, these methods have seen a revival due, in part, to the appearance of mathematical analysis, as well as to interest in parallel and distributed computing. This review begins by briefly summarizing the history of direct search methods and considering the special properties of problems for which they are well suited. Our focus then turns to a broad class of methods for which we provide a unifying framework that lends itself to a variety of convergence results. The underlying principles allow generalization to handle bound constraints and linear constraints. We also discuss extensions to problems with nonlinear constraints.

This study reports how the differential evolution (DE) algorithm performed on the test bed developed for the CEC05 contest for real parameter optimization. The test bed includes 25 scalable functions, many of which are both non-separable and highly multi-modal. Results include DE's performance on the 10 and 30-dimensional versions of each function

Structure and Algorithms Jakob Puchinger NICTA Victoria Laboratory Department of Computer Science & Software Engineering University of Melbourne, Australia jakobp@csse.unimelb.edu.au Günther R. Raidl Institute of Computer Graphics and Algorithms Vienna University of Technology, Austria raidl@ads.tuwien.ac.at Ulrich Pferschy Department of Statistics and Operations Research University of Graz, Austria pferschy@uni-graz.at

Firefly algorithm is one of the new metaheuristic algorithms for optimization problems. The algorithm is inspired by the flashing behavior of fireflies. In the algorithm, randomly generated solutions will be considered as fireflies, and brightness is assigned depending on their performance on the objective function. One of the rules used to construct the algorithm is, a firefly will be attracted to a brighter firefly, and if there is no brighter firefly, it will move randomly. In this paper we modify this random movement of the brighter firefly by generating random directions in order to determine the best direction in which the brightness increases. If such a direction is not generated, it will remain in its current position. Furthermore the assignment of attractiveness is modified in such a way that the effect of the objective function is magnified. From the simulation result it is shown that the modified firefly algorithm performs better than the standard one in finding the best solution with smaller CPU time.


Question: Bat algorithm and cuckoo search: an instructional guide.

This document offers a comprehensive guide to the application of Genetic Algorithms (GAs) in search, optimization, and machine learning, presented in an accessible and tutorial-like manner. It consolidates the essential computer science methodologies, mathematical instruments, and findings from research that equip students and professionals alike with the skills necessary to deploy GAs effectively in solving a wide array of problems. The text aims to demystify the complex mechanisms of GAs, making them more approachable for beginners while also providing deeper insights for those already familiar with the field. Through a blend of theory and practical examples, the document ensures that readers gain a solid understanding of how GAs can be utilized in search and optimization tasks, as well as in the rapidly evolving domain of machine learning. This resource is particularly valuable for anyone looking to harness the power of GAs to tackle complex computational challenges across various disciplines.

This paper puts forward two useful methods for self-adaptation of the mutation distribution - the concepts of derandomization and cumulation. Principle shortcomings of the concept of mutative strategy parameter control and two levels of derandomization are reviewed. Basic demands on the self-adaptation of arbitrary (normal) mutation distributions are developed. Applying arbitrary, normal mutation distributions is equiv-alent to applying a general, linear problem encoding. The underlying objective of mutative strategy parameter control is roughly to favor previously selected mutation steps in the future. If this objective is pursued rigor-ously, a completely derandomized self-adaptation scheme results, which adapts arbitrary normal mutation distributions. This scheme, called covariance matrix adaptation (CMA), meets the previously stated demands. It can still be considerably improved by cumulation - utilizing an evolution path rather than single search steps. Simulations on various test functions reveal local and global search properties of the evolution strategy with and without covariance matrix adaptation. Their performances are comparable only on perfectly scaled functions. On badly scaled, non-separable functions usually a speed up factor of several orders of magnitude is ob-served. On moderately mis-scaled functions a speed up factor of three to ten can be expected.

Specifically, we apply genetic algorithms that include diploid genotypes and dominance operators to a simple nonstationary problem in function optimization: an oscillating, blind knapsack problem. In doing this, we find that diploidy and dominance induce a form of long term distributed memory that stores and occasionally remembers good partial solutions that were once desirable. This memory permits faster adaptation to drastic environmental shifts than is possible without the added structures and operators. This paper investigates the use of diploid representations and dominance operators in genetic algorithms (GAs) to improve performance in environments that vary with time. The mechanics of diploidy and dominance in natural genetics are briefly discussed, and the usage of these structures and operators in other GA investigations is reviewed. An extension of the schema theorem is developed which illustrates the ability of diploid GAs with dominance to hold alternative alleles in abeyance. Both haploid and diploid GAs are applied to a simple time varying problem: an oscillating, blind knapsack problem. Simulation results show that a diploid GA with an evolving dominance map adapts more quickly to the sudden changes in this problem environment than either a haploid GA or a diploid GA with a fixed dominance map. These proof-of-principle results indicate that diploidy and dominance can be used to induce a form of long term distributed memory within a population of structures. In the remainder of this paper, we explore the mechanism, theory, and implementation of dominance and diploidy in artificial genetic search. We start by examining the role of diploidy and dominance in natural genetics, and we briefly review examples of their usage in genetic algorithm circles. We extend the schema theorem to analyze the effect of these structures and mechanisms. We present results from computational experiments on a 17-object, oscillating, blind 0-1 knapsack problem. Simulations with adaptive dominance maps and diploidy are able to adapt more quickly to sudden environmental shifts than either a haploid genetic algorithm or a diploid genetic algorithm with fixed dominance map. These results are encouraging and suggest the investigation of dominance and diploidy in other GA applications in search and machine learning. INTRODUCTION Real world problems are seldom independent of time. If you don't like the weather, wait five minutes and it will change. If this week gasoline costs $1.30 a gallon, next week it may cost $0.89 a gallon or perhaps $2.53 a gallon. In these and many more complex ways, real world environments are both nonstationary and noisy. Searching for good solutions or good behavior under such conditions is a difficult task; yet, despite the perpetual change and uncertainty, all is not lost. History does repeat itself, and what goes around does come around. The horrors of Malthusian extrapolation rarely come to pass, and solutions that worked well yesterday are at least somewhat likely to be useful when circumstances are somewhat similar tomorrow or the day after. The temporal regularity implied in these observations places a premium on search augmented by selective memory. In other words, a system which does not learn the lessons of its history is doomed to repeat its mistakes. THE MECHANICS OF NATURAL DOMINANCE AND DIPLOIDY It is surprising to some genetic algorithm newcomers that tpe most commonly used GA is modeled after the mechanics of haploid genetics. After all, don't most elementary genetics textbooks start off with a discussion of Mendel's pea plants and some mention of diploidy and dominance? The reason for this disparity between genetic algorithm practice and genetics textbook coverage is due to the success achieved by early GA investigators (Hollstien, 1971; De Jong, 1975) using haploid chromosome models on stationary problems. It was found that surprising efficacy and efficiency could be obtained using single stranded (haploid) chromosomes under the action of reproduction and crossover. As a result, later investigators of artificial genetic search have tended to ignore diploidy and dominance. In this section we examine the mechanics of diploidy and dominance to understand their roles in shielding alternate In this paper, we investigate the behavior of a genetic algorithm augmented by structures and operators capable of exploiting the regularity and repeatability of many nonstationary environments.

Computational intelligence is a branch of artificial intelligence that comprises algorithms inspired by nature. The common characteristics of all these algorithms is their collective intelligence and adaptability to a changing environment. Due to their efficiency and simplicity, these algorithms have been employed for problem solving across social and natural sciences. The aim of this paper is to demonstrate that nature-inspired algorithms are also useful within the domain of sport, in particular for obtaining safe and effective training plans targeting various aspects of performance. We outline the benefits and opportunities of applying computational intelligence in sports, and we also comment on the pitfalls and challenges for the future development of this emerging research domain. © 2015 Elsevier Inc. All rights reserved.

Bio-inspired evolutionary algorithms are probabilistic search methods that mimic natural biological evolution. They show the behavior of the biological entities interacting locally with one another or with their environment to solve complex problems. This paper aims to analyze the most predominantly used bio-inspired optimization techniques that have been used for stock market prediction and hence present a comparative study between them.

The particle swarm is an algorithm for finding optimal regions of complex search spaces through the interaction of individuals in a population of particles. Even though the algorithm, which is based on a metaphor of social interaction, has been shown to perform well, researchers have not adequately explained how it works. Further, traditional versions of the algorithm have had some undesirable dynamical properties, notably the particles’ velocities needed to be limited in order to control their trajectories. The present paper analyzes a particle’s trajectory as it moves in discrete time (the algebraic view), then progresses to the view of it in continuous time (the analytical view). A five-dimensional depiction is developed, which describes the system completely. These analyses lead to a generalized model of the algorithm, containing a set of coefficients to control the system’s convergence tendencies. Some results of the particle swarm optimizer, implementing modifications derived from the analysis, suggest methods for altering the original algorithm in ways that eliminate problems and increase the ability of the particle swarm to find optima of some well-studied test functions.

The task of optimizing a complex system presents at least two levels of problems for the system designer. First, a class of optimization algorithms must be chosen that is suitable for application to the system. Second, various parameters of the optimization algorithm need to be tuned for efficiency. A class of adaptive search procedures called genetic algorithms (GA) has been used to optimize a wide variety of complex systems. GA's are applied to the second level task of identifying efficient GA's for a set of numerical optimization problems. The results are validated on an image registration problem. GA's are shown to be effective for both levels of the systems optimization problem.

In contrast to the conventional role of evolution in evolutionary computation (EC) as an optimization algorithm, a new class of evolutionary algorithms has emerged in recent years that instead aim to accumulate as diverse a collection of discoveries as possible, yet where each variant in the collection is as fit as it can be. Often applied in both neuroevolution and morphological evolution, these new quality diversity (QD) algorithms are particularly well-suited to evolution's inherent strengths, thereby offering a promising niche for EC within the broader field of machine learning. However, because QD algorithms are so new, until now no comprehensive study has yet attempted to systematically elucidate their relative strengths and weaknesses under different conditions. Taking a first step in this direction, this paper introduces a new benchmark domain designed specifically to compare and contrast QD algorithms. It then shows how the degree of alignment between the measure of quality and the behavior characterization (which is an essential component of all QD algorithms to date) impacts the ultimate performance of different such algorithms. The hope is that this initial study will help to stimulate interest in QD and begin to unify the disparate ideas in the area.

Many software engineering problems have been addressed with search algorithms. Search algorithms usually depend on several parameters (e.g., population size and crossover rate in genetic algorithms), and the choice of these parameters can have an impact on the performance of the algorithm. It has been formally proven in the No Free Lunch theorem that it is impossible to tune a search algorithm such that it will have optimal settings for all possible problems. So, how to properly set the parameters of a search algorithm for a given software engineering problem? In this paper, we carry out the largest empirical analysis so far on parameter tuning in search-based software engineering. More than one million experiments were carried out and statistically analyzed in the context of test data generation for object-oriented software using the EvoSuite tool. Results show that tuning does indeed have impact on the performance of a search algorithm. But, at least in the context of test data generation, it does not seem easy to find good settings that significantly outperform the “default” values suggested in the literature. This has very practical value for both researchers (e.g., when different techniques are compared) and practitioners. Using “default” values is a reasonable and justified choice, whereas parameter tuning is a long and expensive process that might or might not pay off in the end.


Question: Applying deep reinforcement learning to dialogue generation tasks.

This paper introduces a novel approach to natural language generation (NLG) for spoken dialogue systems, focusing on the creation of more human-like and varied responses. Current NLG systems, often based on rules and heuristics, can result in rigid and formulaic language, lacking the natural diversity found in human conversation. They also struggle to adapt to various domains and languages. To address these limitations, we propose a statistical NLG method that utilizes a semantically conditioned Long Short-term Memory (LSTM) network. This LSTM-based generator can learn from unaligned data by simultaneously optimizing sentence planning and surface realization, achieving this through a straightforward cross entropy training criterion. Language diversity is introduced by sampling from a range of output candidates, resulting in more natural and varied responses. The proposed method, with its reduced reliance on heuristics, was evaluated objectively in two different domains, demonstrating improved performance over previous methods. In subjective evaluations, human judges rated the LSTM system higher on criteria such as informativeness and naturalness, and generally preferred it over alternative systems. This research showcases the potential of deep learning techniques, specifically LSTMs, in enhancing the naturalness and adaptability of NLG in dialogue systems.

We propose Neural Responding Machine (NRM), a neural networ k-based response generator for Short-Text Conversation. NRM takes the general encoder-de coder framework: it formalizes the generation of response as a decoding process based on the lat ent representation of the input text, while both encoding and decoding are realized with recurren t n ural networks (RNN). The NRM is trained with a large amount of one-round conversation dat a collected from a microblogging service. Empirical study shows that NRM can generate gramma tically correct and content-wise appropriate responses to over 75% of the input text, outperf orming state-of-the-arts in the same setting, including retrieval-based and SMT-based models.

Speaker: Mrinmaya Sachan Thesis Committee: Eric P. Xing, Chair Jaime Carbonell Tom Mitchell Dan Roth (University of Pennsylvania) Thesis Proposal Jan. 22, 2018 1:00pm 8102 GHC Link to draft document: http://www.cs.cmu.edu/~mrinmays/thesis_proposal.pdf Standardized tests are often used to test students as they progress in the formal education system. These tests are widely available and measurable with clear evaluation procedures and metrics. Hence, these can serve as good tests for AI. We propose approaches for solving some of these tests. We broadly categorize these tests into two categories: open domain question answering tests such as reading comprehensions and elementary school science tests, and closed domain question answering tests such as intermediate or advanced math and science tests. We present alignment based approach with multi-task learning for the former. For closed domain tests, we propose a parsing to programs approach which can be seen as a natural language interface to expert systems. We also describe approaches for question generation based on instructional material in both open domain as well as closed domain settings. Finally, we show that we can improve both the question answering and question generation models by learning them jointly. This mechanism also allows us to leverage cheap unlabelled data for learning the two models. Our work can potentially be applied for the social good in the education domain. We perform studies on human subjects who found our approaches useful as assistive tools in education.

Generic generation and manipulation of text is challenging and has limited success compared to recent deep generative modeling in visual domain. This paper aims at generating plausible natural language sentences, whose attributes are dynamically controlled by learning disentangled latent representations with designated semantics. We propose a new neural generative model which combines variational auto-encoders and holistic attribute discriminators for effective imposition of semantic structures. With differentiable approximation to discrete text samples, explicit constraints on independent attribute controls, and efficient collaborative learning of generator and discriminators, our model learns highly interpretable representations from even only word annotations, and produces realistic sentences with desired attributes. Quantitative evaluation validates the accuracy of sentence and attribute generation.

Learning goal-directed behavior in environments with sparse feedback is a major challenge for reinforcement learning algorithms. One of the key difficulties is insufficient exploration, resulting in an agent being unable to learn robust policies. Intrinsically motivated agents can explore new behavior for their own sake rather than to directly solve external goals. Such intrinsic behaviors could eventually help the agent solve tasks posed by the environment. We present hierarchicalDQN (h-DQN), a framework to integrate hierarchical action-value functions, operating at different temporal scales, with goal-driven intrinsically motivated deep reinforcement learning. A top-level q-value function learns a policy over intrinsic goals, while a lower-level function learns a policy over atomic actions to satisfy the given goals. h-DQN allows for flexible goal specifications, such as functions over entities and relations. This provides an efficient space for exploration in complicated environments. We demonstrate the strength of our approach on two problems with very sparse and delayed feedback: (1) a complex discrete stochastic decision process with stochastic transitions, and (2) the classic ATARI game – ‘Montezuma’s Revenge’.

This paper proposes a new dataset, Frames, composed of 1369 human-human dialogues with an average of 15 turns per dialogue. This corpus contains goal-oriented dialogues between users who are given some constraints to book a trip and assistants who search a database to find appropriate trips. The users exhibit complex decision-making behaviour which involve comparing trips, exploring different options, and selecting among the trips that were discussed during the dialogue. To drive research on dialogue systems towards handling such behaviour, we have annotated and released the dataset and we propose in this paper a task called frame tracking. This task consists of keeping track of different semantic frames throughout each dialogue. We propose a rule-based baseline and analyse the frame tracking task through this baseline.

One of the long standing goals of Artificial Intelligence (AI) is to build cognitive agents which can perform complex tasks from raw sensory inputs without explicit supervision (Lake et al. 2016). Recent progress in combining Reinforcement Learning objective functions and Deep Learning architectures has achieved promising results for such tasks. An important aspect of such sequential decision making problems, which has largely been neglected, is for the agent to decide on the duration of time for which to commit to actions. Such action repetition is important for computational efficiency, which is necessary for the agent to respond in real-time to events (in applications such as self-driving cars). Action Repetition arises naturally in real life as well as simulated environments. The time scale of executing an action enables an agent (both humans and AI) to decide the granularity of control during task execution. Current state of the art Deep Reinforcement Learning models, whether they are off-policy (Mnih et al. 2015; Wang et al. 2015) or on-policy (Mnih et al. 2016), consist of a framework with a static action repetition paradigm, wherein the action decided by the agent is repeated for a fixed number of time steps regardless of the contextual state while executing the task. In this paper, we propose a new framework Dynamic Action Repetition which changes Action Repetition Rate (the time scale of repeating an action) from a hyper-parameter of an algorithm to a dynamically learnable quantity. At every decision-making step, our models allow the agent to commit to an action and the time scale of executing the action. We show empirically that such a dynamic time scale mechanism improves the performance on relatively harder games in the Atari 2600 domain, independent of the underlying Deep Reinforcement Learning algorithm used.

Statistical Spoken Dialogue Systems have been around for many years. However, access to these systems has always been difficult as there is still no publicly available end-to-end system implementation. To alleviate this, we present PyDial, an opensource end-to-end statistical spoken dialogue system toolkit which provides implementations of statistical approaches for all dialogue system modules. Moreover, it has been extended to provide multidomain conversational functionality. It offers easy configuration, easy extensibility, and domain-independent implementations of the respective dialogue system modules. The toolkit is available for download under the Apache 2.0 license.

Humans have the remarkable ability of imagination, where within the human mind virtual simulations are done of scenarios whether visual, auditory or any other senses. These imaginations are based on the experiences during interaction with the real world, where human senses help the mind understand their surroundings. Such level of imagination has not yet been achieved using current algorithms, but a current trend in deep learning architectures known as Generative Adversarial Networks (GANs) have proven capable of generating new and interesting images or videos based on the training data. In that way, GANs can be used to mimic human imagination, where the resulting generated visuals of GANs are based on the data used during training. In this paper, we use a combination of Long Short-Term Memory (LSTM) Networks and 3D GANs to generate videos. We use a 3D Convolutional GAN to generate new human action videos based on trained data. The generated human action videos are used to generate longer videos consisting of a sequence of short actions combined creating longer and more complex activities. To generate the sequence of actions needed we use an LSTM network to translate a simple input description text into the required sequence of actions. The generated chunks are then concatenated using a motion interpolation scheme to form a single video consisting of many generated actions. Hence a visualization of the input text description is generated as a video of a subject performing the activity described.

The General Video Game AI (GVGAI) competition and its associated software framework provides a way of benchmarking AI algorithms on a large number of games written in a domain-specific description language. While the competition has seen plenty of interest, it has so far focused on online planning, providing a forward model that allows the use of algorithms such as Monte Carlo Tree Search. In this paper, we describe how we interface GVGAI to the OpenAI Gym environment, a widely used way of connecting agents to reinforcement learning problems. Using this interface, we characterize how widely used implementations of several deep reinforcement learning algorithms fare on a number of GVGAI games. We further analyze the results to provide a first indication of the relative difficulty of these games relative to each other, and relative to those in the Arcade Learning Environment under similar conditions.

We propose detecting and responding to humor in spoken dialogue by extracting language and audio cues and subsequently feeding these features into a combined recurrent neural network (RNN) and logistic regression model. In this paper, we parse Switchboard phone conversations to build a corpus of punchlines and unfunny lines where punchlines precede laughter tokens in Switchboard transcripts. We create a combined RNN and logistic regression model that uses both acoustic and language cues to predict whether a conversational agent should respond to an utterance with laughter. Our model achieves an F1-score of 63.2 and accuracy of 73.9. This model outperforms our logistic language model (F1-score 56.6) and RNN acoustic model (59.4) as well as the final RNN model of D. Bertero, 2016 (52.9). Using our final model, we create a “laughbot” that audibly responds to a user with laughter when their utterance is classified as a punchline. A conversational agent outfitted with a humorrecognition system such as the one we present in this paper would be valuable as these agents gain utility in everyday life. Keywords—Chatbots; spoken natural language processing; deep learning; machine learning


Question: Applying deep reinforcement learning to dialogue generation tasks.

This paper explores the application of reinforcement learning for enhancing dialogue management in conversational systems, specifically through the development and testing of the NJFun system. The design of a dialogue policy for a spoken dialogue interface involves complex decision-making processes. To tackle these intricacies, we propose a reinforcement learning methodology to automatically refine dialogue policies, overcoming the technical obstacles typically encountered when integrating reinforcement learning into real-world dialogue systems that interact with human users. We detail the conception, implementation, and comprehensive evaluation of NJFun, a prototype spoken dialogue system designed to offer users information on entertainment options in New Jersey. Our findings reveal that by leveraging reinforcement learning to optimize its performance, NJFun achieves significant improvements in system efficiency and user satisfaction.

**Question**: The Role of Natural Language Processing in Enhancing Healthcare Delivery

**Documents**: Natural Language Processing for Healthcare: Opportunities and Challenges
Healthcare is a complex field with a vast amount of unstructured textual data that can be difficult to analyze. This paper discusses the opportunities and challenges of using natural language processing (NLP) in healthcare. NLP can be used to extract information from medical records, such as symptoms, diagnoses, and treatments, and can also be used to improve patient care by providing real-time decision support. However, there are also challenges to using NLP in healthcare, such as the need for high-quality data, the need for accurate and reliable NLP algorithms, and the need for privacy and security.

**Similar Documents**: This paper examines the potential benefits and hurdles associated with implementing natural language processing (NLP) techniques in the healthcare sector. Given the intricate nature of healthcare and the abundance of unstructured text data, NLP offers a promising avenue for data analysis. By processing medical records, NLP can identify critical information including symptoms, diagnoses, and prescribed treatments, which can subsequently enhance patient care through real-time decision-making support. Despite these prospects, integrating NLP into healthcare is not without challenges. Key issues include the requirement for high-quality data, the necessity for precise and dependable NLP algorithms, and the imperative to maintain patient privacy and data security.

Deep Neural Networks (DNNs) can be used as function approximators in Reinforcement Learning (RL). One advantage of DNNs is that they can cope with large input dimensions. Instead of relying on feature engineering to lower the input dimension, DNNs can extract the features from raw observations. The drawback of this end-to-end learning is that it usually requires a large amount of data, which for real-world control applications is not always available. In this paper, a new algorithm, Model Learning Deep Deterministic Policy Gradient (ML-DDPG), is proposed that combines RL with state representation learning, i.e., learning a mapping from an input vector to a state before solving the RL task. The ML-DDPG algorithm uses a concept we call predictive priors to learn a model network which is subsequently used to pre-train the first layer of the actor and critic networks. Simulation results show that the ML-DDPG can learn reasonable continuous control policies from high-dimensional observations that contain also task-irrelevant information. Furthermore, in some cases, this approach significantly improves the final performance in comparison to end-to-end learning.

We propose the “advanced” n-grams as a new technique for simulating user behaviour in spoken dialogue systems, and we compare it with two methods used in our prior work, i.e. linear feature combination and “normal” n-grams. All methods operate on the intention level and can incorporate speech recognition and understanding errors. In the linear feature combination model user actions (lists of〈 speech act, task 〉 pairs) are selected, based on features of the current dialogue state which encodes the whole history of the dialogue. The user simulation based on “normal” n-grams treats a dialogue as a sequence of lists of 〈 speech act, task 〉 pairs. Here the length of the history considered is restricted by the order of the n-gram. The “advanced” n-grams are a variation of the normal ngrams, where user actions are conditioned not only on speech acts and tasks but also on the current status of the tasks, i.e. whether the information needed by the application (in our case flight booking) has been provided and confirmed by the user. This captures elements of goal-directed user behaviour. All models were trained and evaluated on the C OMMUNICATOR corpus, to which we added annotations for user actions and dialogue context. We then evaluate how closely the synthetic responses resemble the real user responses by comparing the user response generated by each user simulation model in a given dialogue context (taken from the annotated corpus) with the actual user response. We propose the expected accuracy, expected precision, and expected recall evaluation metrics as opposed to standard precision and recall used in prior work. We also discuss why they are more appropriate metrics for evaluating user simulation models compared to their standard counterparts. The advanced n-grams produce higher scores than the normal n-grams for small values of n, which proves their strength when little amount of data is available to train larger ngrams. The linear model produces the best expected accuracy but with respect to expected precision and expected recall it is outperformed by the large n-grams even though it is trained using more information. As a task-based evaluation, we also run each of the user simulation models against a system policy trained on the same corpus. Here the linear feature combination model outperforms the other methods and the advanced n-grams outperform the normal ngrams for all values of n, which again shows their potential. We also calculate the perplexity of the different user models.

Human generates responses relying on semantic and functional dependencies, including coreference relation, among dialogue elements and their context. In this paper, we investigate matching a response with its multi-turn context using dependency information based entirely on attention. Our solution is inspired by the recently proposed Transformer in machine translation (Vaswani et al., 2017) and we extend the attention mechanism in two ways. First, we construct representations of text segments at different granularities solely with stacked self-attention. Second, we try to extract the truly matched segment pairs with attention across the context and response. We jointly introduce those two kinds of attention in one uniform neural network. Experiments on two large-scale multi-turn response selection tasks show that our proposed model significantly outperforms the state-of-the-art models.

In recent years deep reinforcement learning (RL) systems have attained superhuman performance in a number of challenging task domains. However, a major limitation of such applications is their demand for massive amounts of training data. A critical present objective is thus to develop deep RL methods that can adapt rapidly to new tasks. In the present work we introduce a novel approach to this challenge, which we refer to as deep meta-reinforcement learning. Previous work has shown that recurrent networks can support meta-learning in a fully supervised context. We extend this approach to the RL setting. What emerges is a system that is trained using one RL algorithm, but whose recurrent dynamics implement a second, quite separate RL procedure. This second, learned RL algorithm can differ from the original one in arbitrary ways. Importantly, because it is learned, it is configured to exploit structure in the training domain. We unpack these points in a series of seven proof-of-concept experiments, each of which examines a key aspect of deep meta-RL. We consider prospects for extending and scaling up the approach, and also point out some potentially important implications for neuroscience.

Neural networks have achieved state-ofthe-art performance on several structuredoutput prediction tasks, trained in a fully supervised fashion. However, annotated examples in structured domains are often costly to obtain, which thus limits the applications of neural networks. In this work, we propose Maximum Margin Reward Networks, a neural networkbased framework that aims to learn from both explicit (full structures) and implicit supervision signals (delayed feedback on the correctness of the predicted structure). On named entity recognition and semantic parsing, our model outperforms previous systems on the benchmark datasets, CoNLL-2003 and WebQuestionsSP.

Generic generation and manipulation of text is challenging and has limited success compared to recent deep generative modeling in visual domain. This paper aims at generating plausible natural language sentences, whose attributes are dynamically controlled by learning disentangled latent representations with designated semantics. We propose a new neural generative model which combines variational auto-encoders and holistic attribute discriminators for effective imposition of semantic structures. With differentiable approximation to discrete text samples, explicit constraints on independent attribute controls, and efficient collaborative learning of generator and discriminators, our model learns highly interpretable representations from even only word annotations, and produces realistic sentences with desired attributes. Quantitative evaluation validates the accuracy of sentence and attribute generation.

Inquiry is fundamental to communication, and machines cannot effectively collaborate with humans unless they can ask questions. In this thesis work, we explore how can we teach machines to ask clarification questions when faced with uncertainty, a goal of increasing importance in today’s automated society. We do a preliminary study using data from StackExchange, a plentiful online resource where people routinely ask clarifying questions to posts so that they can better offer assistance to the original poster. We build neural network models inspired by the idea of the expected value of perfect information: a good question is one whose expected answer is going to be most useful. To build generalizable systems, we propose two future research directions: a template-based model and a sequence-to-sequence based neural generative model.

Recently a variety of LSTM-based conditional language models (LM) have been applied across a range of language generation tasks. In this work we study various model architectures and different ways to represent and aggregate the source information in an endto-end neural dialogue system framework. A method called snapshot learning is also proposed to facilitate learning from supervised sequential signals by applying a companion cross-entropy objective function to the conditioning vector. The experimental and analytical results demonstrate firstly that competition occurs between the conditioning vector and the LM, and the differing architectures provide different trade-offs between the two. Secondly, the discriminative power and transparency of the conditioning vector is key to providing both model interpretability and better performance. Thirdly, snapshot learning leads to consistent performance improvements independent of which architecture is used.

We propose a general class of language models that treat reference as explicit stochastic latent variables. This architecture allows models to create mentions of entities and their attributes by accessing external databases (required by, e.g., dialogue generation and recipe generation) and internal state (required by, e.g. language models which are aware of coreference). This facilitates the incorporation of information that can be accessed in predictable locations in databases or discourse context, even when the targets of the reference may be rare words. Experiments on three representative applications show our model variants outperform models based on deterministic attention.

Doodling is a useful and common intelligent skill that people can learn and master. In this work, we propose a two-stage learning framework to teach a machine to doodle in a simulated painting environment via Stroke Demonstration and deep Q-learning (SDQ). The developed system, Doodle-SDQ, generates a sequence of pen actions to reproduce a reference drawing and mimics the behavior of human painters. In the first stage, it learns to draw simple strokes by imitating in supervised fashion from a set of strokeaction pairs collected from artist paintings. In the second stage, it is challenged to draw real and more complex doodles without ground truth actions; thus, it is trained with Qlearning. Our experiments confirm that (1) doodling can be learned without direct stepby-step action supervision and (2) pretraining with stroke demonstration via supervised learning is important to improve performance. We further show that Doodle-SDQ is effective at producing plausible drawings in different media types, including sketch and watercolor. A short video can be found at https://www.youtube.com/watch? v=-5FVUQFQTaE.


Question: Deep reinforcement learning applied to dialogue generation.

This paper presents a novel deep neural network architecture designed to integrate loose-structured knowledge into Long Short-Term Memory (LSTM) networks through a Recall gate mechanism for enhanced conversation modeling. The essence of creating engaging and contextually aware chatbots lies in effectively modeling human conversations, a task that can be significantly improved by leveraging domain-specific knowledge. This knowledge not only clarifies the relationships between sentences but also introduces semantic nuances that enrich the conversational context. To achieve this, the proposed model employs a Recall gate, which serves as a bridge for incorporating domain knowledge as an additional global memory component within the LSTM framework. This global memory collaborates with the LSTM's inherent local memory to capture and represent the subtle semantic connections between sentences in multi-turn dialogues. Furthermore, the paper highlights the use of a loose structured domain knowledge base, which can be constructed with minimal manual effort and seamlessly integrated into the Recall gate. The effectiveness of the proposed model is demonstrated through evaluations on a context-oriented response selection task. Results obtained from testing on two datasets showcase the model's potential for advancing conversation modeling and serving as a key component in the development of sophisticated automatic chatting systems.

Neural networks have achieved state-ofthe-art performance on several structuredoutput prediction tasks, trained in a fully supervised fashion. However, annotated examples in structured domains are often costly to obtain, which thus limits the applications of neural networks. In this work, we propose Maximum Margin Reward Networks, a neural networkbased framework that aims to learn from both explicit (full structures) and implicit supervision signals (delayed feedback on the correctness of the predicted structure). On named entity recognition and semantic parsing, our model outperforms previous systems on the benchmark datasets, CoNLL-2003 and WebQuestionsSP.

Intelligent control of robotic arms has huge potential over the coming years, but as of now will often fail to adapt when presented with new and unfamiliar environments. Recent trends to solve this problem have seen a shift to endto-end solutions using deep reinforcement learning to learn policies from visual input, rather than relying on a handcrafted, modular pipeline. Building upon the recent success of deep Q-networks, we present an approach which uses threedimensional simulations to train a 7-DOF robotic arm in a robot arm control task without any prior knowledge. Policies accept images of the environment as input and output motor actions. However, the high-dimensionality of the policies as well as the large state space makes policy search difficult. This is overcome by ensuring interesting states are explored via intermediate rewards that guide the policy towards higher reward states. Our results demonstrate that deep Q-networks can be used to learn policies for a task that involves locating a cube, grasping, and then finally lifting. The agent is able to learn to deal with a range of starting joint configurations and starting cube positions when tested in simulation. Moreover, we show that policies trained via simulation have the potential to be directly applied to real-world equivalents without any further training. We believe that robot simulations can decrease the dependency on physical robots and ultimately improve productivity of training robot control tasks.

We introduce a general strategy for improving neural sequence generation by incorporating knowledge about the future. Our decoder combines a standard sequence decoder with a ‘soothsayer’ prediction function Q that estimates the outcome in the future of generating a word in the present. Our model draws on the same intuitions as reinforcement learning, but is both simpler and higher performing, avoiding known problems with the use of reinforcement learning in tasks with enormous search spaces like sequence generation. We demonstrate our model by incorporating Q functions that incrementally predict what the future BLEU or ROUGE score of the completed sequence will be, its future length, and the backwards probability of the source given the future target sequence. Experimental results show that future prediction yields improved performance in abstractive summarization and conversational response generation and the state-of-the-art in machine translation, while also enabling the decoder to generate outputs that have specific properties.

Learning goal-directed behavior in environments with sparse feedback is a major challenge for reinforcement learning algorithms. One of the key difficulties is insufficient exploration, resulting in an agent being unable to learn robust policies. Intrinsically motivated agents can explore new behavior for their own sake rather than to directly solve external goals. Such intrinsic behaviors could eventually help the agent solve tasks posed by the environment. We present hierarchicalDQN (h-DQN), a framework to integrate hierarchical action-value functions, operating at different temporal scales, with goal-driven intrinsically motivated deep reinforcement learning. A top-level q-value function learns a policy over intrinsic goals, while a lower-level function learns a policy over atomic actions to satisfy the given goals. h-DQN allows for flexible goal specifications, such as functions over entities and relations. This provides an efficient space for exploration in complicated environments. We demonstrate the strength of our approach on two problems with very sparse and delayed feedback: (1) a complex discrete stochastic decision process with stochastic transitions, and (2) the classic ATARI game – ‘Montezuma’s Revenge’.

We investigate how reinforcement learning agents can learn to cooperate. Drawing inspiration from human societies, in which successful coordination of many individuals is often facilitated by hierarchical organisation, we introduce Feudal Multiagent Hierarchies (FMH). In this framework, a ‘manager’ agent, which is tasked with maximising the environmentally-determined reward function, learns to communicate subgoals to multiple, simultaneously-operating, ‘worker’ agents. Workers, which are rewarded for achieving managerial subgoals, take concurrent actions in the world. We outline the structure of FMH and demonstrate its potential for decentralised learning and control. We find that, given an adequate set of subgoals from which to choose, FMH performs, and particularly scales, substantially better than cooperative approaches that use a shared reward function.

Reinforcement learning is considered to be a strong AI paradigm which can be used to teach machines through interaction with the environment and learning from their mistakes. Despite its perceived utility, it has not yet been successfully applied in automotive applications. Motivated by the successful demonstrations of learning of Atari games and Go by Google DeepMind, we propose a framework for autonomous driving using deep reinforcement learning. This is of particular relevance as it is difficult to pose autonomous driving as a supervised learning problem due to strong interactions with the environment including other vehicles, pedestrians and roadworks. As it is a relatively new area of research for autonomous driving, we provide a short overview of deep reinforcement learning and then describe our proposed framework. It incorporates Recurrent Neural Networks for information integration, enabling the car to handle partially observable scenarios. It also integrates the recent work on attention models to focus on relevant information, thereby reducing the computational complexity for deployment on embedded hardware. The framework was tested in an open source 3D car racing simulator called TORCS. Our simulation results demonstrate learning of autonomous maneuvering in a scenario of complex road curvatures and simple interaction of other vehicles. INTRODUCTION A robot car that drives autonomously is a long-standing goal of Artificial Intelligence. Driving a vehicle is a task that requires high level of skill, attention and experience from a human driver. Although computers are more capable of sustained attention and focus than humans, fully autonomous driving requires a level of intelligence that surpasses that achieved so far by AI agents. The tasks involved in creating an autonomous driving agent can be divided into 3 categories, as shown in Figure 1: 1) Recognition: Identifying components of the surrounding environment. Examples of this are pedestrian detection, traffic sign recognition, etc. Although far from trivial, recognition is a relatively easy task nowadays thanks to advances in Deep Learning (DL) algorithms, which have reached human level recognition or above at several object detection and classification problems [8] [2]. Deep learning models are able to learn complex feature representations from raw input data, omitting the need for handcrafted features [15] [2] [7]. In this regard, Convolutional Neural Networks (CNNs) are probably the most successful deep learning model, and have formed the basis of every winning entry on the ImageNet challenge since AlexNet [8]. This success has been replicated in lane & vehicle detection for autonomous driving [6]. 2) Prediction: It is not enough for an autonomous driving agent to recognize its environment; it must also be able to build internal models that predict the future states of the environment. Examples of this class of problem include building a map of the environment or tracking an object. To be able to predict the future, it is important to integrate past information. As such, Recurrent Neural Networks (RNNs) are essential to this class of problem. Long-Short Term Memory (LSTM) networks [5] are one such category of RNN that have been used in end-to-end scene labeling systems [14]. More recently, RNNs have also been used to improve object tracking performance in the DeepTracking model [13]. 3) Planning: The generation of an efficient model that incorporates recognition and prediction to plan the future sequence of driving actions that will enable the vehicle to navigate successfully. Planning is the hardest task of the three. The difficulty lies in integrating the ability of the model to understand the environment (recognition) and its dynamics (prediction) in a way that enables it to plan the future actions such that it avoids unwanted situations (penalties) and drives safely to its destination (rewards). Figure 1: High level autonomous driving tasks The Reinforcement Learning (RL) framework [17] [20] has been used for a long time in control tasks. The mixture of RL with DL was pointed out to be one of the most promising approaches to achieve human-level control in [9]. In [12] and [11] this humanlevel control was demonstrated on Atari games using the Deep Q Networks (DQN) model, in which RL is responsible for the planning part while DL is responsible for the representation learning part. Later, RNNs were integrated in the mixture to account for partial observable scenarios [4]. Autonomous driving requires the integration of information ar X iv :1 70 4. 02 53 2v 1 [ st at .M L ] 8 A pr 2 01 7 from multiple sensors. Some of them are low dimensional, like LIDAR, while others are high dimensional, like cameras. It is noteworthy in this particular example, however, that although raw camera images are high dimensional, the useful information needed to achieve the autonomous driving task is of much lower dimension. For example, the important parts of the scene that affect driving decisions are limited to the moving vehicle, free space on the road ahead, the position of kerbs, etc. Even the fine details of vehicles are not important, as only their spatial location is truly necessary for the problem. Hence the memory bandwidth for relevant information is much lower. If this relevant information can be extracted, while the other non-relevant parts are filtered out, it would improve both the accuracy and efficiency of autonomous driving systems. Moreover, this would reduce the computation and memory requirements of the system, which are critical constraints on embedded systems that will contain the autonomous driving control unit. Attention models are a natural fit for such an information filtering process. Recently, these models were successfully deployed for image recognition in [23] and [10], wherein RL was mixed with RNNs to obtain the parts of the image to attend to. Such models are easily extended and integrated to the DQN [11] and Deep Recurrent Q Networks (DRQN) [4] models. This integration was performed in [16]. The success of attention models drives us to propose them for the extraction of low level information from the raw sensory information to perform autonomous driving. In this paper, we propose a framework for an end-end autonomous driving model that takes in raw sensor inputs and outputs driving actions. The model is able to handle partially observable scenarios. Moreover, we propose to integrate the recent advances in attention models in order to extract only relevant information from the received sensors data, thereby making it suitable for real-time embedded systems. The main contributions of this paper: 1) presenting a survey of the recent advances of deep reinforcement learning and 2) introducing a framework for endend autonomous driving using deep reinforcement learning to the automotive community. The rest of the paper is divided into two parts. The first part provides a survey of deep reinforcement learning algorithms, starting with the traditional MDP framework and Q-learning, followed by the the DQN, DRQN and Deep Attention Recurrent Q Networks (DARQN). The second part of the paper describes the proposed framework that integrates the recent advances in deep reinforcement learning. Finally, we conclude and suggest directions for future work. REVIEW OF REINFORCEMENT LEARNING For a comprehensive overview of reinforcement learning, please refer to the second edition of Rich Sutton’s textbook [18]. We provide a short review of important topics in this section. The Reinforcement Learning framework was formulated in [17] as a model to provide the best policy an agent can follow (best action to take in a given state), such that the total accumulated rewards are maximized when the agent follows that policy from the current and until a terminal state is reached. Motivation for RL Paradigm Driving is a multi-agent interaction problem. As a human driver, it is much easier to keep within a lane without any interaction with other cars than to change lanes in heavy traffic. The latter is more difficult because of the inherent uncertainty in behavior of other drivers. The number of interacting vehicles, their geometric configuration and the behavior of the drivers could have large variability and it is challenging to design a supervised learning dataset with exhaustive coverage of all scenarios. Human drivers employ some sort of online reinforcement learning to understand the behavior of other drivers such as whether they are defensive or aggressive, experienced or in-experienced, etc. This is particularly useful in scenarios which need negotiation, namely entering a roundabout, navigating junctions without traffic lights, lane changes during heavy traffic, etc. The main challenge in autonomous driving is to deal with corner cases which are unexpected even for a human driver, like recovering from being lost in an unknown area without GPS or dealing with disaster situations like flooding or appearance of a sinkhole on the ground. The RL paradigm models uncharted territory and learns from its own experience by taking actions. Additionally, RL may be able to handle non-differentiable cost functions which can create challenges for supervised learning problems. Currently, the standard approach for autonomous driving is to decouple the system into isolated sub-problems, typically supervised-learning-like object detection, visual odometry, etc and then having a post processing layer to combine all the results of the previous steps. There are two main issues with this approach: Firstly, the sub-problems which are solved may be more difficult than autonomous driving. For example, one might be solving object detection by semantic segmentation which is both challenging and unnecessary. Human drivers don’t detect and classify all visible objects while driving, only the most relevant ones. Secondly, the isolated sub-problems may not combine coherently to achieve

Search engines play an important role in our everyday lives by assisting us in finding the information we need. When we input a complex query, however, results are often far from satisfactory. In this work, we introduce a query reformulation system based on a neural network that rewrites a query to maximize the number of relevant documents returned. We train this neural network with reinforcement learning. The actions correspond to selecting terms to build a reformulated query, and the reward is the document recall. We evaluate our approach on three datasets against strong baselines and show a relative improvement of 5-20% in terms of recall. Furthermore, we present a simple method to estimate a conservative upperbound performance of a model in a particular environment and verify that there is still large room for improvements.

In this article we introduce the Arcade Learning Environment (ALE): both a challenge problem and a platform and methodology for evaluating the development of general, domain-independent AI technology. ALE provides an interface to hundreds of Atari 2600 game environments, each one different, interesting, and designed to be a challenge for human players. ALE presents significant research challenges for reinforcement learning, model learning, model-based planning, imitation learning, transfer learning, and intrinsic motivation. Most importantly, it provides a rigorous testbed for evaluating and comparing approaches to these problems. We illustrate the promise of ALE by developing and benchmarking domain-independent agents designed using well-established AI techniques for both reinforcement learning and planning. In doing so, we also propose an evaluation methodology made possible by ALE, reporting empirical results on over 55 different games. All of the software, including the benchmark agents, is publicly available.

This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time. The approach is demonstrated for text (where the data are discrete) and online handwriting (where the data are real-valued). It is then extended to handwriting synthesis by allowing the network to condition its predictions on a text sequence. The resulting system is able to generate highly realistic cursive handwriting in a wide variety of styles.

extraordinarily difficult and poses challenges for both integration and grounding. Here, I propose a unifying idea that meets both challenges: language games. A language game is a sequence of verbal interactions between two agents situated in a specific environment. Language games both integrate the various activities required for dialogue and ground unknown words or phrases in a specific context, which helps constrain possible meanings. Over the past five years, I have been working with a team to develop and test language games on progressively more sophisticated systems, from relatively simple camera-based systems to humanoid robots. The results of our work show that language games are a useful way to both understand and design human–robot interaction.