Question: Reinforcement learning methods using actor-critic strategies.

This paper delves into the intricacies of learning in environments where rewards are not immediate, but rather delayed, presenting a significant challenge for reinforcement learning algorithms. In such settings, the temporal credit assignment problem becomes critical, as the agent must learn to associate its actions with rewards that may not manifest until much later. The document discusses various strategies and algorithms designed to tackle this issue, focusing on the actor-critic framework. Actor-critic methods combine the strengths of value-based and policy-based approaches, with the 'actor' learning the policy and the 'critic' estimating the value function to provide feedback on the quality of the actor's decisions. The text explores different variants of actor-critic algorithms, detailing their mechanisms for handling delayed rewards effectively. Key among these is the use of temporal difference learning, which allows the algorithm to learn from delayed rewards by updating the value function based on the difference between the expected and received rewards. The document also touches upon the role of eligibility traces in capturing the credit assignment over time, enabling the agent to learn long-term dependencies between actions and rewards. Through a series of experiments and theoretical analyses, the paper demonstrates the efficacy of these methods in scenarios characterized by delayed rewards, highlighting their potential for enhancing the performance of reinforcement learning in complex, real-world environments.

**Question**: A Direct Search Method to solve Economic Dispatch Problem with Valve-Point Effect

**Documents**: A novel metaheuristic algorithm for solving the economic dispatch problem

**Similar Documents**: In this paper, a groundbreaking metaheuristic algorithm is proposed for addressing the economic dispatch problem, specifically focusing on scenarios involving the valve-point effect. The valve-point effect refers to the non-smooth cost function that results from the operation of thermal power plants, presenting a significant hurdle in optimizing the dispatch of power generation. The proposed algorithm is designed to efficiently navigate the complex landscape of this problem, seeking to find the optimal allocation of power generation among available units while minimizing the total cost. The algorithm incorporates advanced search strategies and mechanisms to avoid getting trapped in local optima, a common pitfall in solving non-linear optimization problems. It is capable of handling the discontinuities and non-convexities introduced by the valve-point effect, thereby offering a robust solution to the economic dispatch problem. The paper details the algorithm's architecture, including its initialization phase, iterative search process, and convergence criteria. Through comprehensive simulations and case studies, the effectiveness of the proposed algorithm is demonstrated, showcasing its ability to outperform existing methods in terms of solution quality and computational efficiency. This work represents a significant advancement in

Programming robot behavior remains a challenging task. While it is often easy to abstractly define or even demonstrate a desired behavior, designing a controller that embodies the same behavior is difficult, time consuming, and ultimately expensive. The machine learning paradigm offers the promise of enabling “programming by demonstration” for developing high-performance robotic systems. Unfortunately, many “behavioral cloning” (Bain & Sammut, 1995; Pomerleau, 1989; LeCun et al., 2006) approaches that utilize classical tools of supervised learning (e.g. decision trees, neural networks, or support vector machines) do not fit the needs of modern robotic systems. These systems are often built atop sophisticated planning algorithms that efficiently reason far into the future; consequently, ignoring these planning algorithms in lieu of a supervised learning approach often leads to myopic and poor-quality robot performance. While planning algorithms have shown success in many real-world applications ranging from legged locomotion (Chestnutt et al., 2003) to outdoor unstructured navigation (Kelly et al., 2004; Stentz, 2009), such algorithms rely on fully specified cost functions that map sensor readings and environment models to quantifiable costs. Such cost functions are usually manually designed and programmed. Recently, a set of techniques has been developed that explore learning these functions from expert human demonstration. These algorithms apply an inverse optimal control approach to find a cost function for which planned behavior mimics an expert’s demonstration. The work we present extends the Maximum Margin Planning (MMP) (Ratliff et al., 2006a) framework to admit learning of more powerful, non-linear cost functions. These algorithms, known collectively as LEARCH (LEArning to seaRCH ), are simpler to implement than most existing methods, more efficient than previous attempts at non-linearization (Ratliff et al., 2006b), more naturally satisfy common constraints on the cost function, and better represent our prior beliefs about the function’s form. We derive and discuss the framework both mathematically and intuitively, and demonstrate practical realworld performance with three applied case-studies including legged locomotion, grasp planning, and autonomous outdoor unstructured navigation. The latter study includes hundreds of kilometers of autonomous traversal through complex natural environments. These case-studies address key challenges in applying the algorithm in practical settings that utilize state-of-the-art planners, and which may be constrained by efficiency requirements and imperfect expert demonstration.

We investigate projection methods, for evaluating a linear approximation of the value function of a policy in a Markov Decision Process context. We consider two popular approaches, the one-step Temporal Difference fix-point computation (TD(0)) and the Bellman Residual (BR) minimization. We describe examples, where each method outperforms the other. We highlight a simple relation between the objective function they minimize, and show that while BR enjoys a performance guarantee, TD(0) does not in general. We then propose a unified view in terms of oblique projections of the Bellman equation, which substantially simplifies and extends the characterization of Schoknecht (2002) and the recent analysis of Yu & Bertsekas (2008). Eventually, we describe some simulations that suggest that if the TD(0) solution is usually slightly better than the BR solution, its inherent numerical instability makes it very bad in some cases, and thus worse on average.

Recent studies have shown the vulnerability of reinforcement learning (RL) models in noisy settings. The sources of noises differ across scenarios. For instance, in practice, the observed reward channel is often subject to noise (e.g., when observed rewards are collected through sensors), and thus observed rewards may not be credible as a result. Also, in applications such as robotics, a deep reinforcement learning (DRL) algorithm can be manipulated to produce arbitrary errors. In this paper, we consider noisy RL problems where observed rewards by RL agents are generated with a reward confusion matrix. We call such observed rewards as perturbed rewards. We develop an unbiased reward estimator aided robust RL framework that enables RL agents to learn in noisy environments while observing only perturbed rewards. Our framework draws upon approaches for supervised learning with noisy data. The core ideas of our solution include estimating a reward confusion matrix and defining a set of unbiased surrogate rewards. We prove the convergence and sample complexity of our approach. Extensive experiments on different DRL platforms show that policies based on our estimated surrogate reward can achieve higher expected rewards, and converge faster than existing baselines. For instance, the state-of-the-art PPO algorithm is able to obtain 67.5% and 46.7% improvements in average on five Atari games, when the error rates are 10% and 30% respectively.

General Inverse Kinematic (IK) solvers may not guarantee real-time control of the end-effectors in external coordinates along with maintaining stability. This work addresses this problem by using Reinforcement Learning (RL) for learning an inverse kinematics solver for reachability tasks which ensures stability and self-collision avoidance while solving for end effectors. We propose an actor-critic based algorithm to learn joint space trajectories of stable configuration for solving inverse kinematics that can operate over continuous action spaces. Our approach is based on the idea of exploring the entire workspace and learning the best possible configurations. The proposed strategy was evaluated on the highly articulated upper body of a 27 degrees of freedom (DoF) humanoid for learning multi-goal reachability tasks of both hands along with maintaining stability in double support phase. We show that the trained model was able to solve inverse kinematics for both the hands, where the articulated torso contributed to both the tasks.

Techniques for both exploratory and known item search tend to direct only to more specific subtopics or individual documents, as opposed to allowing directing the exploration of the information space. We present an interactive information retrieval system that combines Reinforcement Learning techniques along with a novel user interface design to allow active engagement of users in directing the search. Users can directly manipulate document features (keywords) to indicate their interests and Reinforcement Learning is used to model the user by allowing the system to trade off between exploration and exploitation. This gives users the opportunity to more effectively direct their search nearer, further and following a direction. A task-based user study conducted with 20 participants comparing our system to a traditional query-based baseline indicates that our system significantly improves the effectiveness of information retrieval by providing access to more relevant and novel information without having to spend more time acquiring the information.

High-level spinal cord injury (SCI) in humans causes paralysis below the neck. Functional electrical stimulation (FES) technology applies electrical current to nerves and muscles to restore movement, and controllers for upper extremity FES neuroprostheses calculate stimulation patterns to produce desired arm movement. However, currently available FES controllers have yet to restore natural movements. Reinforcement learning (RL) is a reward-driven control technique; it can employ user-generated rewards, and human preferences can be used in training. To test this concept with FES, we conducted simulation experiments using computer-generated “pseudo-human” rewards. Rewards with varying properties were used with an actor-critic RL controller for a planar two-degree-of-freedom biomechanical human arm model performing reaching movements. Results demonstrate that sparse, delayed pseudo-human rewards permit stable and effective RL controller learning. The frequency of reward is proportional to learning success, and human-scale sparse rewards permit greater learning than exclusively automated rewards. Diversity of training task sets did not affect learning. Long-term stability of trained controllers was observed. Using human-generated rewards to train RL controllers for upper-extremity FES systems may be useful. Our findings represent progress toward achieving human-machine teaming in control of upper-extremity FES systems for more natural arm movements based on human user preferences and RL algorithm learning capabilities.

Balancing the ratio between exploration and exploitation is an important problem in reinforcement learning. This paper evaluates four different exploration strategies combined with Q-learning using random stochastic mazes to investigate their performances. We will compare: UCB-1, softmax, ∈-greedy, and pursuit. For this purpose we adapted the UCB-1 and pursuit strategies to be used in the Q-learning algorithm. The mazes consist of a single optimal goal state and two suboptimal goal states that lie closer to the starting position of the agent, which makes efficient exploration an important part of the learning agent. Furthermore, we evaluate two different kinds of reward functions, a normalized one with rewards between 0 and 1, and an unnormalized reward function that penalizes the agent for each step with a negative reward. We have performed an extensive grid-search to find the best parameters for each method and used the best parameters on novel randomly generated maze problems of different sizes. The results show that softmax exploration outperforms the other strategies, although it is harder to tune its temperature parameter. The worst performing exploration strategy is ∈-greedy.

In this work, we explore the possibility of training agents which are able to show intelligent behaviour in many different scenarios. We present the effectiveness of different machine learning algorithms in the StarCraft II Learning Environment [1] and their performance in different scenarios and compare them. In the end, we recreate DeepMind’s FullyConv Agent with slightly better results.

In the model-based policy search approach to reinforcement learning (RL), policies are found using a model (or "simulator") of the Markov decision process. However, for high-dimensional continuous-state tasks, it can be extremely difficult to build an accurate model, and thus often the algorithm returns a policy that works in simulation but not in real-life. The other extreme, model-free RL, tends to require infeasibly large numbers of real-life trials. In this paper, we present a hybrid algorithm that requires only an approximate model, and only a small number of real-life trials. The key idea is to successively "ground" the policy evaluations using real-life trials, but to rely on the approximate model to suggest local changes. Our theoretical results show that this algorithm achieves near-optimal performance in the real system, even when the model is only approximate. Empirical results also demonstrate that---when given only a crude model and a small number of real-life trials---our algorithm can obtain near-optimal performance in the real system.

This paper investigates how adaptive software agents may utilize reinforcement learning algorithms such as Q-learning to make economic decisions such as setting prices in a competitive marketplace. For a single adaptive agent facing fixed-strategy opponents, ordinary Q-learning is guaranteed to find the optimal policy. However, for a population of agents each trying to adapt in the presence of other adaptive agents, the problem becomes non-stationary and history dependent, and it is not known whether any global convergence will be obtained, and if so, whether such solutions will be optimal. In this paper, we study simultaneous Q-learning by two competing seller agents in three moderately realistic economic models. This is the simplest case in which interesting multi-agent phenomena can occur, and the state space is small enough so that lookup tables can be used to represent the Q-functions. We find that, despite the lack of theoretical guarantees, simultaneous convergence to self-consistent optimal solutions is obtained in each model, at least for small values of the discount parameter. In some cases, exact or approximate convergence is also found even at large discount parameters. We show how the Q-derived policies increase profitability and damp out or eliminate cyclic price “wars” compared to simpler policies based on zero lookahead or short-term lookahead. In one of the models (the “Shopbot” model) where the sellers' profit functions are symmetric, we find that Q-learning can produce either symmetric or broken-symmetry policies, depending on the discount parameter and on initial conditions.


Question: Reinforcement learning methods using actor-critic strategies.

This paper explores and presents results for two integrated architectures, Dyna PI and Dyna Q, which are based on approximating dynamic programming (DP) methods, designed for intelligent systems that can learn, plan, and react. Dyna PI, rooted in DP's policy iteration method, relates to concepts such as evaluation functions, universal plans, and reactive systems in artificial intelligence (AI). Demonstrated through a navigation task, a basic Dyna PI system simultaneously learns through trial and error, constructs a world model, and plans optimal routes using this evolving model. Dyna Q, based on Q-learning, a form of reinforcement learning, employs less conventional data structures than Dyna PI but is potentially simpler to implement and utilize, particularly in dynamic environments.

Dyna architectures address the question of how a robot should decide on actions by integrating planning, learning, and reacting. Traditionally, AI advocated planning based on goals and world models. However, planning's utility is constrained by computational complexity and reliance on accurate world models. An alternative is to pre-plan and compile results into rapid reactions or situation-action rules for real-time decision-making. A third approach is learning good reactions by trial and error, which bypasses the need for a world model.

Dyna integrates these approaches, using machine learning algorithms to approximate DP techniques. DP is not a learning method but a computational approach for determining optimal behavior given a complete task model. It is similar to state space search but is more incremental, focusing on single actions rather than sequences, making it suitable for incremental planning at execution time and stochastic environments. Dyna architectures learn a world model online while using DP approximations to learn and plan optimal behavior.

Dyna PI, based on approximating policy iteration, comprises a policy function that maps current states to actions, a world model that mimics one-step input-output behavior, and an evaluation function that maps states to values. These components are updated through separate learning processes. Dyna Q, using Q-learning, is adaptable for use in changing environments and offers a simpler framework for implementing and using DP approximations. The architectures are promising for learning systems due to their capability to handle stochastic and uncertain world models.

While Reinforcement Learning (RL) is not traditionally designed for interactive supervisory input from a human teacher, several works in both robot and software agents have adapted it for human input by letting a human trainer control the reward signal. In this work, we experimentally examine the assumption underlying these works, namely that the human-given reward is compatible with the traditional RL reward signal. We describe an experimental platform with a simulated RL robot and present an analysis of real-time human teaching behavior found in a study in which untrained subjects taught the robot to perform a new task. We report three main observations on how people administer feedback when teaching a robot a task through Reinforcement Learning: (a) they use the reward channel not only for feedback, but also for future-directed guidance; (b) they have a positive bias to their feedback — possibly using the signal as a motivational channel; and (c) they change their behavior as they develop a mental model of the robotic learner. Given this, we made specific modifications to the simulated RL robot, and analyzed and evaluated its learning behavior in four additional experiments with human trainers. We report significant improvements on several learning measures. This work demonstrates the importance of understanding the human-teacher/robot-learner partnership in order to design algorithms that support how people want to teach while simultaneously improving the robot’s learning behavior.

Learning Automata (LA) are adaptive decision making devices suited for operation in unknown environments [12]. Originally they were developed in the area of mathematical psychology and used for modeling observed behavior. In its current form, LA are closely related to Reinforcement Learning (RL) approaches and most popular in the area of engineering. LA combine fast and accurate convergence with low computational complexity, and have been applied to a broad range of modeling and control problems. However, the intuitive, yet analytically tractable concept of learning automata makes them also very suitable as a theoretical framework for Multi agent Reinforcement Learning (MARL). Reinforcement Learning (RL) is already an established and profound theoretical framework for learning in stand-alone or single-agent systems. Yet, extending RL to multi-agent systems (MAS) does not guarantee the same theoretical grounding. As long as the environment an agent is experiencing is Markov, and the agent can experiment sufficiently, RL guarantees convergence to the optimal strategy. In a MAS however, the reinforcement an agent receives, may depend on the actions taken by the other agents acting in the same environment. Hence, the Markov property no longer holds. And as such, guarantees of convergence are lost. In the light of the above problem it is important to fully understand the dynamics of multi-agent reinforcement learning. Although, they are not fully recognized as such, LA are valuable tools for current MARL research. LA are updated strictly on the basis of the response of the environment, and not on the basis of any knowledge regarding other automata, i.e. nor their strategies, nor their feedback. As such LA agents are very simple. Moreover, LA can be treated analytically. Convergence proofs do exist for a variety of settings ranging from a single automaton model acting in a simple stationary random environment to a distributed automata model interacting in a complex environment. In this paper we argue that LA are very interesting building blocks for learning in multi agent systems. The LA can be viewed as policy iterators, who update their action probabilities based on private information only. This is a very attractive property in applications where communication is expensive. LA are in particular appealing in games with stochastic payoffs. Then we move to collections of learning automata, that can independently converge to interesting solution concepts. We study the single stage setting, including the analytical results. Then we generalize to interconnected learning automata, that can deal with multi agent multi-stage problems. We also show how Ant Colony Optimization can be mapped to the interconnected Learning Automata setting. This extended abstract is a summary of an article published earlier [14].

Exploration remains a significant challenge to reinforcement learning methods, especially in environments where reward signals are sparse. Recent methods of learning from demonstrations have shown to be promising in overcoming exploration difficulties but typically require considerable highquality demonstrations that are difficult to collect. We propose to effectively leverage available demonstrations to guide exploration through enforcing occupancy measure matching between the learned policy and current demonstrations, and develop a novel Policy Optimization from Demonstration (POfD) method. We show that POfD induces implicit dynamic reward shaping and brings provable benefits for policy improvement. Furthermore, it can be combined with policy gradient methods to produce state-of-the-art results, as demonstrated experimentally on a range of popular benchmark sparse-reward tasks, even when the demonstrations are few and imperfect.

Temporal difference methods are theoretically grounded and empirically effective methods for addressing reinforcement learning problems. In most real-world reinforcement learning tasks, TD methods require a function approximator to represent the value function. However, using function approximators requires manually making crucial representational decisions. This thesis investigates evolutionary function approximation, a novel approach to automatically selecting function approximator representations that enable efficient individual learning. This method evolves individuals that are better able to learn. I present a fully implemented instantiation of evolutionary function approximation which combines NEAT, a neuroevolutionary optimization technique, with Q-learning, a popular TD method. The resulting NEAT+Q algorithm automatically discovers effective representations for neural network function approximators. This thesis also presents on-line evolutionary computation, which improves the on-line performance of evolutionary computation by borrowing selection mechanisms used in TD methods to choose individual actions and using them in evolutionary computation to select policies for evaluation. I evaluate these contributions with extended empirical studies in two domains: 1) the mountain car task, a standard reinforcement learning benchmark on which neural network function approximators have previously performed poorly and 2) server job scheduling, a large probabilistic domain drawn from the field of autonomic computing. The results demonstrate that evolutionary function approximation can significantly improve the performance of TD methods and on-line evolutionary computation can significantly improve evolutionary methods.

Reinforcement Learning (RL) algorithms allow artificial agents to improve their selection of actions so as to increase rewarding experiences in their environments. The learning can become intractably slow as the state space of the environment grows. This has motivated methods that learn internal representations of the agents state by a function approximator. Impressive results have been produced by using deep artificial neural networks to learn the state representations. However, deep reinforcement learning algorithms require solving a non-convex and nonlinear unconstrained optimization problem. Methods for solving the optimization problems in deep RL are restricted to the class of first-order algorithms, like stochastic gradient descent (SGD). The major drawback of the SGD methods is that they have the undesirable effect of not escaping saddle points. Furthermore, these methods require exhaustive trial and error to fine-tune many learning parameters. Using second derivative information can result in improved convergence properties, but computing the Hessian matrix for large-scale problems is not practical. QuasiNewton methods, like SGD, require only first-order gradient information, but they can result in superlinear convergence, which makes them attractive alternatives. The limitedmemory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) approach is one of the most popular quasi-Newton methods that construct positive definite Hessian approximations. In this paper, we introduce an efficient optimization method, based on the limited memory BFGS quasi-Newton method using line search strategy – as an alternative to SGD methods. Our method bridges the disparity between first order methods and second order methods by continuing to use gradient information to calculate a low-rank Hessian approximations. We provide empirical results on variety of the classic ATARI 2600 games. Our results show a robust convergence with preferred generalization characteristics, as well as fast training time and no need for the experience replaying mechanism.

The acquisition and improvement of motor skills and control policies for robotics from trial and error is of essential importance if robots should ever leave precisely pre-structured environments. However, to date only few existing reinforcement learning methods have been scaled into the domains of high-dimensional robots such as manipulator, legged or humanoid robots. Policy gradient methods remain one of the few exceptions and have found a variety of applications. Nevertheless, the application of such methods is not without peril if done in an uninformed manner. In this paper, we give an overview on learning with policy gradient methods for robotics with a strong focus on recent advances in the field. We outline previous applications to robotics and show how the most recently developed methods can significantly improve learning performance. Finally, we evaluate our most promising algorithm in the application of hitting a baseball with an anthropomorphic arm

We introduce a hybrid CPU/GPU version of the Asynchronous Advantage ActorCritic (A3C) algorithm, currently the state-of-the-art method in reinforcement learning for various gaming tasks. We analyze its computational traits and concentrate on aspects critical to leveraging the GPU’s computational power. We introduce a system of queues and a dynamic scheduling strategy, potentially helpful for other asynchronous algorithms as well. Our hybrid CPU/GPU version of A3C, based on TensorFlow, achieves a significant speed up compared to a CPU implementation; we make it publicly available to other researchers at https://github.com/NVlabs/GA3C.

Model-based control strategies for robot manipulators can present numerous performance advantages when an accurate model of the system dynamics is available. In practice, obtaining such a model is a challenging task which involves modeling such physical processes as friction, which may not be well understood and difficult to model. Furthermore, uncertainties in the physical parameters of a system may be introduced from significant discrepancies between the manufacturer data and the actual system. Traditionally, adaptive and robust control strategies have been developed to deal with parametric uncertainty in the dynamic model, but often require knowledge of the structure of the dynamics. Recent approaches to model-based manipulator control involve data-driven learning of the inverse dynamics relationship, eliminating the need for any a-priori knowledge of the system model. Locally Weighted Projection Regression (LWPR) has been proposed for learning the inverse dynamics function of a manipulator. Due to its use of simple local, linear models, LWPR is suitable for online and incremental learning. Although global regression techniques such as Gaussian Process Regression (GPR) have been shown to outperform LWPR in terms of accuracy, due to its heavy computational requirements, GPR has been applied mainly to offline learning of inverse dynamics. More recent efforts in making GPR computationally tractable for real-time control have resulted in several approximations which operate on a select subset, or sparse representation of the entire training data set. Despite the significant advancements that have been made in the area of learning control, there has not been much work in recent years to evaluate these newer regression techniques against traditional model-based control strategies such as adaptive control. Hence, the first portion of this thesis provides a comparison between a fixed model-based control strategy, an adaptive controller and the LWPR-based learning controller. Simulations are carried out in order to evaluate the position and orientation tracking performance of each controller under varied end effector loading, velocities and inaccuracies in the known dynamic parameters. Both the adaptive controller and LWPR controller are shown to have comparable performance in the presence of parametric uncertainty. However, it is shown that the learning controller is unable to generalize well outside of the regions in which it has been trained. Hence, achieving good performance requires significant amounts of training in the anticipated region of operation. In addition to poor generalization performance, most learning controllers commence learning entirely from ‘scratch,’ making no use of any a-priori knowledge which may be available from the well-known rigid body dynamics (RBD) formulation. The second portion of this thesis develops two techniques for online, incremental learning algorithms which incorporate prior knowledge to improve generalization performance. First, prior knowledge

The goal of reinforcement learning (RL) is to let an agent learn an optimal control policy in an unknown environment so that future expected rewards are maximized. The model-free RL approach directly learns the policy based on data samples. Although using many samples tends to improve the accuracy of policy learning, collecting a large number of samples is often expensive in practice. On the other hand, the model-based RL approach first estimates the transition model of the environment and then learns the policy based on the estimated transition model. Thus, if the transition model is accurately learned from a small amount of data, the model-based approach is a promising alternative to the model-free approach. In this paper, we propose a novel model-based RL method by combining a recently proposed model-free policy search method called policy gradients with parameter-based exploration and the state-of-the-art transition model estimator called least-squares conditional density estimation. Through experiments, we demonstrate the practical usefulness of the proposed method.

We propose a new approach to value function approximation which combines linear temporal difference reinforcement learning with subspace identification. In practical applications, reinforcement learning (RL) is complicated by the fact that state is either high-dimensional or partially observable. Therefore, RL methods are designed to work with features of state rather than state itself, and the success or failure of learning is often determined by the suitability of the selected features. By comparison, subspace identification (SSID) methods are designed to select a feature set which preserves as much information as possible about state. In this paper we connect the two approaches, looking at the problem of reinforcement learning with a large set of features, each of which may only be marginally useful for value function approximation. We introduce a new algorithm for this situation, called Predictive State Temporal Difference (PSTD) learning. As in SSID for predictive state representations, PSTD finds a linear compression operator that projects a large set of features down to a small set that preserves the maximum amount of predictive information. As in RL, PSTD then uses a Bellman recursion to estimate a value function. We discuss the connection between PSTD and prior approaches in RL and SSID. We prove that PSTD is statistically consistent, perform several experiments that illustrate its properties, and demonstrate its potential on a difficult optimal stopping problem.


Question: The completely knowledgeable particle swarm: potentially superior, possibly more effective.

This paper explores the concept of computable numbers, defined as real numbers whose decimal representations can be calculated through finite methods. While ostensibly focused on computable numbers, the investigation can be extended with minimal additional effort to computable functions of integral or real variables, as well as computable predicates. Despite the various forms these entities may take, the underlying issues and challenges are fundamentally similar. The choice of computable numbers for detailed examination is due to the simplicity of the techniques involved. An upcoming publication is planned to explore the interrelations between computable numbers, functions, and predicates, encompassing a theoretical framework for functions of a real variable grounded in the realm of computable numbers. According to the proposed definition, a number qualifies as computable if its decimal expression can be systematically generated by a machine.

In a similar vein, this paper delves into the concept of computable numbers, characterized as real numbers whose decimal expansions can be systematically derived using finite procedures. Although the primary focus is on computable numbers, the methodologies and analyses are readily adaptable to computable functions of integral, real, or computable variables, as well as computable predicates. The core complexities and inquiries are consistent across these categories. The decision to concentrate on computable numbers is motivated by the streamlined techniques required for their study. An upcoming work will elaborate on the interconnections among computable numbers, functions, and predicates, incorporating a theoretical analysis of functions of a real variable based on the principles of computable numbers. A number is deemed computable if its decimal can be algorithmically produced by a machine.

Harmony search (HS) algorithm was applied to solving Sudoku puzzle. The HS is an evolutionary algorithm which mimics musicians’ behaviors such as random play, memory-based play, and pitch-adjusted play when they perform improvisation. Sudoku puzzles in this study were formulated as an optimization problem with number-uniqueness penalties. HS could successfully solve the optimization problem after 285 function evaluations, taking 9 seconds. Also, sensitivity analysis of HS parameters was performed to obtain a better idea of algorithm parameter values.

Evolutionary computation (EC) is a kind of optimization methodology inspired by the mechanisms of biological evolution and behaviors of living organisms. In the literature, the terminology evolutionary algorithms is frequently treated the same as EC. This article focuses on making a survey of researches based on using ML techniques to enhance EC algorithms. In the framework of an ML-technique enhanced-EC algorithm (MLEC), the main idea is that the EC algorithm has stored ample data about the search space, problem features, and population information during the iterative search process, thus the ML technique is helpful in analyzing these data for enhancing the search performance. The paper presents a survey of five categories: ML for population initialization, ML for fitness evaluation and selection, ML for population reproduction and variation, ML for algorithm adaptation, and ML for local search.

In this work, performance of the Artificial Bee Colony Algorithm which is a recently proposed algorithm, has been tested on fuzzy clustering. We applied the Artificial Bee Colony (ABC) Algorithm fuzzy clustering to classify different data sets; Cancer, Diabetes and Heart from UCI database, a collection of classification benchmark problems. The results indicate that the performance of Artificial Bee Colony Optimization Algorithm is successful in fuzzy clustering.

We are developing a probabilistic technique for performing multiple target detection and localization based on data from a swarm of flying sensors, for example to be mounted on a group of micro-UAVs (unmanned aerial vehicles). Swarms of sensors can facilitate detecting and discriminating low signal-to-clutter targets by allowing correlation between different sensor types and/or different aspect angles. However, for deployment of swarms to be feasible, UAVs must operate more autonomously. The current approach is designed to reduce the load on humans controlling UAVs by providing computerized interpretation of a set of images from multiple sensors. We consider a complex case in which target detection and localization are performed concurrently with sensor fusion, multi-target signature association, and improved UAV navigation. This method yields the bonus feature of estimating precise tracks for UAVs, which may be applicable for automatic collision avoidance. We cast the problem in a probabilistic framework known as modeling field theory (MFT), in which the pdf of the data is composed of a mixture of components, each conditional upon parameters including target positions as well as sensor kinematics. The most likely set of parameters is found by maximizing the log-likelihood function using an iterative approach related to expectation-maximization. In terms of computational complexity, this approach scales linearly with number of targets and sensors, which represents an improvement over most existing methods. Also, since data association is treated probabilistically, this method is not prone to catastrophic failure if data association is incorrect. Results from computer simulations are described which quantitatively show the advantages of increasing the number of sensors in the swarm, both in terms of clutter suppression and more accurate target localization. 2005 Elsevier B.V. All rights reserved.

We considered non-clairvoyant multiprocessor scheduling of jobs with arbitrary arrival times and changing execution characteristics. The problem has been studied extensively when either the jobs all arrive at time zero, or when all the jobs are fully parallelizable, or when the scheduler has considerable knowledge about the jobs. This paper considers for the first time this problem without any of these three restrictions yet when our algorithm is given more resources than the adversary. We provide new upper and lower bound techniques applicable in this more difficult scenario. The results are of both theoretical and practical interest. In our model, a job can arrive at any arbitrary time and its execution characteristics can change through the life of the job from being anywhere from fully parallelizable to completely sequential. We assume that the scheduler has no knowledge about the jobs except for knowing when a job arrives and knowing when it completes. (This is why we say that the scheduler is completely in the dark.) Given all this, we prove that the scheduler algorithm Equi-partition, though simple, performs within a constant factor as well as the optimal scheduler as long as it is given at least twice as many processors. More over, we prove that if none of the jobs are ”strictly” fully parallelizable, then Equi-partition performs competitively with no extra processors. Author is supported by NSERC Canada.

The goal of giving a well-defined meaning to information is currently shared by endeavors such as the Semantic Web as well as by current trends within Knowledge Management. They all depend on the large-scale formalization of knowledge and on the availability of formal metadata about information resources. However, the question how to provide the necessary formal metadata in an effective and efficient way is still not solved to a satisfactory extent. Certainly, the most effective way to provide such metadata as well as formalized knowledge is to let humans encode them directly into the system, but this is neither efficient nor feasible. Furthermore, as current social studies show, individual knowledge is often less powerful than the collective knowledge of a certain community.As a potential way out of the knowledge acquisition bottleneck, we present a novel methodology that acquires collective knowledge from the World Wide Web using the GoogleTM API. In particular, we present PANKOW, a concrete instantiation of this methodology which is evaluated in two experiments: one with the aim of classifying novel instances with regard to an existing ontology and one with the aim of learning sub-/superconcept relations.

Operations managers and scholars in their search for fast and good solutions to real-world problems have applied genetic algorithms to many problems. While genetic algorithms are promising tools for problem solving, future research will benefit from a review of the problems that have been solved and the designs of the genetic algorithms used to solve them. This paper provides a review of the use of genetic algorithms to solve operations problems. Reviewed papers are classified according to the problems they solve. The basic design of each genetic algorithm is described, the shortcomings of the current research are discussed and directions for future research are suggested.

In many applications of black-box optimization, one can evaluate multiple points simultaneously, e.g. when evaluating the performances of several different neural networks in a parallel computing environment. In this paper, we develop a novel batch Bayesian optimization algorithm — the parallel knowledge gradient method. By construction, this method provides the one-step Bayes optimal batch of points to sample. We provide an efficient strategy for computing this Bayes-optimal batch of points, and we demonstrate that the parallel knowledge gradient method finds global optima significantly faster than previous batch Bayesian optimization algorithms on both synthetic test functions and when tuning hyperparameters of practical machine learning algorithms, especially when function evaluations are noisy.

In this work, a proposed hybrid dragonfly algorithm (DA) with extreme learning machine (ELM) system for prediction problem is presented. ELM model is considered a promising method for data regression and classification problems. It has fast training advantage, but it always requires a huge number of nodes in the hidden layer. The usage of a large number of nodes in the hidden layer increases the test/evaluation time of ELM. Also, there is no guarantee of optimality of weights and biases settings on the hidden layer. DA is a recently promising optimization algorithm that mimics the moving behavior of moths. DA is exploited here to select less number of nodes in the hidden layer to speed up the performance of the ELM. It also is used to choose the optimal hidden layer weights and biases. A set of assessment indicators is used to evaluate the proposed and compared methods over ten regression data sets from the UCI repository. Results prove the capability of the proposed DA-ELM model in searching for optimal feature combinations in feature space to enhance ELM generalization ability and prediction accuracy. The proposed model was compared against the set of commonly used optimizers and regression systems. These optimizers are namely, particle swarm optimization (PSO) and genetic algorithm (GA). The proposed DA-ELM model proved an advance overall compared methods in both accuracy and generalization ability.

This paper investigates the permutation flowshop scheduling problem (PFSP) with the objectives of minimizing the makespan and the total flowtime and proposes a hybrid metaheuristic based on the particle swarm optimization (PSO). To enhance the exploration ability of the hybrid metaheuristic, a simulated annealing hybrid with a stochastic variable neighborhood search is incorporated. To improve the search diversification of the hybrid metaheuristic, a solution replacement strategy based on the pathrelinking is presented to replace the particles that have been trapped in local optimum. Computational results on benchmark instances show that the proposed PSO-based hybrid metaheuristic is competitive with other powerful metaheuristics in the literature.


Question: The fully informed particle swarm optimization: simpler, possibly more effective.

In this paper, we delve into the particle swarm optimization (PSO) algorithm, a technique for locating optimal areas within intricate search spaces by emulating social interaction among individuals in a particle swarm. Despite its proven effectiveness, the inner workings of PSO have remained somewhat enigmatic, and previous versions have exhibited certain unfavorable dynamics, such as the necessity to cap particle velocities to manage their paths. This study offers a comprehensive analysis of a particle's trajectory, initially in discrete time (the algebraic perspective), and subsequently in continuous time (the analytical perspective). We construct a five-dimensional model that fully captures the system's behavior. This analysis results in a generalized formulation of the algorithm, introducing a set of coefficients to govern the system's convergence properties. The outcomes of implementing modifications to the particle swarm optimizer, based on our analysis, suggest methods for refining the original algorithm. These improvements not only resolve existing issues but also enhance the particle swarm's capacity to identify optima of several established test functions, thereby underscoring the potential for simpler and more effective versions of the particle swarm optimization technique.

Abstract: This paper generalises the golden section optimal search method to higher dimensional optimisation problem. The method is applicable to a strict quasi-convex function of N-variables over an N-dimensional hyper rectangle. An algorithm is proposed in N-dimension. The algorithm is illustrated graphically in two dimensions and verified through several test functions in higher dimension using MATLAB.

We solve a problem of non-convex stochastic optimisation with help of simulated annealing of Lévy flights of a variable stability index. The search of the ground state of an unknown potential is non-local due to big jumps of the Levy flights process. The convergence to the ground state is fast due to a polynomial decrease rate of the temperature.

In recent years there has been growing interest in algorithms inspired by the observation of natural phenomena to define computational procedures which can solve complex problems. In this article we introduce a distributed heuristic algorithm which was inspired by the observation of the behavior of ant colonies and we propose its use for the Quadratic Assignment Problem. Finally the results obtained in solving some classical instances of the problem are compared with those obtained from other evolutionary heuristics to evaluate the quality of the system proposed. ^Dipartimento di Scienze dell'Informazione, Università di Bologna, Via Sacchi 3, 47023, Cesena, Italy, EU, maniezzo@csr.unibo.it & Dipartimento di Elettronica e Informazione, Politecnico di Milano, Via Ponzio 34/5, 20133, Milano, Italy, EU, colorni@elet.polimi.it #IRIDIA, Université Libre de Bruxelles, Av. Franklin Roosevelt 50, CP 194/6, 1050 Brussels, Belgium, EU, mdorigo@ulb.ac.be

While evolutionary computing inspired approaches to multi-objective optimization have many advantages over conventional approaches; they generally do not explicitly exploit directional/gradient information. This can be inefficient if the underlying objectives are reasonably smooth, and this may limit the application of such approaches to real-world problems. This paper develops a local framework for such problems by geometrically analyzing the multiobjective concepts of descent, diversity and convergence/optimality. It is shown that locally optimal, multi-objective descent direction can be calculated that maximally reduce all the objectives and a local sub-space also exists that is a basis for diversity updates. Local convergence of a point towards the optimal Pareto set is therefore assured. The concept of a population of points is also considered and it is shown that it can be used to locally increase the diversity of the population while still ensuring convergence and a method to extract the local directional information from the population is also described. The paper describes and introduces the basic theoretical concepts as well as demonstrating how they are used on simple test problems.

Simulated annealing (SA) presents an optimization technique with several striking positive and negative features. Perhaps its most salient feature, statistically promising to deliver an optimal solution, in current practice is often spurned to use instead modified faster algorithms, “simulated quenching” (SQ). Using the author’s Adaptive Simulated Annealing (ASA) code, some examples are given which demonstrate how SQ can be much faster than SA without sacrificing accuracy.

In this paper, we propose an effective approach to the nearly untouched problem, still photograph auto cropping, which is one of the important features to automatically enhance photographs. To obtain an optimal result, we first formulate auto cropping as an optimization problem by defining an energy function, which consists of three sub models: composition sub model, conservative sub model, and penalty sub model. Then, particle swarm optimization (PSO) is employed to obtain the optimal solution by maximizing the objective function. Experimental results and user studies over hundreds of photographs show that the proposed approach is effective and accurate in most cases, and can be used in many practical multimedia applications.

Summary
PESTO is a widely applicable and highly customizable toolbox for parameter estimation in MathWorks MATLAB. It offers scalable algorithms for optimization, uncertainty and identifiability analysis, which work in a very generic manner, treating the objective function as a black box. Hence, PESTO can be used for any parameter estimation problem, for which the user can provide a deterministic objective function in MATLAB.

Availability and implementation
PESTO is a MATLAB toolbox, freely available under the BSD license. The source code, along with extensive documentation and example code, can be downloaded from https://github.com/ICB-DCM/PESTO/.

Contact
jan.hasenauer@helmholtz-muenchen.de.

Supplementary information
Supplementary data are available at Bioinformatics online.

This paper reports a Fast Local Search (FLS) algorithm which helps to improve the efficiency of hill climbing and a Guided Local Search (GLS) Algorithm which is developed to help local search to escape local optima and distribute search effort. To illustrate how these algorithms work, this paper describes their application to British Telecom’s workforce scheduling problem, which is a hard real life problem. The effectiveness of FLS and GLS are demonstrated by the fact that they both out-perform all the methods applied to this problem so far, which include simulated annealing, genetic algorithms and constraint logic programming.

The No Free Lunch (NFL) theorems for search and optimization are reviewed and their implications for the design of metaheurist ics are discussed. The theorems state that any two search or optimization algorith ms are equivalent when their performance is averaged across all possible problems and even over subsets of problems fulfilling certain constraints. The NFL results show that if there is no assumption regarding the relation between visited and unse en arch points, efficient search and optimization is impossible. There is no wel l performing universal metaheuristic, but the heuristics must be tailored to the pr oblem class at hand using prior knowledge. In practice, it is not likely that the pr econditions of the NFL theorems are fulfilled for a problem class and thus differenc es between algorithms exist. Therefore, tailored algorithms can exploit structu re nderlying the optimization problem. Given full knowledge about the problem class, it i in theory possible to construct an optimal algorithm.

Swarm Intelligence is the part of Artificial Intel ligence based on study of actions of individuals in various decentralized sys tems. The Bee Colony Optimization (BCO) metaheuristic has been introduce fairly recently as a new direction in the field of Swarm Intelligence. Artif icial bees represent agents, which collaboratively solve complex combinatorial optimiz ation problem. The chapter presents a classification and analysis of the resul ts achieved using Bee Colony Optimization (BCO) to model complex engineering and management processes. The primary goal of this chapter is to acquaint readers with the basic principles of Bee Colony Optimization, as well as to indicate potenti al BCO applications in engineering and management.


Question: The fully informed particle swarm optimization: simpler, possibly more effective.

This paper delves into the relationship between effective optimization algorithms and the problems they are designed to solve, unveiling a series of No Free Lunch (NFL) theorems. These theorems highlight the principle that any algorithm's superior performance on one set of problems is invariably offset by poorer performance on another set. This trade-off is illustrated geometrically, offering a visual representation of algorithmic suitability for specific optimization challenges. The NFL theorems are further applied to information-theoretic dimensions of optimization and benchmark performance measures. The paper also discusses time-varying optimization problems and explores a priori head-to-head minimax distinctions between optimization algorithms. Despite the NFL theorems imposing a semblance of uniformity across all algorithms, these distinctions reveal that certain algorithms can indeed outperform others in specific contexts. The fully informed particle swarm optimization (PSO) approach is examined within this theoretical framework, potentially revealing its comparative advantages and limitations in various optimization scenarios.

Artificial bee colony algorithm is a new population-based evolutionary method based on the intelligent behavior of honey bee swarm. It has shown more effective than other biological-inspired algorithms. However, there are still insufficiencies in ABC algorithm, which is good at exploration but poor at exploitation and its convergence speed is also an issue in some cases. For these insufficiencies, we propose a novel artificial bee colony algorithm (NABC) for numerical optimization problems in this paper to improve the exploitation capability by incorporating the current best solution into the search procedure. Experiments are conducted on a set of unimodal/multimodal benchmark functions. The experiments results of NABC have been compared with Gbest-guided artificial bee colony algorithm (G-ABC), improved artificial bee colony algorithm (I-ABC), Elitist artificial bee colony algorithm (E-ABC). The results show that NABC is superior to those algorithms in most of the tested functions.

This article gives a comprehensive introduction into one of the main branches of evolutionary computation – the evolution strategies (ES) the history of which dates back to the 1960s in Germany. Starting from a survey of history the philosophical background is explained in order to make understandable why ES are realized in the way they are. Basic ES algorithms and design principles for variation and selection operators as well as theoretical issues are presented, and future branches of ES research are discussed.

With the rapid growth of data and computational needs, distributed systems and computational Grids are gaining more and more attention. The huge amount of computations a Grid can fulfill in a specific amount of time cannot be performed by the best supercomputers. However, Grid performance can still be improved by making sure all the resources available in the Grid are utilized optimally using a good load balancing algorithm. This research proposes two new distributed swarm intelligence inspired load balancing algorithms. One algorithm is based on ant colony optimization and the other algorithm is based on particle swarm optimization. A simulation of the proposed approaches using a Grid simulation toolkit (GridSim) is conducted. The performance of the algorithms are evaluated using performance criteria such as makespan and load balancing level. A comparison of our proposed approaches with a classical approach called State Broadcast Algorithm and two random approaches is provided. Experimental results show the proposed algorithms perform very well in a Grid environment. Especially the application of particle swarm optimization, can yield better performance results in many scenarios than the ant colony approach.

Advance in information and communication technologies, force us to keep most of the information electronically, consequently, the security of information has become a fundamental issue. The traditional intrusion detection systems look for unusual or suspicious activity, such as patterns of network traffic that are likely indicators of unauthorized activity. However, normal operation often produces traffic that matches likely "attack signature", resulting in false alarms. One main drawback is the inability of detecting new attacks which do not have known signatures. In this paper particle swarm optimization (PSO) is used to implement a feature selection, and support vector machine (SVMs) with the one-versus-rest method serve as a fitness function of PSO for classification problems from the literature. Experimental result shows that our method allows us to recognize not only known attacks but also to detect suspicious activity that may be the result of a new, unknown attack. Our method simplifies features effectively and obtains a higher classification accuracy compared to other methods

This paper investigates the permutation flowshop scheduling problem (PFSP) with the objectives of minimizing the makespan and the total flowtime and proposes a hybrid metaheuristic based on the particle swarm optimization (PSO). To enhance the exploration ability of the hybrid metaheuristic, a simulated annealing hybrid with a stochastic variable neighborhood search is incorporated. To improve the search diversification of the hybrid metaheuristic, a solution replacement strategy based on the pathrelinking is presented to replace the particles that have been trapped in local optimum. Computational results on benchmark instances show that the proposed PSO-based hybrid metaheuristic is competitive with other powerful metaheuristics in the literature.

Since the advent of computers, many tasks which required humans to spend a lot of time and energy have been trivialized by the computers' ability to perform repetitive tasks extremely quickly. However there are still many areas in which humans excel in comparison with the machines. One such area is chess. Even with great advances in the speed and computational power of modern machines, Grandmasters often beat the best chess programs in the world with relative ease1. This may be due to the fact that a game of chess cannot be won by pure calculation. There is more to the goodness of a chess position than some numerical value which apparently can be seen only by the human brain. Here an effort has been made to improve current chess engines by letting themselves evolve over a period of time. Firstly, the problem of learning is reduced into an optimization problem by defining Position Evaluation in terms of Positional Value Tables (PVTs). Next, the PVTs are optimized using Multi-Niche Crowding which successfully identifies the optima in a multimodal function, thereby arriving at distinctly different solutions which are close to the global optimum.

Geographic Information Systems have gradually acquired greater relevance as tools to support decision-making processes, and during the last decades they have been used in conjunction with Multi-Criteria Decision Analysis techniques (GIS-MCDA) to solve real-world spatial problems. GIS-MCDA can be generally divided in two main approaches: Multi-Attribute and MultiObjective techniques. Until now most of the applications of GIS-MCDA have been focused only on using the multi-attribute approach, and less than 10% of the research has been related to a specific type of multi-objective technique: the use of heuristic/meta-heuristic algorithms. The present study explores how different heuristic/meta-heuristic methods perform on solving a spatial multi-objective optimisation problem. To achieve this, four algorithms representing different types of heuristics methods were implemented, and applied to solve the same spatial multi-objective optimisation problem related with an evacuation planning situation. The implemented algorithms were Standard Particle Swarm Optimisation (SPSO), Non-dominated Sorting Genetic Algorithm II (NSGA-II), Archived Multi-Objective Simulated Annealing (AMOSA) and Multi-Objective Grey Wolf Optimiser (MOGWO). The results show that the four algorithms were effective on solving the given problem, although in general AMOSA and MOGWO had a higher performance for the evaluated aspects (number of solutions, effectiveness of the optimisation, diversity, execution time and repeatability). However, the differences in the results obtained from each algorithm were not clear enough to state that one type of heuristic is superior than others. Since AMOSA and MOGWO are the most recent algorithms among the implemented ones, they include several improvements achieved by the latest research, and their superior performance could be connected to these improvements more than to the specific type of algorithms they belong to. Further research is suggested to explore the suitability of these methods for many-objectives spatial problems, to consider the temporal variability and dynamism of real-world situations, to create a standard set of algorithms to be used for benchmarking, and to integrate them with the currently available GIS-MCDA tools. Despite this, from the performed research it is possible to conclude that heuristics methods are reliable techniques for solving spatial problems with multiple and conflictive objectives, and future research and practical implementations in this field can strengthen the capacities of GIS as a multi-criteria decisionmaking support tool.

We show that all algorithms that search for an extremum of a cost function perform exactly the same when averaged over all possible cost functions In particular if algorithm A outperforms algorithm B on some cost functions then loosely speaking there must exist exactly as many other functions where B outperforms A Starting from this we analyze a number of the other a priori characteristics of the search problem like its geometry and its information theoretic aspects This analysis allows us to derive mathematical benchmarks for assessing a particular search algo rithm s performance We also investigate minimax aspects of the search problem the validity of using characteristics of a partial search over a cost function to predict future behavior of the search algorithm on that cost function and time varying cost functions We conclude with some discussion of the justi ability of biologically inspired search methods

Find the secret to improve the quality of life by reading this evolutionary algorithms in theory and practice evolution strategies evolutionary programming genetic algorithms. This is a kind of book that you need now. Besides, it can be your favorite book to read after having this book. Do you ask why? Well, this is a book that has different characteristic with others. You may not need to know who the author is, how well-known the work is. As wise word, never judge the words from who speaks, but make the words as your good value to your life.


Question: The fully informed particle swarm optimization: simpler, possibly more effective.

In the realm of optimization algorithms, the concept of "No Free Lunch Theorems" for search algorithms has been rigorously explored. These theorems posit that when averaged across all conceivable cost functions, all search algorithms exhibit identical performance. This implies that if one algorithm, designated as A, outperforms another algorithm, B, on a subset of cost functions, then there exists an equal number of functions on which B surpasses A. Stemming from this foundational principle, we delve into the intrinsic characteristics of the search problem, encompassing its geometric properties and information theoretic facets. This comprehensive analysis facilitates the establishment of mathematical benchmarks to evaluate the performance of specific search algorithms. Additionally, we scrutinize the minimax dimensions of the search problem, the legitimacy of utilizing partial search outcomes over a cost function to forecast future algorithmic behavior on the same function, and the nuances of time-varying cost functions. Our exploration concludes with a critical assessment of the rationale behind employing biologically inspired search methods in the optimization domain. This discourse underscores the importance of understanding the inherent limitations and potential advantages of different search algorithms in the context of optimization problems.

We describe algorithmic enhancements to a decision-support tool that residential consumers can utilize to optimize their acquisition of electrical energy services. The decision-support tool optimizes energy services provision by enabling end users to first assign values to desired energy services, and then scheduling their available distributed energy resources (DER) to maximize net benefits. We chose particle swarm optimization (PSO) to solve the corresponding optimization problem because of its straightforward implementation and demonstrated ability to generate near-optimal schedules within manageable computation times. We improve the basic formulation of cooperative PSO by introducing stochastic repulsion among the particles. The improved DER schedules are then used to investigate the potential consumer value added by coordinated DER scheduling. This is computed by comparing the end-user costs obtained with the enhanced algorithm simultaneously scheduling all DER, against the costs when each DER schedule is solved separately. This comparison enables the end users to determine whether their mix of energy service needs, available DER and electricity tariff arrangements might warrant solving the more complex coordinated scheduling problem, or instead, decomposing the problem into multiple simpler optimizations.

Genetic algorithm is a stochastic parallel beam search that can be applied to many typical search problems. This paper describes a genetic algorithmic approach to a problem in artificial intelligence. During the process of evolution, the environment cooperates with the population by continuously making itself friendlier so as to lower the evolutionary pressure. Evaluations show the performance of this approach seems considerably effective in solving this type of board games. Game-playing programs are often described as being a combination of search and

In this paper an algorithm based on the concepts of genetic algorithms that uses an estimation of a probability distribution of promising solutions in order to generate new candidate solutions is proposed To esti mate the distribution techniques for model ing multivariate data by Bayesian networks are used The proposed algorithm identi es reproduces and mixes building blocks up to a speci ed order It is independent of the ordering of the variables in the strings rep resenting the solutions Moreover prior in formation about the problem can be incor porated into the algorithm However prior information is not essential Preliminary ex periments show that the BOA outperforms the simple genetic algorithm even on decom posable functions with tight building blocks as a problem size grows

In this paper, we propose a differential evolution (DE) algorithm variant with a combination of multiple mutation strategies based on roulette wheel selection, which we call MMRDE. We first propose a new, reflection-based mutation operation inspired by the reflection operations in the Nelder–Mead method. We design an experiment to compare its performance with seven mutation strategies, and we prove its effectiveness at balancing exploration and exploitation of DE. Although our reflection-based mutation strategy can balance exploration and exploitation of DE, it is still prone to premature convergence or evolutionary stagnation when solving complex multimodal optimization problems. Therefore, we add two basic strategies to help maintain population diversity and increase the robustness. We use roulette wheel selection to arrange mutation strategies based on their success rates for each individual. MMRDE is tested with some improved DE variants based on 28 benchmark functions for real-parameter optimization that have been recommended by the Institute of Electrical and Electronics Engineers CEC2013 special session. Experimental results indicate that the proposed algorithm shows its effectiveness at cooperative work with multiple strategies. It can obtain a good balance between exploration and exploitation. The proposed algorithm can guide the search for a global optimal solution with quick convergence compared with other improved DE variants.

This paper discusses the relationship between data science and population-based algorithms, which include swarm intelligence and evolutionary algorithms. We reviewed two categories of literature, which include population-based algorithms solving data analysis problem and utilizing data analysis methods in population-based algorithms. With the exponential increment of data, the data science, or more specifically, the big data analytics has gained increasing attention among researchers. New and more efficient algorithms should be designed to handle this massive data problem. Based on the combination of population-based algorithms and data mining techniques, we understand better the insights of data analytics, and design more efficient algorithms to solve real-world big data analytics problems. Also, the weakness and strength of population-based algorithms could be analyzed via the data analytics along the optimization process, a crucial entity in population-based algorithms.

The genetic algorithm includes some parameters that should be adjusted, so as to get reliable results. Choosing a representation of the problem addressed, an initial population, a method of selection, a crossover operator, mutation operator, the probabilities of crossover and mutation, and the insertion method creates a variant of genetic algorithms. Our work is part of the answer to this perspective to find a solution for this combinatorial problem. What are the best parameters to select for a genetic algorithm that creates a variety efficient to solve the Traveling Salesman Problem (TSP)? In this paper, we present a comparative analysis of different mutation operators, surrounded by a dilated discussion that justifying the relevance of genetic operators chosen to solving the TSP problem.

This paper propose Firefly Algorithm (FA) new method for tuning the membership function of fuzzy logic controller for Maximum Power Point Tracker (MPPT) system Photovoltaic solar car, that consist of hybrid Fuzzy Logic Controller - Firefly Algorithm (FLC-FA) for the parameter. There are many MPPT methods for photovoltaic (PV) system, Perturbation and Observation (PnO), fuzzy logic controller (FLC) standard, and hybrid Fuzzy logic controller firefly algorithm (FLC-FA) is compared in this paper. The proposed FLC-FA algorithm is to obtain the optimal solution for MPPT for photovoltaic (PV) systems for solar cars. The result Fuzzy logic controller firefly (FLC-FA) of the proposed method, the highest maximum strength and efficiency generated is PnO = 96.31%, Standard FLC = 99.88% and Proposed FLC-FA = 99.98%. better than PnO and Fuzzy logic controller standard method. The main advantage of the proposed FLC-FA is more efficient and accurate than the still fuzzy logic controller standard.

In this study, a novel clustering-based selection strategy of nondominated individuals for evolutionary multi-objective optimization is proposed. The new strategy partitions the nondominated individuals in current Pareto front adaptively into desired clusters. Then one representative individual will be selected in each cluster for pruning nondominated individuals. In order to evaluate the validity of the new strategy, we apply it into one state of the art multi-objective evolutionary algorithm. The experimental results based on thirteen benchmark problems show that the new strategy improves the performance obviously in terms of breadth and uniformity of nondominated solutions.

Asurveyofcurrentcontinuousnonlinearmultiobjective optimization (MOO) concepts and methods is presented. It consolidates and relates seemingly different terminology and methods. The methods are divided into three major categories: methods with a priori articulation of preferences, methods with a posteriori articulation of preferences, and methods with no articulation of preferences. Genetic algorithms are surveyed as well. Commentary is provided on three fronts, concerning the advantages and pitfalls of individual methods, the different classes of methods, and the field of MOO as a whole. The Characteristics of the most significant methods are summarized. Conclusions are drawn that reflect often-neglected ideas and applicability to engineering problems. It is found that no single approach is superior. Rather, the selection of a specific method depends on the type of information that is provided in the problem, the user’s preferences, the solution requirements, and the availability of software.


Question: The fully informed particle swarm: simpler, possibly superior.

This study explores the efficacy of cross-validation and bootstrap techniques in accuracy estimation and model selection, focusing on the comparison between two prevalent methods: cross-validation and bootstrap. Recent experimental findings on synthetic data and theoretical analyses within constrained scenarios have suggested that, for selecting the most effective classifier from a set of options, ten-fold cross-validation might outperform the computationally more demanding leave-one-out cross-validation. Our investigation is based on extensive empirical analysis, involving over half a million runs of C and Naive Bayes algorithms, to assess the influence of various parameters on these methodologies when applied to real-world datasets. We examine the impact of different numbers of folds in cross-validation, as well as the effect of stratification, and also consider the role of the number of bootstrap samples in the bootstrap method. Our empirical results reveal that, for real-world datasets akin to those in our study, the optimal choice for model selection is ten-fold stratified cross-validation, even when computational resources permit the use of a larger number of folds. This finding underscores the potential superiority of ten-fold stratified cross-validation in practical model selection scenarios.

The influence maximization problem turns up in many online social networks (OSN) in which each participant can potentially influence the decisions made by others in the network. Relationships can be friendships, family relationships or even professional relationships. Such influences can be used to achieve some objective and the influence maximization problem attempts to make decisions so as to maximize the effect of these influences. Past work focused on a static problem whereby one tries to identify the participants who are the most influential. Recently, a multi-stage version of the problem was proposed in which outcomes of influence attempts are observed before additional participants are chosen. For example, in online advertising, one can provide an impression to a particular subject and if that subject clicks on the impression, then their friends are informed in the hope that this information will increase the chances that they also click on the impression and eventually purchase the product. This problem is computationally intensive; in this paper we investigate various optimization methods for finding its solution that yield close to optimal results while taking less computation time. These include greedy and particle swarm optimization algorithms.

A cloud workflow system is a type of platform service which facilitates the automation of distributed applications based on the novel cloud infrastructure. Compared with grid environment, data transfer is a big overhead for cloud workflows due to the market-oriented business model in the cloud environments. In this paper, a Revised Discrete Particle Swarm Optimization (RDPSO) is proposed to schedule applications among cloud services that takes both data transmission cost and computation cost into account. Experiment is conducted with a set of workflow applications by varying their data communication costs and computation costs according to a cloud price model. Comparison is made on make span and cost optimization ratio and the cost savings with RDPSO, the standard PSO and BRS (Best Resource Selection) algorithm. Experimental results show that the proposed RDPSO algorithm can achieve much more cost savings and better performance on make span and cost optimization.

We consider a global optimization problem of a deterministic function f in a semimetric space, given a finite budget of n evaluations. The function f is assumed to be locally smooth (around one of its global maxima) with respect to a semi-metric l. We describe two algorithms based on optimistic exploration that use a hierarchical partitioning of the space at all scales. A first contribution is an algorithm, DOO, that requires the knowledge of l. We report a finite-sample performance bound in terms of a measure of the quantity of near-optimal states. We then define a second algorithm, SOO, which does not require the knowledge of the semimetric l under which f is smooth, and whose performance is almost as good as DOO optimally-fitted.

In this study, a new metaheuristic optimization algorithm, called cuckoo search (CS), is introduced for solving structural optimization tasks. The new CS algorithm in combination with Lévy flights is first verified using a benchmark nonlinear constrained optimization problem. For the validation against structural engineering optimization problems, CS is subsequently applied to 13 design problems reported in the specialized literature. The performance of the CS algorithm is further compared with various algorithms representative of the state of the art in the area. The optimal solutions obtained by CS are mostly far better than the best solutions obtained by the existing methods. The unique search features used in CS and the implications for future research are finally discussed in detail.

In the Bacteria Foraging Optimization Algorithm (BFAO), the chemotactic process is randomly set, imposing that the bacteria swarm together and keep a safe distance from each other. In hybrid bacteria foraging optimization algorithm and particle swarm optimization (hBFOA-PSO) algorithm the principle of swarming is introduced in the framework of BFAO. The hBFOA-PSO algorithm is based on the adjustment of each bacterium position according to the neighborhood environment. In this paper, the effectiveness of the hBFOA-PSO algorithm has been tested for Automatic Generation Control (AGC) of an interconnected power system. A widely used linear model of two area non-reheat thermal system equipped with Proportional-Integral (PI) controller is considered initially for the design and analysis purpose. At first, a conventional Integral Time multiply Absolute Error (ITAE) based objective function is considered and the performance of hBFOA-PSO algorithm is compared with PSO, BFOA and GA. Further a modified objective function using ITAE, damping ratio of dominant eigenvalues and settling time with appropriate weight coefficients is proposed to increase the performance of the controller. Further, robustness analysis is carried out by varying the operating load condition and time constants of speed governor, turbine, tieline power in the range of +50% to -50% as well as size and position of step load perturbation to demonstrate the robustness of the proposed hBFOA-PSO optimized PI controller. The proposed approach is also extended to a nonlinear power system model by considering the effect of governor dead band non-linearity and the superiority of the proposed approach is shown by comparing the results of Craziness based Particle Swarm Optimization (CRAZYPSO) approach for the identical interconnected power system. Finally, the study is extended to a three area system considering both thermal and hydro units with different PI coefficients and comparison between ANFIS and proposed approach has been provided.

We propose a computationally efficient limited memory Covariance Matrix Adaptation Evolution Strategy for large scale optimization, which we call the LM-CMA-ES. The LM-CMA-ES is a stochastic, derivative-free algorithm for numerical optimization of non-linear, non-convex optimization problems in continuous domain. Inspired by the limited memory BFGS method of Liu and Nocedal (1989), the LM-CMA-ES samples candidate solutions according to a covariance matrix reproduced from m direction vectors selected during the optimization process. The decomposition of the covariance matrix into Cholesky factors allows to reduce the time and memory complexity of the sampling to O(mn), where $n$ is the number of decision variables. When $n$ is large (e.g., n > 1000), even relatively small values of $m$ (e.g., m=20,30) are sufficient to efficiently solve fully non-separable problems and to reduce the overall run-time.

Many software engineering problems have been addressed with search algorithms. Search algorithms usually depend on several parameters (e.g., population size and crossover rate in genetic algorithms), and the choice of these parameters can have an impact on the performance of the algorithm. It has been formally proven in the No Free Lunch theorem that it is impossible to tune a search algorithm such that it will have optimal settings for all possible problems. So, how to properly set the parameters of a search algorithm for a given software engineering problem? In this paper, we carry out the largest empirical analysis so far on parameter tuning in search-based software engineering. More than one million experiments were carried out and statistically analyzed in the context of test data generation for object-oriented software using the EvoSuite tool. Results show that tuning does indeed have impact on the performance of a search algorithm. But, at least in the context of test data generation, it does not seem easy to find good settings that significantly outperform the “default” values suggested in the literature. This has very practical value for both researchers (e.g., when different techniques are compared) and practitioners. Using “default” values is a reasonable and justified choice, whereas parameter tuning is a long and expensive process that might or might not pay off in the end.

In this paper we introduce a restart-CMA-evolution strategy, where the population size is increased for each restart (IPOP). By increasing the population size the search characteristic becomes more global after each restart. The IPOP-CMA-ES is evaluated on the test suit of 25 functions designed for the special session on real-parameter optimization of CEC 2005. Its performance is compared to a local restart strategy with constant small population size. On unimodal functions the performance is similar. On multi-modal functions the local restart strategy significantly outperforms IPOP in 4 test cases whereas IPOP performs significantly better in 29 out of 60 tested cases.

There are many variations of local search: Hill-climbing: The name for the basic method described above. Metropolis: Pick a random y ∈ N(x). If f(y) < f(x), set x = y. If f(y) > f(x), set x = y with some probability. Simulated annealing: Like Metropolis, but where the probability of moving to a higher-cost neighbor decreases over time. Tabu search: Like the previous two, but with memory in order to avoid getting stuck in suboptimal regions or in plateaus (“taboo” areas). Parallel search: Do more than one search, and occasionally replace versions that are doing poorly with copies of versions that are doing well. Genetic algorithm: Keep a population of searches that changes over time via “breeding” of the best-performing searches.