Question: A microfluidically reconfigurable dual-band slot antenna offering a 3:1 frequency coverage ratio.

This paper introduces a novel type of low-profile frequency selective surface (FSS) with a third-order bandpass frequency response, achieving an overall thickness of just lambda/24. The design is a three-layer structure, featuring three metal layers interspersed with two thin dielectric substrates. Each layer consists of a two-dimensional periodic structure with sub-wavelength unit cell dimensions and periodicity. The unit cell integrates both resonant and non-resonant elements, functioning as a spatial bandpass filter with a third-order response. Unlike conventional third-order bandpass FSSs, which are typically formed by cascading three identical first-order bandpass FSSs spaced a quarter wavelength apart and having thicknesses around lambda/2, this new structure boasts an exceptionally low profile and an overall thickness of approximately lambda/24, making it ideal for conformal FSS applications. Due to the miniaturized unit cells and reduced overall thickness, the proposed FSS exhibits decreased sensitivity to the angle of incidence of the electromagnetic wave compared to traditional third-order frequency selective surfaces. This paper outlines the operational principles and design guidelines for the proposed FSS. A prototype of the third-order bandpass FSS is fabricated and tested using a free space measurement system in the C band, demonstrating its practicality and performance.

This paper presents the design and development of a dual-band switched-beam microstrip array for global navigation satellite system (GNSS) applications such as ocean reflectometry and remote sensing. In contrast to the traditional Butler matrix, a simple, low cost, broadband and low insertion loss beam switching feed network is proposed, designed and integrated with a dual band antenna array to achieve continuous beam coverage of ±25° around the boresight at the L1 (1.575 GHz) and L2 (1.227 GHz) bands. To reduce the cost, microstrip lines and PIN diode based switches are employed. The proposed switched-beam network is then integrated with dual-band step-shorted annular ring (S-SAR) antenna elements in order to produce a fully integrated compact-sized switched-beam array. Antenna simulation results show that the switched-beam array achieves a maximum gain of 12 dBic at the L1 band and 10 dBic at the L2 band. In order to validate the concept, a scaled down prototype of the simulated design is fabricated and measured. The prototype operates at twice of the original design frequency, i.e., 3.15 GHz and 2.454 GHz and the measured results confirm that the integrated array achieves beam switching and good performance at both bands.

In this letter, a new omnidirectional circularly polarized (CP) antenna is proposed. By adopting a coaxial probe, in which a metal sleeve acts as a monopole, and printed spoke-like metal strips fabricated on two substrates, which act as a loop, this antenna exhibits an omnidirectional circularly polarized radiation in the azimuthal plane. Theory, design, simulation, and experiment of the antenna are presented. Measurement results show that the proposed antenna can achieve a -10-dB impedance bandwidth of 13.8% (ranges from 1.55 to 1.78 GHz) and a 3-dB axial-ratio (AR) bandwidth of 17.5% (ranges from 1.46 to 1.74 GHz) in the azimuthal plane. At the frequency of 1.575 GHz, the measured average axial ratio and right-hand CP (RHCP) gain are 1.91 dB and 0.86 dBi, respectively, in the azimuthal plane.

In this paper a multiband fractal based rectangular microstrip patch antenna is designed. FR4 substrate having thickness of 1.58 mm is used as substrate material for the design of proposed antenna and microstrip feed line provides the excitation to the antenna. The antenna operating frequency range is from 1 to 10 GHz. The proposed antenna resonate at twelve different frequencies as 1.86, 2.33, 3.67, 4.57, 5.08, 6.06, 7.03, 7.75, 8.08, 8.84, 9.56 and 10 GHz and the return losses are -15.39, -16.48, -10.02, -17.29, -13.15, -23.41, -10.22, -11.28, -17.02, -10.94, -15.15 and -15.48 dB respectively. The proposed antenna is designed and simulated by using the Ansoft HFSS V13 (high frequency structure simulator) software.

A new methodology to design a dual-band 3-way Bagley power divider is presented. The general method of using π-network as well as the proposed modified π-network has been discussed. The proposed π-network mimics a λ/2 line at two different frequencies. Design equations in closed form are obtained by using transmission line concepts and simple network analysis techniques. The proposed design is validated using Agilent ADS. This modification also leads to wider bandwidth as observed by a careful simulation.

In this research article, a compact dual band asymmetric coplanar strip-fed printed antenna is designed and presented for Bluetooth, WLAN/WiMAX and public safety applications. The dual frequency operating bands (2.45 GHz and 5.25 GHz) have been achieved by attaching two simple meander shaped radiating strips to the ACS feed line. The proposed antenna geometry is printed on a low cost FR4 substrate having thickness of 1.6mm with overall dimensions of 13 × 21.3m including uniplanar ground plane. The -10 dB impedance bandwidth of the meandered ACS-fed dual band monopole antenna is about 140MHz from 2.36-2.5 GHz, and 2500MHz from 4.5-7.0 GHz respectively, which can cover 2.4 GHz Bluetooth/WLAN, 5.2/5.8 GHz WLAN, 5.5 GHz WiMAX and 4.9 GHz US public safety bands. In addition to the simple geometry and wide impedance bandwidth features, proposed structure perform bidirectional and omnidirectional radiation pattern in both E and H-plane respectively.

Dual-polarized antenna is widely used in communication systems such as ECM and DF systems. In this paper a novel doubleridged horn antenna with dual polarizations is introduced for frequency range of 8–18 GHz. Common double ridged horn antennas have single polarization over the operating frequency. We have used five layers polarizer to provide dual polarizations performance of the doubleridged horn antenna. In order to achieve dual polarizations the strips width, strips spacing and layers distances are optimized. It is worth mentioning that the corresponding VSWR of the antenna during the optimization process should be maintain below a certain value (VSWR< 2). Simulation results show that the proposed antenna yields dual polarizations performance and low VSWR over the operating frequency. We have used CST software for antenna simulation which is based on the finite integral technique.

As the demand of UWB system increases nowadays, several types of hybrid technique on antenna design are studied, modified, and proposed for UWB applications. In this work, an enhanced performance of modified antipodal Vivaldi antenna (AVA) for targeting ultra-wideband (UWB) frequency range between 2.17 GHz and 10.6 GHz has been proposed. This proposed UWB antenna is using FR-4 substrate with dielectric constant, εr = 4.4 and the electrical conductivity tangent loss, tanΔ = 0.019. This antenna design is based on the modified from the dual exponential tapered slot antenna (DETSA) with varying of the exponential flares design for obtain the best return loss of the antenna within the frequency bandwidth of the UWB. It has two exponential flares that follow the characteristic of exponential curve. Next, an elliptical slotting edge (ESE) with fixed horizontal radius and varied vertical radius is implemented to the antenna which effectively increases the performance of the antenna at lower frequencies. These modifications are done without changing the antenna size, which is 70 mm × 90 mm in dimension, which is approximately 0.5λ × 0.6λ, where λ is the wavelength of 2 GHz. The effects of the parameters of the antenna characteristic are also studied. In this case, the parametric study on varying distance between elliptical slots, Ks. It shows that, the higher value of Ks will effect to shift to the resonant frequency and increase the return loss of the antenna. At 9.104 GHz of resonant frequencies, it shows of the best return loss of 48.809 dB. Rather than that, the side lobe levels of the radiation pattern of the antenna are reduced in the lower frequencies.

Modern personal communication handsets are shrinking in size and are required to operate at multiple frequency bands for enhanced functionality and performance. This poses an important challenge for antenna designers to build multiband antennas within the limited allowable space. In this paper, an internal antenna covering seven frequency bands is presented for personal communication handsets. The proposed antenna operates at GSM (880-960 MHz), DCS (1710-1880 MHz), PCS (1880-1990 MHz), UMTS (1900-2170 MHz), WiBro (2300-2390 MHz), Bluetooth (2.4-2.48 GHz), and WLAN (5.0-5.5 GHz) frequency bands. Measured input return loss of the antenna is better than dB at all the frequency bands with reasonable radiation performance. Antenna volume is 30 mm times15 mm times 4.0 mm (1.8 cm) that makes it attractive for modern multiband and multifunctional slim handsets.

A compact circularly polarized (CP) co-designed filtering antenna is reported. The device is based on a patch radiator seamlessly integrated with a bandpass filter composed of coupled stripline open-loop resonators, which are designed together as a system. In the proposed design, the patch functions simultaneously as the radiator and the last stage resonator of the filter, resulting in a low-profile integrated radiating and filtering module with a small overall form factor of 0.53λ0 × 0.53λ0 × 0.07λ0. It is shown that the filtering circuit not only ensures frequency selectivity but also provides impedance matching functionality, which serves to broaden both the impedance and axial ratio bandwidths. The designed filtering antenna was fabricated and measured, experimentally achieving an S11 <; -13.5 dB, an axial ratio of less than 3 dB and a gain higher than 5.2 dBi over a bandwidth from 3.77 to 4.26 GHz, i.e., around 12.2%, which makes it an excellent candidate for integration into a variety of wireless systems. A linearly polarized version of the integrated filtering antenna was also demonstrated. In addition, further full-wave simulations and experiments were carried out to verify that the designed CP filtering antenna maintains its properties even when mounted on different positions of the human body with various body gestures. The stable impedance and radiation properties also make it a suitable candidate as a wearable antenna for off-body wireless communications.

An ultra-wideband horizontally polarized omnidirectional antenna is proposed. The connected Vivaldi antenna array mechanism is exploited to achieve the ultra-wide bandwidths of both the impedance and the radiation patterns simultaneously. Sixteen Vivaldi antenna elements are connected with each other and distributed along a circular substrate to form the proposed antenna. The designed antenna exhibits a relative impedance bandwidth of 159.97% (1.28–11.51 GHz) for <inline-formula> <tex-math notation="LaTeX">$\vert S_{11}\vert <-10$ </tex-math></inline-formula> dB. Meanwhile, uniform omnidirectional radiation patterns with a gain variation of less than 1.5 dB have been obtained within the frequency band of 1.25–7.6 GHz (142.73%).


Question: A microfluidically reconfigurable dual-band slot antenna offering a frequency coverage ratio of 3:1.

This paper presents the development of a coplanar reconfigurable folded slot antenna designed specifically for Wireless Local Area Network (WLAN) applications, eliminating the need for a bias network. The antenna's design incorporates metal strips within the slot, which effectively alter the ground size surrounding the slot, leading to a change in the slot's perimeter and, consequently, a shift in the resonant frequency. Initially, the design of a conventional single folded slot antenna is detailed. Subsequently, the design is enhanced to enable reconfigurability. The resonant frequencies for this reconfigurable design are strategically selected to align with the frequency bands utilized in WLAN applications. The paper provides a comprehensive description of the antenna's design, simulation process, and measurement results. The simulation outcomes for both return loss and maximum gain closely match the measured values. Notably, the antenna maintains consistent radiation patterns across both bands. A key benefit of this design is that the length of the bias lines does not influence antenna performance, significantly simplifying the design, feeding, and matching process, and reducing costs for antenna engineers.

A new miniaturized antenna for 433MHz frequency band is introduced for sensor networks. The antenna is a magnetic monopole loaded with a capacitor, which can significantly decrease the resonant frequency of the structure. It is designed to operate in the ISM 433MHz band on a small PCB (103mm⋆55mm) and has at least 3MHz bandwidth. Moreover, its radiation efficiency is suitable for the possible applications and its pattern is omnidirectionnal.

The notch element is a classic example of a wideband antenna that functions very well as a stand-alone radiator and also as an element in wideband phased arrays [1, 2]. Also known as a Vivaldi Aerial or tapered-slot antenna, the Vivaldi has seen perhaps the most widespread use in ultra-high bandwidth applications (≫ 3:1 Bandwidth) of any type of element due to its robustness in design and range of manufacturability. Part of its popularity stems from the ease with which inexpensive printed-circuit board manufacturing can be used to create large arrays. It is also popular in high-power radar applications as well as ultra-wideband military applications, where all-metal designs are typically preferred. In this paper, a 12:1 bandwidth all-metal Vivaldi array element design is presented that can be easily machined from metal sheets and fed via standard SMA coax connectors. No soldering is required and assembly is simple and modular.

Microstrip patch antennas represent one family of compact antennas that offer a conformal nature and the capability of ready integration with communication system’s printed circuitry. In this project, a 2.4 GHz circular polarization microstrip antenna is designed, constructed and measured. The selected microstrip antenna is a dual –fed circular polarized microstrip antenna. The antenna consists of square patch and 3 dB hybrid coupler. The dual – fed circular polarized microstrip antenna is etched on a FR4 with dielectric substrate of 4.6 with the height of 1.6 mm. Circular polarization is obtained when two orthogonal modes are equally excited with 90° phase difference between them. Circular polarization is important because regardless of the receiver orientation, it will always be receiving a component of the signal. This is due to the resulting wave having an angular variation. KEYWORD:MICROSTRIP,CIRCULAR POLARIZATION

Microwave-based head imaging systems demand wideband antennas with compact size. To ensure high scattered signals from the intended target point, the antennas need to have directional radiation patterns. Designing an antenna by addressing all these issues is challenging as the head imaging system utilizes low microwave frequencies (around 1-3 GHz) to maintain high signal penetration in the lossy human head. This paper reports a coplanar waveguide-fed design that overcomes all these limitations. The antenna is constructed in a three-dimensional manner by combining a dipole antenna with a folded parasitic structure printed on two substrate blocks. The antenna mechanism is explained with the description of its design parameters. The antenna covers 101% fractional bandwidth with a stable gain, good radiation efficiency and well directional radiation characteristics. The overall volume of the antenna is 0.24 × 0.1 × 0.05 λm where λm is the wavelength at the lowest frequency of operation.

In this paper, we present an offset microstrip-fed ultrawideband antenna with band notched characteristics. The antenna structure consists of rectangular radiating patch and ground plane with rectangular shaped slot, which increases impedance bandwidth upto 123.52%(2.6–11GHz). A new modified U slot is etched in the radiating patch to create band-notched properties in the WiMAX (3.3–3.7GHz) and C-band satellite communication (3.7–4.15GHz). Furthermore, parametric studies have been conducted using EM simulation software CADFEKO suite(7.0). A prototype of antenna is fabricated on 1.6mm thick FR-4 substrate with dielectric constant of 4.4 and loss tangent of 0.02. The proposed antenna exhibits directional and omnidirectional radiation patterns along E and H-plane with stable efficiency over the frequency band from 2.6GHz to 11GHz with VSWR less than 2, except 3.3–4.15GHz notched frequency band. The proposed antenna shows good time domain analysis.

An investigation into the design of compact patch antennas loaded with complementary split-ring resonators (CSRRs) and reactive impedance surface (RIS) is presented in this study. The CSRR is incorporated on the patch as a shunt LC resonator providing a low resonance frequency and the RIS is realized using the two-dimensional metallic patches printed on a metal-grounded substrate. Both the meta-resonator (CSRR) and the meta-surface (RIS) are able to miniaturize the antenna size. By changing the configuration of the CSRRs, multi-band operation with varied polarization states can be obtained. An equivalent circuit has been developed for the CSRR-loaded patch antennas to illustrate their working principles. Six antennas with different features are designed and compared, including a circularly-polarized antenna, which validate their versatility for practical applications. These antennas are fabricated and tested. The measured results are in good agreement with the simulation.

A novel design of a high-gain reconfigurable sectoral antenna using an active cylindrical frequency selective surface (FSS) structure is presented. The FSS structure consists of metallic discontinuous strips with PIN diodes in their discontinuities, and it is placed cylindrically around an omnidirectional electromagnetically coupled coaxial dipole (ECCD) array. The cylindrical FSS structure is divided into two semi-cylinders. By controlling the state of diodes in each semi-cylinder, a directive radiation pattern is obtained that can be swept in the entire azimuth plane. The effect of the diode-state configuration and the radius of the cylindrical structure are carefully studied to obtain an optimum sectoral radiation pattern. In addition, a solution for increasing the matching bandwidth of the antenna is also proposed. An experimental prototype was fabricated, and the measured results show a beamwidth of 20° in elevation and 70° in the azimuth plane at 2.1 GHz with a gain of 13 dBi. With these features, the proposed antenna is suitable for base-station applications in wireless communication systems.

Abstract— This paper describes 2 x 2 triangular microstrip patch antenna using T-junction with quarter wave transformer. By regulating the distance in patch antenna and adjusting feed position, bandwidth can be obtained and by using an array, directivity is enhanced. The requirement of large bandwidth, high directivity, and minimum size leads to the design of 2 x 2 triangular microstrip patch antenna array feeding with T-junction network operate at 5.5 GHz. An antenna designed on an FR4 substrate that had a dielectric constant (r) 4.4, a loss tangent 0.02 and thickness of 1.6 mm. Simulated results showed that designed antenna has directivity 12.91 dB and bandwidth 173 MHz with VSWR 1.07 using T-junction feeding network. The proposed 2 x 2 triangular array has the benefit of light weight, simplicity of fabrication, single layer structure, and high directivity. Keyword Bandwidth, Corporate feeding, Return Loss, T-junction, VSWR.

Reconfigurable reflectarray antennas operating in Ku-band are presented in this paper. First, a novel multilayer unit-cell based on polarization turning concept is proposed to achieve the single-bit phase shift required for reconfigurable reflectarray applications. The principle of the unit-cell is discussed using the current model and the space match condition, along with simulations to corroborate the design and performance criteria. Then, an offset-fed configuration is developed to verify performance of the unit-cell in antenna application, and its polarization transformation property is elaborated. Finally, an offset-fed reflectarray with 10 × 10 elements is developed and fabricated. The dual-polarized antenna utilizes the control code matrices to accomplish a wide angle beam-scanning. A full-wave analysis is applied to the reflectarray, and detailed results are presented and discussed. This electronically steerable reflectarray antenna has significant potential for satellite applications, due to its wide operating band, simple control and beam-scanning capability.

A design approach for developing microwave bandpass filters (BPFs) with continuous control of the center frequency and bandwidth is presented. The proposed approach exploits a simple loop-shaped dual-mode resonator that is tapped and perturbed with varactor diodes to realize center frequency tunability and passband reconfigurabilty. The even- and odd-mode resonances of the resonator can be predominately controlled via the incorporated varactors, and the passband response reconfiguration is obtained with the proposed tunable external coupling mechanism, which resolves the return losses degradation attributed to conventional fixed external coupling mechanisms. The demonstrated approach leads to a relatively simple synthesis of the microwave BPF with an up to 33% center frequency tuning range, an excellent bandwidth tuning capability, as well as a high filter response reconfigurability, including an all-reject response.


Question: A microfluidically reconfigurable dual-band slot antenna offering a 3:1 frequency coverage ratio.

This paper presents a fresh perspective on the phenomenon of electrowetting, which occurs on dielectric-coated electrodes, highlighting two distinct phenomena: (i) the voltage-induced alteration of the apparent contact angle, a widely recognized effect, and (ii) an electromechanical force that can be utilized to control and manipulate small liquid volumes on a substrate. Notably, the electromechanical force operates independently of any field-induced changes in the contact angle; it remains effective even when the liquid surface is fixed. The forces acting on liquid volumes can be accurately calculated using either capacitance measurements or the Maxwell stress tensor, without the need for detailed information about the liquid surface profile. This interpretation suggests that a non-linear mechanism, characterized by a voltage threshold, is responsible for both the saturation of the contact angle and the observed clamping of the electromechanical force, providing a comprehensive understanding of the underlying physics.

A high gain wideband U-shaped patch antenna with two equal arms on poly tetra fluoro ethylene (PTFE) substrate is presented. An inverted U-shaped slot is introduced on the circular or square shaped ground plane just under the U-shaped patch. In this communication the effect of size and shape of the ground plane on impedance bandwidth is studied. Maximum impedance bandwidth of 86.79% (4.5–11.4 GHz) is obtained with circular shaped ground plane with diameter 36 mm. The highest gain achieved is 4.1 dBi. The simulated results are confirmed experimentally. The proposed antenna is simple in structure compared to the regular stacked or coplanar parasitic patch antennas. It is highly suitable for wireless communications.

Dual-polarized antenna is widely used in communication systems such as ECM and DF systems. In this paper a novel doubleridged horn antenna with dual polarizations is introduced for frequency range of 8–18 GHz. Common double ridged horn antennas have single polarization over the operating frequency. We have used five layers polarizer to provide dual polarizations performance of the doubleridged horn antenna. In order to achieve dual polarizations the strips width, strips spacing and layers distances are optimized. It is worth mentioning that the corresponding VSWR of the antenna during the optimization process should be maintain below a certain value (VSWR< 2). Simulation results show that the proposed antenna yields dual polarizations performance and low VSWR over the operating frequency. We have used CST software for antenna simulation which is based on the finite integral technique.

In this research, a wideband long-term evolution (LTE) base-station antenna with flexible band-notch characteristics is introduced by two window-type slots etched on radiation patches of the magnetoelectric dipole. The presented antenna can provide a band-notch range with an impedance bandwidth of 2.39-2.52 GHz in the band of 1.825-2.925 GHz, which well applies the wideband antenna for LTE with the good anti-interference from wireless local area network narrow communication band. The typical parameters are investigated to show that the antenna with anti-interference characteristics exhibits large scope band-notch adjustability in the whole wideband region. Based on above element, an LTE base station antenna multiple-input, multiple-output system with anti-interference characteristics is investigated and measured to show its good performances: anti-interference, high isolation, high gain, unidirectional radiation, low back radiation, and low cross-polarization.

In this paper a multiband fractal based rectangular microstrip patch antenna is designed. FR4 substrate having thickness of 1.58 mm is used as substrate material for the design of proposed antenna and microstrip feed line provides the excitation to the antenna. The antenna operating frequency range is from 1 to 10 GHz. The proposed antenna resonate at twelve different frequencies as 1.86, 2.33, 3.67, 4.57, 5.08, 6.06, 7.03, 7.75, 8.08, 8.84, 9.56 and 10 GHz and the return losses are -15.39, -16.48, -10.02, -17.29, -13.15, -23.41, -10.22, -11.28, -17.02, -10.94, -15.15 and -15.48 dB respectively. The proposed antenna is designed and simulated by using the Ansoft HFSS V13 (high frequency structure simulator) software.

A new microstrip monopolar patch antenna is proposed and analyzed. The antenna has a wide bandwidth and a monopole like radiation pattern. Such antenna is constructed on a circular patch antenna that is shorted concentrically with a set of conductive vias. The antenna is analyzed using a cavity model. The cavity model analysis not only distinguishes each resonating mode and gives a physical insight into each mode of the antenna, but also provides a guideline to design a broadband monopolar patch antenna that utilizes two modes (TM01 and TM02 modes). Both modes provide a monopole like radiation pattern. The proposed antenna has a simple structure with a low profile of 0.024 wavelengths, and yields a wide impedance bandwidth of 18% and a maximum gain of 6 dBi.

Circularly polarized single-layer U-slot microstrip patch antenna has been proposed. The suggested asymmetrical U-slot can generate the two orthogonal modes for circular polarization without chamfering any corner of the probe-fed square patch microstrip antenna. A parametric study has been carried out to investigate the effects caused by different arm lengths of the U-slot. The thickness of the foam substrate is about 8.5% of the wavelength at the operating frequency. The 3 dB axial ratio bandwidth of the antenna is 4%. Both experimental and theoretical results of the antenna have been presented and discussed. Circular polarization, printed antennas, U-slot.

In this letter, a new omnidirectional circularly polarized (CP) antenna is proposed. By adopting a coaxial probe, in which a metal sleeve acts as a monopole, and printed spoke-like metal strips fabricated on two substrates, which act as a loop, this antenna exhibits an omnidirectional circularly polarized radiation in the azimuthal plane. Theory, design, simulation, and experiment of the antenna are presented. Measurement results show that the proposed antenna can achieve a -10-dB impedance bandwidth of 13.8% (ranges from 1.55 to 1.78 GHz) and a 3-dB axial-ratio (AR) bandwidth of 17.5% (ranges from 1.46 to 1.74 GHz) in the azimuthal plane. At the frequency of 1.575 GHz, the measured average axial ratio and right-hand CP (RHCP) gain are 1.91 dB and 0.86 dBi, respectively, in the azimuthal plane.

A compact circularly polarized (CP) co-designed filtering antenna is reported. The device is based on a patch radiator seamlessly integrated with a bandpass filter composed of coupled stripline open-loop resonators, which are designed together as a system. In the proposed design, the patch functions simultaneously as the radiator and the last stage resonator of the filter, resulting in a low-profile integrated radiating and filtering module with a small overall form factor of 0.53λ0 × 0.53λ0 × 0.07λ0. It is shown that the filtering circuit not only ensures frequency selectivity but also provides impedance matching functionality, which serves to broaden both the impedance and axial ratio bandwidths. The designed filtering antenna was fabricated and measured, experimentally achieving an S11 <; -13.5 dB, an axial ratio of less than 3 dB and a gain higher than 5.2 dBi over a bandwidth from 3.77 to 4.26 GHz, i.e., around 12.2%, which makes it an excellent candidate for integration into a variety of wireless systems. A linearly polarized version of the integrated filtering antenna was also demonstrated. In addition, further full-wave simulations and experiments were carried out to verify that the designed CP filtering antenna maintains its properties even when mounted on different positions of the human body with various body gestures. The stable impedance and radiation properties also make it a suitable candidate as a wearable antenna for off-body wireless communications.

A novel single-fed low-cost wideband and high-gain slotted cavity antenna based on substrate integrated waveguide (SIW) technology using high-order cavity modes is demonstrated in this paper. High-order resonant modes (TE130, TE310, TE330) inside the cavity are simply excited by a coaxial probe which is located at the center of the antenna. Energy is coupled out of the cavity by a 3 × 3 slot array etched on the top surface of the cavity. Two antennas with different polarizations are designed and tested. Measured results show that the linearly polarized prototype achieves an impedance bandwidth (|S11| <; -10 dB) of > 26% (28 to 36.6 GHz), and a 1-dB gain bandwidth of 14.1% (30.3 to 34.9 GHz). In addition, a measured maximum gain of 13.8 dBi and radiation efficiency of 92% are obtained. To generate circularly polarized radiation, a rotated dipole array is placed in front of the proposed linearly polarized antenna. Measured results show that the circularly polarized antenna exhibits a common bandwidth (10-dB return loss bandwidth, 3-dB axial ratio bandwidth, and 1-dB gain bandwidth) of 11%.

A new technique to design single-feed circularly polarized (CP) microstrip antenna is proposed. The CP radiation is obtained by adjusting the dimension of the etched fractal defected ground structure (FDGS) in the ground plane. Parameter studies of the FDGS are given to illustrate the way to achieve CP radiation. The CP microstrip antennas with the second and third iterative FDGS are fabricated and measured. The measured 10-dB return-loss bandwidth of the CP microstrip antenna is about 30 MHz (1.558 to 1.588 GHz), while its 3-dB axial-ratio bandwidth is 6 MHz (1.572 to 1.578 GHz). The gain across the CP band is between 1.7 and 2.2 dBic.


Question: StarCraft micromanagement utilizing reinforcement learning and curriculum transfer learning.

This paper delves into the transformative impact of deep learning on various facets of modern technology and society, from web search algorithms to content moderation on social media platforms, e-commerce recommendation systems, and the integration into consumer electronics like cameras and smartphones. Deep learning, a subset of machine-learning techniques, has revolutionized the processing of natural data in its raw form, overcoming the limitations of conventional machine-learning approaches that required extensive domain expertise to design feature extractors for transforming raw data into a usable feature vector. Representation learning, a key component of deep learning, enables machines to automatically discover the necessary representations for pattern detection or classification directly from raw data. Deep-learning methods, characterized by their multi-level representation learning, involve the composition of simple, non-linear modules that progressively transform data representations, allowing for the learning of highly complex functions. For classification tasks, these methods amplify crucial aspects of the input while suppressing irrelevant variations. For instance, in image recognition, the first layer of representation typically identifies edges, the second layer detects motifs by recognizing specific edge arrangements, and subsequent layers assemble these motifs into recognizable object parts and complete objects. The uniqueness of deep learning lies in its ability to learn these layers of features without human intervention, using a general-purpose learning procedure. This technology has made significant strides in solving long-standing problems in artificial intelligence, excelling in high-dimensional data analysis across diverse scientific, business, and governmental domains. Deep learning has set new benchmarks in image and speech recognition, drug molecule activity prediction, particle accelerator data analysis, brain circuit reconstruction, and the prediction of non-coding DNA mutation effects on gene expression and disease. Remarkably, it has also shown great promise in natural language processing tasks such as topic classification, sentiment analysis, question answering, and language translation. The future of deep learning appears bright, given its minimal reliance on manual engineering, enabling it to capitalize on the increasing availability of computational resources and data. Innovations in deep neural network learning algorithms and architectures are poised to further accelerate its progress.

Learning, planning, and representing knowledge at multiple levels of temporal abstraction are key, longstanding challenges for AI. In this paper we consider how these challenges can be addressed within the mathematical framework of reinforcement learning and Markov decision processes (MDPs). We extend the usual notion of action in this framework to include options—closed-loop policies for taking action over a period of time. Examples of options include picking up an object, going to lunch, and traveling to a distant city, as well as primitive actions such as muscle twitches and joint torques. Overall, we show that options enable temporally abstract knowledge and action to be included in the reinforcement learning framework in a natural and general way. In particular, we show that options may be used interchangeably with primitive actions in planning methods such as dynamic programming and in learning methods such as Q-learning. Formally, a set of options defined over an MDP constitutes a semi-Markov decision process (SMDP), and the theory of SMDPs provides the foundation for the theory of options. However, the most interesting issues concern the interplay between the underlying MDP and the SMDP and are thus beyond SMDP theory. We present results for three such cases: (1) we show that the results of planning with options can be used during execution to interrupt options and thereby perform even better than planned, (2) we introduce new intra-option methods that are able to learn about an option from fragments of its execution, and (3) we propose a notion of subgoal that can be used to improve the options themselves. All of these results have precursors in the existing literature; the contribution of this paper is to establish them in a simpler and more general setting with fewer changes to the existing reinforcement learning framework. In particular, we show that these results can be obtained without committing to (or ruling out) any particular approach to state abstraction, hierarchy, function approximation, or the macroutility problem. ! 1999 Published by Elsevier Science B.V. All rights reserved. ∗ Corresponding author. 0004-3702/99/$ – see front matter ! 1999 Published by Elsevier Science B.V. All rights reserved. PII: S0004-3702(99)00052 -1 182 R.S. Sutton et al. / Artificial Intelligence 112 (1999) 181–211

This paper describes quasi-online reinforcement learning: while a robot is exploring its environment, in the background a probabilistic model of the environment is built on the fly as new experiences arrive; the policy is trained concurrently based on this model using an anytime algorithm. Prioritized sweeping, directed exploration, and transformed reward functions provide additional speed-ups. The robot quickly learns goal-directed policies from scratch, requiring few interactions with the environment and making efficient use of available computation time. From an outside perspective it learns the behavior online and in real time. We describe comparisons with standard methods and show the individual utility of each of the proposed techniques

Temporal difference methods are theoretically grounded and empirically effective methods for addressing reinforcement learning problems. In most real-world reinforcement learning tasks, TD methods require a function approximator to represent the value function. However, using function approximators requires manually making crucial representational decisions. This thesis investigates evolutionary function approximation, a novel approach to automatically selecting function approximator representations that enable efficient individual learning. This method evolves individuals that are better able to learn. I present a fully implemented instantiation of evolutionary function approximation which combines NEAT, a neuroevolutionary optimization technique, with Q-learning, a popular TD method. The resulting NEAT+Q algorithm automatically discovers effective representations for neural network function approximators. This thesis also presents on-line evolutionary computation, which improves the on-line performance of evolutionary computation by borrowing selection mechanisms used in TD methods to choose individual actions and using them in evolutionary computation to select policies for evaluation. I evaluate these contributions with extended empirical studies in two domains: 1) the mountain car task, a standard reinforcement learning benchmark on which neural network function approximators have previously performed poorly and 2) server job scheduling, a large probabilistic domain drawn from the field of autonomic computing. The results demonstrate that evolutionary function approximation can significantly improve the performance of TD methods and on-line evolutionary computation can significantly improve evolutionary methods.

This work studies how an AI-controlled dog-fighting agent with tunable decisionmaking parameters can learn to optimize performance against an intelligent adversary, as measured by a stochastic objective function evaluated on simulated combat engagements. Gaussian process Bayesian optimization (GPBO) techniques are developed to automatically learn global Gaussian Process (GP) surrogate models, which provide statistical performance predictions in both explored and unexplored areas of the parameter space. This allows a learning engine to sample full-combat simulations at parameter values that are most likely to optimize performance and also provide highly informative data points for improving future predictions. However, standard GPBO methods do not provide a reliable surrogate model for the highly volatile objective functions found in aerial combat, and thus do not reliably identify global maxima. These issues are addressed by novel Repeat Sampling (RS) and Hybrid Repeat/Multipoint Sampling (HRMS) techniques. Simulation studies show that HRMS improves the accuracy of GP surrogate models, allowing AI decision-makers to more accurately predict performance and efficiently tune parameters.

Machine Learning is the study of methods for programming computers to learn. Reinforcement Learning is a type of Machine Learning and refers to learning problem which allows machines and software agents to automatically determine the ideal behaviour within a specific context, in order to maximize its performance. RL is inspired by behaviourist psychology based on the mechanism of learning from rewards and does not require prior knowledge and automatically get optimal policy with the help of knowledge obtained by trial-and-error and continuous interaction with the dynamic environment. This paper provides the overview of Reinforcement Learning from Machine learning perspective. It discusses the fundamental principles and techniques used to solve RL problems. It presents nature of RL problems, with focus on some influential model free RL algorithms, challenges and recent trends in theory and practice of RL. It concludes with the future scope of RL.

A novel class of Approximate Policy Iteration (API) algorithms have recently demonstrated impressive practical performance (e.g., ExIt [1], AlphaGo-Zero [2]). This new family of algorithms maintains, and alternately optimizes, two policies: a fast, reactive policy (e.g., a deep neural network) deployed at test time, and a slow, non-reactive policy (e.g., Tree Search), that can plan multiple steps ahead. The reactive policy is updated under supervision from the non-reactive policy, while the non-reactive policy is improved via guidance from the reactive policy. In this work we study this class of Dual Policy Iteration (DPI) strategy in an alternating optimization framework and provide a convergence analysis that extends existing API theory. We also develop a special instance of this framework which reduces the update of non-reactive policies to model-based optimal control using learned local models, and provides a theoretically sound way of unifying model-free and model-based RL approaches with unknown dynamics. We demonstrate the efficacy of our approach on various continuous control Markov Decision Processes.

In large multiagent games, partial observability , coordination, and credit assignmentpersistently plague attempts to design good learning algorithms. We provide a simple and efficient algorithm that in part uses a linear system to model the world fr om a single agent’s limited perspective, and takes advantage of Kalman filtering to allow an agent to construct a good training signal and effectively learn a near-optimal policy in a wide variety of settings.A sequenceof increasinglycomplex empirical tests verifies the efficacy of this technique.

Common approaches to control a data-center cooling system rely on approximated system/environment models that are built upon the knowledge of mechanical cooling and electrical and thermal management. These models are difficult to design and often lead to suboptimal or unstable performance. In this paper, we show how deep reinforcement learning techniques can be used to control the cooling system of a simulated data center. In contrast to common control algorithms, those based on reinforcement learning techniques can optimize a system’s performance automatically without the need of explicit model knowledge. Instead, only a reward signal needs to be designed. We evaluated the proposed algorithm on the open source simulation platform EnergyPlus. The experimental results indicate that we can achieve 22% improvement compared to a model-based control algorithm built into the EnergyPlus. To encourage the reproduction of our work as well as future research, we have also publicly released an open-source EnergyPlus wrapper interface 1 directly compatible with existing reinforcement learning frameworks.

A parameterized skill is a mapping from multiple goals/task parameters to the policy parameters to accomplish them. Existing works in the literature show how a parameterized skill can be learned given a task space that defines all the possible achievable goals. In this work, we focus on tasks defined in terms of final states (goals), and we face on the challenge where the agent aims to autonomously acquire a parameterized skill to manipulate an initially unknown environment. In this case, the task space is not known a priori and the agent has to autonomously discover it. The agent may posit as a task space its whole sensory space (i.e. the space of all possible sensor readings) as the achievable goals will certainly be a subset of this space. However, the space of achievable goals may be a very tiny subspace in relation to the whole sensory space, thus directly using the sensor space as task space exposes the agent to the curse of dimensionality and makes existing autonomous skill acquisition algorithms inefficient. In this work we present an algorithm that actively discovers the manifold of the achievable goals within the sensor space. We validate the algorithm by employing it in multiple different simulated scenarios where the agent actions achieve different types of goals: moving a redundant arm, pushing an object, and changing the color of an object.

Human machine interaction is a field where machine learning is present at almost any level, from human activity recognition to natural language generation. The interaction manager is probably one of the latest components of an interactive system that benefited from machine learning techniques. In the late 90's, sequential decision making algorithms like reinforcement learning have been introduced in the field with the aim of making the interaction more natural in a measurable way. Yet, these algorithms require providing the learning agent with a reward after each interaction. This reward is generally handcrafted by the system designer who introduces again some expertise in the system. In this paper, we will discuss a method for learning a reward function by observing expert humans, namely inverse reinforcement learning (IRL). IRL will then be applied to several steps of the spoken dialogue management design such as user simulation and clustering but also to co-adaptation of human user and machine.


Question: StarCraft micromanagement utilizing reinforcement learning and curriculum transfer learning.

This paper explores the utilization of Markov Games as a foundational framework for multi-agent reinforcement learning, particularly focusing on scenarios involving two agents with opposing objectives. Traditionally, reinforcement learning is framed within the context of a Markov decision process (MDP), where a solitary adaptive agent interacts with an environment characterized by a probabilistic transition function. However, this perspective confines secondary agents to the role of environmental components, thus limiting their behavioral flexibility. Markov games, on the other hand, offer a broader perspective, enabling multiple adaptive agents to coexist within the same environment, each with potentially conflicting goals. The paper outlines a Q-learning inspired algorithm designed to discover optimal policies in such adversarial scenarios and illustrates its application through a simple two-player game. In this game, the optimal policy is probabilistic, showcasing the algorithm's capability to handle complex decision-making processes in multi-agent settings.

**Question**: StarCraft Micromanagement With Reinforcement Learning and Curriculum Transfer Learning

**Documents**: Transfer Learning in StarCraft Micromanagement: A Reinforcement Learning Approach
In the realm of StarCraft micromanagement, reinforcement learning (RL) has shown promise in developing sophisticated AI agents capable of making strategic decisions in real-time. However, the process of training these agents is often time-consuming and resource-intensive, requiring a significant amount of data to achieve high performance. Transfer learning, a technique that leverages knowledge gained from one task to improve performance on another, has the potential to accelerate this training process. This paper presents a novel method for applying transfer learning in the context of StarCraft micromanagement. Specifically, it focuses on transferring knowledge from simpler, related tasks to more complex ones, a concept known as curriculum transfer learning. The proposed approach is evaluated using a variety of StarCraft micromanagement scenarios, demonstrating its effectiveness in enhancing the learning efficiency and overall performance of RL agents.

**Similar Documents**: This document introduces a pioneering technique that integrates transfer learning into the training of AI agents for StarCraft micromanagement, utilizing reinforcement learning (RL) as the foundational learning methodology. RL has proven its efficacy in equipping agents with the ability to make strategic choices in dynamic, real-world environments. Yet, the training phase can be excessively demanding in terms of computational resources and data requirements. Transfer learning, which capitalizes on previously acquired knowledge to expedite learning in new contexts, is a promising solution to this challenge. The document details a unique strategy that employs curriculum transfer learning, a method that facilitates the transfer of knowledge from simpler tasks to more intricate ones. This approach is thoroughly assessed

In this paper we describe a self-adjusting algorithm for packet routing, in which a reinforcement learning module is embedded into each node of a switching network. Only local communication is used to keep accurate statistics at each node on which routing policies lead to minimal delivery times. In simple experiments involving a 36-node, irregularly connected network, this learning approach proves superior to a nonadaptive algorithm based on precomputed shortest paths. The authors would like to thank for their support the Bellcore Cognitive Science Research Group, the National Defense Science and Engineering Graduate fellowship program, and National Science Foundation Grant IRI-9214873. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the o cial policies, either expressed or implied, of Bellcore, the National Science Foundation or the U.S. Government.

Deep reinforcement learning (DRL) has achieved outstanding results in recent years. This has led to a dramatic increase in the number of applications and methods. Recent works have explored learning beyond single-agent scenarios and have considered multiagent scenarios. Initial results report successes in complex multiagent domains, although there are several challenges to be addressed. In this context, first, this article provides a clear overview of current multiagent deep reinforcement learning (MDRL) literature. Second, it provides guidelines to complement this emerging area by (i) showcasing examples on how methods and algorithms from DRL and multiagent learning (MAL) have helped solve problems in MDRL and (ii) providing general lessons learned from these works. We expect this article will help unify and motivate future research to take advantage of the abundant literature that exists in both areas (DRL and MAL) in a joint effort to promote fruitful research in the multiagent community.

We consider the task of driving a remote control car at high speeds through unstructured outdoor environments. We present an approach in which supervised learning is first used to estimate depths from single monocular images. The learning algorithm can be trained either on real camera images labeled with ground-truth distances to the closest obstacles, or on a training set consisting of synthetic graphics images. The resulting algorithm is able to learn monocular vision cues that accurately estimate the relative depths of obstacles in a scene. Reinforcement learning/policy search is then applied within a simulator that renders synthetic scenes. This learns a control policy that selects a steering direction as a function of the vision system's output. We present results evaluating the predictive ability of the algorithm both on held out test data, and in actual autonomous driving experiments.

Minimizing job scheduling time is a fundamental issue in data center networks that has been extensively studied in recent years. The incoming jobs require different CPU and memory units, and span different number of time slots. The traditional solution is to design efficient heuristic algorithms with performance guarantee under certain assumptions. In this paper, we improve a recently proposed job scheduling algorithm using deep reinforcement learning and extend it to multiple server clusters. Our study reveals that deep reinforcement learning method has the potential to outperform traditional resource allocation algorithms in a variety of complicated environments.

This paper introduces a fully automatic method for generating video game tutorials. The AtDELFI system (Automatically DEsigning Legible, Full Instructions for games) was created to investigate procedural generation of instructions that teach players how to play video games. We present a representation of game rules and mechanics using a graph system as well as a tutorial generation method that uses said graph representation. We demonstrate the concept by testing it on games within the General Video Game Artificial Intelligence (GVG-AI) framework; the paper discusses tutorials generated for eight different games. Our findings suggest that a graph representation scheme works well for simple arcade style games such as Space Invaders and Pacman, but it appears that tutorials for more complex games might require higher-level understanding of the game than just single mechanics.

Techniques for both exploratory and known item search tend to direct only to more specific subtopics or individual documents, as opposed to allowing directing the exploration of the information space. We present an interactive information retrieval system that combines Reinforcement Learning techniques along with a novel user interface design to allow active engagement of users in directing the search. Users can directly manipulate document features (keywords) to indicate their interests and Reinforcement Learning is used to model the user by allowing the system to trade off between exploration and exploitation. This gives users the opportunity to more effectively direct their search nearer, further and following a direction. A task-based user study conducted with 20 participants comparing our system to a traditional query-based baseline indicates that our system significantly improves the effectiveness of information retrieval by providing access to more relevant and novel information without having to spend more time acquiring the information.

In this work, we propose a novel robot learning framework called Neural Task Programming (NTP), which bridges the idea of few-shot learning from demonstration and neural program induction. NTP takes as input a task specification (e.g., video demonstration of a task) and recursively decomposes it into finer sub-task specifications. These specifications are fed to a hierarchical neural program, where bottom-level programs are callable subroutines that interact with the environment. We validate our method in three robot manipulation tasks. NTP achieves strong generalization across sequential tasks that exhibit hierarchal and compositional structures. The experimental results show that NTP learns to generalize well towards unseen tasks with increasing lengths, variable topologies, and changing objectives.stanfordvl.github.io/ntp/.

This manuscript surveys reinforcement learning from the perspective of optimization and control with a focus on continuous control applications. It surveys the general formulation, terminology, and typical experimental implementations of reinforcement learning and reviews competing solution paradigms. In order to compare the relative merits of various techniques, this survey presents a case study of the Linear Quadratic Regulator (LQR) with unknown dynamics, perhaps the simplest and best-studied problem in optimal control. The manuscript describes how merging techniques from learning theory and control can provide non-asymptotic characterizations of LQR performance and shows that these characterizations tend to match experimental behavior. In turn, when revisiting more complex applications, many of the observed phenomena in LQR persist. In particular, theory and experiment demonstrate the role and importance of models and the cost of generality in reinforcement learning algorithms. This survey concludes with a discussion of some of the challenges in designing learning systems that safely and reliably interact with complex and uncertain environments and how tools from reinforcement learning and control might be combined to approach these challenges.

Humans are able to understand and perform complex tasks by strategically structuring the tasks into incremental steps or subgoals. For a robot attempting to learn to perform a sequential task with critical subgoal states, such states can provide a natural opportunity for interaction with a human expert. This paper analyzes the benefit of incorporating a notion of subgoals into Inverse Reinforcement Learning (IRL) with a Human-In-The-Loop (HITL) framework. The learning process is interactive, with a human expert first providing input in the form of full demonstrations along with some subgoal states. These subgoal states define a set of subtasks for the learning agent to complete in order to achieve the final goal. The learning agent queries for partial demonstrations corresponding to each subtask as needed when the agent struggles with the subtask. The proposed Human Interactive IRL (HI-IRL) framework is evaluated on several discrete path-planning tasks. We demonstrate that subgoal-based interactive structuring of the learning task results in significantly more efficient learning, requiring only a fraction of the demonstration data needed for learning the underlying reward function with the baseline IRL model.

A critical component to enabling intelligent reasoning in partially observable environments is memory. Despite this importance, Deep Reinforcement Learning (DRL) agents have so far used relatively simple memory architectures, with the main methods to overcome partial observability being either a temporal convolution over the past k frames or an LSTM layer. More recent work (Oh et al., 2016) has went beyond these architectures by using memory networks which can allow more sophisticated addressing schemes over the past k frames. But even these architectures are unsatisfactory due to the reason that they are limited to only remembering information from the last k frames. In this paper, we develop a memory system with an adaptable write operator that is customized to the sorts of 3D environments that DRL agents typically interact with. This architecture, called the Neural Map, uses a spatially structured 2D memory image to learn to store arbitrary information about the environment over long time lags. We demonstrate empirically that the Neural Map surpasses previous DRL memories on a set of challenging 2D and 3D maze environments and show that it is capable of generalizing to environments that were not seen during training.


Question: StarCraft micromanagement utilizing reinforcement learning and curriculum transfer learning techniques.

In this paper, we present an innovative method for achieving mastery in the game of Go using deep neural networks and tree search techniques. Historically, Go has been considered the ultimate challenge for artificial intelligence due to its vast search space and the complexity of evaluating board positions and moves. Our approach involves the utilization of 'value networks' for board position evaluation and 'policy networks' for move selection. These networks are deep learning architectures that are trained through a unique combination of supervised learning from human expert gameplay and reinforcement learning from self-play games. Remarkably, without employing any lookahead search, the neural networks can play Go at a level comparable to advanced Monte Carlo tree search programs that simulate thousands of random self-play games. We further introduce an enhanced search algorithm that integrates Monte Carlo simulation with both value and policy networks. This algorithm empowers our program, AlphaGo, to achieve a 99.8% winning rate against other Go programs and to triumph over the European Go champion in a 5-0 victory. This milestone marks the first instance of a computer program defeating a professional human player in the full-sized game of Go, an achievement that was previously anticipated to be a decade away. This breakthrough demonstrates the potential of combining deep learning with tree search strategies to conquer complex games, setting a precedent for future advancements in artificial intelligence and game theory.

Cooperative games are those in which both agents share the same payoff structure. Valuebased reinforcement-learning algorithms, such as variants of Q-learning, have been applied to learning cooperative games, but they only apply when the game state is completely observable to both agents. Policy search methods are a reasonable alternative to value-based methods for partially observable environments. In this paper, we provide a gradient-based distributed policysearch method for cooperative games and compare the notion of local optimum to that of Nash equilibrium. We demonstrate the effectiveness of this method experimentally in a small, partially observable simulated soccer domain.

Classical methods to control heating systems are often marred by suboptimal performance, inability to adapt to dynamic conditions and unreasonable assumptions e.g. existence of building models. This paper presents a novel deep reinforcement learning algorithm which can control space heating in buildings in a computationally efficient manner, and benchmarks it against other known techniques. The proposed algorithm outperforms rule based control by between 5-10% in a simulation environment for a number of price signals. We conclude that, while not optimal, the proposed algorithm offers additional practical advantages such as faster computation times and increased robustness to non-stationarities in building dynamics.

Hierarchical reinforcement learning (HRL) is a promising approach to extend traditional reinforcement learning (RL) methods to solve more complex tasks. Yet, the majority of current HRL methods require careful task-specific design and on-policy training, making them difficult to apply in real-world scenarios. In this paper, we study how we can develop HRL algorithms that are general, in that they do not make onerous additional assumptions beyond standard RL algorithms, and efficient, in the sense that they can be used with modest numbers of interaction samples, making them suitable for real-world problems such as robotic control. For generality, we develop a scheme where lower-level controllers are supervised with goals that are learned and proposed automatically by the higher-level controllers. To address efficiency, we propose to use off-policy experience for both higherand lower-level training. This poses a considerable challenge, since changes to the lower-level behaviors change the action space for the higher-level policy, and we introduce an off-policy correction to remedy this challenge. This allows us to take advantage of recent advances in off-policy model-free RL to learn both higherand lower-level policies using substantially fewer environment interactions than on-policy algorithms. We term the resulting HRL agent HIRO and find that it is generally applicable and highly sample-efficient. Our experiments show that HIRO can be used to learn highly complex behaviors for simulated robots, such as pushing objects and utilizing them to reach target locations,1 learning from only a few million samples, equivalent to a few days of real-time interaction. In comparisons with a number of prior HRL methods, we find that our approach substantially outperforms previous state-of-the-art techniques.2

Reinforcement learning has achieved broad and successful application in cognitive science in part because of its general formulation of the adaptive control problem as the maximization of a scalar reward function. The computational reinforcement learning framework is motivated by correspondences to animal reward processes, but it leaves the source and nature of the rewards unspecified. This paper advances a general computational framework for reward that places it in an evolutionary context, formulating a notion of an optimal reward function given a fitness function and some distribution of environments. Novel results from computational experiments show how traditional notions of extrinsically and intrinsically motivated behaviors may emerge from such optimal reward functions. In the experiments these rewards are discovered through automated search rather than crafted by hand. The precise form of the optimal reward functions need not bear a direct relationship to the fitness function, but may nonetheless confer significant advantages over rewards based only on fitness.

Reinforcement learning provides a powerful and flexible framework for automated acquisition of robotic motion skills. However, applying reinforcement learning requires a sufficiently detailed representation of the state, including the configuration of task-relevant objects. We present an approach that automates state-space construction by learning a state representation directly from camera images. Our method uses a deep spatial autoencoder to acquire a set of feature points that describe the environment for the current task, such as the positions of objects, and then learns a motion skill with these feature points using an efficient reinforcement learning method based on local linear models. The resulting controller reacts continuously to the learned feature points, allowing the robot to dynamically manipulate objects in the world with closed-loop control. We demonstrate our method with a PR2 robot on tasks that include pushing a free-standing toy block, picking up a bag of rice using a spatula, and hanging a loop of rope on a hook at various positions. In each task, our method automatically learns to track task-relevant objects and manipulate their configuration with the robot's arm.

The popular Q-learning algorithm is known to overestimate action values under certain conditions. It was not previously known whether, in practice, such overestimations are common, whether they harm performance, and whether they can generally be prevented. In this paper, we answer all these questions affirmatively. In particular, we first show that the recent DQN algorithm, which combines Q-learning with a deep neural network, suffers from substantial overestimations in some games in the Atari 2600 domain. We then show that the idea behind the Double Q-learning algorithm, which was introduced in a tabular setting, can be generalized to work with large-scale function approximation. We propose a specific adaptation to the DQN algorithm and show that the resulting algorithm not only reduces the observed overestimations, as hypothesized, but that this also leads to much better performance on several games. The goal of reinforcement learning (Sutton and Barto 1998) is to learn good policies for sequential decision problems, by optimizing a cumulative future reward signal. Q-learning (Watkins 1989) is one of the most popular reinforcement learning algorithms, but it is known to sometimes learn unrealistically high action values because it includes a maximization step over estimated action values, which tends to prefer overestimated to underestimated values. In previous work, overestimations have been attributed to insufficiently flexible function approximation (Thrun and Schwartz 1993) and noise (van Hasselt 2010, 2011). In this paper, we unify these views and show overestimations can occur when the action values are inaccurate, irrespective of the source of approximation error. Of course, imprecise value estimates are the norm during learning, which indicates that overestimations may be much more common than previously appreciated. It is an open question whether, if the overestimations do occur, this negatively affects performance in practice. Overoptimistic value estimates are not necessarily a problem in and of themselves. If all values would be uniformly higher then the relative action preferences are preserved and we would not expect the resulting policy to be any worse. Furthermore, it is known that sometimes it is good to be optimistic: optimism in the face of uncertainty is a well-known Copyright c © 2016, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. exploration technique (Kaelbling et al. 1996). If, however, the overestimations are not uniform and not concentrated at states about which we wish to learn more, then they might negatively affect the quality of the resulting policy. Thrun and Schwartz (1993) give specific examples in which this leads to suboptimal policies, even asymptotically. To test whether overestimations occur in practice and at scale, we investigate the performance of the recent DQN algorithm (Mnih et al. 2015). DQN combines Q-learning with a flexible deep neural network and was tested on a varied and large set of deterministic Atari 2600 games, reaching human-level performance on many games. In some ways, this setting is a best-case scenario for Q-learning, because the deep neural network provides flexible function approximation with the potential for a low asymptotic approximation error, and the determinism of the environments prevents the harmful effects of noise. Perhaps surprisingly, we show that even in this comparatively favorable setting DQN sometimes substantially overestimates the values of the actions. We show that the Double Q-learning algorithm (van Hasselt 2010), which was first proposed in a tabular setting, can be generalized to arbitrary function approximation, including deep neural networks. We use this to construct a new algorithm called Double DQN. This algorithm not only yields more accurate value estimates, but leads to much higher scores on several games. This demonstrates that the overestimations of DQN indeed lead to poorer policies and that it is beneficial to reduce them. In addition, by improving upon DQN we obtain state-of-the-art results on the Atari domain.

Creating animation of a character putting on clothing is challenging due to the complex interactions between the character and the simulated garment. We take a model-free deep reinforcement learning (deepRL) approach to automatically discovering robust dressing control policies represented by neural networks. While deepRL has demonstrated several successes in learning complex motor skills, the data-demanding nature of the learning algorithms is at odds with the computationally costly cloth simulation required by the dressing task. This paper is the first to demonstrate that, with an appropriately designed input state space and a reward function, it is possible to incorporate cloth simulation in the deepRL framework to learn a robust dressing control policy. We introduce a salient representation of haptic information to guide the dressing process and utilize it in the reward function to provide learning signals during training. In order to learn a prolonged sequence of motion involving a diverse set of manipulation skills, such as grasping the edge of the shirt or pulling on a sleeve, we find it necessary to separate the dressing task into several subtasks and learn a control policy for each subtask. We introduce a policy sequencing algorithm that matches the distribution of output states from one task to the input distribution for the next task in the sequence. We have used this approach to produce character controllers for several dressing tasks: putting on a t-shirt, putting on a jacket, and robot-assisted dressing of a sleeve.

While Reinforcement Learning (RL) is not traditionally designed for interactive supervisory input from a human teacher, several works in both robot and software agents have adapted it for human input by letting a human trainer control the reward signal. In this work, we experimentally examine the assumption underlying these works, namely that the human-given reward is compatible with the traditional RL reward signal. We describe an experimental platform with a simulated RL robot and present an analysis of real-time human teaching behavior found in a study in which untrained subjects taught the robot to perform a new task. We report three main observations on how people administer feedback when teaching a robot a task through Reinforcement Learning: (a) they use the reward channel not only for feedback, but also for future-directed guidance; (b) they have a positive bias to their feedback — possibly using the signal as a motivational channel; and (c) they change their behavior as they develop a mental model of the robotic learner. Given this, we made specific modifications to the simulated RL robot, and analyzed and evaluated its learning behavior in four additional experiments with human trainers. We report significant improvements on several learning measures. This work demonstrates the importance of understanding the human-teacher/robot-learner partnership in order to design algorithms that support how people want to teach while simultaneously improving the robot’s learning behavior.

The paper discusses the augmentation of the Concurrent Individual and Social Learning (CISL) mechanism with a new Human Advice Layer (HAL). The new layer is characterized by a Gaussian Mixture Model (GMM), which is trained on human experience data. The CISL mechanism consists of the Individual Performance and Task Allocation Markov Decision Processes (MDP), and the HAL can provide preferred action selection policies to the individual agents. The data utilized for training the GMM is collected using a heterogeneous team foraging simulation. When leveraging human experience in the multi-agent learning process, the team performance is enhanced significantly.

Techniques for both exploratory and known item search tend to direct only to more specific subtopics or individual documents, as opposed to allowing directing the exploration of the information space. We present an interactive information retrieval system that combines Reinforcement Learning techniques along with a novel user interface design to allow active engagement of users in directing the search. Users can directly manipulate document features (keywords) to indicate their interests and Reinforcement Learning is used to model the user by allowing the system to trade off between exploration and exploitation. This gives users the opportunity to more effectively direct their search nearer, further and following a direction. A task-based user study conducted with 20 participants comparing our system to a traditional query-based baseline indicates that our system significantly improves the effectiveness of information retrieval by providing access to more relevant and novel information without having to spend more time acquiring the information.


Question: StarCraft micromanagement utilizing reinforcement learning and curriculum transfer learning.

This paper presents an advanced approach to experience replay in reinforcement learning, focusing on prioritizing experiences for more efficient learning. In previous methods, transitions from the replay memory were sampled uniformly, disregarding the potential significance of individual experiences. The proposed framework introduces a prioritization mechanism that enables the replay of crucial transitions at a higher frequency, thereby enhancing learning efficiency. This enhancement is particularly notable when applied to the Deep Q-Network (DQN) algorithm, known for its human-level performance in Atari games. By integrating prioritized experience replay into DQN, the algorithm achieves a new benchmark, outperforming its uniform replay counterpart in 42 out of 57 games, setting a new state-of-the-art standard in reinforcement learning performance.

Adapting this concept to StarCraft micromanagement tasks, prioritized experience replay could significantly improve the learning curve of reinforcement learning agents. By emphasizing the replay of pivotal combat scenarios and micromanagement decisions, agents could learn more effectively, leading to superior performance in complex real-time strategy game environments. This strategy not only accelerates learning but also ensures that the agent's decision-making process is finely tuned to handle critical in-game situations, thus enhancing its overall capability in managing units during battles.

Creating animation of a character putting on clothing is challenging due to the complex interactions between the character and the simulated garment. We take a model-free deep reinforcement learning (deepRL) approach to automatically discovering robust dressing control policies represented by neural networks. While deepRL has demonstrated several successes in learning complex motor skills, the data-demanding nature of the learning algorithms is at odds with the computationally costly cloth simulation required by the dressing task. This paper is the first to demonstrate that, with an appropriately designed input state space and a reward function, it is possible to incorporate cloth simulation in the deepRL framework to learn a robust dressing control policy. We introduce a salient representation of haptic information to guide the dressing process and utilize it in the reward function to provide learning signals during training. In order to learn a prolonged sequence of motion involving a diverse set of manipulation skills, such as grasping the edge of the shirt or pulling on a sleeve, we find it necessary to separate the dressing task into several subtasks and learn a control policy for each subtask. We introduce a policy sequencing algorithm that matches the distribution of output states from one task to the input distribution for the next task in the sequence. We have used this approach to produce character controllers for several dressing tasks: putting on a t-shirt, putting on a jacket, and robot-assisted dressing of a sleeve.

Exploration remains a significant challenge to reinforcement learning methods, especially in environments where reward signals are sparse. Recent methods of learning from demonstrations have shown to be promising in overcoming exploration difficulties but typically require considerable highquality demonstrations that are difficult to collect. We propose to effectively leverage available demonstrations to guide exploration through enforcing occupancy measure matching between the learned policy and current demonstrations, and develop a novel Policy Optimization from Demonstration (POfD) method. We show that POfD induces implicit dynamic reward shaping and brings provable benefits for policy improvement. Furthermore, it can be combined with policy gradient methods to produce state-of-the-art results, as demonstrated experimentally on a range of popular benchmark sparse-reward tasks, even when the demonstrations are few and imperfect.

Games are considered important benchmark tasks of artificial intelligence research. Modern strategic board games can typically be played by three or more people, which makes them suitable test beds for investigating multi-player strategic decision making. Monte-Carlo Tree Search (MCTS) is a recently published family of algorithms that achieved successful results with classical, two-player, perfect-information games such as Go. In this paper we apply MCTS to the multi-player, non-deterministic board game Settlers of Catan. We implemented an agent that is able to play against computer-controlled and human players. We show that MCTS can be adapted successfully to multi-agent environments, and present two approaches of providing the agent with a limited amount of domain knowledge. Our results show that the agent has considerable playing strength when compared to existing heuristics for the game. We conclude that MCTS is suitable for implementing a strong Settlers of Catan player.

This paper extends previous work with Dyna a class of architectures for intelligent systems based on approximating dynamic program ming methods Dyna architectures integrate trial and error reinforcement learning and execution time planning into a single process operating alternately on the world and on a learned model of the world In this paper I present and show results for two Dyna archi tectures The Dyna PI architecture is based on dynamic programming s policy iteration method and can be related to existing AI ideas such as evaluation functions and uni versal plans reactive systems Using a nav igation task results are shown for a simple Dyna PI system that simultaneously learns by trial and error learns a world model and plans optimal routes using the evolving world model The Dyna Q architecture is based on Watkins s Q learning a new kind of rein forcement learning Dyna Q uses a less famil iar set of data structures than does Dyna PI but is arguably simpler to implement and use We show that Dyna Q architectures are easy to adapt for use in changing environments Introduction to Dyna How should a robot decide what to do The traditional answer in AI has been that it should deduce its best action in light of its current goals and world model i e that it should plan However it is now widely recognized that planning s usefulness is limited by its computational complexity and by its dependence on an accurate world model An alternative approach is to do the planning in advance and compile its result into a set of rapid reactions or situation action rules which are then used for real time decision making Yet a third approach is to learn a good set of reactions by trial and error this has the advantage of eliminating the dependence on a world model In this paper I brie y introduce Dyna a class of simple architectures integrating and permitting tradeo s among these three approaches Dyna architectures use machine learning algo rithms to approximate the conventional optimal con trol technique known as dynamic programming DP Bellman Ross DP itself is not a learn ing method but rather a computational method for determining optimal behavior given a complete model of the task to be solved It is very similar to state space search but di ers in that it is more incremental and never considers actual action sequences explicitly only single actions at a time This makes DP more amenable to incremental planning at execution time and also makes it more suitable for stochastic or in completely modeled environments as it need not con sider the extremely large number of sequences possi ble in an uncertain environment Learned world mod els are likely to be stochastic and uncertain making DP approaches particularly promising for learning sys tems Dyna architectures are those that learn a world model online while using approximations to DP to learn and plan optimal behavior Intuitively Dyna is based on the old idea that planning is like trial and error learning from hypothet ical experience Craik Dennett The theory of Dyna is based on the theory of DP e g Ross and on DP s relationship to reinforcement learning Watkins Barto Sutton Watkins to temporal di erence learning Sutton and to AI methods for planning and search Korf Werbos has previously argued for the general idea of building AI systems that approx imate dynamic programming and Whitehead and others Sutton Barto Sutton Pinette Rumelhart et al have presented results for the speci c idea of augmenting a reinforcement learning system with a world model used for planning Dyna PI Dyna by Approximating Policy Iteration I call the rst Dyna architecture Dyna PI because it is based on approximating a DP method known as pol icy iteration Howard The Dyna PI architec ture consists of four components interacting as shown in Figure The policy is simply the function formed by the current set of reactions it receives as input a description of the current state of the world and pro duces as output an action to be sent to the world The world represents the task to be solved prototypi cally it is the robot s external environment The world receives actions from the policy and produces a next state output and a reward output The overall task is de ned as maximizing the long term average reward per time step cf Russell The architecture also includes an explicit world model The world model is intended to mimic the one step input output behavior of the real world Finally the Dyna PI architecture in cludes an evaluation function that rapidly maps states to values much as the policy rapidly maps states to actions The evaluation function the policy and the world model are each updated by separate learning processes WORLD Action Reward (scalar) Heuristic Reward (scalar) State EVALUATION FUNCTION

Providing Reinforcement Learning agents with expert advice can dramatically improve various aspects of learning. Prior work has developed teaching protocols that enable agents to learn efficiently in complex environments; many of these methods tailor the teacher’s guidance to agents with a particular representation or underlying learning scheme, offering effective but specialized teaching procedures. In this work, we explore protocol programs, an agent-agnostic schema for Human-in-the-Loop Reinforcement Learning. Our goal is to incorporate the beneficial properties of a human teacher into Reinforcement Learning without making strong assumptions about the inner workings of the agent. We show how to represent existing approaches such as action pruning, reward shaping, and training in simulation as special cases of our schema and conduct preliminary experiments on simple domains.

Classic approaches to game AI require either a high quality of domain knowledge, or a long time to generate effective AI behaviour. These two characteristics hamper the goal of establishing challenging game AI. In this paper, we put forward Monte-Carlo Tree Search as a novel, unified framework to game AI. In the framework, randomized explorations of the search space are used to predict the most promising game actions. We will demonstrate that Monte-Carlo Tree Search can be applied effectively to (1) classic board-games, (2) modern board-games, and (3) video games.

How do we learn to think better, and what can we do to promote such metacognitive learning? Here, we propose that cognitive growth proceeds through metacognitive reinforcement learning. We apply this theory to model how people learn how far to plan ahead and test its predictions about the speed of metacognitive learning in two experiments. In the first experiment, we find that our model can discern a reward structure that promotes metacognitive reinforcement learning from one that hinders it. In the second experiment, we show that our model can be used to design a feedback mechanism that enhances metacognitive reinforcement learning in an environment that hinders learning. Our results suggest that modeling metacognitive learning is a promising step towards promoting cognitive growth.

Cooperative games are those in which both agents share the same payoff structure. Valuebased reinforcement-learning algorithms, such as variants of Q-learning, have been applied to learning cooperative games, but they only apply when the game state is completely observable to both agents. Policy search methods are a reasonable alternative to value-based methods for partially observable environments. In this paper, we provide a gradient-based distributed policysearch method for cooperative games and compare the notion of local optimum to that of Nash equilibrium. We demonstrate the effectiveness of this method experimentally in a small, partially observable simulated soccer domain.

A novel class of Approximate Policy Iteration (API) algorithms have recently demonstrated impressive practical performance (e.g., ExIt [1], AlphaGo-Zero [2]). This new family of algorithms maintains, and alternately optimizes, two policies: a fast, reactive policy (e.g., a deep neural network) deployed at test time, and a slow, non-reactive policy (e.g., Tree Search), that can plan multiple steps ahead. The reactive policy is updated under supervision from the non-reactive policy, while the non-reactive policy is improved via guidance from the reactive policy. In this work we study this class of Dual Policy Iteration (DPI) strategy in an alternating optimization framework and provide a convergence analysis that extends existing API theory. We also develop a special instance of this framework which reduces the update of non-reactive policies to model-based optimal control using learned local models, and provides a theoretically sound way of unifying model-free and model-based RL approaches with unknown dynamics. We demonstrate the efficacy of our approach on various continuous control Markov Decision Processes.

Though successful in high-dimensional domains, deep reinforcement learning exhibits high sample complexity and suffers from stability issues as reported by researchers and practitioners in the field. These problems hinder the application of such algorithms in real-world and safety-critical scenarios. In this paper, we take steps towards stable and efficient reinforcement learning by following a model-based approach that is known to reduce agent-environment interactions. Namely, our method augments deep Q-networks (DQNs) with model predictions for transitions, rewards, and termination flags. Having the model at hand, we then conduct a rigorous theoretical study of our algorithm and show, for the first time, convergence to a stationary point. En route, we provide a counterexample showing that ’vanilla’ DQNs can diverge confirming practitioners’ and researchers’ experiences. Our proof is novel in its own right and can be extended to other forms of deep reinforcement learning. In particular, we believe exploiting the relation between reinforcement (with deep function approximators) and online learning can serve as a recipe for future proofs in the domain. Finally, we validate our theoretical results in 20 games from the Atari benchmark. Our results show that following the proposed model-based learning approach not only ensures convergence but leads to a reduction in sample complexity and superior performance. Introduction Model-free deep reinforcement learning methods have recently demonstrated impressive performance on a range of complex learning tasks (Hessel et al. 2018; Lillicrap et al. 2017; Jaderberg et al. 2017). Deep Q-Networks (DQNs) (Mnih et al. 2015), in particular, stand out as a versatile and effective tool for a wide range of applications. DQNs offer an end-to-end solution for many reinforcement learning problems by generalizing tabular Q-learning to highdimensional input spaces. Unfortunately, DQNs still suffer from a number of important drawbacks that hinder wide-spread adoption. There is a general lack of theoretical understanding to guide the use of deep Q-learning algorithms. Combining non-linear function approximation with off-policy reinforcement learning is known to lead to possible divergence issues, even in very simple tasks (Boyan and Preliminary work. Moore 1995). Despite improvements introduced by deep Qnetworks (Mnih et al. 2015), Q-learning approaches based on deep neural networks still exhibit occasional instability during the learning process, see for example (van Hasselt, Guez, and Silver 2016). In this paper, we aim to address the instability of current deep RL approaches by augmenting model-free value learning with a predictive model. We propose a hybrid model and value learning objective, which stabilises the learning process and reduces sample complexity when compared to model-free algorithms. We attribute this reduction to the richer feedback signal provided by our method compared to standard deep Q-networks. In particular, our model provides feedback at every time step based on the current prediction error, which in turn eliminates one source of sample inefficiency caused by sparse rewards. These conclusions are also in accordance with previous research conducted on linear function approximation in reinforcement learning (Parr et al. 2008; Sutton et al. 2012; Song et al. 2016). While linear function approximation results provide a motivation for model-based stabilisation, such theories fail to generalise to deep non-linear architectures (Song et al. 2016). To close this gap in the literature and demonstrate stability of our method, we prove convergence in the general deep RL setting with deep value function approximators. Theoretically analysing deep RL algorithms, however, is challenging due to nonlinear functional dependencies and objective non-convexities prohibiting convergence to globally optimal policies. As such, the best one would hope for is a stationary point (e.g., a first-order one) that guarantees vanishing gradients. Even understanding gradient behavior for deep networks in RL can be difficult due to explorationbased policies and replay memory considerations. To alleviate some of these problems, we map deep reinforcement learning to a mathematical framework explicitly designed for understanding the exploration/exploitation trade-off. Namely, we formalise the problem as an online learning game the agent plays against an adversary (i.e., the environment). Such a link allows us to study a more general problem combining notions of regret, reinforcement learning, and optimisation. Given such a link, we prove that for any > 0, regret vanishes as O ( T 1−2 lnT ) with T being the total number of rounds. This, in turn, guarantees convergence to a stationary point. ar X iv :1 80 9. 01 90 6v 1 [ cs .L G ] 6 S ep 2 01 8 Guided by the above theoretical results, we validate that our algorithm leads to faster learning by conducting experiments on 20 Atari games. Due to the high computational and budgeting demands, we chose a benchmark that is closest to our setting. Concretely, we picked DQNs as our template algorithm to improve on both theoretically and empirically. It is worth noting that extending our theoretical results to more complex RL algorithms is an orthogonal future direction. Background: Reinforcement Learning We consider a discrete time, infinite horizon, discounted Markov Decision Process (MDP) 〈 S,A, P , P , γ 〉 . Here S denotes the state space, A the set of possible actions, P (R) : S × A × R → [0, 1] is the reward function, P (S) : S × A × S → [0, 1] is the state transition function, and γ ∈ (0, 1) is a discount factor. An agent, being in state St ∈ S at time step t, executes an action at ∈ A sampled from its policy π(at|St), a conditional probability distribution where π : S × A → [0, 1]. The agent’s action elicits from the environment a reward signal rt ∈ R indicating instantaneous reward, a terminal flag ft ∈ {0, 1} indicating a terminal event that restarts the environment, and a transition to a successor state St+1 ∈ S. We assume that the sets S, A, and R are discrete. The reward rt is sampled from the conditional probability distribution P (rt|St,at). Similarly, ft ∼ P (F (ft|St,at) with P (F ) : S × A × {0, 1} → [0, 1], where a terminal event (ft = 1) restarts the environment according to some initial state distribution S0 ∼ P (S) 0 (S0). The state transition to a successor state is determined by a stochastic state transition function according to St+1 ∼ P (St+1|St,at). The agent’s goal is to maximise future cumulative reward E P (S) 0 ,π,P (R),P (F ),P (S) [ ∞ ∑ t=0 ( t ∏ t′=0 (1− ft′−1) ) γrt ] with respect to the policy π. An important quantity in RL are Q-values Q(St,at), which are defined as the expected future cumulative reward value when executing action at in state St and subsequently following policy π. Qvalues enable us to conveniently phrase the RL problem as maxπ ∑ S p (S) ∑ a π(a|S)Q(S,a), where p(S) is the stationary state distribution obtained when executing π in the environment (starting from S0 ∼ P (S) 0 (S0)). Deep Q-Networks: Value-based reinforcement learning approaches identify optimal Q-values directly using parametric function approximators Qθ(S,a), where θ ∈ Θ represents the parameters (Watkins 1989; Busoniu et al. 2010). Optimal Q-value estimates then correspond to an optimal policy π(a|S) = δa,argmaxa′Qθ(S,a′). Deep Q-networks (Mnih et al. 2015) learn a deep neural network based Qvalue approximation by performing stochastic gradient descent on the following training objective: L(θ) = ES,a,r,f,S′ [( r + (1− f)γmax a′ Qθ−(S ′,a′) −Qθ(S,a) )2] . (1) The expectation ranges over transitions S,a, r, f,S′ sampled from an experience replay memory (S′ denotes the state at the next time step). Use of this replay memory, together with the use of a separate target network Qθ− (with different parameters θ− ∈ Θ) for calculating the bootstrap values maxa′ Qθ−(S ′,a′), helps stabilise the learning process. Background: Online Learning In this paper, we employ a special form of regret minimisation games that we briefly review here. A regret minimisation game is a triple 〈Θ,F , T 〉, where Θ is a non-empty decision set, F is the set of moves of the adversary which contains bounded convex functions from R to R and T is the total number of rounds. The game commences in rounds, where at round j = 1, . . . , T , the agent chooses a prediction θj ∈ Θ and the adversary a loss function Lj ∈ F . At the end of the round, the adversary reveals its choice and the agent suffers a loss Lj(θj). In this paper, we are concerned with the full-information case where the agent can observe the complete loss function Lj at the end of each round. The goal of the game is for the agent to make successive predictions to minimise cumulative regret defined as: