\section*{Extensions to PCGNN\xspace}
\label{sec:appdx}
We extend standard PCGNN\xspace by adding in additional features useful for \texttt{MCHAMR}\xspace. The specific features we use are as follows:
\begin{LaTeXdescription}
    \item[3D] We simply generalise the generation procedure to three dimensions, by considering a level as a $W\times H\times D$ block of tiles, and iterating through all of the tiles.
    \item[Multiple Iterations] Using multiple iterations is also a relatively simple change: after performing the generation once, we use this as a starting point, and repeat the process for multiple iterations. The intuition here is that performing this process multiple times has the effect of increasing the effective \textit{receptive field} of the network. This is similar to how deep convolutional neural networks are able to focus on long-range dependencies by stacking local interactions~\citep{luo2017Understanding}. An approach like this has been used successfully in PCG before; specifically, \citet{earle2021Illuminating} perform multiple processing iterations of a neural cellular automaton to generate levels.
    \item[Input Center Tile] This adds in the center tile the model is currently focusing on as an input. This is beneficial in the case of multiple iterations, so that the network can pass along information to the subsequent iterations.

    \item[Various Starting Points] While we mostly still use random initialisations, some experiments instead initialised the level with a specific ``default'' tile, so that the starting points are all the same.
\end{LaTeXdescription}
\section*{Reproducibility Details}
As mentioned in the main text, we publicly release our source code at \url{https://github.com/Michael-Beukman/MCHAMR}. Implementation details, as well as instructions to reproduce our results, can be found there.
\section*{Experimental Setup}
When plotting the fitness value in our experiments, we use the maximum fitness in the current population. This is done because we use the best individual at the end of generation as our chosen generator. Furthermore, all of our fitness functions are scaled between 0 and 1, so $1.0$ corresponds to the optimal fitness.
\section*{Hyperparameters}
Here we detail the hyperparameters and exact setups used to generate the images in the main paper. If a particular parameter is not mentioned, the default value from \citet{beukman2022Procedural} was used; for example, all of our experiments padded the boundaries with $-1$. When we used novelty, we used the Hamming distance, which was denoted as \textit{Visual Diversity} by \citet{beukman2022Procedural}.
\label{sec:hierarchy_vals}
In this experiment, we compared the results when using different logical window sizes, from $1\times 1$, representing a composed approach, to $10\times 10$. 
For all of these experiments, we used the hyperparameters shown in \autoref{tab:appdx:hierarchy}, with \autoref{fig:appx:tile_sizes_examples} containing some examples of generated levels. We calculated the individual's fitness as the average of the following two fitness functions:
\label{sec:reach_fit}
\begin{LaTeXdescription}
    \item[Probability]: We compute the level's tile distribution, representing the frequency of each tile type in the level. This fitness is the square root of the Jensen-Shannon divergence~\citep{fuglede2004JensenShannon} between the level's tile distribution and an ``ideal'' distribution of 40\% house, 30\% garden and 30\% road.
    \item[Reachability]: All houses must be reachable (with allowable moves being up, down, left and right) from all other houses via roads without being directly connected. This fitness is calculated as $a \times b$, where $a = \min{(\frac{H}{20}, 1)}$, with $H$ being the number of houses that have between 1 and 3 (inclusive) neighbouring road tiles. $b = \frac{1}{(d_{house,road} + 1)(d_{road} + 1)}$, where $d_i = \min{(|1 - c_i|, 10)}$ and $c_i$ is the number of disconnected regions of tile type $i$. For example, if a level consists entirely of roads, with a vertical line of gardens through the middle of the level, then there would be two disconnected regions of road (to the left and right of the vertical line).
     When calculating $c_{house,road}$, we label houses and roads as the same tile, and find the number of disconnected regions (i.e. those regions separated by gardens). All of these operations can be efficiently implemented using morphological image operations, notably image labelling.\footnote{\url{https://scikit-image.org/docs/stable/api/skimage.morphology.html\#label}} Here, $a$ incentivises each house to be reachable by a road, without being completely isolated from gardens or other houses; $b$ specifies that there must be one connected road region, and that each house must be reachable from every other house (as $b$ is maximised when $c_i = 1$, corresponding to there only being one connected region).
\end{LaTeXdescription}


\begin{figure*}[h]
    \centering
    \begin{subfigure}{0.49\linewidth}
        \includegraphics[width=1\linewidth]{hierarchy_examples/example_levels_0_True.jpg}
        \caption{}
    \end{subfigure}
    \begin{subfigure}{0.49\linewidth}
        \includegraphics[width=1\linewidth]{hierarchy_examples/example_levels_1_True.jpg}
        \caption{}
    \end{subfigure}

    \caption{Showcasing example levels generated by the networks with different logical tile sizes for two different seeds.}
    \label{fig:appx:tile_sizes_examples}
\end{figure*}

\begin{figure*}[h!]
    \centering
    \begin{minipage}[t]{.47\textwidth}
        \centering
        \begin{subfigure}{0.32\linewidth}
            \includegraphics[width=1\linewidth]{hand_designed_correct.pdf}
            \caption{}
        \end{subfigure}
        \begin{subfigure}{0.32\linewidth}
            \includegraphics[width=1\linewidth]{hand_designed_compose.pdf}
            \caption{}
        \end{subfigure}
        \begin{subfigure}{0.32\linewidth}
            \includegraphics[width=1\linewidth]{hand_designed_flat.pdf}
            \caption{}
        \end{subfigure}
        \caption{}
        \label{fig:appx:compose_examples:hand}
    \end{minipage}
    \hfill
    \begin{minipage}[t]{.47\textwidth}
    \centering
    \begin{subfigure}{0.32\linewidth}
        \includegraphics[width=1\linewidth]{random_correct.pdf}
        \caption{}
    \end{subfigure}
    \begin{subfigure}{0.32\linewidth}
        \includegraphics[width=1\linewidth]{random_compose.pdf}
        \caption{}
    \end{subfigure}
    \begin{subfigure}{0.32\linewidth}
        \includegraphics[width=1\linewidth]{random_flat.pdf}
        \caption{}
    \end{subfigure}
    \caption{}
    \label{fig:appx:compose_examples:random}
    \end{minipage}
    \caption*{The (a) desired, (b) composed and (c) flat levels for a (\autoref{fig:appx:compose_examples:hand}) hand-designed or (\autoref{fig:appx:compose_examples:random}) randomly generated target layout. Note that in the main text, we did not use the fitness results from the hand-designed layout and just show it here for comparison.}
\end{figure*}


\begin{table}[h]
    \centering
    \caption{Hyperparameters where we experimented with different window sizes, i.e. differing complexity levels of the base structures.}
    \label{tab:appdx:hierarchy}
    \begin{adjustbox}{width=1\linewidth}
    \begin{tabular}{ll}
        \toprule
        Name & Value \\
        \midrule
        One Hot Encoding & True \\
        Generations & 50 \\
        Population Size & 50 \\
        Number of Random Variables & 1 \\
        Random Perturb Size & 0 \\
        Number of Iterations & 3 \\
        Input Center Tile & Yes \\
        Default Tile & Garden \\
        Novelty & No Novelty \\
        Level Starting Point & Default Tile (Garden) \\
        Number of levels for fitness calculations & 5 \\
        \bottomrule
    \end{tabular}
\end{adjustbox}
\end{table}



\subsection*{Composition vs flat}
\autoref{tab:appdx:compose_vs_flat} shows the hyperparameters we used when directly comparing \texttt{MCHAMR}\xspace and PCGNN\xspace.

In this experiment, we have 20 random, but fixed town layouts. Thus, for each method (\texttt{MCHAMR}\xspace and PCGNN\xspace), we have the same desired levels.

\Autoref{fig:appx:compose_examples:hand,fig:appx:compose_examples:random} show the desired layout and the result from both the composed and flat methods. While the composed method does not generate perfect levels, they are significantly closer to the desired layouts compared to the flat generator. When using the flat method, the levels are very poor, not containing house structures at all.

\begin{table*}[h]
    \centering
    \caption{Hyperparameters used for the Composed vs Flat Towns}
    \label{tab:appdx:compose_vs_flat}
    \begin{adjustbox}{width=1\linewidth}
    \begin{tabular}{llll}
        \toprule
        Name & Flat (PCGNN\xspace) & Composed Town & Composed House\\
        \midrule
        Level Size                                  & $25\times 25$     & $5\times 5$             & $5\times 5$\\
        One Hot Encoding                            & False             & False                   & False\\
        Generations                                 & 150               & 150                     & 150\\
        Population Size                             & 50                & 50                      & 50\\
        Number of Random Variables                  & 1                 & 1                       & 1\\
        Random Perturb Size                         & 0                 & 0                       & 0\\
        Number of Iterations                        & 5                & 5                      & 5\\
        Input Center Tile                           & Yes               & Yes                     & Yes\\
        Novelty                                     & No Novelty & No Novelty & No Novelty \\
        Level Starting Point                        & Default (Road)             & Default (Road)                   & Default (Air)\\
        \# of levels for fitness calculations       & 5                 & 5                       & 5\\
        \bottomrule
    \end{tabular}
\end{adjustbox}
\end{table*}

\subsection*{Showcase}
\autoref{tab:appdx:showcase} details the hyperparameters used for each showcase result. The generator that we used to generate both towns and cities is denoted as ``Town \& City''.
We used the following fitness functions for these experiments:
\begin{LaTeXdescription}
    \item[House] The house was incentivised to be a hollow cube, with roof tiles at the top by setting the fitness to the average overlap between the generated house and the ``ideal'', hollow cube. Here we weighted novelty, intra-generator novelty and this fitness using the ratio $1:1:8$.
    \item[Garden] The garden had multiple fitness functions, incentivising it to have at least one tree and one flower, contain some water tiles (less than 5\%) and not be too dense (the fraction of grass tiles must be between $0.2$ and $0.7$ and trees should not be too close to each other). We weighted novelty, intra-generator novelty and this fitness using the ratio $1:1:4$.
    \item[Town] This incentivised the distribution of tiles to be 40\% house, 30\% garden and 30\% road (by computing the square root of the Jensen-Shannon divergence between this distribution and the actual one), in addition to the houses being reachable via roads. This was calculated as $\frac{a + b}{2}$, with $a$ and $b$ defined in the \hyperref[sec:reach_fit]{Hyperparameters Section}.
    \item[City] The average of two fitnesses, the first one incentivising an equal number of houses, roads and gardens, calculated as $\frac{1}{3}\sum_{i \in \{ house, road, garden\}}\max(1 - 10 (n_i - \frac{1}{3})^2, 0)$ where $n_i$ is the fraction of tiles of type $i$. The second fitness is \textit{reachability}, again calculated as $\frac{a + b}{2}$.
    \item[Town \& City] This fitness incentivises the boundary of the town to be houses, with an inner ring of roads, and a large interior section of gardens. This fitness was calculated as the average overlap between the ideal layout and the actual level for each section. This generator also used a context size of $2$, i.e. it took in a $5\times 5$ block around the current tile, instead of the normal $3\times 3$.
\end{LaTeXdescription}
\begin{table*}
    \centering
    \caption{This table details the hyperparameters used for the showcase images shown in the main text.}
    \label{tab:appdx:showcase}
    \begin{adjustbox}{width=1\linewidth}
    \begin{tabular}{llllll}
        \toprule
        Name                                   & House & Garden & Town & City  & Town  \& City \\
        \midrule
        One Hot Encoding                       & False            & False           & False            & False             & False\\
        Generations                            & 500              & 100             & 150              & 150               & 350\\
        Population Size                        & 20               & 20              & 50               & 50                & 30\\

        Number of Random Variables             & 1                & 2               & 1                & 1                 & 1\\
        Random Perturb Size                    & 0                & 0.1565          & 0                & 0                 & 0.1\\
        Number of Iterations                   & 10               & 1               & 10               & 10                & 5\\
        Input Center Tile                      & Yes              & No              & Yes              & Yes               & Yes\\
        Level Starting Point                   & Random           & Random          & Default (Road)          & Default (Road)           & Random \\
        \# of levels for fitness calculations  & 15               & 15              & 5                & 5                & 15\\
        Novelty Distance Function                & Hamming          & Hamming         & No Novelty       & No Novelty        & No Novelty\\
        \bottomrule
    \end{tabular}
\end{adjustbox}
\end{table*}

\section{}\label{}




\begin{figure}[<options>]
	\centering
		\includegraphics[<options>]{}
	  \caption{}\label{fig1}
\end{figure}


\begin{table}[<options>]
\caption{}\label{tbl1}
\begin{tabular*}{\tblwidth}{@{}LL@{}}
\toprule
  &  \\
\midrule
 & \\
 & \\
 & \\
 & \\
\bottomrule
\end{tabular*}
\end{table}




\section{}\label{}

\printcredits

\bibliographystyle{cas-model2-names}




\section{Introduction}
\label{chap:intro}

Procedural content generation (PCG) is a research field focused on generating game content, such as levels, maps, music, etc., in an automated way~\citep{togelius2011Searchbased}. PCG has numerous benefits, such as being cheaper to use than the alternative of manually designing content~\citep{hendrikx2013Procedural} and being able to generate significantly more content than would be possible through manual creation~\citep{smith2017Procedural,korn2017Procedural}. 
PCG has recently garnered more attention~\citep{brewer2017computerized} as video games have become more popular~\citep{esa_2022,esa_2023}, leading to impressive results in numerous games~\citep{volz2018Evolving,gisslen2021Adversarial,earle2021Illuminating}.

However, much of the research done in the field of PCG focuses on simple, 2D tilemap games, such as \textit{Super Mario Bros}, \textit{The Legend of Zelda}, and maze games~\citep{beukman2022Procedural,khalifa2020Pcgrl,earle2021Illuminating,ferreira2014Multipopulation,shaker2011Mario}. While these are useful test domains, the games themselves are often very simple, and do not exhibit complex structures. This focus on simplicity makes it non-trivial to generalise these methods to more complex and modern games. Furthermore, few methods focus on, or excel at, generating large and complex levels, which would be necessary for PCG to become more mainstream~\citep{togelius2013Procedural}.

This has led to recent research that applies PCG to more complex titles~\citep{grbic2020Evocraft,salge2018Generative,jiang2022Learning,barthet2022openended}, one notable example being Minecraft\xspace. Minecraft\xspace is a 3D, discrete voxel-based game, making it amenable to grid-based PCG methods originally developed for 2D tilemaps. It is also more complex than these aforementioned games, and thus provides a useful domain to bridge the gap between traditional and modern games.

There are still numerous problems with current methods, however. In particular, some methods rely on significant hand-designed effort~\citep{salge2021Settlement}. These approaches carry many disadvantages, primarily cost~\citep{hendrikx2013Procedural} and being hard to generalise to new games~\citep{khalifa2020Pcgrl}. Other approaches generate levels that lack some functional requirements, such as being playable~\citep{grbic2020Evocraft}, which is a core part of PCG. Lastly, some methods generate relatively small and simple structures~\citep{jiang2022Learning,barthet2022openended}, limiting their applicability to complex games.


Our work aims to address these limitations by proposing an approach that breaks the problem of generating a complex and large-scale level down into smaller, but manageable pieces. For instance, it is simple to generate an abstract, high-level layout of a town (by considering houses, roads and gardens as the individual building blocks) while ensuring all houses are reachable via roads. Similarly, it is relatively straightforward to learn to generate a single house, garden, or farm. Composed together, however, these simple generators can easily generate a fully-fledged, and functional, town.

In particular, we develop a compositional approach to level generation, where multiple generators are hierarchically composed to generate large and complex structures. This approach is inspired by \textit{hierarchical reinforcement learning}~\citep{al2015Hierarchical,pateria2021Hierarchical}, where a complex task is broken down into smaller components and an agent learns how to optimally perform each subcomponent. We break the construction of a high-level structure (e.g. a town) into simple subtasks (e.g. houses, gardens, etc.) and train agents to generate these individual components.

This approach has three main benefits: (1) the level of control a user has over each component, being able to change each component individually (at different levels of abstraction) without modifying the others; (2) the combinatorial explosion in the number of final structures that can be generated~\citep{tasse2020Boolean,tasse2022Generalisation}; and (3) being able to specify the desired structure in an interpretable way by referencing subcomponents, allowing easier generation of complex levels compared to normal, non-hierarchical PCG methods.

Our approach builds on a recent method, PCGNN\xspace~\citep{beukman2022Procedural}, which uses neuroevolution and novelty search to evolve diverse level generators. We extend this method by hierarchically composing these generators to create large and complex structures.

We demonstrate that using composition makes it significantly easier to generate complex structures compared to using a single, non-compositional generator, an effect which becomes more pronounced as we increase the complexity of the low-level structures. Furthermore, we can automatically generate large-scale towns and cities by specifying a few, simple objectives. We finally provide a qualitative showcase of the generated levels.\footnote{Source code is publicly released at \url{https://github.com/Michael-Beukman/MCHAMR}}


\section{Background}
\label{chapter:background}
\label{sec:bg}

\subsection{Hierarchical and Compositional Reinforcement Learning}
Reinforcement learning (RL)~\citep{rlbookold} is a field focused on solving sequential decision-making problems, where an agent interacts with an environment (e.g. by placing specific tiles in a tilemap level) and receives a scalar reward signal indicating how good an action was (e.g. obtaining a high reward when removing a tile that creates a path in a maze~\citep{khalifa2020Pcgrl}).

Generally, however, \textit{long-horizon tasks}, i.e. those that require a long sequence of actions to solve~\citep{pateria2021Hierarchical} (e.g. generating a video game level by sequentially placing a large number of tiles), are often quite challenging~\citep{bellemare2016Unifying,nachum2019Does,pathak2017Curiositydriven} for the normal RL paradigm. Hierarchical RL (HRL) has emerged as a solution to this, where a long and complex task is decomposed into smaller and simpler individual tasks~\citep{pateria2021Hierarchical}. Each subtask can itself be solved using an RL policy, and the top-level policy simply chooses which subtask to perform~\citep{hengst2010Hierarchical,hengst2012Hierarchical}.

Relatedly, there has recently been more interest in \textit{compositional RL}, where agents, having learned some low-level skills, can compose them together to perform more complex behaviours~\citep{niekerk2019Composing,tasse2020Boolean,tasse2022Generalisation}. This has the additional benefit of allowing tasks to be specified in an understandable way, by referencing the lower-level skills.
\subsection{Evolutionary Algorithms}
Many PCG methods~\citep{togelius2011Searchbased,ferreira2014Multipopulation,earle2021Illuminating} use some aspects of evolutionary computing, which is a subclass of optimisation algorithms that attempt to mimic natural selection, as found in the natural world~\citep{goldberg1989genetic}.

Genetic algorithms usually consist of a collection (or population) of individuals. Each of these individuals possesses a \textit{genome}, which describes the individual (e.g. an integer vector representing the height of a platform~\citep{ferreira2014Multipopulation}). These genomes are mapped to \textit{phenotypes}, which can be thought of as the result or instantiation of this individual in the actual problem domain (e.g. a level generated using the given height information).

To obtain the next generation, the high-performing individuals (according to the \textit{fitness function}) in the current population are combined using crossover -- combining multiple parents to form a single child. These children are also mutated slightly to form the new population.
\subsubsection{NeuroEvolution of Augmenting Topologies (NEAT)}
NEAT~\citep{neat} is a genetic algorithm that evolves the weights and structure of neural networks. Generally, NEAT starts with a simple neural network with no hidden layers. Through the process of crossover and mutation, however, these networks can become more complex, thereby becoming fitter, and thus solving the problem better.
\subsubsection{Novelty Search}
Novelty search~\citep{novelty_search} uses a different idea of fitness compared to traditional genetic algorithms. Instead of rewarding individuals based on their objective performance, this algorithm instead rewards individuals for being novel compared to the rest of the population. This results in improved exploration and can help to solve hard or deceptive\footnote{Deceptive problems are those where directly maximising fitness can lead to being stuck in local minima~\citep{whitley1991Fundamental,novelty_search}.} problems.
\subsection{PCGNN\xspace}
PCGNN\xspace~\citep{beukman2022Procedural} is a method that uses NEAT~\citep{neat} and novelty search~\citep{novelty_search} to evolve level generators. The main goal of this work is to learn reusable level generators that could quickly generate multiple different levels, as opposed to searching for single levels each time one is desired.  Each individual in this population is a neural network, which can be used to generate multiple levels. To obtain a level from a network, an initial, random, tilemap is created. Then, for each tile, the neural network receives as input the surrounding tiles and outputs the current center tile. This process is repeated, sequentially, for each tile. PCGNN\xspace generally uses multiple fitness functions: (1) the normal novelty search objective, to incentivise exploration; (2) intra-generator novelty, which incentivises one neural network to generate multiple diverse levels; and (3) one or more \textit{feasibility} fitness functions, which describe the feasibility criteria for the specific game (e.g. solvability in a maze). PCGNN\xspace generates levels with comparable metrics to baselines, while being significantly faster. Additionally, the generators can generalise to unseen level sizes, allowing the generation of arbitrarily-sized levels without retraining.

\section{Related Work}

\subsection{General PCG}
While there are numerous approaches to PCG, many of them focus on generating simple levels, and are therefore difficult to generalise to more complex settings, such as modern games. Many methods use some form of search~\citep{togelius2011Searchbased}, often evolutionary-based algorithms, to maximise a specific objective function~\citep{beukman2022Procedural,earle2021Illuminating,ferreira2014Multipopulation,barthet2022openended}. However, it is challenging to design an objective function that is optimisable while also incentivising the generation of complex structures~\citep{togelius2013Procedural}. Other methods rely on a dataset of existing content (which may not be available for many games), applying machine learning to generate novel content~\citep{summerville2017Procedural,liu2021Deep,shu2021Experiencedriven}. Finally, the recent paradigm of Procedural Content Generation via Reinforcement Learning (PCGRL) generates levels using RL, training an agent policy to sequentially place tiles to maximise some reward~\citep{khalifa2020Pcgrl,earle2021Learning,jiang2022Learning}. Similarly to search-based methods, however, designing a monolithic reward function that leads to complex and functional levels can also be challenging.
\subsubsection{Generating Complex Levels}

While many approaches focus on simple games, such as \textit{Super Mario Bros}, \textit{The Legend of Zelda}, etc., there is research that rather aims to generate more complex content. 
One notable example of this is the \textit{Generative Design in Minecraft (GDMC)} competition, where the task is to generate an entire settlement in Minecraft\xspace~\citep{salge2018Generative, salge2019Generative, salge2021Settlement}. This competition aims to improve the generation of content while taking the external environment into account, as well as generating holistic and more complex constructions (i.e. an entire, coherent town instead of just a single building). Many submissions to this competition, however, focused on more hand-designed, rule-based~\citep{barthet2022openended} methods specifically designed for settlement generation, which may require much effort to generalise to other scenarios.
This has led to work that focuses on automatically generating lower-level structures one would find in a settlement, such as buildings. For instance, \citet{green2019Organic} leverage constrained growth algorithms to generate useful and natural floor plans for buildings in Minecraft\xspace. They employ a modular approach, generating the building separately from the floor plan, which can be useful when customisation is desired. \citet{barthet2022openended} use a recent method, DeLeNoX~\citep{liapis2021Transforming}, to evolve diverse building generators for Minecraft\xspace using an autoencoder-based novelty score. Here the focus was on generating creative buildings, as opposed to complex and functional structures such as settlements. Additionally, these methods do not focus on how to combine these generated buildings into an actual settlement.
\subsubsection{Adversarial and Open-Ended Approaches}
Recently, PCG has also been used to generate environments on which machine learning agents can be trained. Training on a large collection of diverse and complex levels often has the benefit of making agents more robust and generalisable~\citep{justesen2018illuminating,cobbe2019quantifying}.

In some cases, level generators and game-playing agents can work in tandem, where the generator generates levels that challenge the agent. As the agent improves, the level generator must also improve to generate more complex and harder levels. Examples of this include work by \citet{gisslen2021Adversarial}, where two RL agents (one generator and one player) compete against each other; and \citet{bontrager2020Learning}, where a game-playing agent provides the learning signal to the level generator, enabling them to generate levels without explicit domain knowledge.

Some {Open-Ended Learning} methods aim to generate increasingly complex environments, alongside increasingly capable agents in an ever-evolving adversarial game. For instance, POET~\citep{wang2019POET} evolves a population of terrain layouts and agents that traverse these layouts. \citet{holder2022Evolving} instead train a single, robust agent by leveraging the automated generation of increasingly complex mazes, resulting in a natural curriculum.

Most of these methods, however, focus on simple games (such as mazes~\citep{holder2022Evolving}, 2D terrains~\citep{wang2019POET} and \textit{The Legend of Zelda}~\citep{bontrager2020Learning}), making it non-obvious how to generalise them to more complex games. 
Furthermore, the focus here is often on obtaining robust and high-performing artificial agents~\citep{holder2022Evolving} instead of generating video game levels to be played and enjoyed by humans. These two goals may have different priorities in general, and focusing on the former could result in levels that are too difficult for humans. 
These levels may also have different characteristics compared to human-designed ones~\citep{dharna2020Cogeneration}, making it more difficult to use these levels as an alternative to manually created content. 
Finally, while considering the generation process as one large problem makes sense as an initial solution, this may limit the ability of the generator to generate levels with enough complexity to challenge the agent.
Instead, having multiple generators that solve simpler problems at different levels of abstraction could enable the generation of more complex and larger-scale structures~\citep{snodgrass2015Hierarchical}.
\subsection{Compositional PCG}
Much work has also been done on combining different PCG methods~\citep{togelius2012Compositional,liapis2019Orchestrating} to generate more complex pieces of content, or even entire games. For instance, \citet{togelius2012Compositional} combine an Answer Set Programming (ASP)-based approach~\citep{smith2011Answer,smith2010Variations} with a search-based genetic algorithm that searches over variables for this ASP. This allowed them to benefit from the advantages of each method while mitigating the disadvantages.
\citet{liapis2019Orchestrating} advocate for game design orchestration, where different content generators are combined to generate full, coherent, games. One benefit of this compositional approach is allowing algorithms to specialise in (and thus excel at generating) specific pieces of content. There are still numerous open problems in this field, however, such as how to best coordinate automated generators and human designers.


Hierarchy can also be used to improve the generation speed of PCG methods. For example, \citet{smith2014Logical} find that using ASPs to generate large levels is infeasibly slow but that using hierarchy can lead to significantly faster generation. In particular, they first generate the high-level structure of a level, followed by filling in the details within the already-defined structure. However, this work focused on using hand-designed high-level structures, which may not be easily generalisable to new games~\citep{togelius2013Procedural}.

\citet{snodgrass2015Hierarchical} also leverage hierarchy to generate structures at different scales. From a set of example levels, they extract a set of high-level and low-level patterns and fit two multi-dimensional Markov chains to this data. To generate a level, then, the high-level model generates the structure while the low-level generator fills in the details. This method results in improved generation compared to a non-hierarchical approach, but requires example levels as training data, which may not always be available~\citep{summerville2017Procedural}.

\section{Hierarchically Composing Level Generators}\label{sec:method:mcc}
This section describes our approach, \textbf{M}ulti-level \textbf{C}omposition of \textbf{H}ierarchical and \textbf{A}daptive \textbf{M}odels via \textbf{R}ecursion (\texttt{MCHAMR}\xspace), in depth.

To illustrate and explain our method, we first consider a straightforward example, generating a simple medieval town. At a high level, we can represent it as a 2D tilemap, shown in \autoref{fig:mcc:we:1-town}, with some houses, a large castle, and roads. Each of these components, however, can be composed of simpler components. For instance, a house that is a single tile in \autoref{fig:mcc:we:1-town} actually consists of external walls and an empty interior, as shown in \autoref{fig:mcc:we:2-components:house}. Using our composition algorithm (described below), we can use the definitions of these smaller structures together with the high-level map to obtain a fully-fledged and detailed town, shown in \autoref{fig:mcc:we:3-composed}. Furthermore, the user has control over each of these components. Should they wish to make a similar town but would prefer to replace or modify a specific structure, this would be possible without retraining all of the generators.

To implement this method in an automatic way, we require the concept of a \textit{generator} $g \in G$. This object can generate a tilemap $W$, which is represented as a 2- or 3-dimensional matrix, where each element is taken from some tile set $T$. Each generator optionally has a \textit{tile mapping} $M_g: T \to G$, that controls which subgenerators are used when this generator places a specific tile. For instance, in the above example, the town generator's mapping specifies that the ``house'' tile actually maps to the generator shown in \autoref{fig:mcc:we:2-components:house}. In essence, we construct a tree (\autoref{fig:mcc:high_level_tree} shows the tree for the previous example), where each node is a level generator, with parent nodes generating placeholders to be filled by their children and leaf nodes producing the lowest-level, non-abstract, structures. To generate the level, we start at the root and traverse through the tree. Finally, for each generator $g$, we also have $S_g$, which indicates the size of the subtiles of this generator, i.e. the size at which the next level of the hierarchy will be generated. As an example of this, if our town generator $g_{town}$ produces an abstract tilemap of size $10\times 10$, and has an associated $S_{g_{town}}$ of $3\times 3$, then the final level will have size $30\times 30$, with each tile in the abstract map corresponding to a $3\times 3$ block.


Given a set of generators, our algorithm (detailed in \autoref{alg:mcc}) works as follows: We start with the top-level generator, which generates a tilemap (line 4 in \autoref{alg:mcc}). In standard PCG, the procedure would terminate here. We instead go further and interpret each tile as an instruction to use another generator and place its output at that location (lines 14-16). Each of these lower-level generators can also be composed of generators. If at any point, though, the generator does not contain a \textit{tile mapping} $M_g$, we end the recursion (lines 5-6). A summary of the overall generation process is illustrated in \autoref{fig:mcc:high_level_tldr}.

This approach generally creates somewhat blocky structures, due to each lower-level tile being the same size. What we do to mitigate this is to coalesce separate connected tiles of the same type into one larger, ``logical'', tile of that type (line 12). Then, at generation time, we generate one large structure of this size, instead of many small ones.
Coalescing works by finding contiguous rectangles and merging them. We intuitively start with each tile and attempt to enlarge its dimensions, one at a time until we cannot do so any longer. This additionally allows the top-level generator to effectively control the scale at which the lower-level components get generated. We coalesce connected tiles at every level of the hierarchy before we recursively call the subgenerators. \autoref{fig:mcc:coalesce} shows an example of this process.

Now, if we are interested in using \texttt{MCHAMR}\xspace with actual, automatic level generators, they must satisfy some requirements. Specifically, the generators must generate levels quickly (as we need to generate many low-level structures); be able to generate structures of arbitrary sizes (as the sizes of the final level and its components are not fixed beforehand); and generate many diverse levels (to prevent the different structures from looking identical, and thus uninteresting to a player).






\begin{figure*}[h!]
    \captionsetup{width=.35\linewidth}
    \centering
    \begin{minipage}[t]{.45\textwidth}
        \centering
        \includegraphics[width=0.85\linewidth]{composition_we_town.pdf}
        \caption{High-level view of a town}
        \label{fig:mcc:we:1-town}
    \end{minipage}
    \begin{minipage}[t]{.45\textwidth}
        \centering
        \includegraphics[width=0.85\linewidth]{composition_we_final.pdf}
        \caption{The town after composition}
        \label{fig:mcc:we:3-composed}
    \end{minipage}
\end{figure*}

\newcommand{0.24}{0.24}
\begin{figure}
    \centering
    \begin{subfigure}{0.24\linewidth}
        \includegraphics[width=1\linewidth]{images/composition_we_house.pdf}
        \caption{House}
        \label{fig:mcc:we:2-components:house}
    \end{subfigure}
    \begin{subfigure}{0.24\linewidth}
        \includegraphics[width=1\linewidth]{composition_we_road.pdf}
        \caption{Road}
    \end{subfigure}
    \begin{subfigure}{0.24\linewidth}
        \includegraphics[width=1\linewidth]{composition_we_castle.pdf}
        \caption{Castle}
    \end{subfigure}
    \begin{subfigure}{0.24\linewidth}
        \includegraphics[width=1\linewidth]{composition_we_farm.pdf}
        \caption{Farm}
    \end{subfigure}
    \caption{Individual components. Note that the castle is composed of houses, whereas the other components consist only of low-level tiles.}
    \label{fig:mcc:we:2-components}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=1\linewidth]{model_tree.pdf}
    \caption{An example tree structure, representing the generators involved in generating \autoref{fig:mcc:we:3-composed}.}
    \label{fig:mcc:high_level_tree}
\end{figure}




\begin{figure*}
    \centering
    \includegraphics[width=1\linewidth]{mc/model_new.pdf}
    \caption{A high-level illustration of \texttt{MCHAMR}\xspace. We first start with a top-level generator, which generates an abstract tilemap. Each tile in this map is an instruction to use a lower-level generator and to place its output at that location. Each of these generators can be further composed of other generators. In the end, we obtain a fully-fledged level.}
    \label{fig:mcc:high_level_tldr}
\end{figure*}






\begin{figure}
    \centering
    \begin{subfigure}{0.45\linewidth}
        \includegraphics[width=1\linewidth]{images/coalesce_True.pdf}
        \caption{}
    \end{subfigure}
    \begin{subfigure}{0.45\linewidth}
        \includegraphics[width=1\linewidth]{images/coalesce_False.pdf}
        \caption{}
    \end{subfigure}
    \caption{This plot shows a generated level in (a), and the result after coalescing connected tiles in (b). Cyan indicates those tiles that were coalesced into one larger tile.}
    \label{fig:mcc:coalesce}
\end{figure}



One specific method that is particularly well-suited to act as the low-level generator is PCGNN\xspace, which satisfies all of our requirements. Additional benefits include that the training time is generally quite low~\citep{beukman2022Procedural} and, while we can include detailed domain knowledge, this is not required.

\begin{algorithm}
    \caption{\texttt{MCHAMR}\xspace}\label{alg:mcc}
    \begin{algorithmic}[1]
    \State
        \Comment{We call this initially with the top-level generator, as well as the desired size of the final level.}
        \Procedure{Generate}{$g \in G, \text{size} \in \mathbb{Z}^2 \cup \mathbb{Z}^3$}
          \State \Comment{Generate a map, possibly abstract, using $g$}
          \State map = \Call{MakeMap}{g, $size / S_g$}
          \If {$M_g$ is null}
                \State return map \Comment{Return map if this is the lowest level}
          \EndIf
          \State \Comment{Otherwise recurse to the next level in the hierarchy}
          \State \Comment{Create an empty map of sufficient size}
          \State FilledOutMap = Empty(size)
          \State \Comment{For each position, tile and coalesced size in the map}
          \ForAll {$pos, tile, {S}_{c} \in \Call{Coalesce}{\text{map}}$} 
            \State   \Comment{Recursively generate using the appropriate size}
            \State $\text{NewMap} = \Call{Generate}{M_g(tile), \text{size}=S_g \times {S}_{c}}$
            \State   \Comment{Place this generated map at the correct location}
            \State $\text{FilledOutMap}[pos \times S_g:(pos + {S_{c}}) \times S_g] = \text{NewMap}$
          \EndFor
          \State \Return FilledOutMap
        \EndProcedure
    \end{algorithmic}
\end{algorithm}

\subsection{Using PCGNN\xspace in \texttt{MCHAMR}\xspace}\label{sec:method:pcgnn_plus_mcc}
Having presented the workings of \texttt{MCHAMR}\xspace in \autoref{sec:method:mcc}, we next describe a concrete approach that uses PCGNN\xspace to train the low-level generators.

Firstly, we train each component separately, using some objective function that describes a feasible individual. Once each of these generators has been trained, we can compose them together, needing only to specify which tiles map to which generators. This has multiple benefits; for instance, we can train and design multiple generators in parallel, obtaining a repertoire of components that can be composed later on. The approach is also modular, so we could leverage the same generator for different purposes (e.g. a house could be used in many different scenarios). 

Additionally, we can also train generators faster in this factored way compared to training one monolithic generator. 
Say we want to build an $n \times n$ level, consisting of some high-level structure (e.g. an abstract map consisting of single-tile houses and roads) and low-level components (e.g. a generated house).
Let us compose the $n\times n$ level out of a high-level structure of size $\sqrt{n} \times \sqrt{n}$ and low-level components of size $\sqrt{n} \times \sqrt{n}$.
Now, during training, the flat method must generate levels of size $n \times n$. For the composed method, however, we have two separate training procedures, each generating levels of size $\sqrt{n} \times \sqrt{n}$. 
Thus, the overall time complexity\footnote{We omit factors such as the population size and the number of generations as they are kept constant for both approaches.} of training the flat method would be $O((n \times n)) = O(n^2)$, whereas the composed method would be $O(2(\sqrt{n} \times \sqrt{n})) = O(n)$.
For large levels, this speedup in training time can be quite impactful. During inference, however, each method generates the same amount of tiles.


Furthermore, as there is no dependence between different level generators during training, we can redefine the meaning of a specific generator. For instance, a town generates a tilemap containing houses, roads and gardens. We could also change the interpretation of the individual tiles, resulting in this generator generating cities, consisting of towns, highways and parks. This effectively constructs multiple different tree structures, with the same generator as the root and different child generators. In general, this approach would work as long as the same feasibility criteria are valid in both cases.

Finally, while PCGNN\xspace is particularly suited to be used as the low-level generator in \texttt{MCHAMR}\xspace, this is by no means a requirement. We could use other suitable level generators (e.g. approaches such as \citep{earle2021Illuminating,shu2021Experiencedriven,earle2021Learning}), hand-designed components, or a combination of all of these. This provides a significant amount of control and flexibility, which is useful in designing a game~\citep{lai2020Towards}.

\section{Experiments and Results}
\label{chap:experiments}
\subsection{Experimental Setup}
\subsubsection{Domains}
We consider two benchmark domains to demonstrate the capabilities of \texttt{MCHAMR}\xspace. The first, where we quantitatively demonstrate the effects of composition, is a simple 2D game, where we build towns. The second domain is Minecraft\xspace, to demonstrate that our approach can work equally well in 3 dimensions, and to show some of the complex structures that were generated.
\subsubsection{Experiments}
We perform three main experiments in this section. In \autoref{sec:exp:hierarchical}, we first illustrate the benefits of having a hierarchical generation approach, as the complexity of the lower-level structures increases. The second experiment, in \autoref{sec:exp:comp_vs_flat}, directly compares our hierarchical method against a non-compositional, flat approach. Finally, in \autoref{sec:showcase}, we qualitatively evaluate our method by generating complex towns and cities in Minecraft\xspace. More information about the different experiments, including hyperparameters and fitness functions, is listed in the \refappendix{sec:appdx}.
\subsection{Hierarchical Generation}\label{sec:exp:hierarchical}
In this experiment, we show the benefits of using composition compared to a non-compositional approach. To do this, we consider a simple town level, with houses, gardens and roads. To simulate our compositional approach, where each high-level structure is simple to build, we generate a normal $10\times 10$ level using PCGNN\xspace. To simulate the effect of a non-compositional method, where each component (e.g. a house) is not trivial, we generate $20\times 20$ levels, where each non-overlapping $2 \times 2$ window is considered as a single tile. If the window consists of 4 matching tiles, a single tile of this type is placed. Otherwise, the tile is replaced with a ``default'' tile -- gardens in this case. We repeat this with window sizes of $3\times 3$, $4\times 4$, $5\times 5$ and $10\times 10$, to evaluate how the complexity of the low-level structures influences our final performance. 
The window size effectively controls how many coordinated actions are required to build a low-level structure. For example, $1\times 1$ represents our composed approach where a house can be built using one top-level action, whereas the larger window sizes represent non-compositional approaches, where building a house requires multiple coordinated actions.

This experimental setup has the added benefit of being comparable, as each method uses the same fitness functions, and the only difference is in the size of the generated structures.

The fitness functions used here are (1) reachability, where houses must be reachable via roads and (2) a probability fitness, incentivising a roughly equal number of house, garden and road tiles. See the \refappendix{sec:appdx} for more details.

We compare the fitness of each method over time in \autoref{fig:mcc:results:composition}, where we demonstrate that using the compositional approach results in a higher final fitness value compared to the other, non-compositional methods. The performance decreases significantly when we have a much larger, $10\times 10$ window size (corresponding to an overall level size of $100\times 100$), where each house requires $100$ coordinated actions to build. Even this is still significantly fewer actions than a real Minecraft\xspace house. Overall, this experiment shows that simplifying the task of the top-level generator by abstracting away the details of the lower-level components is beneficial.
\begin{figure}
  \centering
  \includegraphics[width=1\linewidth]{3551-hl_all.pdf}
  \caption{A plot showing the maximum fitness value in the population (i.e., how ``correct'' the generated levels are) over time for different window sizes (corresponding to the complexity of the lower-level structures). The mean across 10 seeds is shown, with the standard deviation shaded.}
  \label{fig:mcc:results:composition}
\end{figure}


\subsection{Composition vs Flat Generation}\label{sec:exp:comp_vs_flat}
Our next experiment directly compares \texttt{MCHAMR}\xspace against a non-compositional baseline that generates the entire level using PCGNN\xspace. In particular, we are again interested in building towns, this time attempting to replicate a specific, randomly generated, town layout.

For the compositional approach, we learn two separate generators, to generate houses and towns respectively, while the non-compositional method generates the entire level at once. In particular, we have a $25\times 25$ level, consisting of a collection of $5\times 5$ houses. The high-level town structure also has a size of $5\times 5$. The fitness functions used did not include novelty, and measured the average overlap between the desired level and the generated one. The desired levels use one set layout for houses, where walls surround empty space, and a randomly generated town layout. We generate 20 of these random town layouts, train a separate generator on each of them, and average the fitness values obtained. 

The fitness results over time are shown in \autoref{fig:mcc:results:composed_vs_not}, and it is clear that PCGNN\xspace's fitness cannot touch that of \texttt{MCHAMR}\xspace. Our composed method achieves significantly higher fitness, and generates much more accurate levels than the flat method. See the \refappendix{sec:appendix} for examples of the generated levels.

\begin{figure}
  \centering
  \includegraphics[width=1\linewidth]{1/composed_more_structures_better.pdf}
  \caption{Illustrating the fitness over time for \texttt{MCHAMR}\xspace (consisting of the composed town and house generators) compared to the flat, non-compositional method, corresponding to vanilla PCGNN\xspace. We run this over 20 different random layouts, and the mean is shown, with the standard deviation shaded. Each run was averaged over 10 random seeds.}
  \label{fig:mcc:results:composed_vs_not}
\end{figure}

\subsection{Showcase}\label{sec:showcase}
Finally, we evaluate our method on Minecraft\xspace, and show a few qualitative examples of interesting and complex structures that were generated by \texttt{MCHAMR}\xspace. We use the Evocraft~\citep{grbic2020Evocraft} library to place and visualise the generated structures in game.\footnote{\url{https://github.com/real-itu/Evocraft-py}}


Here, we generate 3D settlements, where each town consists of houses, gardens and roads. Additionally, to demonstrate the notion of reusing a single generator, we further generate \textit{cities} using the town generator, by redefining houses to be towns. Concretely, we use the following generators, with more information about the fitness functions used in the \refappendix{sec:appendix}:
\begin{LaTeXdescription}
  \item[Town] A high level 2D tilemap of a town.
  \item[House] A 3D house, consisting of a roof, walls and doors.
  \item[Garden] A 2D tilemap with flowers, grass, ponds, and trees.
\end{LaTeXdescription}

Some generated cities are shown in \autoref{fig:mcc:showcase:subfigs}. In \autoref{fig:mcc:showcase:town1} we have a generated city, consisting of lower-level towns, and gardens. \autoref{fig:mcc:showcase:town2} shows the same generation, without using the abovementioned coalescing technique. Here all the towns are the same size, and are not joined together.

In \autoref{fig:mcc:showcase:town3}, we use the same generator for the city and the town. We also show a first-person view of a specific section to illustrate the detail in each part of the generated settlement.


\autoref{fig:mcc:showcase:annotate} shows an annotated example of a generated city. We have the large, high-level structure of the city being generated by the top-level generator. Then, each tile that it places is transformed into a town by the town generator. We see that the coalescing here results in longer towns and the towns themselves, while not identical, contain similar repeating structures.

Finally, we note that the structures we generate are very large in the scale of standard PCG. For instance, levels in \textit{Super Mario Bros.} have size $114 \times 14$. Much of the work in Minecraft\xspace also generates relatively small structures, such as $7\times 7\times 7$ mazes~\citep{jiang2022Learning} or $20\times 20 \times 20$ lattices~\citep{barthet2022openended}. The composed towns we generate can easily be $200\times 5\times 200$ or larger, totalling more than $100\text{ }000$ blocks. Despite this massive scale, we are still able to generate coherent structures and sensible towns. Large settlements are also generated for the GDMC competition~\citep{salge2018Generative}, but unlike our approach, these generations often require significant hand-designing effort~\citep{salge2021Settlement}.


\section{Discussion and Future Work}
The approach introduced here, \texttt{MCHAMR}\xspace, follows a recent trend of emphasising the importance of more complex and open-ended creations in PCG~\citep{salge2018Generative,grbic2020Evocraft,barthet2022openended}.
Our method has numerous applications; in particular, it could be (partially) leveraged for the generation of large open-world games, e.g. Massively Multiplayer Online (MMO) games. This would allow designers freedom in specifying certain, reusable low-level components that can be composed in numerous different ways, thereby enticing players towards exploration.

Furthermore, in contrast to many other methods, we abstract away details relevant to a particular game by having separate high-level and low-level generators. Separating these responsibilities could enable the abstract generators to be used in a variety of games, needing only to swap out the low-level generators for each specific game.

Relatedly, since we independently train the generators, it would be possible to have a community-driven database of interesting and useful generators, which can then be seamlessly composed together to generate novel artifacts. This could be similar to \textit{Picbreeder}~\citep{secretan2011Picbreeder},\footnote{\url{https://nbenko1.github.io/}} where users collectively contribute to the generation of many novel and interesting images.

Our method is just a first step in this direction, and there are numerous avenues for future work. For instance, we could use a combination of different PCG methods as low-level generators~\citep{khalifa2020Pcgrl,earle2021Illuminating,jiang2022Learning}, instead of just PCGNN\xspace. Additionally, we would like to explore the use of quality-diversity algorithms~\citep{mouret2015Illuminating} to obtain a collection of diverse and high-performing low-level generators, and use these interchangeably to obtain more diverse content. Furthermore, the generated levels do not have large vertical components, due to using a 2D level for the root generator. Expanding upon this by adding more verticality (e.g. mountains, hills, multi-story houses, etc.) to the generators would also be valuable. Finally, we would like to explore more open-ended avenues, where the structures we generate increase in complexity over time, as opposed to reaching an endpoint when the feasibility fitness is maximised~\citep{grbic2020Evocraft,barthet2022openended}. This could, for example, be achieved by dynamically learning to compose different generators, or using learned objective functions.

\section{Conclusion}
We introduce \texttt{MCHAMR}\xspace, a compositional approach to procedural level generation, which leverages multiple simple generators to generate large and complex structures. Our approach has multiple benefits, such as (1) being configurable, allowing the designer to choose which low-level generators are used; (2) it being straightforward to combine different generators, without needing to specify one monolithic fitness function; and (3) by composing multiple levels of generators, each individual generator's task is much simpler, which allows for potentially less training or smaller (and thus faster) models. We demonstrate that our approach improves generation compared to a non-compositional approach, and that it is able to generate large-scale and complex settlements in Minecraft\xspace. 
Ultimately, we hope that this work is the first step towards more complex and large-scale generations, which will be necessary for the adoption of PCG in the modern gaming industry.


\begin{figure*}[h!]
  \centering
  \begin{subfigure}{0.45\linewidth}
    \includegraphics[width=1\linewidth]{mc/show1.jpg}
      \caption{}
      \label{fig:mcc:showcase:town1}
  \end{subfigure}
  \begin{subfigure}{0.45\linewidth}
      \includegraphics[width=1\linewidth]{mc/show2.jpg}
      \caption{}
      \label{fig:mcc:showcase:town2}
  \end{subfigure}



  \begin{subfigure}{0.9\linewidth}
    \includegraphics[width=1\linewidth]{mc/mincraft_picture_zoom_figure.jpg}
    \caption{}
    \label{fig:mcc:showcase:town3}
  \end{subfigure}
  \caption{Demonstrating some generated cities. In (a) and (b) we use the same generators, but (a) uses coalescing whereas (b) does not. In (c) we show the result when the city generator and town generator are the same, as well as a zoomed in portion of an alleyway.}
  \label{fig:mcc:showcase:subfigs}
\end{figure*}

\begin{figure*}[h!]
  \centering
  \includegraphics[width=0.9\linewidth]{mc/minecraft_small.jpg}
  \caption{An annotated example of a generator similar to that in \autoref{fig:mcc:showcase:town1}}
  \label{fig:mcc:showcase:annotate}
\end{figure*}


