\section{Introduction}

Recent research on the lottery ticket hypothesis \citep[LTH,][]{frankle_2019, frankle_2019b} in deep learning has demonstrated the existence of very sparse neural networks that train to performance levels comparable to those of their dense counterparts.
These results challenge the role of overparameterization in supervised learning and provide a new perspective on the emergence of stable learning dynamics \citep{ frankle_2020b, frankle_2020a}.
Recently these results have been extended to various domains beyond supervised image classification. These include self-supervised learning \citep{chen_2020}, natural language processing \citep{yu_2019, chen_2020b} and semantic segmentation \citep{girish_2020}.
But how does the lottery ticket ticket phenomenon transfer to reinforcement learning agents?
One key challenge may be the inherent non-stationarity of the optimization problem in deep reinforcement learning (DRL): The data-generation process is not static, but depends on the changing state of the neural network.
Furthermore, a weight may serve different roles at different stages of learning (e.g.\ during exploration and exploitation). It is not obvious how a simple weight magnitude-based pruning heuristic acting on a well-performing policy shapes the learning process of the agent.
In this work, we therefore investigate winning tickets in reinforcement learning and their underlying contributing factors. We compare supervised behavioral cloning with DRL, putting a special emphasis on the resulting input representations used for prediction and control. 
Thereby, we connect the statistical perspective of sparse structure discovery \citep[e.g.][]{hastie_2019} with the iterative magnitude pruning \citep[IMP,][]{han_2015} procedure in the context of Markov decision processes (MDPs). The contributions of this work are summarized as follows:

\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{figures/figure_1_new.png}%
\caption{Representation compression, disentangling baselines and lottery ticket effects in continuous control tasks. 
\textbf{Left.} IMP successively prunes task-irrelevant outer rim pixels in a MNIST digit-classification task. A similar representation compression emerges for an IMP-masked agent solving a visual navigation task in DRL. The channel that encodes the patrolling enemy is pruned up to the point where only potential enemy locations are considered.
\textbf{Middle.} To disentangle the contributions of mask, initialization and layer-wise pruning ratio to the winning lottery ticket (corresponding to condition \textit{mask/weights}), we compare three baselines: After each IMP iteration, we permute either only the remaining initial weights (\textit{mask/permuted}) or also the sparsity mask (\textit{permuted/permuted}). The third baseline is created by randomly sampling a sparse mask and random re-initialization of the weights (\textit{random/re-init}). It no longer preserves the layer-wise pruning ratio of the lottery ticket.
\textbf{Right.} Average normalized performance of policies at different sparsity levels and for different baseline conditions as well as four PyBullet \citep{benelot2018} environments. For both behavioral cloning and PPO agents most of the ticket effect can be attributed to the IMP-derived mask as compared to the weight initialization. Furthermore, agents trained with explicit supervision can be pruned to higher sparsity levels before performance deteriorates.}
\label{fig1:conceptual}
\end{figure}

\begin{enumerate}
    \item We show that winning tickets exist in both high-dimensional visual and control tasks (continuous/discrete). A positive lottery ticket effect is robustly observed for both off-policy DRL algorithms, including Deep-Q-Networks \citep[DQN,][]{mnih_2015} and on-policy policy-gradient methods \citep[PPO,][]{schulman2015trust, schulman_2017}, providing evidence that the lottery ticket effect is a universal phenomenon across optimization formulations in DRL.
    \item By comparing reinforcement learning to supervised behavioral cloning (BC), we show that networks trained with explicit supervision can be pruned to higher sparsity levels before performance starts to degrade, indicating that the RL-specific distributional shift requires a larger amount of parameters in order to be addressed (section \ref{sec3a:distillrl}).
   
    \item By introducing a set of lottery ticket baselines (section \ref{sec3:baselines}; figure \ref{fig1:conceptual}, middle column), we disentangle the contributions of the mask, weight initialization and layer-wise pruning ratio. We demonstrate that the mask explains most of the ticket effect in behavioral cloning and reinforcement learning for MLP-based agents, whereas the associated weight initialization is less important (section \ref{sec3a:distillation} and \ref{sec3b:deeprl}). For CNN-based agents the weight initialization contributes more.

	\item By visualizing the sparsified weights for each layer, we find that early network layers are pruned more. Entire input dimensions can be rendered invisible to MLP-based agents by the pruning procedure. By this mechanism, IMP compresses the input representation of the MDP (e.g. figure \ref{fig1:conceptual}, left column, bottom row) and reveals a minimal task representation for the underlying control problems (section \ref{sec4:representations}).
	\item The IMP input layer mask not only eliminates obviously redundant dimensions, but also identifies complex relationships between input features and the task of the agent (section \ref{sec4a:repr_vision}), e.g. the proximity of an enemy or the speed of an approaching object. We show that the input masking can be transferred to train dense agents with sparse inputs at lower costs.
	\item We show that the weight initialization scheme is important for discovering minimal representations.
	Depending on the input size of different layers of the network, global magnitude-based pruning can introduce a strong layer-specific pruning bias. We compare initializations and show that a suitable initialization scheme enables the removal of task-irrelevant dimensions (section \ref{sec4b:repr_control}).
\end{enumerate}

\section{Background and Related Work}
\label{sec2:background}

\textbf{Iterative Magnitude Pruning.} We use the iterative pruning procedure outlined in \citet{frankle_2019} to identify winning tickets. We train DRL agents for a previously calibrated number of transitions and track the best performing network checkpoint throughout. Performance is measured by the average return on a set of evaluation episodes. Afterwards, we prune $20\%$ of the weights with smallest magnitude globally (across all layers). The remaining weights are reset to their initial values and we iterate this procedure (train $\to$ prune $\to$ reset).\footnote{In supervised learning, the pruning mask is often constructed based on an early stopping criterion and the final network. We instead track the best performing agent.
Thereby, we reduce noise introduced by unstable learning dynamics and exploit that the agent is trained and evaluated on the same environment.
We found that late rewinding to a later checkpoint \citep{frankle_2019b} is not necessary for obtaining tickets (SI figure \ref{figsupp:regularization}).}
The \textit{lottery ticket effect} refers to the performance gap between the sparse network obtained via IMP and a randomly initialized network with sparsity-matched random pruning mask.

\textbf{Lottery Tickets in Deep Reinforcement Learning.}
\citet{yu_2019} previously demonstrated the existence of tickets in DRL that outperform parameter-matched random initializations.
They obtained tickets for a distributed on-policy actor-critic agent on a subset of environments in the ALE benchmark \citep{bellemare_2013} as well as a set of discrete control tasks.
While they provide empirical evidence for the existence of lottery tickets in DRL, they did not investigate the underlying mechanisms. Here, we aim to unravel these mechanisms. To this end, we focus on a diverse set of environments and provide a detailed comparison between supervised behavioral cloning and on-/off-policy Deep RL with a set of carefully designed ticket baselines.
We analyze the resulting masked representations that the agent learns to act upon and the impact of specific weight initializations on the resulting sparse networks.

\textbf{Lottery Tickets with Non-Stationary Data Distributions.} \citet{desai_2019} investigated whether trained lottery tickets overfit the training data distribution under which they were obtained. Using transfer learning tasks on natural language data, they showed that lottery tickets provide general inductive biases. Similar ticket transfer results were reported by \citet{morcos_2019} and \citet{mehta_2019} in the context of optimizers and vision datasets. Unlike our work, these studies do not investigate within-training covariate shift, but instead focus on transferring ticket initializations after a full IMP run.
\citet{chen_2021}, on the other hand, investigate the ticket phenomenon in the context of lifelong learning and class-incremental image classification. They propose new pruning strategies to overcome the sequential nature of tasks and need for increased model capacity. Compared to the DRL setting, the covariate shift is here determined by the curriculum schedule of tasks and not the exploration behaviour of the network-parameterized agent.

\textbf{Deep Reinforcement Learning Background.} In our off-policy DRL experiments, we train Deep-Q-Networks \citep[DQN,][]{mnih_2015} with double Q-learning loss \citep{van_hasselt_2016} and prioritized experience replay \citep{schaul_2015}.
As a representative on-policy algorithm, we chose Proximal Policy Optimization \citep[PPO,][]{schulman2015trust, schulman_2017}. 
PPO is a baseline-corrected policy gradient algorithm which uses a clipping strategy to approximate a computationally expensive trust-region optimization method.
For illustrative purposes, we train DQN agents on a visual navigation task, in which an agent has to collect coins in a grid while avoiding poison and two patrollers that are moving in restricted parts of the grid (figure \ref{fig1:conceptual}, left column, bottom row; SI \ref{SI:environments}).
We scale our results to four PyBullet \citep{benelot2018} continuous control and a subset of ALE benchmark \citep{bellemare_2013} environments. Due to computational considerations we limit each individual IMP iteration for the ATARI environments to 2.5 million frames. All other tasks were trained for a pre-calibrated generous amount of transitions. 
We focus on feedforward value estimators and policies (MLP \& CNN) and used default hyperparameters with little tuning (SI \ref{SI:hyperparameters}).

\textbf{Supervised Behavioral Cloning.}
While most supervised learning relies on a stationary data distribution provided by a static dataset, reinforcement learning agents have to acquire their training data in an action-perception loop.
Since the agent's behavioural policy is learned over time, the data distribution used in optimization undergoes covariate shift. To study how the covariate shift influences winning tickets, we mimic the supervised learning case by training agents via supervised policy distillation \citep{rusu_2015, schmitt_2018}. We roll out a pre-trained expert policy and train the student agent by minimizing the KL divergence between the student's and teacher's policies. 

\section{Disentangling Ticket Contributions in BC and Deep RL} 
\label{sec3:baselines}

\begin{wraptable}{r}{10cm}
\begin{tabular}{ |p{2.8cm}||p{1.5cm}|p{1.5cm}| p{2cm}| }
 \hline \hline
 \multicolumn{4}{|c|}{Examined Sparsity-Generating IMP Variants} \\
 \hline
 & Retain weights & Retain \newline mask &  Retain layer \newline pruning ratio  \\
 \hline
 \textit{mask/weights}  & \cmark    & \cmark & \cmark \\
\textit{mask/permuted}  & \xmark    & \cmark & \cmark\\
\textit{permuted/permuted}    & \xmark & \xmark & \cmark \\
\textit{random/re-init} & \xmark & \xmark & \xmark\\
 \hline \hline
\end{tabular}
\caption{Proposed Baselines for Disentangling Ticket Contributions}
\label{table1:baselines}
\end{wraptable} 
There are two contributing factors to the lottery ticket effect: The IMP-identified binary mask and the preserved initialized weights that remain after pruning (\textit{mask/weights}).
We aim to disentangle the contributions by introducing a set of counterfactual baselines, which modify the original IMP procedure (figure \ref{fig1:conceptual}, middle column; table \ref{table1:baselines}).
A first baseline estimates how much of the performance of the ticket can be attributed to the initial weights, by means of a layer-specific permutation of the weights that remain after masking (\textit{mask/permuted}).
A second, weaker baseline estimates the contribution of the mask, by also permuting the layer-specific masks (\textit{permuted/permuted}). 
Finally, we consider the standard \textit{random/re-init} baseline, which samples random binary masks -- discarding layer-specific pruning ratios -- and re-initializes all weights at each IMP iteration.
Throughout the next sections we use these baselines to analyze and compare the factors that give rise to the lottery ticket effect in different control settings.

\vspace{-0.1cm}
\subsection{Comparing Winning Tickets in Supervised Behavioral Cloning and Deep RL}
\label{sec3a:distillrl}

Does the covariate shift in DRL affect the existence and nature of lottery tickets?
Weights pruned for their small magnitude at the end of the learning process might be needed at earlier stages, e.g., during exploration. If this were the case, the performance of the ticket in DRL should degrade for lower levels of sparsity than in a corresponding supervised task. To investigate this question, we turn to a behavioral cloning setting, in which a student agent is trained to imitate the stochastic policy of a pre-trained expert and apply magnitude pruning to the student network after each training run iteration. By collecting transitions based on the static behavioral policy of the expert, we avoid the need for exploration and the effect of an otherwise non-stationary data distribution.
For both the discrete action space and all continuous control tasks we find that agents trained with RL and the supervised settings start to degrade in performance at different sparsity levels (figure \ref{fig2:distillation_a}). More specifically, the maximal return of agents that were trained with reinforcement learning starts to drop at significantly earlier stages of the IMP procedure. Hence, the covariate shift inherent to the RL problem increases the minimal required size of a winning ticket in the tasks studied here: More parameters are required for an agent that has to solve the additional exploration problem.

\vspace{-0.1cm}
\subsection{Disentangling Ticket Contributions in Supervised Behavioral Cloning}
\label{sec3a:distillation}
To disentangle the contributions of the initial weights and the weight mask to the ticket effect in supervised behavioral cloning, we next compared the three baselines to the full non-randomized ticket (figure \ref{fig1:conceptual}, top, right and \ref{fig3:distillation_b}). 
For most agents, training performance does not degrade substantially when the weights are permuted but the mask is kept intact (no significant gap between red and green curves). But we observe a strong decrease in the ability to prune the policy to high sparsity levels if one additionally permutes the IMP-derived weight mask (gap between green and blue curves).
This observation holds for the cloning of expert policies in both discrete and continuous control tasks. Hence, the ticket effect can be mainly attributed to the mask rather than the weights.
Finally, the traditional \textit{random/re-init} baseline performs worse already for moderate levels of sparsity. The resulting performance gap (blue and yellow curves) indicates the contribution of the layer-wise pruning ratio to the mask effect.  
These insights emphasize the importance of strong and nuanced baselines to understand the contributions to the full lottery ticket effect.

\vspace{-0.1cm}
\subsection{Disentangling Ticket Contributions in Deep Reinforcement Learning}
\label{sec3b:deeprl}

The observation that ticket information is mostly carried by the mask rather than the initial weights carries over to the full DRL setups (figure \ref{fig1:conceptual}, bottom, right and figure \ref{fig4:learning_algos}).

\begin{figure}[H]
\centering
\includegraphics[width=\textwidth]{figures/figure_2_new.png}%
\caption{Comparing lottery tickets in DRL and supervised behavioral cloning. Networks trained with explicit supervision can be pruned to higher sparsity levels before performance starts to degrade. Results are averaged over 15 runs for both the Cart-Pole and Acrobot and 10 runs for PyBullet environments. We plot mean best performance and one standard deviation.}
\label{fig2:distillation_a}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=\textwidth]{figures/figure_3_new.png}%
\caption{Disentangling baselines for lottery tickets in supervised behavioral cloning. The gap between the ticket (\textit{mask/weights}) and weight-permuted baseline (\textit{mask/permuted}) is small, indicating a strong contribution of the mask. Results are averaged over 15 runs for both Cart-Pole and Acrobot and 10 runs for PyBullet environments. We plot mean best performance and one standard deviation.}
\label{fig3:distillation_b}
\end{figure}


When preserving the information contained in the mask, agents can be pruned to much higher levels of sparsity as compared to the case in which the mask information is distorted (permuted or resampled).
This effect holds for both PPO- and DQN-based agents and across a wide range of qualitatively different tasks.\footnote{For further experiments investigating the role of different network capacities (SI figures \ref{figsupp:network_capacity}, \ref{figsupp:network_capacity_2}), state encoding (SI figure \ref{figsupp:representations}) and regularization (SI figure \ref{figsupp:regularization}) we refer the interested reader to the SI.} 
Interestingly, ticket weights are more important for CNN-based agents (e.g., see ATARI plots). We provide further experimental evidence for this observation on four MinAtar games \citep{young_2019} in the supplementary material (SI figure \ref{figsupp:learning_algos_b}), where we compare MLP- and CNN-based agents.
Unlike the behavioral cloning experiments, for some tasks the \textit{permuted/permuted} baseline performs worse than the \textit{random/reinit} configuration. This observation only occurs for the PyBullet control tasks in which both the separate critic and actor networks were pruned. If instead, one only prunes the actor network, as was done for the PPO agents trained on Cart-Pole, Acrobot and ATARI, the \textit{permuted/permuted} baseline performs stronger (see SI figure \ref{figsupp:no_critic_prune}). This result highlights the importance of differences in IMP-derived pruning ratios between agent modules, i.e. separate actor and critic networks. 


\begin{figure}[H]
\centering
\includegraphics[width=\textwidth]{figures/figure_4_new.png}%
\caption{Tickets in on- and off-policy deep reinforcement learning.  The disentangling baselines for tickets in on-policy (PPO) and off-policy (DQN) DRL for a set of continuous control, a visual navigation task and a subset of ATARI environments reveal the consistent importance of the IMP-extracted mask. The results are averaged over 5 independent runs on the Gridworld and ATARI environments and 10 runs on the continuous control environments. We plot mean best performance and one standard deviation.}
\label{fig4:learning_algos}
\end{figure}



\section{Minimally Task-Sufficient Representations via IMP}
\label{sec4:representations}

The previous results establish the importance of the information contained in the mask for the success of winning tickets. To better understand this phenomenon and the resulting inductive biases, we analyzed the sparsified weight matrices, specifically between the input and the units in layer one. The next section sets out to answer the following questions: \textbf{1)} The derived mask can be thought of as a pair of goggles which guide the processing of state information. What do the masked MLP-based agents observe? \textbf{2)} Can we transfer the input layer mask and re-use it as an inductive bias for otherwise dense agents? \textbf{3)} How much influence do weight initialization and input dimensionality have on the discovered input layer mask?

\subsection{Minimal Task Representations in High-Dimensional Visual Tasks}
\label{sec4a:repr_vision}

In the visual navigation experiments, the pruning primarily affects the input layer (figure \ref{fig6:representation_grid}, top right column).
When visualizing the cumulative absolute weights of the input layer for an IMP-derived DQN agent, we find that IMP deletes entire dimensions from the observation vector (figure \ref{fig6:representation_grid}, left column) by removing all connections between those dimensions and the first hidden layer.
These eliminated input dimensions are not task-relevant. A fully-connected network trained on the remaining dimensions learns the task as quickly as when trained on the complete set of input dimensions (figure \ref{fig6:representation_grid}, bottom right column).
Hence, the IMP-derived mask provides a compressed representation of a high-dimensional observation space, enabling successful training even at high levels of sparsity. 

\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{figures/f6_representation_grid.png}%
\caption{IMP eliminates task-irrelevant observation dimensions for a high-dimensional visual navigation task ($o_t \in \mathbb{R}^{6 \times 10 \times 20}$). 
\textbf{Left.} Channel-/pixel-wise cumulative weight magnitudes. IMP successively prunes redundant input pixels which are not necessary to solve the navigation task. All of the pruned enemy channel pixels encode locations which the patrolling enemy cannot access. 
\textbf{Right, Top.} Pruning affects the layers and object channels differentially. The input layer is pruned most strongly, while the agent channel is pruned the least.
\textbf{Right, Bottom.} The input layer pruning mask (at moderate sparsity levels) can be used as an inductive bias for training a dense network.}
\label{fig6:representation_grid}
\end{figure}

This result extends to masked input layers of MLP-agents trained on the MinAtar environments (figure \ref{fig7:representation_minatar} and SI).
For the Freeway task we find that the IMP mask encodes the notion of velocity of moving cars. The channel encoding the slowest moving car is pruned the most, maintaining only the pixels needed to react in time.
The same principle applies to the enemy bullet channel in the SpaceInvaders task. The agent only needs to know about a potential collision at the next time step in order to avoid being hit. Therefore, IMP prunes all information about the enemy bullet that is more than one step away from the agent.
For all visualized MinAtar tasks, only the actionable row/column of the agent channel is preserved. In Breakout, additionally the bottom row of the ball and trail channel is pruned since the game terminates once the ball or trail reach it. There is no action-relevant information encoded and it can be discarded. 
In summary, we find that IMP-derived input layer mask provides a visually interpretable and physically meaningful compression of the observation space.

\begin{figure}[H]
\centering
\includegraphics[width=\textwidth]{figures/f7_representation_minatar.png}%
\caption{IMP eliminates task-irrelevant observation dimensions for a selection of MinAtar environments \citep{young_2019}. The environment depicting figures were adapted from \citet{young_2019}.
\textbf{Left.} Freeway. IMP provides an inductive bias by differentially pruning object channels with different velocities (e.g.\ car speeds). 
\textbf{Middle.} SpaceInvaders. IMP only preserves pixels of enemy objects which encode actionable proximity information (e.g. bullets being close to the agent).
\textbf{Right.} Breakout. IMP discards pixels which only change when it is too late to act (e.g.\ when the ball has left the display and the episode terminates).}
\label{fig7:representation_minatar}
\end{figure}

\subsection{Minimal Representations in Low-Dimensional Control Tasks and the Issue of Initialization Scales}
\label{sec4b:repr_control}

The continuous and discrete control tasks, on the other hand, rely on low-dimensional state representations. At first sight, they do not contain obviously uninformative input dimensions that lend themselves to be pruned.
We would expect the cumulative weight strength of each input dimension to decrease at equal speed over the course of iterative pruning.
Contrary to this initial intuition, IMP still aggressively prunes core dimensions while yielding trainable agents (figure \ref{fig7:representation_pybullet}, \ref{fig8:representation_cartpole}).

For a subset of continuous control tasks, we find that one can prune 20 and up to 50 percent of the input dimensions, while still being able to train the agent to the performance level of a dense counterpart (figure \ref{fig7:representation_pybullet}). The exact amount depends on the considered environment indicating a varying degree of observation over-specification across task formulations. For example, the Walker2D environment can train to full performance with only 11 out of 22 observation dimensions, while the Ant environment requires 22 out of 28 dimensions. We note an initial positive effect of the weight pruning on the performance of all agents indicating the effectiveness of the implied regularization.

\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{figures/figure_7_new.png}%
\caption{IMP eliminates task-irrelevant observation dimensions for a set of continuous control tasks. For all three tasks there exists a moderate sparsity level (approximately 10\% non-sparse weights) at which entire input units (columns of the first linear layer) are pruned while the agents still train to the performance level of their dense counterparts. The extent of this observation strongly depends on the considered environment indicating a varying extent of observation over-specification.}
\label{fig7:representation_pybullet}
\end{figure}

In the Cart-Pole task the cart position and its velocity are pruned (figure \ref{fig8:representation_cartpole}, top). Newton's laws are invariant with regard to the choice of the inertial frame of reference, so the input channels encoding the cart position and velocity do not provide additional information over the pole's angle and velocity.
The Acrobot task requires the agent to swing-up a two-link pendulum across a line as fast as possible. Here, IMP eliminates all sine and cosine transformations of the rotational angles of the links (figure \ref{fig8:representation_cartpole}, middle row, right column). Since the swing-up can be achieved by coordinating and increasing the angular velocities of the two links, only these two dimensions have to be preserved from pruning. 
These experiments demonstrate that IMP can yield fundamental insights into what physical information is sufficient to solve a task and leverages them to prune the input representation without impairing performance.

During our analysis we discovered that the core dimension discovery of IMP crucially depends on the initialization of the input layer weights. Popular weight initialization heuristics such as the Kaiming family \citep{he_2015} aim to preserve activation magnitudes across network layers. As a result, layers that receive high-dimensional input are initialized at lower values and are therefore prone to more aggressive pruning by IMP. This bias does not occur for Xavier initializations \citep[figure \ref{fig8:representation_cartpole}, top rows, right column]{glorot_2010}. Furthermore, we experimented with different approaches to combat this initialization bias. We find that downscaling the Kaiming initialization of the input layer by a factor of ten leads to a robust discovery after few IMP iterations (figure \ref{fig8:representation_cartpole}, top rows, middle column). While smaller rescaling factors lead to a slower separation of dimensions, larger factors result in overly aggressive pruning of the input layer. In summary, we have shown that the input layer initialization can shape the resulting lottery ticket mask and provide a first initialization heuristic promoting efficient minimal task representation discovery.


\begin{figure}
\centering
\includegraphics[width=\textwidth]{figures/f8_representation_cartpole.png}%
\caption{IMP eliminates task-irrelevant observation dimensions for low-dimensional control tasks. Suitable initialization heuristics can promote and accelerate the discovery of minimal representations.
\textbf{Top.} IMP identifies that the Cart-Pole task can be solved with only two dimensions: The pole angle and angular velocity.
\textbf{Middle.} IMP identifies that the Acrobot task can be solved with only two dimensions: The angular velocities of the two pendulum links.
\textbf{Bottom.} Agents can successfully be trained on the subset of IMP-derived dimensions alone. We plot mean performance and one standard deviation across 10 independent runs.}
\label{fig8:representation_cartpole}
\end{figure}

\section{Conclusion}
\label{section5:conclusion}

\textbf{Summary.} This work has investigated the mechanisms underlying the lottery ticket effect in behavioral cloning and DRL. 
We provide evidence that agents which are trained on a stationary data distribution and do not face the exploration problem, can be pruned to higher levels of sparsity while successfully performing their task.
For both setups we find that the effect can be attributed to the identified mask rather than to the corresponding weight initialization. 
We have shown that the mask compresses the input representation by removing task-irrelevant information in the form of visual object pixels or entire control observation dimensions.
Finally, we found that the pruning process is sensitive to the layer-specific scale of the weight initialization. The interpretability of the resulting minimal representations can be enhanced by heuristically re-scaling the weights in different layers.

\textbf{Limitations.} While our observations are qualitatively robust, they quantitatively vary across learning paradigms, algorithms, architectures and different tasks. Compared to our results, weight values have been claimed to play a more central role in the context of lottery tickets in supervised image classification \citep{frankle_2020b}. Future work will need to investigate under which conditions the initialization of the weight values significantly contributes to the ticket effect in RL. 
Furthermore, whether the interpretation of the mask effect as a representation regularizer also holds for intermediate and higher layer masks is hard to assess and remains an open question. Finally, this study is empirical in its nature and will require further theoretical guidance and foundations.

\textbf{Future Work.} In future work we want to investigate layer-wise effects, e.g., by studying layer-specific pruning ratios, normalization schemes and fine-grained baseline analyses, in the hope of identifying sparser, more efficient and interpretable winning tickets. 
Furthermore, our proposed baseline analysis can be applied more generally to all studies considering the lottery ticket procedure.
Finally, we believe that there are many opportunities to modify the original IMP procedure to simplify the discovery of minimal representations. For example, pruning entire units of intermediate layers may improve the interpretability of hidden representations. 



\section*{Acknowledgements}

We thank Jonathan Frankle for initial discussions and feedback on the first manuscript draft. Furthermore, we thank Joram Keijser for reviewing the manuscript. This work is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy - EXC 2002/1 “Science of Intelligence” - project number 390523135.

\newpage

