
\section{Introduction}
\label{sec:introduction}
\input{introduction.tex}

\section{Related Work}
\label{sec:relatedwork}
\input{relatedwork.tex}

\section{Methodology}
\label{sec:methodology}
\input{methodology.tex}

\section{Experiments}
\label{sec:experiments}
\input{experiment.tex}


\section{Discussion}\label{conclusion}

We have proposed a simple yet effective attention distillation approach, \ie, SAD, to improve the representation learning of CNN-based lane detection models. SAD~is validated in various models (\ie, ENet, ResNet-18, ResNet-34, and ResNet-101) and achieves consistent performance gains in three popular benchmarks (\ie, TuSimple, CULane and BDD100K), demonstrating the effectiveness of SAD. The results show that SAD~can generally improve the visual attention of different layers in various networks. 
It would be interesting to extend this idea to other tasks that demands fine-grained attention to details, such as image saliency detection and image matting. 


\noindent
\textbf{Acknowledgement:} This work is supported by SenseTime Group Limited, the General Research Fund sponsored by the Research Grants Council of the Hong Kong SAR (CUHK 14241716), Singapore MOE AcRF Tier 1 (M4012082.020), NTU SUG, and NTU NAP.

{\small
\bibliographystyle{ieee}

\subsection{Dataset}

\begin{table*}[!t]
\caption{Basic information of three lane detection datasets.}
\vskip 0.1cm
\label{dataset_table}
\centering
\small{
\begin{tabular}{c|c|c|c|c|c|c|c}
\hline
Name & \# Frame & Train & Validation & Test & Resolution & Road Type & \# Lane $>$ 5 ? \\
\hline
\hline
TuSimple~\cite{tusimple} & 6, 408 & 3, 268 & 358 & 2, 782 & 1280 $\times$ 720 & highway & $\times$ \\
CULane~\cite{pan2017spatial} & 133, 235 & 88, 880 & 9, 675 & 34, 680 & 1640 $\times$ 590 & urban, rural and highway & $\times$ \\
BDD100K~\cite{yu2018bdd100k} & 80, 000 & 60, 000 & 10, 000 & 10, 000 & 1280 $\times$ 720 & urban, rural and highway &  $\surd$ \\
\hline
\end{tabular}
}
\vspace{-3ex}
\end{table*}

\begin{figure}[t]
  \centering
  \includegraphics[width=1.0\linewidth]{./dataset.pdf}
  \vskip -0.2cm
  \caption{Typical video frames of TuSimple, CULane and BDD100K datasets. Ground-truth lanes are marked in green color.}
  \centering
  \vskip -0.2cm
  \label{fig:dataset}
\end{figure}

\vspace{0.1cm}
\noindent
\textbf{Datasets}.
Figure~\ref{fig:dataset} shows several video frames of three datasets that we use in our experiments. They are TuSimple~\cite{tusimple}, CULane~\cite{pan2017spatial} and BDD100K~\cite{yu2018bdd100k}. TuSimple and CULane are widely used in the literature. Many algorithms~\cite{pan2017spatial, neven2018towards, ghafoorian2018gan} have been tested in TuSimple since it was the largest lane detection dataset before 2018. As to CULane, it contains many challenging driving scenarios like crowded road conditions or roads under poor lighting (see Fig.~\ref{fig:dataset}). BDD100K is originally designed for lane instance classification. However, since there are typically multiple lanes in an image and these lanes are usually very close to each other, using instance segmentation algorithms will yield inferior performance. Therefore, we choose to only detect lanes without differentiating lane instances for BDD100K.
We discuss the details of transforming the original ground truths for our task in the following section on implementation details. 
Table \ref{dataset_table} summarizes their details. Note that the last column of Table~\ref{dataset_table} shows that TuSimple and CULane have no more than 5 lanes in a video frame while BDD100K typically contains more than 8 lanes in a video frame. Besides, TuSimple is relatively easy while CULane and BDD100K are more challenging considering the total number of video frames and road types. Note that the original BDD100K dataset provides 100K video frames, in which 70K are used for training, 10K for validation and 20K for testing. However, since the ground-truth labels of the testing partition are not publicly available, we keep the training set unchanged but use the original validation set for testing. 
A new validation set is allocated separately from the training set, as shown in Table~\ref{dataset_table}.


\noindent
\textbf{Evaluation metrics}. To facilitate comparisons against previous studies, we follow the literature and use the corresponding evaluation metrics for each particular dataset.  

\noindent \textit{1) TuSimple.} We use the official metric (accuracy) as the evaluation criterion. Besides, false positive ($FP$) and false negative ($FN$) are also reported. Accuracy is computed as~\cite{tusimple}: $Accuracy = \frac{N_{pred}}{N_{gt}}$,
where $N_{pred}$ is the number of correctly predicted lane points and $N_{gt}$ is the number of ground-truth lane points.

\noindent \textit{2) CULane.} Following~\cite{pan2017spatial}, to judge whether a lane is correctly detected, we treat each lane as a line with 30 pixel width and compute the intersection-over-union (IoU) between labels and predictions. Predictions whose IoUs are larger than 0.5 are considered as true positives (TP). Then, we use $F_{1}$ measure as the evaluation metric, which is defined as: $F_{1} = \frac{2 \times Precision \times Recall}{Precision + Recall}$,
where $Precision = \frac{TP}{TP + FP}$ and $Recall = \frac{TP}{TP + FN}$.

\noindent \textit{3) BDD100K.} Since there are typically more than 8 lanes in an image, we decide to use pixel accuracy and IoU of lanes to evaluate the performance of different models. 

\noindent
\textbf{Implementation details}.
Following~\cite{pan2017spatial}, we resize the images of TuSimple and CULane to 368$\times$640 and 288$\times$800, respectively. As to BDD100K, we resize the image to 360$\times$640 to save memory usage. The lanes of BDD100K are labelled by two lines. Training the networks using the provided labels is tricky. Therefore, based on these two lines, we calculate the center lines as new targets. We dilate ground-truth lanes of the training set of BDD100K as 8 pixels to provide denser targets while keeping these of testing set unchanged (2 pixels). We use SGD~\cite{bottou2010large} to train our models and the learning rate is set to 0.01. Batch size is set as 12 and the total number of training episodes is set as 1800 for TuSimple and 60K for CULane and BDD100K. The cross entropy loss of background pixels is multiplied by 0.4. Loss coefficients $\alpha$, $\beta$, and $\gamma$ are set as 0.1. Since we select lane pixel accuracy and IoU as the evaluation criterion for BDD100K dataset, we alter the original segmentation branch to output binary segmentation maps to facilitate the evaluation on BDD100K. The lane existence prediction branch is also removed for the BDD100K evaluation.  


We empirically found that several practical techniques, \ie, data augmentation and IoU loss, can considerably enhance the performance of CNN-based lane detection models. As to data augmentation, we use random rotation, random cropping and horizontal flipping to process the input images.  
In our experiments, we apply the same segmentation losses and augmentation strategy to our method, SCNN, ResNet baselines, and deep supervision methods, to ensure a fair comparison. Since the source codes of LaneNet~\cite{neven2018towards} and EL-GAN~\cite{ghafoorian2018gan} are not available, we use their results reported in their papers.


\subsection{Results}

\begin{table}[!t]
\caption{Performance of different algorithms on TuSimple testing set. Here "R-18-SAD~" denotes ResNet-18 + SAD~and we use the same abbreviation in the following sections.}
\label{tusimple_table}
\centering
\small{
\begin{tabular}{c|c|c|c}
\hline
Algorithm & Accuracy & FP & FN \\
\hline \hline
ResNet-18~\cite{he2016deep} & 92.69\% & 0.0948 & 0.0822 \\
ResNet-34~\cite{he2016deep} & 92.84\% & 0.0918 & 0.0796 \\
ENet~\cite{paszke2016enet} & 93.02\% & 0.0886 & 0.0734 \\
LaneNet~\cite{neven2018towards} & 96.38\% & 0.0780 & 0.0244 \\
EL-GAN~\cite{ghafoorian2018gan} & 96.39\% & \textbf{0.0412} & 0.0336 \\
SCNN~\cite{pan2017spatial} & 96.53\% & 0.0617 & \textbf{0.0180} \\
\hline \hline
\textbf{R-18-SAD~(ours)} & 96.02\% & 0.0786 & 0.0451 \\
\textbf{R-34-SAD~(ours)} & 96.24\% & 0.0712 & 0.0344 \\
\textbf{ENet-SAD~(ours)} & \textbf{96.64\%} & 0.0602 & 0.0205 \\
\hline
\end{tabular}
}
\vspace{-2ex}
\end{table}

\begin{table*}[!t]
\caption{Performance ($F_{1}$-measure) of different algorithms on CULane testing set. For crossroad, only FP is shown. The second column denotes the proportion of each scenario in the testing set.}
\label{culane_table}
\centering
\small{
\begin{tabular}{c|c|c|c|c|c||c|c}
\hline
Category & Proportion & \textbf{ENet-SAD~} & \textbf{R-18-SAD~} & \textbf{R-34-SAD~} & \textbf{R-101-SAD~} & ResNet-101~\cite{he2016deep} & SCNN~\cite{pan2017spatial} \\
\hline \hline
Normal & 27.7\% & 90.1 & 89.8 & 89.9 & \textbf{90.7} & 90.2 & 90.6 \\
Crowded & 23.4\% & 68.8 & 68.1 & 68.5 & \textbf{70.0} & 68.2 & 69.7 \\
Night & 20.3\% & 66.0 & 64.2 & 64.6 & \textbf{66.3} & 65.9 & 66.1 \\
No line & 11.7\% & 41.6 & 42.5 & 42.2 & \textbf{43.5} & 41.7 & 43.4 \\
Shadow & 2.7\% & 65.9 & 67.5 & \textbf{67.7} & 67.0 & 64.6 & 66.9 \\
Arrow & 2.6\% & 84.0 & 83.9 & 83.8 & \textbf{84.4} & 84.0 & 84.1 \\
Dazzle light & 1.4\% & \textbf{60.2} & 59.8 & 59.9 & 59.9 & 59.8 & 58.5 \\
Curve & 1.2\% & 65.7 & 65.5 & \textbf{66.0} & 65.7 & 65.5 & 64.4 \\
Crossroad & 9.0\% & 1998 & 1995 & \textbf{1960} & 2052 & 2183 & 1990 \\
Total & -- & 70.8 & 70.5 & 70.7 & \textbf{71.8} & 70.8 & 71.6 \\
\hline \hline
Runtime (ms) & -- & \textbf{13.4} & 25.3 & 50.5 & 171.2 & 171.2 & 133.5  \\
Parameter (M) & -- & \textbf{0.98} & 12.41 & 22.72 & 52.53 & 52.53 & 20.72  \\
\hline
\end{tabular}
}
\vspace{-3ex}
\end{table*}

\begin{table}[!t]
\caption{Comparative results on BDD100K test set.}
\label{bdd100k_table}
\centering
\small{
\begin{tabular}{c|c|c}
\hline
Algorithm & Accuracy & IoU \\
\hline \hline
ResNet-18~\cite{he2016deep} & 30.66\% & 11.07 \\
ResNet-34~\cite{he2016deep} & 30.92\% & 12.24 \\
ResNet-101~\cite{he2016deep} & 34.45\% & 15.02 \\
ENet~\cite{paszke2016enet} & 34.12\% & 14.64 \\
SCNN~\cite{pan2017spatial} & 35.79\% & 15.84 \\
\hline \hline
\textbf{R-18-SAD~(ours)} & 31.10\% & 13.29 \\
\textbf{R-34-SAD~(ours)} & 32.68\% & 14.56 \\
\textbf{R-101-SAD~(ours)} & 35.56\% & 15.96 \\
\textbf{ENet-SAD~(ours)} & \textbf{36.56\%} & \textbf{16.02} \\
\hline
\end{tabular}
}
\vspace{-2ex}
\end{table}

Tables \ref{tusimple_table}-\ref{bdd100k_table} summarize the performance of our methods, \ie, ResNet-18-SAD, ResNet-34-SAD, and ENet-SAD~against state-of-the-art algorithms on the testing set of TuSimple, CULane and BDD100K datasets.
We also report the runtime and parameter count of different algorithm in Table~\ref{culane_table} so that we can compare the performance with the complexity of the model taken into account. The runtime is recorded using a single GPU (GeForce GTX TITAN X) and the final value of runtime is obtained after averaging the runtime of 100 samples. 

It is observed that ENet-SAD~outperforms all baselines in BDD100K while achieving compelling performance in TuSimple and CULane. Considering that ENet-SAD~has 20 $\times$ fewer parameters and runs 10 $\times$ faster compared with SCNN on CULane testing set, the performance strongly suggests the effectiveness of SAD.
It is observed that ResNet-18-SAD and ResNet-34-SAD achieve slightly inferior performance to ENet-SAD~despite their larger model capacity. The is because ResNet-18 and ResNet-34 only use spatial upsampling as the decoder while ENet has a specially designed decoder for the task. 
It is noteworthy that SAD also helps given a deeper model. Specifically, we apply SAD~to ResNet-101, and find that it increases the F$_{1}$-measure from 70.8 to 71.8 in CULane and the accuracy increases from 34.45\% to 35.56\% in BDD100K. 

  









We show some qualitative results of our algorithm and some baselines in these three benchmarks. As can be seen in Fig.~\ref{fig:tusimple_result}, ENet-SAD~can detect lanes more precisely than ENet~\cite{paszke2016enet} in TuSimple and CUlane.
As can be seen in Fig. ~\ref{fig:bdd100k_result}, the output probability maps of ENet-SAD~are more compact and contain less noise compared with those of vanilla ENet and SCNN in poor lighting conditions. However, since many images in BDD100K contain more than 8 lanes and are collected in challenging scenarios like severe occlusion and poor lighting conditions, the performance of all algorithms is unsatisfactory and needs further improvement. In general, SAD~can improve the visual attention as well as the detection performance in challenging conditions like crowded roads and poor light conditions. 

We also perform experiments that apply SAD and remove the effect of the P1 branch by blocking the gradient of the P1 branch from the main branch. Results show that ENet-SAD (without supervision from P1 branch) can still achieve 96.61$\%$ on TuSimple, 70.8 on CULane and 36.54$\%$ on BDD100K, which means the performance gains come mainly from SAD~itself.
        

\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{./tusimple_culane_result.pdf}
  \vskip -0.1cm
  \caption{Performance of different algorithms on (a) TuSimple and (b) CULane testing sets. The number below each image denotes the accuracy. Ground-truth lanes are drawn on the input image.}
  \centering
  \vskip -0.25cm
  \label{fig:tusimple_result}
\end{figure}

\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{./bdd100k_result.pdf}
  \vskip -0.2cm
  \caption{Performance of different algorithms on BDD100K testing set. We visualize the probability maps to better showcase the effect of adding self attention distillation. The brightness of the pixel indicates the probability of this pixel belonging to lanes. The number below each image denotes the pixel accuracy of lanes. Ground-truth lanes are drawn on the input image.}
  \centering
  \vskip -0.5cm
  \label{fig:bdd100k_result}
\end{figure}






\subsection{Ablation Study}

We investigate the effects of different factors, e.g., the mimicking path, on the final performance. Besides, we also perform extensive experiments to investigate the timepoint to introduce SAD~in the training process.


\noindent \textbf{Distillation paths of SAD.} We summarize the performance of performing SAD~between different blocks of ENet in Table \ref{attention_ablation_table}. We have a few observations. (1) SAD~works well in the middle and high-level layers. (2) Adding SAD~in low level layers will degrade the performance. 
The reason why SAD~does not work in low-level layers is that these layers are originally designated to detect low-level details of the scene. Making them to mimic the attention maps of later layers will inevitably harm their ability of detecting local features since later layers encode more global information. 
Besides, we also find that mimicking the attention maps of the neighbouring layer successively brings more performance gains compared with mimicking those of non-adjacent layers ($P_{23}$ + $P_{34}$ outperforms $P_{24}$ + $P_{34}$). 
We conjecture that attention maps of neighbouring layers are closer from the semantic perspective compared with those of non-neighbouring layers (see Fig.~\ref{fig:attention_map}).  

\noindent \textbf{Backward distillation.} We also tested another distillation scheme that makes higher layers to mimic lower layers. It decreases the performance of ENet from 93.02\% to 91.26\% in TuSimple dataset. This is not surprising as low-level attention maps contain more details and are more noisy. Having higher-level layers to mimic lower layers will inevitably interfere the global information captured in higher layers, hampering the crucial clues for the lane detection task.
 
\begin{table}[!t]
\caption{Performance of different variants of ENet-SAD~on TuSimple testing set. $P_{ij}$ denotes that the output of the $i$-$th$ block of ENet mimics the output of the $j$-th block.}
\label{attention_ablation_table}
\centering
\small{
\begin{tabular}{c|c||c|c||c|c}
\hline
Path & Accuracy & Path & Accuracy & Path & Accuracy \\
\hline \hline
$P_{12}$ & 91.22\% & $P_{23}$ & 94.72\% & $P_{23}$, $P_{24}$ & 95.38\% \\
$P_{13}$ & 91.36\% & $P_{24}$ & 94.63\% & \textbf{$P_{23}$, $P_{34}$} & \textbf{96.64\%} \\
$P_{14}$ & 91.47\% & $P_{34}$ & 95.29\% & $P_{24}$, $P_{34}$ & 96.52\% \\
\hline
\end{tabular}
}
\vspace{-2ex}
\end{table}

\noindent \textbf{SAD~ v.s. Deep Supervision.} We also compare SAD~with deep supervision~\cite{Xie_2015_ICCV}. Here, deep supervision denotes the algorithm that uses the labels directly as supervision for each layer in the network. More specifically, we use 1x1 convolution and bilinear upsampling to obtain the prediction of intermediate layers and use the cross entropy loss to train the intermediate outputs of the model. We empirically find that adding deep supervision in blocks 2 to 4 obtains the most significant performance gains. 
As can be seen in Table~\ref{deep_ablation_table}, SAD~brings more performance gains than deep supervision in all three benchmarks. 
We attribute this to the following reasons. Firstly, compared with labels that are considered sparse and rigid, SAD~provides softer attention targets that capture more contextual information that indicate the 
scene structure. Distilling information from attention maps of later layers helps previous layers to grasp the contextual signals. Secondly, a SAD path offers a feedback connection from deeper layers to shallower layers. The connection helps facilitate reciprocal learning between successive layers through attention distillation.

\begin{table}[!t]
\caption{Performance of SAD~and deep supervision applied to different base models on TuSimple, CULane and BDD100K testing sets.}
\label{deep_ablation_table}
\centering
\small{
\begin{tabular}{c|c|c|c|c}
\hline
\multirow{2}*{Algorithm} & TuSimple & CULane & \multicolumn{2}{|c}{BDD100K} \\
\cline{2-5}
~ & Accuracy & Total & Accuracy & IoU \\
\hline \hline
ENet & 93.02\% & 68.4 & 34.12\% & 14.64 \\
ENet-Deep & 94.69\% & 69.6 & 35.61\% & 15.38 \\
ENet-SAD & \textbf{96.64\%} & \textbf{70.8} & \textbf{36.56\%} & \textbf{16.02} \\
\hline
R-18 & 92.69\% & 67.9 & 30.66\% & 11.07 \\
R-18-Deep & 94.14\% & 68.8 & 30.95\% & 12.23 \\
R-18-SAD & \textbf{96.02\%} & \textbf{70.5} & \textbf{31.10\%} & \textbf{13.29} \\
\hline
R-34 & 92.84\% & 68.6 & 30.92\% & 12.24 \\
R-34-Deep & 94.52\% & 69.2 & 31.72\% & 13.59 \\
R-34-SAD & \textbf{96.24\%} & \textbf{70.7} & \textbf{32.68\%} & \textbf{14.56} \\
\hline
\end{tabular}
}
\vspace{-3ex}
\end{table}

\noindent \textbf{When to add SAD.} Recall that we assume a half-trained model before we add SAD into the training. Here, we investigate the different timepoints to add SAD. As can be seen in Fig.~\ref{fig:when_add_SAD}, although different timepoints of introducing SAD  achieve almost the same performance in the end, the time to add SAD~ has an effect on the convergence speed of the networks. We attribute the phenomenon to the quality of the target attention maps produced by later layers. In earlier training stage, deeper layers have not been trained well and therefore the distillation targets produced by these layers are of low quality. Introducing SAD at these earlier stages is not as fruitful. Conversely, adding SAD in later training stage would benefit the representation learning of the previous layers. 

\begin{figure}[t]
  \centering
 
  \includegraphics[width=0.8\linewidth]{./when_add_SAD_culane_new.pdf}
  \vskip -0.2cm
  \caption{Performance of adding self attention distillation on the ENet model at different training episodes on the CULane validation set. The number in the legend denotes the episode when self attention distillation is added. "Baseline" denotes the ENet model without self attention distillation.}
  \vskip -0.1cm
  \centering
  \label{fig:when_add_SAD}
\end{figure}

\subsection{Dataset}

\begin{table*}[!t]
\caption{A brief description about three lane detection datasets.}
\label{dataset_table}
\centering
\begin{tabular}{c|c|c|c|c|c|c|c}
\hline
Name & \# Frame & Train & Validation & Test & Resolution & Road Type & \# Lane $>$ 5 ? \\
\hline
TuSimple & 6, 408 & 3, 626 & -- & 2, 782 & 1280 $\times$ 720 & highway & $\times$ \\
\hline
CULane & 133, 235 & 88, 880 & 9, 675 & 34, 680 & 1640 $\times$ 590 & urban, rural and highway & $\times$ \\
\hline
BDD100K & 80, 000 & 70, 000 & -- & 10, 000 & 1280 $\times$ 720 & urban, rural and highway &  $\surd$ \\
\hline
\end{tabular}
\end{table*}

\begin{figure}[t]
  \centering
  \includegraphics[width=1.0\linewidth]{./figure/dataset.pdf}
  \vskip -0.2cm
  \caption{Typical video frames of TuSimple, CULane and BDD100K datasets.}
  \centering
  \label{fig:dataset}
\end{figure}

Table \ref{dataset_table} records the basic information of three lane detection datasets. Note that the last column of Table \ref{dataset_table} shows that TuSimple and CULane have no more than 5 lanes in a video frame while BDD100K typically has more than 8 lanes in a video frame. Besides, TuSimple is relatively easy while CULane and BDD100K are more challenging considering the total number of video frames and road types. Note that original BDD100K dataset provides 100 k video frames, 70 k for training, 10 k for validation and 20 k for testing. However, since they do not provide ground-truth labels for testing set, we keep the training set unchanged but use the original validation set for testing.   


\subsection{Evaluation Criterion}

\noindent \textbf{TuSimple:} in TuSimple, we use the official metric (accuracy) as the evaluation criterion. Besides, false positive ($FP$) and false negative ($FN$) are also listed. The following is the equation to compute accuracy~\cite{tusimple}: 

\begin{equation}
Accuracy = \frac{N_{pred}}{N_{gt}} ,
\end{equation}

where $N_{pred}$ is the number of correctly predicted lane points and $N_{gt}$ is the number of ground-truth lane points.

\noindent \textbf{CULane and BDD100K:} to judge whether a lane is correctly detected, we treat each lane as a line with fixed pixel width (30 for CULane and 8 for BDD100K) and compute the intersection-over-union (IoU) between labels and predictions. Predictions whose IoUs are larger than 0.5 are considered as true positives (TP). Then, we use $F_{1}-measure$ as the evaluation metric formulated as follows:

\begin{equation}
F_{1}-measure = \frac{2 \times Precision \times Recall}{Precision + Recall} ,
\end{equation}

where $Precision = \frac{TP}{TP + FP}$ and $Recall = \frac{TP}{TP + FN}$.


\subsection{Methods}

We give a brief introduction to several CNN-based lane detection models. Note that the following approaches all adopt the encoder and decoder structure as the main network.

\noindent \textbf{SCNN:} SCNN uses VGG-16 as the encoder, bilinear upsampling as the decoder and performs message passing to better exploit the long, thin prior of lanes. Besides, the lane existence prediction branch is added to provide extra performance gains as well as facilitate the evaluation process.  

\noindent \textbf{ResNet-101:} similar to SCNN except that the encoder is ResNet-101 and no message passing is used, it achieves compelling performance in CULane. 

\noindent \textbf{LaneNet:} it is mainly comprised of two sub-networks, i.e., LaneNet and H-Net. The former is used to generate the segmentation maps differentiating lane instances and background. The later is utilized to obtain the lane points from original images and segmentation maps to facilitate evaluation.

\noindent \textbf{EL-GAN:} it adopts U-Net~\cite{ronneberger2015u} as the generator (i.e., the encoder and decoder) and DenseNet~\cite{huang2017densely} as the discriminator. The introduction of GAN aims at generating more realistic segmentation maps as well as easing the complex post-processing.

\noindent \textbf{GC-LD:} it uses VGG-16 as the backbone and performs lane and road detection simultaneously. Both geometric prior and inner relationship between two detection tasks are exploited to boost the overall performance of their models.

\subsection{Preliminary}

Lane detection is commonly formulated as a semantic segmentation task. More specifically, given an input image \textbf{X}, the objective is to assign a label $l_{ij}$ ($l_{ij}$ = 1, ..., $N_{c}$) to each pixel $(i, j)$ of \textbf{X}, comprising the segmentation map $s$. Here, $N_{c}$ is the number of classes. The objective is to learn a mapping $\mathcal{F}$: \textbf{X} $\mapsto$ $s$. Recent studies use CNN as $\mathcal{F}$ for end-to-end prediction. The task of lane existence prediction is also introduced to facilitate the evaluation process. We use $b$ to represent the binary labels that indicate the existence of lanes. Then, the function becomes $\mathcal{F}$: \textbf{X} $\mapsto$ ($s$, $b$). 


\subsection{Self Attention Distillation}



Apart from training our lane detection network with the aforementioned semantic segmentation and lane existence prediction losses, we aim to perform layer-wise and top-down attention distillation to enhance the representation learning process. The proposed SAD does not require any external supervision or additional labels since attention maps are derived from the network itself.

In general, attention maps can be divided into two categories, \ie, activation-based attention maps~\cite{zagoruyko2016paying} and gradient-based attention maps~\cite{zagoruyko2016paying}. The activation-based attention maps are obtained via processing the activation output of a specific layer while the gradient-based ones are obtained via using the layer's gradient output. In the experiment, we empirically find that activation-based attention distillation yields considerable performance gains while gradient-based attention distillation barely works. Hence, in the following sections we only discuss the activation-based attention distillation.

\noindent \textbf{Activation-based attention distillation.} We use $A_{m} \in R^{C_{m} \times H_{m} \times W_{m}}$ to denote the activation output of the $m$-th layer of the network, where $C_{m}$, $H_{m}$ and $W_{m}$ denote the channel, height and width, respectively. Let $M$ denote the number of layers in the network. The generation of the attention map is equivalent to finding a mapping function $\mathcal{G}$: $R^{C_{m} \times H_{m} \times W_{m}} \to R^{H_{m} \times W_{m}}$. The absolute value of each element in this map represents the importance of this element on the final output. Therefore, this mapping function can be constructed via computing statistics of these values across the channel dimension. More specifically, the following three operations~\cite{zagoruyko2016paying} can serve as the mapping function: $\mathcal{G}_{sum}(A_{m}) = \sum_{i=1}^{C_{m}}| A_{mi}|$, $\mathcal{G}_{sum}^{p}(A_{m}) = \sum_{i=1}^{C_{m}}| A_{mi}|^{p}$ and $\mathcal{G}_{\max}^{p}(A_{m}) = \max_{i=1, C_{m}}| A_{mi}|^{p}$. Here, $p > 1$ and $A_{mi}$ denotes the $i$-th slice of $A_{m}$ in the channel dimension.

\begin{figure}[b]
  \centering
  \includegraphics[width=1.0\linewidth]{./compare_func.pdf}
 
  \caption{Attention maps of the block 4 of the ENet model using different mapping functions.}
  \centering
  \vskip -0.2cm
  \label{fig:func}
\end{figure}

\begin{figure*}[t]
  \centering
  \includegraphics[width=0.78\linewidth]{./framework.pdf}
  \vskip -0.2cm
  \caption{An instantiation of using SAD. $E_{1}$ $\sim$ $E_{4}$ comprise the encoder of ENet~\cite{paszke2016enet}, $D_{1}$ and $D_{2}$ comprise the decoder of ENet. Following \cite{pan2017spatial}, we add a small network to predict the existence of lanes, denoted as $P_{1}$. AT-GEN is the attention generator.}
  \centering
  \vskip -0.2cm
  \label{fig:pipeline}
\end{figure*}

The differences between these mapping functions are depicted in Fig.~\ref{fig:func}. Compared with $\mathcal{G}_{sum}(A_{m})$, $\mathcal{G}_{sum}^{p}(A_{m})$ puts more weights to areas with higher activations. The larger the $p$ is, the more focus is placed on these highly activated areas. Compared with $\mathcal{G}_{\max}^{p}(A_{m})$, $\mathcal{G}_{sum}^{p}(A_{m})$ is less biased since it calculates weights across multiple neurons instead of selecting the maximum value of these neuron activations as the weight. In the experiment, we empirically find that using $\mathcal{G}_{sum}^{2}(.)$ as the mapping function yields the most performance gains. 




      
\noindent \textbf{Adding SAD to training.} The intuition behind SAD~is that the attention maps of previous layers can distil useful contextual information from those of successive layers. Following ~\cite{zagoruyko2016paying}, we also perform spatial softmax operation $\Phi(.)$ on $\mathcal{G}_{sum}^{2}(A_{m})$. Bilinear upsampling $\mathcal{B}(.)$ is added before the softmax operation if the size of original attention maps is different from that of targets. However, different from Sergey \etal ~\cite{zagoruyko2016paying} who perform attention distillation within two networks, the proposed self attention distillation is performed within the network itself. 

Adding SAD to an existing network is straight-forward. It is possible to introduce SAD at different timepoint of the training, which could affect the convergence time. We will show an evaluation in the experiment section. Here we assume an ENet half-trained to 40K episodes.
As shown in Fig. \ref{fig:pipeline}, we add an attention generator, abbreviated as AT-GEN, after each $E_2$, $E_3$, and $E_4$ encoder block of ENet. Formally, AT-GEN is represented by a function $\Psi(.) = \Phi(\mathcal{B}(\mathcal{G}_{sum}^{2}(.)))$. A successive layer-wise distillation loss is formulated as follows:
\begin{equation}
\label{eqn:sad_loss}
\begin{split}
\mathcal{L}_\mathrm{distill}(A_{m}, A_{m+1}) =
\sum_{m=1}^{M-1} \mathcal{L}_\mathrm{d} (\Psi(A_{m}), \Psi(A_{m+1})),
\end{split}
\end{equation}  
where $\mathcal{L}_\mathrm{d}$ is typically defined as a $L_{2}$ loss and $\Psi(A_{m+1})$ is the target of the distillation loss. In the example shown in Fig. \ref{fig:pipeline}, we have the number of layers $M = 4$. Note that we do not assign different weights to different SAD paths, although this is possible. We found that this uniform scheme works well in our experiments.

The total loss is comprised of four terms:
\begin{equation}
\label{eqn:total_loss}
\begin{split}
\mathcal{L} = & \underbrace{\mathcal{L}_\mathrm{seg}(s, \hat{s}) + \alpha \mathcal{L}_\mathrm{IoU}(s, \hat{s})}_{\text{segmentation loss}} \\
& + \underbrace{\beta \mathcal{L}_\mathrm{exist}(b, \hat{b})}_{\text{existence loss}} + \underbrace{\gamma \mathcal{L}_\mathrm{distill}(A_{m}, A_{m+1})}_{\text{distillation loss}}.
\end{split}
\end{equation}
Here, the first two terms are segmentation losses that comprise of the standard cross entropy loss $\mathcal{L}_\mathrm{seg}$ and the IoU loss $\mathcal{L}_\mathrm{IoU}$. The IoU loss aims at increasing the intersection-over-union between the predicted lane pixels and ground-truth lane pixels. It is formulated as $\mathcal{L}_{IoU} = 1 - \frac{N_{p}}{N_{p} + N_{g} - N_{o}}$, where $N_{p}$ is the number of predicted lane pixels, $N_{g}$ is the number of ground-truth lane pixels and $N_{o}$ is the number of lane pixels in the overlapped areas between predicted lane areas and ground-truth lane areas.
$\mathcal{L}_\mathrm{exist}$ is the binary cross entropy loss. $\hat{s}$ is the segmentation map produced by the network and $\hat{b}$ is the prediction of the existence of lanes. The parameters $\alpha$, $\beta$, and $\gamma$ balance the influence of segmentation losses, existence loss, and distillation loss on the final task. 



It is noteworthy that the SAD paths can be generalized to dense connections beyond the example shown here. For instance, we can add block~1 $ \xrightarrow[]{\text{mimic}}$ block~3, block~1 $ \xrightarrow[]{\text{mimic}}$ block~4, and block~ 2 $ \xrightarrow[]{\text{mimic}}$ block~4 in addition to the current paths. In general, the number of possible SAD paths for a network with a depth of $M$ layers is $\frac{M(M-1)}{2}$. We will evaluate this possibility in our experiments. 

\noindent \textbf{Visualization of attention maps with and without SAD.} 
We investigate the influence of SAD by studying the attention maps of different blocks in ENet with and without SAD. More results will be reported in Section~\ref{sec:experiments}. Both networks with and without SAD are trained up to 60K episodes. 
We visualize the attention maps of four existing blocks in ENet. As can be observed in Fig.~\ref{fig:learning}, after adding SAD, the attention maps of ENet become more concentrated on task-relevant objects, \eg, lanes, vehicles and road curbs. This would in turn improve the lane detection accuracy, as we will show in the experiments.


\begin{figure*
  \centering
  \includegraphics[width=0.75\linewidth]{./vis_sad_learning.pdf}
 
  \caption{Attention maps of ENet with and without self attention distillation. Both networks with and without SAD are trained up to 60K episodes. SAD is applied to ENet at 40K training episodes.}
  \centering
  \vskip -0.2cm
  \label{fig:learning}
\end{figure*}

\subsection{Lane Prediction}
\label{subsec:inference}

The output of the model is not post-processed for TuSimple and BDD100K except CULane. For CULane, in the inference stage, we feed the image into the ENet model. Then the multi-channel probability maps and the lane existence vector are obtained. Following~\cite{pan2017spatial}, the final output is obtained as follows: First, we use a 9 $\times$ 9 kernel to smooth the probability maps. Then, for each lane whose existence probability is larger than 0.5, we search the corresponding probability map every 20 rows for the position with the highest probability value. In the end, we use cubic splines to connect these positions to get the final output.

\subsection{Architecture Design}
\label{subsec:architecture}

The original ENet model is an encoder-decoder structure comprised of $E_{1} \sim E_{4}$, $D_{1}$ and $D_{2}$. Following \cite{pan2017spatial}, we add a small network $P_{1}$ to predict the existence of lanes. The encoder module is shared to save memory space.
Apart from this modification, we also observed some useful techniques to modify ENet for achieving better performance in the lane detection task.
Dilated convolution \cite{yu2015multi} is added to replace the original convolution layers in the lane existence prediction branch to increase the receptive field of the network without increasing the number of parameters. In the original design, the resolution of feature maps of $E_{4}$ is only 36 $\times$ 100 for CULane. This leads to severe loss of information. Hence, we use feature concatenation to fuse the output of $E_{4}$ with that of $E_{3}$ so that the output of the encoder can benefit from information encoded in previous layers.




