\section{Introduction}

Deep object detection models have demonstrated the ability to find and classify objects of interest - an essential asset to the interest of national security \cite{projectmaven}.
Due to the breadth of the United States coastline, ships and other watercraft are among the highest-priority objects of interest. Automating the task through the application of object detection networks allows for efficient, yet accurate detection at comparable-to-human accuracy. 


Neural networks represent the state-of-the-art solution for identifying objects of interest for UAV operators in the thermal domain, which is commonly used in maritime environments. Deep convolutional neural networks are able to detect objects of interest and segment image into individual classes with high accuracy in the visual spectrum, as well as the infrared spectrum \cite{ObjectDetectionThermal}\cite{birdsAI}. Image segmentation offers greater detail than detection within each prediction, since each pixel has an associated predicted class. In addition to visible spectrum images, neural-based segmentation has shown quality results for thermal images \cite{ImageSegThermal,ThermalSurvey} as well.






Traditionally, segmentation methods require ground truth annotations in the form of masks, marking each pixel in the image with an object classification. However, this approach requires an extensive amount of time and human resources. An example of a full segmentation mask is displayed in Figure \ref{fig:pointvseg}. 
The introduction of weakly-supervised learning \cite{Pointly,whatsthepoint,pointsupercrowd,cropspoint}, allows for annotators to quickly and efficiently perform sparse annotation rather than annotating fully-segmented mask, without compromising on segmentation performance. The success of image segmentation trained on weakly-supervised annotations dispels standard of requiring fully-segmented mask annotations. 




\begin{figure}[t]
\centering
  \begin{subfigure}[c]{.45\linewidth}
  \centering
    \includegraphics[width=\linewidth]{graphics/sketches/boatpoint.png}%
    \end{subfigure}
    ~
  \begin{subfigure}[c]{.45\linewidth}
  \centering
    \includegraphics[width=\linewidth]{graphics/sketches/boatmask.png}%
    \end{subfigure}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  \caption
    {%
    \small
    Sparse annotations (left) versus dense annotations (right). Sparse annotation are far easier to generate.
     
      \label{fig:pointvseg}%
    }
\end{figure}

This paper expands on the usefulness of the proposed methods --- namely, point-wise and squiggle-wise annotation --- to annotate and train a model for segmentation using weakly-labeled infrared images. By utilizing a modified U-Net segmentation architecture and point supervision on thermal images, we illustrate the effectivity of using point-wise annotations to utilize object segmentation with minimal truth labels, and therefore minimal annotation work. We also demonstrate that this method is robust to constraints presented in our study, including but not limited to poor image quality, noise due to varying environmental conditions, a minimal number of annotations, and limited training times


\section{Related Work}

\subsection{Infrared Domain Image Segmentation}
Image segmentation is among the most popular and widely-researched computer vision problems, across a wide variety of imaging domains, including the visible spectrum (RGB) and infrared (IR) domains.  State-of-the-art neural-based image segmentation solutions generally make use of an encoder-decoder architecture, often with additional feature fusion modifications such as skip connections and residuals. In the encoder, input images are reduced to a low-resolution feature space, with generalized features learned earlier and image-specific features learned further into the network. The decoder learns to up-sample latent features learned by the encoder and project them back into a segmentation mask, for which each pixel is assigned a class probability. To preserve spatial information within the network, architectures such as U-Net \cite{unet} and SegNet \cite{segnet} utilize skip connections, concatenating previously-learned features to latter layer inputs. Other state-of-the-art segmentation networks, such as Mask R-CNN \cite{maskrcnn}, use a pre-trained backbone convolutional neural network in tandem with a region proposal network to learn latent feature mappings, which are then used to generate a segmentation mask.

Specifically for images in the IR domain, skip-enabled encoder-decoder architectures have a history of success for various segmentation tasks. \cite{photovoltaicSeg} uses a modified U-Net architecture with residual learning capability for the task of segmenting photovoltaic panels within thermal images, scoring a Jaccard index score of 92.65\% with a standard U-Net and over 94\% using the modified architecture.  Furthermore, \cite{ImageSegThermal} provides a comprehensive comparative analysis across various thermal segmentation datasets, along with performance metrics from U-Net and other similar segmentation architectures (SegNet, MFNet). 

While state-of-the-art approaches generally perform well on most image segmentation datasets, even with a limited number of samples, annotation of segmentation masks is significantly more costly when considering the time required to annotate a single image. Large, generalizable segmentation datasets that are fully and accurately painted with labels, such as MS-COCO \cite{MSCOCO}, demonstrably allow a network to learn large-scale features that can be used to pre-train for a smaller, more specific task, as shown in \cite{Pointly}. However, annotating for a custom segmentation task can take exponentially longer amounts of time as the number of classes and complexity of the images increases.
\subsection{Weakly-Supervised Segmentation}
Due to the significant increase in cost of full segmentation mask annotation compared to other vision-based tasks, existing studies have sought to mitigate the time required to annotate while simultaneously not compromising performance. Bearman et al. introduce the concept of image-level, squiggle-level, and point-level supervised segmentation, along with the average annotation time and evaluation performance for each type \cite{whatsthepoint}. The study performed three-types of weakly-supervised annotation schemes on the Pascal-VOC dataset, and compared their annotation times and performance to full segmentation masks. Point-level supervision requires one or more points annotated per class object, as opposed to full pixel-accurate painting, taking an average of 22.1 seconds per image. Squiggle-level supervision, similarly, requires one or more “squiggles”, or continuous series of points per image, averaging 34.9 seconds per image. In comparison, full segmentation mask painting took an average of 239.7 seconds per image - nearly 7 times slower than squiggle-level annotation and over 10 times slower than point-level annotation.
\subsection{Point-Supervised Segmentation}
One method of weakly-supervised segmentation only requires the annotation of only a small, variable number of points to be annotated, rather than painted over entirely. Point-supervised segmentation, namely, has been supported by recent research to be a viable solution to the costly process of segmentation annotation. In addition to the study performed by Bearman et al. \cite{whatsthepoint}, Cheng et al. outline a simple, yet effective point annotation scheme, where only 10 points are selected and classified within a bounding box of an object \cite{Pointly}. Trained on Mask R-CNN and evaluated only on annotated points, this annotation scheme achieved up to 98\% of its fully-supervised performance, despite being nearly 5 times faster to annotate point-wise. Cheng et al. further show that applying the paradigm of transfer learning by pretraining on existing large-scale segmentation data prior to training on point annotations is equally effective - both in the case of full masks of the larger dataset being available, as well as when only point annotations of said dataset are available \cite{Pointly}.
\section{Method}

\subsection{Datasets}\label{section:dataset}
     We use two primary datasets, depicted in Table \ref{tab:dataset breakdown}, to explore weakly-supervised learning and its application to ship segmentation:
        \begin{table}[ht]
        \centering
        \begin{tabular}{c|c|c|c|c}
             Dataset&Domain&\# Images& Resolution&Label Scheme\\
             \hline
             Airbus&Visible&~200,000&780x780&Full mask  \\
             NAWCAD&Infrared&~7,000&640x480&Unlabeled 
        \end{tabular}
        \caption{\small Details of the two datasets used during training.}
        \label{tab:dataset breakdown}
    \end{table}\\
     The Airbus dataset \cite{airbus}, consists of clear, high-resolution RGB images taken from a flat aerial perspective, shown in Figure \ref{fig:AIRBUSSAMPLES}. The NAWCAD (Naval Air Warfare Center Aircraft Division) dataset, however, includes comparatively-lower resolution infrared images, with only a fraction of the volume of the Airbus set. The NAWCAD dataset contains 7055 unlabeled infrared images taken from an unmanned aerial vehicle, in both white-hot and black-hot formats. In contrast to the Airbus images, NAWCAD images were captured with varying angles and environmental conditions. All NAWCAD images were collected from the bay area of the Patuxent River in Maryland. A sample of the dataset is displayed in Figure \ref{fig:NAWCADSAMPLES}. 
        \begin{figure*}[ht]
    \centering
    	\begin{subfigure}[]{0.2\textwidth}
    		\includegraphics[width=\linewidth]{graphics/AIRBUS SAMPLES/0a1a7f395.jpg}
   
   
    	\end{subfigure}\hfill
    	\begin{subfigure}[]{0.2\textwidth}
    		\includegraphics[width=\linewidth]{graphics/AIRBUS SAMPLES/0a1b86392.jpg}
    	\end{subfigure}\hfill
    	\begin{subfigure}[]{0.2\textwidth}
     		\includegraphics[width=\linewidth]{graphics/AIRBUS SAMPLES/0a3b48a9c.jpg}
    	\end{subfigure}\hfill
    	\begin{subfigure}[]{0.2\textwidth}
     		\includegraphics[width=\linewidth]{graphics/AIRBUS SAMPLES/0a7b289e5.jpg}
    	\end{subfigure}\\
    		\caption{\small Example images from the Airbus dataset~\cite{AIRBUS}. These satellite images are clean of any post-processing markings, ships are visually identifiable, and the intereference from structures or landmasses is low.}
    		\label{fig:AIRBUSSAMPLES}
\vspace{25pt}
    \begin{subfigure}[]{0.2\textwidth}
        \includegraphics[width=\linewidth]{graphics/NAWCAD SAMPLES/Image196.jpg}
   
   
    \end{subfigure}\hfill
    \begin{subfigure}[]{0.2\textwidth}
        \includegraphics[width=\linewidth]{graphics/NAWCAD SAMPLES/Image1033.jpg}
    \end{subfigure}\hfill
    \begin{subfigure}[]{0.2\textwidth}
        \includegraphics[width=\linewidth]{graphics/NAWCAD SAMPLES/Image1034.jpg}
    \end{subfigure}\hfill
    \begin{subfigure}[]{0.2\textwidth}
        \includegraphics[width=\linewidth]{graphics/NAWCAD SAMPLES/Image1132.jpg}
    \end{subfigure}\hfill
    \caption{\small Example images from the NAWCAD dataset. These images contain post processing marks such as scales, crosshairs, and time stamps imposed onto the image.}
    \label{fig:NAWCADSAMPLES}
\end{figure*} 
    Given the generally-limited availability of infrared images, along with the additional limitations of the NAWCAD images, we aimed to utilize the external Airbus dataset to supplement the weakly-supervised learning process, via transfer learning. 
   
    In order to determine a baseline potential for weakly-supervised segmentation, we train with a random sample of mask pixels from the fully-segmented Airbus set.  Each pixel is labeled as either boat or background. The dataset contains some overlap between nearby boats, landmasses, or other objects, but the majority of objects are isolated in open water.
   
    We randomly mask \(n\%\) of all pixel labels, at both the  both at n=95\% and n=90\% levels.
    The generated masks were then sent through the data pipeline as the ground truth for the model.
   
    A comparison of full labeled data (no mask) to a 95\% mask can be seen in Figure \ref{fig:AirBusReduced}.
   
   
  
   
   
   
   
   
   
   
    \begin{figure}[ht]
    \centering
    \begin{subfigure}[]{0.2\textwidth}
    \centering
        \includegraphics[width=.95\linewidth]{graphics/AIRBUS_COMP/Input_Image.jpg}
        \caption{Input Image}
    \end{subfigure}
    \begin{subfigure}[]{0.2\textwidth}
    \centering
        \includegraphics[width=.95\linewidth]{graphics/AIRBUS_COMP/No_Mask.jpg}
        \caption{No Mask}
    \end{subfigure}\vspace{10pt}
    \begin{subfigure}[]{0.2\textwidth}
    \centering
        \includegraphics[width=.95\linewidth]{graphics/AIRBUS_COMP/90_percent_Mask.jpg}
        \caption{90\% Mask}
    \end{subfigure}
    \begin{subfigure}[]{0.2\textwidth}
    \centering
        \includegraphics[width=.95\linewidth,height=.95\linewidth]{graphics/AIRBUS_COMP/Predicted_Labels.jpg}
        \caption{Prediction}
    \end{subfigure}
    
	\caption{\small We show the different configurations of training labels that were used for testing the hypothesis of using subsampled data to train a segmentation network.}    \label{fig:AirBusReduced}
\end{figure}

\subsection{Data Preparation}
    Inspired by the results using weak supervision on the Airbus dataset, we explore two annotation schemes for labeling the NAWCAD IR dataset, shown in Figure \ref{fig:Selection_Vis}. The first method - \textit{point} annotation - involves selecting 5 points for each class (namely, the ship and background class) to allow for an even sample distribution across all classes. The second method - \textit{squiggle} annotation - involves drawing squiggle lines on each class, from which a weighted random sample of 32 points are selected. Squiggle annotation allows for a larger, more scaled sample of points to be considered while training, while still taking around the same amount of time as point annotation. We annotated 1200 infrared images using the point-supervised and squiggle-supervised annotation schemes. 
    
   
   
   
    \begin{figure}[t]
        \centering
        \begin{subfigure}{.2\textwidth}
            \includegraphics[width=\textwidth]{graphics/pointclick.jpg}
            \caption{N=10 Point Click}

        \end{subfigure}\hfill
        \begin{subfigure}
            {.2\textwidth}
            \includegraphics[width=\textwidth]{graphics/visual.png}
            \caption{N=32 Squiggle}
        \end{subfigure}
        \caption{\small Example displaying the different annotation schemes used to create labels. (a) Shows the point annotation method where 10 points are sampled with an even distribution between ship and background. (b) Shows the use of drawing lines/squiggles on the image, from which we extract 32 random points weighted by the proportion of points drawn per class.}
        \label{fig:Selection_Vis}
    \end{figure}
   
    Once all images were annotated using the respective methods, the new dataset was randomly shuffled and then split into 90\% training / 10\% validation sets, due to limited amount of trainable images.
    Prior to training, the dataset is augmented via standard segmentation transformations, including random shuffling, rotation, scaling, and cropping.
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    \subsection{Loss Function}
   
     
   
   
   
   
   
   
   
   
    
  
                
    Pixel-Wise Cross-Entropy (PWCE) was used to train both annotation schemes. PWCE is an implementation inspired by the original study introducing point-wise supervision \cite{Pointly}, with our method of point sampling in mind. 
    \begin{equation}
        L_{PWCE}(Y, \hat{Y}, m) = -\sum_{i=1}^{n} [(Y_im_i)*log(\hat{Y_i}m_i)]
    \end{equation}
    A diagram of the process is shown in figure \ref{fig:PWCE}.
\begin{figure}[t]
    \centering
    \includegraphics[width=.5\textwidth]{graphics/PCEL_diagram.PNG}
    \caption{\small This diagram visually illustrates how the labels and evaluation mask are applied to the loss function during training.}
    \label{fig:PWCE}
\end{figure}
    
    During the generation of the dataset, the loader creates three different images; the original image, the ground truth annotated points, and an evaluation mask denoting which points in the ground truth label to evaluate on (represented as a sparse boolean array). This way, points not to be considered in evaluation are zeroed out by the mask, and not factored into the loss function. 
    		


\subsection{Transfer Learning}
    Due to the limitations of the NAWCAD dataset, as mentioned in \ref{section:dataset}, we explore the effectiveness of transfer learning in combination with augmentation techniques. This would ideally allow the network to better generalize previous strongly-learned features of interest to our weakly-supervised task, along with reducing total training time. We applied transfer learning for both point-supervised and squiggle-supervised annotation schemes. The model was first pretrained on the full Airbus dataset, with grayscaling and random inversion augmentations applied to account for the domain change to both formats of thermal images. The goal of doing this was to assist the model with identifying boat shapes and edges in contrast with the background scenerary. 
   
   
   
   
   
   
   
    
   
   


\section{Results}
             
    
            
           
This section demonstrates the difference in performance when utilizing 10 evenly distributed samples versus 32 randomly distributed samples. For our anaylsis we created a subset of fully segmented images that were not used during training for inference. The predictions are scored against the inference labels by measuring the precision, recall and Jaccard index score. The Jaccard score gives us quantitave metric of accuracy that measures the ratio of correct pixel-wise classifications to the total number of classifications. It is defined as:

\[ J(A, B) = \frac{|A\cup B|}{|A|+|B|-|A \cap B|}\]




    

        
    
        
 
\subsection{32-point Subsample Results}
The success cases from the squiggle annotations of the IR dataset are displayed in the last row of Table \ref{tab:images}. An outline of the training performance is shown in Tables \ref{tab:pretrained} and \ref{tab:not-pretrained}. We observe that even with a massive reduction in trainable labels, the performance of the model is still admirable. However, as displayed in the last two columns of Table \ref{tab:images}, due to the low amount of overall training image variety, eliminating false positives completely and achieving full segmentation from the model still pose a problem, especially in heavy atmospheric conditions.  

\subsection{10-point Subsample Results}
The analysis for this training scheme is depicted in Tables \ref{tab:pretrained} and \ref{tab:not-pretrained}. These tables depict that with further reduction of the trainable labels, the model still performs relatively similar to the 32-point training regime. The second row of Table \ref{tab:images} displays a visual representation of predictions generated during inference. The first three columns show the success cases which our model perfomed optimally; these images show that with little to no environmental noise, a model trained on only pointly-annotated labels performs well. However, as seen in the last two columns, the model begins to falter and over segment in the presence of boat wake trails and cloud coverage. In addition, we observe that segmentation tends to falter on large ships that extend beyond the frame of the image.

\section{Conclusion}
In this work, we illustrate that a weakly-supervised annotation scheme can be effectively used to perform thermal image segmentation in high-noise, maritime environments, from only several hundred training samples. Despite taking substantially less time for annotation compared to full segmentations, point-supervised segmentation can be used to successfully and robustly detect and segment both large and small ships. We also investigate the effect of transfer learning from full segmentations in the visible domain (RGB) on weakly-supervised segmentation, finding that pre-training on similar ship image data marginally improves performance after training for the same number of epochs. While we still observed certain phenomena (such as clouds or boat trails) affecting detection performance on occasion, this was not the case the majority of the time. As a part of future work, annotating and training on more images containing prevalent cloud coverage and boat trails would likely improve detection accuracy. \footnote{Link to code and data: \href{https://github.com/rushiljoshi/Weakly-Supervised-Ship-Seg}{Github}}
\newcommand{.17}{.17}

\section*{Acknowledgements}
This work was supported by the University of Maryland Applied Research Lab for Intelligence and Security (ARLIS) and the Naval Air Warfare Center Aircraft Division (NAWCAD), as a part of the Research for Intelligence and Security Challenges (RISC) internship program. Metzler was supported in part by the AFOSR Young Investigator Program.
\nocite{*}
\printbibliography

\begin{table*}[ht]

\hskip-1cm
\begin{tabular}{|l|lll|ll|}
\multicolumn{1}{c}{}&\multicolumn{3}{c}{\large\textbf{Success}}&\multicolumn{2}{c}{\large\textbf{Failure}}\\
\hline
Ground Truth\label{tab:GT}&\includegraphics[width=.17\linewidth,valign=m]{graphics/Results/PixelCross/N10/Success/6921_GROUND_TRUTH.jpg} & \includegraphics[width=.17\linewidth,valign=m]{graphics/Results/PixelCross/N10/Success/524_GROUND_TRUTH.jpg} &\includegraphics[width=.17\linewidth,valign=m]{graphics/Results/PixelCross/N10/Success/8412_GROUND_TRUTH.jpg}  & \includegraphics[width=.17\linewidth,valign=m]{graphics/Results/PixelCross/N10/Failure/9008_GROUND_TRUTH.jpg}&\includegraphics[width=.17\linewidth,valign=m]{graphics/Results/PixelCross/N10/Failure/191_GROUND_TRUTH.jpg}\\
N=10 &\includegraphics[width=.17\linewidth,valign=m]{graphics/Results/PixelCross/N10/Success/6921_PREDICTED.jpg} & \includegraphics[width=.17\linewidth,valign=m]{graphics/Results/PixelCross/N10/Success/524_PREDICTED.jpg} &\includegraphics[width=.17\linewidth,valign=m]{graphics/Results/PixelCross/N10/Success/8412_PREDICTED.jpg}& \includegraphics[width=.17\linewidth,valign=m]{graphics/Results/PixelCross/N10/Failure/9008_PREDICTED.jpg}&\includegraphics[width=.17\linewidth,valign=m]{graphics/Results/PixelCross/N10/Failure/191_PREDICTED.jpg}
\\
N=32 &\includegraphics[width=.17\linewidth,valign=m]{graphics/Results/PixelCross/N32/Success/Pred1.jpg} & \includegraphics[width=.17\linewidth,valign=m]{graphics/Results/PixelCross/N32/Success/Pred4.jpg} &\includegraphics[width=.17\linewidth,valign=m]{graphics/Results/PixelCross/N32/Success/Pred2.jpg}& \includegraphics[width=.17\linewidth,valign=m]{graphics/Results/PixelCross/N32/Failure/Pred4.jpg}&\includegraphics[width=.17\linewidth,valign=m]{graphics/Results/PixelCross/N32/Failure/Pred9.jpg}\\
\hline
\end{tabular}
\caption{\small Success and failure cases of segmentation mask predictions, using the two different annotation schemes.}
\label{tab:images}

\end{table*}
\begin{table*}[ht]
    \centering
    \begin{tabular}{|c|c|c|c|c|}
       
       
       
         \hline
         \textbf{Supervision}&\textbf{Augmentations}&\textbf{Precision}&\textbf{Recall}&\textbf{Jaccard Score}\\
         \hline
         Squiggle (n=32)&None&.925&.948&.612\\
         \hline
         Squiggle (n=32)&Grayscale and Inversion&.979&.978&.756\\
         \hline
         Point (n=10)&Grayscale and Inversion&.976&.976&.692\\
        \hline

    \end{tabular}
    
    \caption{\small Calculated Precision, Recall and Jaccard scores of the inference image mask predictions made by a network pre-trained on the Airbus dataset, scored against a lmited set of test images with full segmentation maps.}\vspace{10pt}
    \label{tab:pretrained}
    \vspace{100pt}
    \begin{tabular}{|c|c|c|c|c|}
       
       
         \hline
         \textbf{Supervision}&\textbf{Augmentations}&\textbf{Precision}&\textbf{Recall}&\textbf{Jaccard Score}\\
         \hline
         Squiggle (n=32)&None&.891&.922&.601\\
         \hline
         Squiggle (n=32)&Grayscale and Inversion&.972&.973&.726\\
         \hline
         Point (n=10)&Grayscale and Inversion&.961&.962&.687\\
        \hline
    \end{tabular}
    \caption{\small Calculated Precision, Recall and Jaccard scores of the inference image mask predictions made by a non-pretrained network, scored against a lmited set of test images with full segmentation maps.}
    \label{tab:not-pretrained}
\end{table*}


\newpage


\end{document}
