
\section{Introduction}
\vspace{-1ex}

The United Nations 2020 Sustainable Development Goals report \cite{sustainable_2020} states that \textit{"water scarcity affects more than 40\% of the global population and is projected to increase. Over 1.7 billion people are currently living in river basins where water use exceeds recharge"}.
Small and medium sized streams make up a significant portion of the flowing network.
The receding tips of this network could serve as an early warning for where droughts are born~\cite{steinemann2006developing, hannaford2011examining, svensson2017statistical}.
In the United States, the \ac{nhd}~\cite{geological_survey_2019} contains large rivers, some smaller streams, as well as lakes that have been mapped during aerial and on-ground mapping campaigns since the early 20th century. A subset of these streams are monitored by gages that continually record the water levels at discrete locations.
However, small and intermittent streams (widths < 90 m) are inadequately mapped and not monitored, even though they contribute 48\% of the total surface water extent of the stream network globally~\cite{allen2018global}.

Open data from satellite missions, like Sentinel and Landsat, have been used for water mapping applications \cite{mueller_water_2016,pekel_high-resolution_2016,isikdogan_rivamap_2017,yang_monthly_2020}.
Sentinel and Landsat offer temporally sparse images, in 5 and 16 day intervals, respectively, and at resolutions (10-30m/px) that are too low for detecting small and intermittent streams.
At the opposite extreme, \ac{VHR} imagery, like from Maxar's WorldView satellites, can be processed into very high resolution (<5m) maps over smaller areas.
Such satellites are generally task-based, that is, they acquire images on customer demand, so multi-temporal coverage is not guaranteed.
As a trade-off, the PlanetScope constellation has high-res (3m), daily, wall-to-wall coverage, offering a \ac{VHR}-grade imagery that is temporally dynamic and spatially extensive.



\begin{figure}
    \centering
    \includegraphics[width=.1995\linewidth]{images/SD_1_89_rgb_maxar.jpg}\hspace*{-.1cm}
    \includegraphics[width=.1995\linewidth]{images/SD_1_89_rgb_planet.jpg}\hspace*{-.1cm}
    \includegraphics[width=.194\linewidth]{images/SD_1_89_rgb_Sentinel2.png}\hspace*{-.1cm}
    \includegraphics[width=.1995\linewidth]{images/SD_1_89_dem.jpg}\hspace*{-.1cm}
    \includegraphics[width=.1995\linewidth]{images/SD_1_89_pred_maxar.jpg}
    \caption{\small Narrow (5-2m) river in the South Dakota AoI. \textbf{Left to right:} \ac{wv3} (50cm/px), PlanetScope (3m/px), Sentinel-2 (30m/px), \ac{dem} (1m/px), water probability map (50cm/px) of the same area. \ac{wv3}, All satellite images are acquired on the same day, and are shown with a false RGB composite of \ac{nir}, Red, and Green bands.}
    \label{fig:resolutions}
\end{figure}

\paragraph{From pixels to streams}\footnote{Video on the steps from the pixel-level map, to the \textbf{stream-level map}: \href{http://bit.ly/pix2streams}{bit.ly/pix2streams}
}
We show that, by combining \ac{VHR} with high cadence, we can monitor previously unseen streams at near real-time (daily).
Our \textbf{Pix2streams} pipeline consists of two functional primitives:
\begin{enumerate}[leftmargin=*]
   
    \item {\bf Pixel-level mapping:} A U-Net (SOTA semantic segmentation model~\cite{ronneberger2015u}) creates pixel-level water probability maps.
    Our variant, WasserNetz, fuses multiple sensors at different resolutions and outputs a water probability map at the resolution of the ground-truth annotation layer (0.5m/px).
   
    \item {\bf Stream-level mapping:} We aggregate these probabilities over a synthetic valley network, derived from 1m LiDAR-based digital elevation maps (\ac{dem}), to obtain a stream-level hydrology map.
\end{enumerate}

We apply this pipeline to a 2-year daily time-series of 3m PlanetScope images, in three US watersheds, to obtain a stream-level \textbf{dynamic hydrology map}, that is, a collection of $N$ time-series (for $N$ streams) of the stream's surface \% covered by water, daily.
Through this map, we can now monitor stream flow frequency, and therefore quantify water availability on a daily, monthly, or yearly basis.

\paragraph{Contributions}
\begin{enumerate}[label=\Alph*, leftmargin=*]
    \vspace{-1ex}
    \item We develop WasserNetz, a {\bf water segmentation model for PlanetScope imagery} trained on a custom dataset of manually labeled polygons, derived from same-day very high-resolution \ac{wv3} images (0.5m/px).
   
    We show that water detection accuracy is improved by fusing the PlanetScope imagery with USGS high-resolution \ac{lidar} derived products.
    \item We analyze and compare the performance of WasserNetz {\bf with models trained on higher resolution (30-40cm/px) \ac{wv3} images} on a manually annotated test dataset.
    Also, we study the sensitivity of the models \textbf{w.r.t. the stream width}.
    \item We produce a {\bf fundamentally new hydrology map at the stream level}, by fusing the pixel-level output of WasserNetz with the synthetic valley network, derived from the LiDAR-based \ac{dem}.
\end{enumerate}








\section{Related Work}
\vspace{-1ex}

Most of the work on mapping and monitoring of rivers uses medium-resolution (30m/px) imagery from Landsat or Sentinel-2 \cite{mueller_water_2016,isikdogan_rivamap_2017,yang_monthly_2020}, and only recently has \ac{VHR} (sub-meter) imagery been used, albeit with a broad focus on all water bodies~\cite{randall2019geographic,feng2018water}.
Landsat imagery was used to derive continental and global maps of large river systems, and to estimate river width and surface area at the global scale~\cite{allen2018global,allen2015patterns}.
Through Google's Earth Engine, Landsat was also used to conduct a multi-annual global inventory of inland waters~\cite{pekel_high-resolution_2016}, enabling monthly estimations of water permanence.
Limited by Landsat's \SI{30}{\metre/px} resolution, both studies can reliably detect only streams wider than $60-90$m (2-3 pixels).
Until today, this spatial limitation has been endemic in the detection of narrower streams~\cite{frasson2019global}.

Some aspects of our work are not directly comparable to these studies:
on the data side, due to the \textbf{unique spatial and temporal resolution} of our imagery and LiDAR derivatives.
On the application side, our pipeline for \textbf{daily \ac{VHR} river mapping} is the first of its kind.
On the algorithmic side, the U-Net~\cite{ronneberger2015u}, with its concurrent encoder-decoder bottlenecks of different resolution inputs, learns multi-scale features to ultimately inform a fine-grained segmentation.
This simple and intuitive idea is accepted as \textbf{a staple of semantic segmentation} in remote sensing~\cite{zhang2018road,wei2019multi,yao2018pixel,feng2018water,wang2020weakly}.





Instead of comparing architectures, our ablations in Section~\ref{sec:results} focus on the resolving capacity of similar U-Nets w.r.t. two fronts: i) training on 30-40cm \ac{wv3} vs. 3m PlanetScope + 1m LiDAR derivatives (WasserNetz); ii) stream width.









\section{Data} \label{sec:Data}
\vspace{-1ex}

We study three \ac{aois} across the western United States.
With limited resources for expert labeling, these \ac{aois} are chosen to provide a representative sample of the morphology, geology and vegetation of terrains.
Fig~\ref{fig:resolutions} shows an example of the three types of data used per AoI:

\textbf{\ac{wv3}:} a single mosaic acquired over multiple dates, with 30-50cm ground sample distance (GSD), 8 spectral bands (coastal blue to near-infrared (NIR)), atmospherically corrected and pansharpened.

\textbf{PlanetScope:} 2 years (April 2018 - April 2020) of daily snapshots, with 3m GSD, four bands (rgb + NIR).
We used an experimental version, corrected for the presence of clouds, radiometrically harmonized to Landsat, Sentinel-2 and MODIS information~\cite{houborg2018cubesat}.

\textbf{\ac{lidar} derivatives:} With the TauDEM suite~\cite{taudem_2015} we produce hydrology products from the  bare-earth 1m resolution \ac{3DEP} \ac{dem}, like the synthetic valley network, reach catchments, Strahler stream order, and a stream buffer (the distance from a given pixel to the nearest reach).
See \ref{sec:lidarder} for details on the derivatives used for training.


\textbf{Manually labeled polygons.} To train the model in a supervised manner, a set of georeferenced delineated polygons corresponding to `wet' and `dry' classes is annotated. These annotations focus on narrow streams and contain labels with shadows for both wet and dry, which are difficult to classify. 
Polygons are drawn using the \ac{wv3} images as a reference to exploit its higher resolution. The dataset contains a total of 5664 labels with 46.39$\%$ water polygons, distributed on the three \ac{aois}. We split the dataset into train (5121) and test sets (543), so that they do not overlap spatially.


\section{Methodology}

    \vspace{-1ex}
    \paragraph{Semantic segmentation model}
    We consider the architecture U-Net~\cite{ronneberger2015u} as a starting point.
    We modify it to have two downstream branches (PlanetScope images and \ac{lidar} products) each taking a different size input. The branches are merged when the size of the feature maps is the same. To provide the output at the resolution of the ground-truth (50cm/px) we add an upsampling convolution step. We refer to this network as \textbf{WasserNetz}. See Appendix \ref{sec:WasserNetz} for an illustration.

    \vspace{-1ex}
    \paragraph{Loss function}
    We use the binary cross entropy loss,  back-propagating only the pixels inside the rasterized polygon, where ground-truth is available. This is required since only pixels within those polygons are annotated. Also, the loss contribution of each example is weighted by the number of labeled pixels in its ground truth, i.e polygon-level loss. This seeks to mitigate the bias due to polygon size.
   
    
    \vspace{-1ex}
    \paragraph{Dynamic Hydrology Map}
    The derived synthetic valley network is a vector layer that delineates all the reaches where water can flow. For each reach we consider its catchment area, that is, the hydrological area unit in which all water ends up in the corresponding reach. Then we aggregate the model prediction (pixel-level water probability map) over each catchment area, to give the probability of having water. For an example, see Figure \ref{figurestatic}. This reach-level probability is computed by weighting the pixel-level probability map by the distance to the reach, and it represents the likelihood of finding water at any part of that reach. The reaches and their associated water probabilities give a static snapshot of flowing water in the stream network. Applying this process to a time-series of images we obtain the near-real time stream network\footnote{Video on steps from pixel-level map, to stream-level, to \textbf{dynamic hydrology map}: \href{http://bit.ly/pix2streams}{bit.ly/pix2streams}}.


\begin{figure*}[ht]
    \centering
    \includegraphics[width=0.74\linewidth]{images/figure3_1_nm.pdf} 
   
    \includegraphics[width=0.245\linewidth]{images/figure3_2_nm.pdf}
    \caption{\small \textbf{Left to right:} 1) Synthetic valley network where each reach in the network is shown with a different color. 2) Pixel-level water probability map (colormap: [0-"dry"/ 1-"water"] $\rightarrow$ [brown, blue]) 3) derived stream-level hydrology map for a single day. 4) Annual stream-level hydrology map produced for 2019.}
    \label{figurestatic}
   
\end{figure*}



\section{Results} \label{sec:results}
\vspace{-1ex}
We measure the performance of our water segmentation model with the accuracy at the pixel-level and polygon-level. The latter seek to balance wet and dry classes since dry polygons are usually larger. PR-curves,included in the appendix~\ref{sec:prcurves}, show similar behaviors.
Table (\ref{tab:metrics}) shows the results for different model configurations.
The multi-sensor segmentation (\ac{lidar}+PlanetScope) outperforms by $\textrm{3}\%$ the model trained only on Planet data. This shows that \ac{lidar}-derived products complement 3m imagery yielding a significant improvement.
The U-Net trained on \ac{wv3} (0.5m) outperforms WasserNetz by $\textrm{7}\%$. This is expected, since below a critical width some streams can be resolved at 0.5m while not at 3m resolution. The \ac{ndwi} approach~\cite{McFeeters1996}, shown as a baseline, shows the worst performance, with a 30-40$\%$ loss in accuracy over all models.

Also, we quantify the sensitivity of our pixel-level hydrology maps w.r.t. the width of the streams (Fig. \ref{annotatedpolygons}). We perform this analysis on a subset (280 polygons) of our test set that include width annotations enclosing the opposing banks of a water body.
A positive relationship between width and performance is observed for all models.
In South Dakota, the \ac{wv3}-trained model is the most robust but without a significant performance margin.
Also, South Dakota's landscape conditions seem to be favorable for WasserNetz (\ac{lidar}+PlanetScope), as they allow it to be competitive to the \ac{wv3}-trained U-Net.
Idaho (ID), with extensive shadows and canopy cover, poses an unfavorable environment for PlanetScope-trained models, whose performance degrades significantly on streams narrower than 5m. However, the performance gain of a multi-sensor model over a Planet only model is $18\%$ for narrower streams (<5m). This
suggests that our approach can lead to a large improvement in complex landscapes, especially for narrow streams (<5m).

Fig.~\ref{figurestatic} shows the annual (2019) stream-level water probabilities, produced by averaging daily stream-level probabilities, derived from PlanetScope images through our proposed pipeline.

\begin{figure}
   
    \centering
    \includegraphics[width=1\linewidth]{images/Figure_stream_width.pdf}
    \vspace*{-5mm}
    \caption{\small Models performance on wet polygons of the test dataset stratified by stream width.}
    \label{annotatedpolygons}
    \vspace{-2mm}
\end{figure}

\begin{table}[ht]
\begin{center}
    \vspace{-2mm}
    \footnotesize
    \caption{\small Polygon-level, pixel-level accuracy on test data. Mean (stdev) across 10 runs of different random seeds.}
    \vspace{1mm}
    \label{tab:metrics}
    \begin{tabular}{llllll}
    \toprule
   
   
    \textbf{Data} &\textbf{Model} &\textbf{Input m/px} &\textbf{Output m/px} & \textbf{Polygon-level} & \textbf{Pixel-level} \\
    \midrule
    \ac{lidar}+PlanetScope & WasserNetz & 3 / 1 & 0.5 &  88.24 (0.36) & 94.34 (0.43)  \\ 
    \ac{wv3}  & U-Net & 0.5 & 0.5 & 95.26 (0.23) & 96.87 (0.21) \\ 
    PlanetScope & U-Net &3 & 0.5 & 85.42 (0.24) & 91.60 (0.52)\\ 
    \ac{wv3} & NDWI  & 0.5 &  0.5 & 56.96 & 71.25 \\ 
    \bottomrule
\end{tabular}
\vspace{-5mm}
\end{center}
\end{table}


\section{Conclusions}
\vspace{-1ex}
Our multi-sensor deep-learning model can segment water at unprecedented scales, from a fusion of multispectral satellite imagery and \ac{lidar}-derived products, processed at their native resolutions. Fusing with LiDAR, compared to an optical-only model, improves the detection of narrow streams (< 5m) by $18\%$ in difficult areas (with shadows and canopy cover), and by $8\%$ in optically cooperative areas. We derive the first \textbf{dynamic hydrology map} from the predictions of the semantic segmentation model (WasserNetz) over a time-series of 2 years.
We emphasize the trade-off between spatial and temporal resolution in existing VHR instruments: even though this work pushes the observation frontier down to the scale of 1-3m stream widths (30-60$\times$ over SOTA) with WorldView3 data, our proposed \textbf{pix2streams} pipeline, and the main idea of our paper, rests on the availability of daily VHR imagery, as exemplified by the PlanetScope constellation.

\paragraph{Future work}
We plan to extend our results to other regions that are optically cooperative
and open-source a larger dataset of annotated polygons.
To further ease the interpretation of the stream-level probabilities in our dynamic hydrology maps, more work must be done to disentangle the spatial proportion of water coverage in a stream, from the uncertainty in a model prediction. The latter can be further broken down into \textit{aleatoric} uncertainty (noisy data and annotations) and \textit{epistemic} uncertainty (eg. ambiguity in features of shadow vs. water). Finally, we believe that the connectedness in dendritic stream networks, as an inductive bias, can inspire new architectures and losses that improve the data-efficiency and robustness in stream mapping.

\begin{ack}
This research was conducted during the 2020 Frontier Development Lab (FDL 2020) research accelerator, a public-private partnership with NASA in the USA and ESA in Europe. The authors gratefully acknowledge support from the United States Geological Survey (USGS), Google Cloud, NVIDIA, Planet. We thank the reviewers for their kind and constructive feedback, Yarin Gal for helpful comments on our manuscript, and all the people who supported us through FDL 2020 - alphabetically, Alan Rea, Alison Lowndes, Andrew Annex, Apoorva Shastry, Belina Raffy, Brad Neuberg, Dave Selkowitz, Giovanni Marchisio, Jack Eggleston, James Parr, Jason Stoker, Jodie Hughes, Julie Kiang, Kelsey Doerksen, Leo Silverberg, Massimo Mascaro, Milad Memarzadeh, Oleg Alexandrov, Rachel Sleeter, Richard Strange, Rob Emanuele, Roland Viger, Ross Beyer, Sara Jennnings, Scott Penberthy, Supratik Mukhopadhyay, Tyler Erickson, Yarin Gal, Zhe Jiang.
Gonzalo Mateo-Garcia has been partially supported by the Spanish Ministry of Science, Innovation and Universities under the projects TEC2016-77741-R and PID2019-109026RB-I00 (ERDF).
\end{ack}

\bibliographystyle{unsrt}
