\section{Introduction}

Regular examination and skin monitoring is essential for the early detection of cutaneous malignancies including melanoma. Monitoring these pigmented skin lesions over a long period of time is usually a manual, time-consuming and error-prone task, especially when a patient has hundreds of skin lesions, both the inspection and documentation of these lesions are very laborious and inefficient.

Data-driven approaches using dermoscopic images have been developed in recent decades to assist dermatologists in clinical decisions and to spot extremely suspicious situations~\cite{barata2018survey}.
Recent studies have demonstrated the ability of machine learning to match, if not outperform, clinicians in the diagnosis of individual skin lesion images in controlled reader studies where dermoscopic images contain a single skin lesion. For example, algorithms derived from the ISIC Grand Challenge~\cite{rotemberg2021patient} have shown potential to outperform over 500 clinical experts in such a controlled environment~\cite{tschandl2019diagnostic}. However, these studies did not accurately reflect clinical scenarios where clinicians need to examine all skin lesions of a patient regularly.

Despite the potential for using deep learning in clinical dermatology such as skin cancer risk assessment~\cite{de2019development}, most systems assume that the data were adequately pre-selected with all relevant lesions of a patient that are worthy of examination. Thus, the identification of skin lesions in wide-field images (photographs depicting multiple lesions from large body parts) is needed~\cite{birkenfeld2020computer}.

Contrary to classic investigations, where the attention is focused on a particular lesion and a single image taken using a dermoscope or a high-resolution camera, whole-body imaging systems capture images of the whole body. Such a system allows not only lesion detection, but also the possibility of a longitudinal study of dermatological patients. In this manner, it is possible to detect lesions that evolve over time and to find new lesions that emerge after the previous examination. 

Traditionally, to monitor all skin lesions on the skin surface of a patient, 2D whole-body imaging has been employed. The use of 2D whole body imaging has shown the potential to reduce the number of biopsies taken, and increase the accuracy of diagnosis for patients at high risk of melanoma~\cite{truong2016reduction}.
Recent advances in artificial intelligence have demonstrated its capability to support physicians in detecting outliers or suspicious ``ugly duckling lesions'' in wide-field images~\cite{soenksen2021using,mohseni2021can}.
A few commercial systems such as Dermengine~\cite{dermengine} support total-body image analysis using mobile cameras.
Nevertheless, the data collection is time and resource intensive and requires a photographer to take photos of subjects in different poses. Thus, skin lesion comparison remains challenging due to changes in the pose or illumination between different scans. The majority of previous registration techniques are evaluated only on a small part of the body (\textit{e.g.} the back or front torso).
For example, a device used for capturing images of the whole area of human skin was introduced in~\cite{strzelecki2021skin}, but the entire body is not captured simultaneously.

Investigations toward the identification of lesions that differ from most of the other marks on a patient’s skin using wide-field images have gained attention to flag them for further examination. Nevertheless, such correlation methods that identify particular lesions of the patient’s body that appear in images taken from different views and time are not well studied in the literature. 


Advanced 3D imaging technology, which can be supported by automated analysis, is changing the way to diagnose skin malignancies. 3D imagery has enabled domain researchers to gain new understandings at a faster pace from the image data. 3D data exploration is useful when particular attributes of interests cannot be easily identified in a 2D image, and to accurately inspect the fine topology and textual features of a lesion. 

Digital 3D whole-body images enable a complete and automatic documentation of all skin lesions of a patient, determination of their geometric parameters, and their further analysis by a dermatologist~\cite{rayner2018clinical}.
This technology can be used to provide context to single lesion dermoscopy images~\cite{janda2021describing}, facilitate long-term surveillance, and support the evaluation of raised lesions, as the 3D images can be viewed from multiple angles.
This means that lesions are considered within the context of the surrounding skin, offering a more accurate overall assessment and diagnosis than dermoscopy images alone~\cite{grochulska2021additive}.

Preliminary 3D body scanning for detection of new melanocytic lesions is described in~\cite{bogo2014automated}. Such a solution is based on a multi-camera 3D stereo system to capture body shape and skin texture, and a 3D body model to register scans across time. However, the model is based on expensive scanning devices and limited to higher-resolution RGB imagery. The detection of lesions is also based on handcrafted features.

Although there exist numerous recent articles that demonstrated the benefits of 3D whole body photography~\cite{navarrete2020total,betz2021reproducible,grochulska2021additive}, 3D human representation is obtained with expensive commercial systems such as VECTRA{\tiny{\textregistered}}WB360 (Canfield Scientific Inc)~\cite{canfield}, which simultaneously captures 92 images, using them to reconstruct a 3D avatar.
In contrast, we propose a low-cost mobile health pod equipped with AI based smart tools that can be repurposed from current clinician use case, to help researchers collect and curate data automatically from 3D human models as illustrated in Fig.~\ref{fig:3D_collection}.


\begin{figure}[t!]
\begin{center}
\includegraphics[width=0.95\linewidth]{Images/Fig1.png}
\end{center}
\vspace{-12pt}
   \caption{3D Whole body imaging captures nearly the entire skin surface to map, document and monitor pigmented lesions. Visualisation of the sparse point cloud (left), 3D texture model (middle), and skin lesions detected (right). All lesions detected on 2D images are mapped back to the 3D reconstructed model, improving the ability to track lesions over time.
   }
\label{fig:3D_collection}
\vspace{-3pt}
\end{figure}

\begin{table}[t!]
\centering
\vspace{-9pt}
\caption{Comparison of our imaging system to other related works.}
\resizebox{0.8\textwidth}{!}{%
\begin{tabular}{
l c c c c c c
}
\toprule
& ~\cite{zhao2021detection} & ~\cite{strzelecki2021skin} & ~\cite{navarrete2020total} & ~\cite{betz2021reproducible} & ~\cite{grochulska2021additive} & ours
\\
\midrule
2D whole body photography               &            &            & \checkmark & \checkmark & \checkmark & \checkmark \\ 
Low-cost acquisition system             &            & \checkmark &            &            &            & \checkmark \\ 
3D human reconstruction                 &            &            & \checkmark & \checkmark & \checkmark & \checkmark \\ 
Clinical images used for detection      &            & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark \\ 
Data-driven skin lesion detector        & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark \\ 
Lesion mapped back on 3D body texture                & \checkmark &            & \checkmark & \checkmark & \checkmark & \checkmark \\ 
User Interface data management          &            &            & \checkmark & \checkmark & \checkmark & \checkmark \\ 
\bottomrule
\end{tabular}}
\label{table:literature-summary}
\end{table}

In this paper, we introduce our 3D whole body imaging system to detect and monitor skin lesions. The main specifications of the system compared to the related works are listed in Table~\ref{table:literature-summary}. Our contributions are:
\begin{enumerate}
\item A cost-effective image acquisition system has been designed and constructed, enabling researchers to capture images of the whole body of a patient.
\item A novel 3D pipeline for 3D reconstruction based on open sources tools is proposed to generate high-quality physical-scale 3D body avatars.
\item A processing pipeline that can be used to estimate depth and 3D projection, as well as provide the 3D localisation of skin lesions in whole body images with an optimal balance of inference time and accuracy, has been developed.
\item A custom intuitive and flexible user interface is created that allows users to interact and collaborate with the collected data, the 3D body avatar and detected skin lesions.
\end{enumerate}


\section{Materials and Methods} 

In this manuscript, we introduce the 3DSkin-mapper prototype which enables the detection, monitoring and analysis of skin lesions on the patient's entire body.  An abstract view of the system workflow is illustrated in Fig.~\ref{fig:pipeline}.
First, a device is constructed to capture images of the entire body of a patient simultaneously with commercially available digital cameras mounted on a camera rig. These images are then used in a processing pipeline for the 3D image reconstruction, estimation of the 3D depth and 2D to 3D projection. A trained deep learning model localises the lesions within the 2D domain, which are then mapped back to the 3D image of the human body. Such anatomical correspondence of the 3D localisation of all lesions can be exploited for the longitudinal study of the lesions. All information computed in previous stages is integrated into a user interface (UI) which has the function to seemingly integrate several observations from different views into a single point of interest. A key aspect of developing a 3D data collection and curation tool, such as this, is to create an efficient way to associate where a human expert operator is looking with the data itself.
All hardware components used are commercially available off-the-shelf (COTS) and all algorithms developed are based on open sources to facilitate its reproducibility.
Details of each module of the imaging system is described in the following subsections.


\begin{figure*}[!t]
\centering
\includegraphics[width=1\linewidth]{Images/Fig2.png}
\vspace{-9pt}
   \caption{Overview of the 3DSkin-mapper workflow.
  \textbf{1.} Design and construction of the hardware and software for the data collection (60 cameras).
  \textbf{2.} Collection of diverse human data for the 3D reconstruction, 3D fine-tuning and skin lesion detection.
  \textbf{3.} Reconstruction of the human body from different camera poses and sparse point cloud estimation.
  \textbf{4.} Estimation of the depth, human mask and 2D to 3D projection.
  \textbf{5.} A deep convolutional neural network that examines lesions from given wide-field images.
  \textbf{6.} 3DSkin-mapper user interface for visualisation, curation and data management.
   }
\label{fig:pipeline}
\end{figure*}


\subsection{Image acquisition system: hardware and software}
\label{sec:acquisition}

The acquisition system developed consists of 15 poles approximately 2.1 m tall, arranged in a roughly cylindrical configuration with an approximate diameter of 2.2 m. Four Canon EOS 200D cameras are mounted on each pole at the following distances from the ground: 0.3 m, 0.8 m, 1.3 m, and 1.8 m. Black curtains are placed around the camera rig for privacy, improved lighting as well as blocking out the background of images. The cameras are set up to take portrait images with the following settings: aperture measurement (F-Stop) = 10, exposure time = 1/15; sensitivity to light (ISO) = 800; and focal length = 18 mm.
During examination, the patient stands on a motionless wooden platform at the centre of the system, which is used as a reference dimension. This stand is also used to crop and scale the mesh of a person to the right physical height.
The modular acquisition system is shown in Fig.~\ref{fig:sensors_on_trolley}.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=0.75\linewidth]{Images/Fig3.png}
\end{center}
\vspace{-7pt}
   \caption{Whole body photography structure. Camera cage which consist of 15 poles arranged in a cylindrical configuration, each pole with 4 cameras Canon EOS 200D.
   }
\label{fig:sensors_on_trolley}
\vspace{-4pt}
\end{figure}


The Canon EOS 200D cameras used in the system have a 2.5 mm audio jack port for remote triggering. The jack itself has three terminals, which are ground, shutter release, and autofocus, respectively. Connecting either the shutter release or autofocus terminal to the ground terminal will activate the respective function.
Synchronous image capturing is achieved by sending trigger signals to all cameras at the same time. A control board based on FTDI FT232R microcontroller is used for this purpose. The board has 16 3.5 mm audio jack ports for camera connections, and more cameras can be added through purpose-built repeaters. The autofocus signal and shutter release signal are connected to CBUS bit~2 and CBUS bit~3 respectively of the FT232R microcontroller. The signals are set/reset by a Windows application that communicates with the FT232R chip through a USB connection. 
Fig.~\ref{fig:trigger_conn} (Top) shows the block diagram of the triggering circuit.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=0.85\linewidth]{Images/Fig4.png}
\end{center}
\vspace{-7pt}
   \caption{Top: Block diagram of the camera triggering circuit. Bottom: Screenshot of the camera control application running on Windows OS.}
\label{fig:trigger_conn}
\vspace{-4pt}
\end{figure}
 
The autofocus signal is set first, followed by the shutter release signal. $T_{AF}$, $T_{SR}$, and $D_{SR}$ represent the duration of the autofocus signal, the duration of the shutter release signal, and the delay between asserting the two signals, respectively. 
These parameters can be configured in the Windows application. By default, they are specified to the following values: $T_{AF}$=2,500ms, $T_{SR}$=3,500ms, and $D_{SR}$=2,600ms.
Fig.~\ref{fig:trigger_conn} (Bottom) illustrates the camera control interface. The drop-down menu in the top-left corner of the GUI lists the USB control boards connected to the PC. There should be exactly one device in the list, which is selected automatically when the GUI is launched. The control signals are sent to cameras by clicking the ``Shoot'' button. 
The autofocus signal and shutter-release signal can be enabled or disabled independently by checking/unchecking the ``Focus'' and ``Capture'' boxes correspondingly. This enables different camera settings during image capture. The input boxes next to ``Focus'' and ``Capture'' are used to configure the duration of the autofocus signal and the shutter release signal respectively, while the ``Delay'' input box can be used to configure the delay between asserting the two signals.

The open-source software digiCamControl~\cite{digicamcontrol} is used to change camera settings and transfer captured images from individual cameras to a cloud storage. By launching and properly configuring the software before capturing images, it will automatically download images from all cameras after new images are captured each time. 
All cameras and image files are labelled according to their locations, which allows easy mapping of image files to the corresponding camera locations in the rig.


\subsection{Human dataset compilation}
\label{sec:datasets}

\subsubsection{3D body dataset}
Various datasets of 3D body scans are available for research or commercial applications. 3D body scan images with high-quality skin texture information are used to optimise camera poses, and test the processing pipeline. Texture information combined with the 3D geometry allows for advanced analysis and representations.
For rendering realistic images of human bodies, we use 3D human models from Renderpeople \cite{Renderpeople_bundle_swimwear} and the 3DBodyTex~\cite{saint20183dbodytex,saint2019bodyfitr} dataset. 
Such human models, as shown in Fig.~\ref{fig:3d_human_models}, also contain skin lesions used for the evaluation of the proposed lesion detector. Synthetic images generated from these models for experiments are obtained using Blender software~\cite{Blender} and Pyrender library\cite{Pyrender}.

\begin{figure}[t!]
    \centering
    \includegraphics[width=0.6\linewidth]{Images/Fig5.png}
    \vspace{-7pt}
    \caption{Synthetic 3D human models. Images adapted from \cite{Renderpeople_bundle_swimwear,saint20183dbodytex,saint2019bodyfitr}.}
    \label{fig:3d_human_models}
\vspace{-4pt}
\end{figure}


\subsubsection{Skin image analysis dataset}
\label{sec:dataset-skin}

Skin lesions are detected directly on 2D wide-field images from publicly available sources, rendered from 3D models and captured by our imaging system. For development and validation of our processing pipeline, we curated a ground truth dataset that contains diverse skin lesions. 
Overall, we extracted a total of 24,525 lesion images from the following sources:

\begin{itemize}
    \item We develop an experimental dataset in collaboration with a commercial lab in Australia, which includes de-identified clinical images of skin lesions and ground truth data of the images.
    %
    \item The 2020 ISIC Grand Challenge~\cite{rotemberg2021patient} contains the largest publicly available collection of quality controlled dermoscopic images of skin lesions. This is the official dataset of the SIIM-ISIC Melanoma Classification Challenge hosted on Kaggle~\cite{international2020siim}.
    %
    \item The Edinburgh Dermofit Library~\cite{ballerini2013color} is a collection of focal high quality skin lesion images collected under standardised conditions with internal colour standards. The lesions span across ten different classes including melanomas, seborrhoeic keratosis and basal cell carcinomas.
    %
    \item MED-NODE~\cite{giotis2015med} contains melanoma and nevus images from the digital image archive of the Department of Dermatology of the University Medical Center Groningen (UMCG) in Netherlands.
    %
    \item SD-datasets (SD-198~\cite{sun2016benchmark} and SD-260~\cite{yang2019self}) contains clinical images collected by digital cameras and mobile phones.
    %
    \item PAD-UFES-20 dataset~\cite{pacheco2020pad} consists of samples of six different types of skin lesions. Each sample consists of a clinical image and clinical features including skin lesion location and Fitzpatrick skin type.
    %
    \item Fitzpatrick 17K dataset~\cite{groh2021evaluating} contains clinical images sourced from two online open-source dermatology atlases: DermaAmin~\cite{dermaamin} and Atlas Dermatologico~\cite{dermatologico}.   
    %
    \item MIT dataset~\cite{soenksen2021using} contains wide-field images extracted from open-access dermatology repositories, web scraping outputs, and de-identified clinical images from the hospital Gregorio Mara\~{n}\'on (Madrid, Spain). We adopted the clinical images provided as supplementary material.
    %
    \item DermNet NZ~\cite{dermnet} is supported and contributed by New Zealand Dermatologists on behalf of the New Zealand Dermatological Society Incorporated. 
    %
    \item 3DBodyTex~\cite{saint20183dbodytex,saint2019bodyfitr} consists of 400 high resolution 3D texture scans of 100 male and 100 female subjects captured in two different poses. Some pigmented lesions are visible on the skin.
\end{itemize}

\begin{figure}[t!]
\begin{center}
\includegraphics[width=0.8\linewidth]{Images/Fig6.png}
\end{center}
\vspace{-10pt}
   \caption{
   Selected samples of typical whole-body photography included in the skin image analysis dataset. Images are from our dataset and available in~\cite{rotemberg2021patient,soenksen2021using,dermnet}
   }
\label{fig:dataset-collected}
\vspace{-4pt}
\end{figure}


To improve the performance of the detector, we also adopted GAN-based data augmentation for skin lesion analysis~\cite{bissoto2021gan}. 
To generate synthetic skin lesion images, we adapted the conditional GAN (cGAN) proposed in~\cite{pix2pix2017} which has been successfully used in many cross-domain learning tasks.
Selected samples of raw images that were used to train the skin lesion detector are illustrated in Figure~\ref{fig:dataset-collected}.


\subsubsection{Whole body image captured from participants}
Using the proposed imaging system described in Subsection~\ref{sec:acquisition}, we collected images from participants to test the 3DSkin-mapper workflow.
With a single trigger, sixty images were captured by the 60 cameras from different angles simultaneously. These images represent different views of the human body, and were used to reconstruct a 3D human avatar for visualisation, and 3D mapping of skin lesions for computer-aided skin cancer diagnosis. Selected views of a participant are illustrated in Fig.~\ref{fig:participant_data}.



\begin{figure}[t!]
\begin{center}
\includegraphics[width=1\linewidth]{Images/Fig7-min.png}
\end{center}
\vspace{-14pt}
   \caption{Sample of collected data from a participant with the proposed whole-body imaging system. Selected poles from the imaging system are shown as orange circles.
   }
\label{fig:participant_data}
\vspace{-8pt}
\end{figure}


\subsection{3D human reconstruction}
\label{sec:3Dreconstruction}


\subsubsection{Whole body image rendering from 3D human models}
The viewing angles, positions and field of view (FoV), determined by camera focal length) of captured images could significantly affect the quality of the 3D reconstructed human avatar. Therefore, the angles and positions of the cameras need to be adjusted as much as the setup discussed in Subsection~\ref{sec:acquisition} allows until the best possible quality of images is achieved. To save time, instead of testing different camera positions on the camera rig, we simulated this process by generated synthetic images using computer graphic rendering with similar configuration as our camera rig. For every camera configuration, a set of 60 synthetic images are rendered from an existing 3D human model. The images are processed to reconstruct a 3D human model which is visually inspected for quality. 

There are two conflicting parameters to compromise: i) a wider camera field of view provides a more complete 3D model and more image overlapping between nearby views provides better estimations of camera poses; ii) image resolution for accurate depth estimation for 3D reconstruction.
A camera pose represents the 3D position and orientation of the camera that captures an image. Incorrect camera poses as well as less view overlapping lead to noisy depth and a low-quality 3D reconstructed model.
In practice, we mainly try to make sure all cameras are detected and optimised, and minimise the missing part of the output 3D model, particularly the head, shoulder, hands and feet.

Blender~\cite{Blender}, a popular 3D computer graphics open-source software with a Python API, was firstly used to render a 3D human model in a controlled environment. However, the rendering time was too long and repeated texture of the environment could cause errors in camera pose estimation during 3D reconstruction.
Thus, we adopt Pyrender~\cite{Pyrender}, a light-weight Python package, to render the 3D model only without background scene. This can speed up the rendering while producing good quality rendered images for testing the proposed processing pipeline.



\subsubsection{Image acquisition system evaluation}
To reconstruct a complete and accurate 3D human model, we use optimal positions and angles of the cameras from the virtual cameras used to render synthetic images. 
Besides the attention to the camera positions, angles and FoV, camera settings including ISO (that is, the gain of the camera sensor), exposure time, aperture and focus also affect the quality of the images. 
Higher gain leads to a brighter but also noisier image. Exposure time represent the amount of light captured by the camera sensor. Longer exposure time leads to a brighter image, but subject to motion blurs. Too high ISO or too long exposure time also lead to image saturation, where image details are lost in too bright image regions. As a result, a balance between ISO and exposure time is determined to obtain the best image quality for a given scene and lighting conditions. As depth of focus is finite and therefore only part of the object is in focus, the focus of the camera lens needs to be adjusted to maximise the image sharpness of the body part to be reconstructed. Bright, no saturation, low noise, and sharp images will lead to more successful 3D reconstruction.

By optimising these camera internal settings, we gradually improve the quality of the 3D reconstructed models from real images. Multi-view stereo reconstruction (MVS) successfully detected the camera poses of all 60 images to create a 3D human model as shown in Figure \ref{fig:meshroom_vs_capturereality} by two different software solutions. Although the head of the person is incomplete in regions where no lesions are detectable (hair / scalp), the 3D reconstructed mesh is good enough for skin analysis.

\begin{figure}[t!]
    \centering
    \includegraphics[width=0.75\linewidth]{Images/Fig8.png}
    \caption{Meshes reconstructed by Meshroom (left) and Reality Capture (right).}
    \label{fig:meshroom_vs_capturereality}
    \vspace{-6pt}
\end{figure}

\subsubsection{3D reconstruction approach}
3D avatars of participants are generated from 3D total body photography for lesion identification with deep neural networks.
For this purpose, two 3D reconstruction software packages were evaluated:
i) Meshroom \cite{Meshroom, griwodz2021alicevision} is a powerful open-source software package that provides very flexible environment for customisation running on Linux and Windows environments; 
ii) Reality Capture \cite{RealityCapture} is a well-know commercial software which provides simple and high-resolution 3D reconstruction running on Windows environment only.
Fig.~\ref{fig:meshroom_vs_capturereality} illustrates the 3D meshes reconstructed using Meshroom and Reality Capture, respectively. 

Both software packages produce decent quality meshes, however missing a portion of the top of the head. Reality Capture clearly produces a mesh with higher resolution with slightly more completeness. However, Reality Capture was not able to detect camera poses of four images and excluded them, while Meshroom could detect camera poses for all 60 images. Due to the flexibility and open-source nature of Meshroom, we adopted this tool in the 3DSkin-mapper workflow. For this purpose, we customised a 3D reconstruction pipeline for the implementation of ground plane detection, automatic cropping, scaling and alignment as follows:


\begin{figure}[t!]
    \centering
    \includegraphics[width=1\linewidth]{Images/Fig9.png}
    \caption{Meshroom's standard 3D reconstruction pipeline with solid arrows. The custom reconstruction pipeline is shown by the dotted arrows to align, crop and scale the 3D mesh.}
    \label{fig:pipelines}
    \vspace{-6pt}
\end{figure}


\begin{enumerate}
    \item \textit{Standard reconstruction pipeline}. Such a pipeline is illustrated in Fig.~\ref{fig:pipelines} following the solid arrows. The main steps are structure-from-motion, depth map estimation, meshing and texturing. 
    
    \item \textit{Custom reconstruction pipeline}. The custom 3D reconstruction adds new functionalities including finding the ground (by plane fitting to the reconstructed point cloud), the stand and the scale (by circle fitting) and the transformation of the reconstructed body (by setting the centre and normal vector of the stand to be the world centre and vertical axis respectively).
    %
    Fig.~\ref{fig:overlay} shows the output of structure-from-motion (left), and the filtered non-textured mesh (right) produced by the standard reconstruction. Notice that the non-textured mesh contains the body standing on the floor and the curtain around, and these are removed in the textured mesh of the person.
    %
    The 3D origin is on the ground between the feet, the ground normal represents the y-axis, and the height of the body matches with the physical height of the person. The stand provides the scaling factor from the known diameter, and the origin from the fitted circle center. Such a proposed pipeline is shown as dotted arrows in Fig.~\ref{fig:pipelines}.
    %
    Qualitative results of the 3D whole body skin surface scans and the 3D model with the customised pipeline is depicted in Fig.~\ref{fig:3D_collection}.
\end{enumerate}


\begin{figure}[t!]
    \centering
    \includegraphics[width=1\linewidth]{Images/Fig10.png}    
    \caption{Camera poses and a sparse point cloud estimated (left), and reconstructed and filtered mesh (right) of the whole scene. The finale aligned, cropped, scale and textured mesh of the person is shown as a small upright coloured 3D model near the middle. The final 3D model is overlaid to together with the filtered mesh to show the differences in size, scale and orientation.}
    \label{fig:overlay}
    \vspace{-6pt}
\end{figure}


\subsection{3D Data pre-processing}
\label{sec:preprocessor}

Given the reconstructed 3D model and camera intrinsic and extrinsic parameters, a \textit{preprocessor} renders the 3D models to get the corresponding depth image to each captured image, and a mask indicating which pixels are associated with the subject for each camera view (see Fig.~\ref{fig:preprocessor}). 2D to 3D projection can then be obtained from the depth image and camera parameters. 
The process of 2D to 3D mapping of skin lesions is performed as follows:

\begin{enumerate}
    \item The model view projection (MVP) matrix is calculated from the estimated camera intrinsics / extrinsics
    \item A scene is constructed consisting of the reconstructed 3D mesh and a single ambient light source. 
    \item The scene is rendered using the Pyrender (OpenGL) library to produce a colour and depth image from the camera's viewpoint.
    \item For each pixel in the image, a 3D ray is calculated from the MVP matrix. The ray is combined with its associated depth value to produce the 3D coordinates associated with each pixel.
    \item Every pixel which has an associated 3D coordinate within the predefined capture region (cylinder with origin in the centre) is marked as containing the subject. 
\end{enumerate}


\begin{figure}[t!]
    \centering
    \includegraphics[width=1\linewidth]{Images/Fig11.png}    
    \caption{The input view of preprocessor (left), rendered 3D mesh (middle left), depth image (middle right) and subject mask (right).}
    \label{fig:preprocessor}
    \vspace{-4pt}
\end{figure}

A small mismatch is present between the input camera view and the rendered outputs becoming more pronounced near the edges of the images. This is caused by the estimated camera model not accounting for distortion in the camera lens. Such condition has no effect in this application because the skin lesions are detected using the rendered 2D images.


\subsection{Skin lesion detection}
\label{sec:lesion-detector}

Human skin lesions are not homogeneous, which include structures such as pathological and benign melanocytic lesions, wounds and scars. To identify such lesions, we propose a method based on deep learning object detection as such methods have been useful for practical real-world skin lesion observations~\cite{soenksen2021using}. These models indicate the location and dimension (height and width) within the 2D domain (2D images collected with the whole body imaging system). Then, these detected skin lesions can be mapped back to the 3D surface of the subject, with the associated 3D coordinate of every pixel estimated with the preprocessor described in Subsection~\ref{sec:preprocessor}.

Object detectors have achieved exceptional performance in recent years, thanks to advances in deep convolutional neural networks. The model designs are broadly organised in two categories: one-stage and two-stage detectors.
One-stage detection frameworks, use a single network stage to perform classification and bounding box regression. Two-stage detection frameworks, on the other hand, includes a pre-processing step for generating object proposals to identify the object class and to regress an improved bounding box. 

\begin{figure*}[!t]
\begin{center}
\includegraphics[width=1\linewidth]{Images/Fig12.png}
\end{center}
\vspace{-12pt}
   \caption{Structure of object detectors, highlighting the main components of the Backbone, Neck and Head, and how information flows between these components. Image recreated from~\cite{bochkovskiy2020yolov4}.
   }
\vspace{-4pt}
\label{fig:overview_det}
\end{figure*}

Variants of models are based on changes on the main components of the standard architecture of an object detector. The structure for two-stage and one-stage object detectors are illustrated in Fig.~\ref{fig:overview_det}.
Common components that may be changed are: a \textit{backbone} used as feature extractor, a \textit{neck} used to extract different feature maps from different stages of the backbone, and a \textit{head} which is responsible for the detection of bounding boxes. Such a head can be a dense prediction (one-stage) or a sparse prediction (two-stage)~\cite{liu2020deep,sharma2020comprehensive}.

We opt to evaluate the performance of one-stage detectors as they are better suited and highly viable for real-world practical applications due to the faster inference. 
Among well-known one-stage object detectors, the scaling cross stage partial network (\textit{Scaled-YOLOv4})~\cite{wang2021scaled} from the YOLO family of detectors (You Only Look Once~\cite{redmon2016you}), can achieve an optimal trade-off between speed and accuracy. Another important reason that this model was adopted is that it can be used for commercial purposes.

Scaled-YOLOv4 makes an incremental improvement to YOLOv4~\cite{bochkovskiy2020yolov4} with a network scaling approach that modifies not only the network depth, width, and resolution; but also the structure of the network. 
The model is composed of an optimised CSPDarknet53 as a backbone, which employs a Darknet-53~\cite{redmon2018yolov3} and a CSPNet strategy~\cite{wang2019cspnet} to partition the feature map of the base layer into two parts and then merges them through a cross-stage hierarchy. A spatial pyramid pooling (SPP)~\cite{he2015spatial} additional module and a path aggregation network (PANet)~\cite{liu2018path} with CSP are used as neck. Finally, a YOLOv3~\cite{redmon2018yolov3} is used as the head.

With the refined whole body imaging system, a set of images and their corresponding camera poses are estimated to test the skin lesion detector and 2D-3D mapping. 
The 3D reconstruction from synthetic images provides the estimated camera poses, depth and mask of each input image. These are used to compute the 3D position of each detected lesion from a 2D image onto the 3D model for management and longitudinal tracking.
We map each centre point of the predicted 2D bounding box coordinates to the 3D space. As one lesion can appear multiple times across multiple images, skin lesions that have similar 3D positions on the 3D model are grouped as a specific global lesion ID. For this purpose, a thresholded distance and an agglomerative clustering technique of all 3D lesions are computed. 
Fig.~\ref{fig:global_ID} illustrates the different views of two selected skin lesions and their representations in the 3D space. 

\begin{figure*}[!t]
\begin{center}
\includegraphics[width=0.85\linewidth]{Images/Fig13.png}
\end{center}
\vspace{-12pt}
   \caption{Visualisation of two selected skin lesions and their different views from each camera. Lesions visible from poles C, B, A, O, N, and M.
   }
\label{fig:global_ID}
\vspace{-6pt}
\end{figure*}


\subsection{3DSkin-mapper UI}
\label{sec:ui}

We developed a custom user interface (UI) to visualise our results. The system centres around multiple linked views of the captured and processed data. Such interface is illustrated in Fig.~\ref{fig:qualitative_results_interface}.

A thumbnail grid view of the images that were captured in the imaging system (See Fig.~\ref{fig:participant_data}) appears at the bottom of the central column of the interface. Users can filter the visible thumbnails based on the area and pole position the images were captured from (\textit{e.g.} Front view: Poles C, B, A, O and N). 
The selected image appears in the central top view of the interface and allows the users to zoom and pan around the image for more detailed inspection. The rectangular region of the detected features can be shown as an overlay on the central primary image. An individual detection can be selected within the bounding box region of the detection on the central primary image.


\begin{figure}[!t]
\begin{center}
\includegraphics[width=0.96\linewidth]{Images/Fig14.png}
\end{center}
\vspace{-12pt}
   \caption{Description of the main components in the interface for data management.
   }
\label{fig:qualitative_results_interface}
\vspace{-6pt}
\end{figure}


The left side of the interface shows the reconstructed textured 3D model (See Subsection~\ref{sec:3Dreconstruction}). The view of the 3D model can be altered with the mouse, allowing the model to be rotated around roughly the centre of the model (mid-abdomen), as well as manipulating the camera around the model (pan / tilt as well as moving the camera in/out). There are also user interface options to align the 3D camera view to a fixed predefined position pointing towards the model, as well as moving the model back to the centre of the viewport (while maintaining the current camera position). The model appears with textures by default, but the textures can be turned off to allow more detailed inspection of the 3D mesh without the added visual cues from the textures, which mask some underlying 3D model imperfections.

The right panel highlights the active detection (top-right) showing either the raw image pixels of the detection enlarged with smoothing optionally applied to facilitate clinical use cases. Alternate feature detection crops from the current image appear in the ``Current Image Detections'' image grid. Finally, the ``Current Detection Metadata'' region allows the current selected feature detection to be annotated with metadata (\textit{e.g.} generic textural data that can be expanded to more rich metadata fields).
Further, we allow for curation of the dataset by annotating misclassifications with the ``Remove Detection'' button (which are then removed from visualisation on the user interface). Selecting the active detection (either by selecting within the bounding box region of detection directly from the central primary image or the ``Current Image Detections box'') updates the selection across all views (highlighting on the 3D model, primary image, enlarged image crop and selected current image detection and metadata). 
When the actively selected detection is changed, the virtual camera viewing the 3D model can optionally be moved to a fixed offset from the 3D point on the mesh surface of the model to point directly at the computed 3D points of the detection (focusing the 3D view on the actively selected detection). This is achieved by offsetting the camera a fixed distance from the computed 3D coordinate of the detection on the model surface along the computed normal vector at this location. 

The UI was developed so that it was agnostic to the underlying model generation, skin lesion detection and geometric transformation computations. This completely decouples the data processing from the user interface presentation, by using directories of flat files as the transport mechanism between these separate systems. This means that the UI is a modular product that could be reused for a different application by simply feeding in the appropriate model, images, and detection metadata.

The UI was developed using QT via PySide2 bindings from Python version 5.15.2. The UI takes as input the 60 images saved from the DSLR cameras, the 3D model (stored in an .obj file with an associated single texture image) and a JSON data file which contains the information on the detections (\textit{e.g.} unique identifier, 3D coordinate location and 3D normal direction). Linkage of the data in the JSON file to the underlying image in which the detection was found can be made via filename correspondence (prefixed the same - different file extension) or a metadata field which stores the original image which the detections in the file came from.



\section{Results and discussion} 

\subsection{3D model reconstruction}

Validation of 3D reconstruction accuracy is performed by computing the Hausdorff distance (using MeshLab~\cite{cignoni2008meshlab}) of 3D reconstructed meshes relative to the groundtruth 3D meshes. Hausdorff distance is defined as the largest distance between two meshes. So when comparing two meshes, large Hausdorff distances indicate missing features in the reconstruction results. Once the Hausdorff distance was computed and normalised by the diagonal distance of the 3D mesh bounding box, it is visualised as a heatmap on the ground truth meshes.

The results and their comparison to the ground truth 3D model are shown in Fig.~\ref{fig:comparison_groundtruth_reconstruction}. Significant errors occur in the regions of underarm and upper shoulder due to limited image coverage there.



\subsection{Lesion detection}

\subsubsection{Evaluation protocol and experimental setup}

The reliability of lesion detection is evaluated by comparing the obtained detection results with ground truth images manually annotated with 2D bounding boxes. A scale-invariant feature transformation (SIFT) is used first to detect all blob-like regions (group of connected pixels in an image that share some common property) to accelerate the annotation process. Lesions that occupy an area of less than $25 \times 25$ pixels (approximately $\leq$ 2 mm diameter) are not included, as these lesions cannot be reliably detected with the current quality of the wide-field images. To estimate the diameter, Otsu thresholding is used.
Difficulty in obtaining high enough resolution images for automated detection of small lesions, has also led to issues in obtaining accurately labelled lesions in the training images.

We use the normalised intersection-over-union (IoU) metric to define how well a bounding box overlaps the associated ground truth. We adopt the mAP@IoU[0.5], where the mean average precision requires that the bounding boxes overlap with an IoU $\ge$ 0.5. We also report recall and precision. Given a predefined confidence threshold, precision defines the number of true positives produced relative to the total of the true positive plus false positive, and recall defines how many of the ground truth instances are correctly classified relative to the total number of ground truth instances.
We calculated each metric for all whole body images and report the average over all images.


\begin{figure}[!t]
\begin{center}
\includegraphics[width=1\linewidth]{Images/Fig15.png}
\end{center}
\vspace{-9pt}
   \caption{Comparison between the ground-truth human model (left) with the reconstructed model (middle). The reconstruction error in terms of Hausdorff distance is shown as colour or map and histogram (right). We can see some missing parts on the hands and underarms, and the average error is around 2 mm. 
   }
\label{fig:comparison_groundtruth_reconstruction}
\vspace{-6pt}
\end{figure}


For the training and testing of the skin lesion detector, the whole body images are split into patches of $608 \times 608$ pixels with a 50\% overlap. The detections are then aggregated across all patches and non-max suppression (soft-NMS~\cite{bodla2017soft}) is performed to obtain the final set of lesion detections for the entire input image.

The skin lesion detector is trained using the collected dataset described in Subsection~\ref{sec:dataset-skin} (whole body images spanning various skin-types and body locations) with random split into training (60\%), validation (20\%), and testing (20\%) sets. The model is trained with Mosaic data augmentation, batch size 64, learning rate 1$e^{-3}$, and implemented with the Darknet deep learning framework available in~\cite{darknet-ab}.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=0.9\linewidth]{Images/Fig16.png}
\end{center}
\vspace{-9pt}
   \caption{Qualitative results of the skin lesion detector on clinical images introduced in Subsection~\ref{sec:dataset-skin}.
   }
\label{fig:qualitative_results_skin detector}
\vspace{-6pt}
\end{figure}


\subsubsection{Detecting lesions on 2D images and 3D mapping}

The trained model reaches an mAP@IoU[0.5] of over 92\% with a precision of 0.79 and a recall of 0.88 which is an accurate overlap of bounding boxes by current standards in the literature. Fig.~\ref{fig:qualitative_results_skin detector} illustrates some qualitative results of the skin lesion detection. Our algorithm automatically detects skin lesions larger than around 2.5 mm diameter and provides an opportunity to perform quick screening for skin lesions in whole body images. This is especially useful for patients with many lesions.

As described in Section~\ref{sec:lesion-detector}, the system categorised the same skin lesion detected in multiple 2D images into a unique 3D global lesion ID. This functionality allows the user to visualise the lesion from different perspectives, which would allow operators to overcome a potential shortcoming from any single image (such as a partial occlusion). Fig.~\ref{fig:qualitative_results_3D map} shows the lesions detected on 2D images, as well as the mapping of these lesions onto the 3D model reconstructed.
Limitations of our model include the difficulties to filter the lesion according to more advanced and robust geometrical parameters. By incorporating an additional data-driven segmentation module (lesion / background segmentation) for each lesion contained in a bounding box, the system will be able to rank all the lesions with such conditions. The correct diameter estimation is especially important when comparing the images captured over a period of time. A significant increase in the diameter of a nevus between consecutive examinations is a strong indicator that the melanocytic nevi can be potentially dangerous.


\begin{figure}[!t]
\begin{center}
\includegraphics[width=0.8\linewidth]{Images/Fig17-min.png}
\end{center}
\vspace{-9pt}
   \caption{Visualisation of the selected skin lesions detected in the 2D image (left) and mapped back to the 3D surface of the subject (right). To facilitate visualisation of the 3D model, half of the human body is represented without texture.
   }
\label{fig:qualitative_results_3D map}
\vspace{-6pt}
\end{figure}



\subsubsection{Tracking of skin lesion across temporal scans}
For the purpose of longitudinal tracking of skin lesions (\textit{i.e.} to detect new lesions and temporal changes of prior lesions), a patient can be scanned at different times to form a set of 3D meshes. Thus, variations in the pose across scans are required for the matching of skin lesions.
For this purpose, we first determine anatomical correspondence across 3D human meshes, then, we establish correspondence between the lesions of any two meshes~\cite{bogo2014automated,korotkov2018improved} (\textit{e.g.} we map the centre point of the predicted 2D bounding box coordinates to the corresponding temporal adjacent 3D mesh).  
The 3D information is used for correspondence matching as the visual appearance of the lesions change over time in different scans, which is a limitation of early methods based on only 2D anatomical landmarks and structured graphical models~\cite{mirzaalian2016skin}.


\begin{figure}[!t]
\begin{center}
\includegraphics[width=0.9\linewidth]{Images/Fig18.png}
\end{center}
\vspace{-13pt}
   \caption{Skin lesion matching across scans with different poses.
   Top: Skin lesion detection in the 2D image (t=15 days).
   Bottom: 3D meshes and longitudinal tracking of selected skin lesions across different sessions (t=0; t=15 days; t=30 days). The coloured line between each lesion indicates the unique lesion correspondence across different images.
   }
\label{fig:skin_tracking}
\vspace{-6pt}
\end{figure}


Learning-based deformation methods can be exploited for 3D correspondence between two 3D meshes~\cite{deprelle2019learning}. This means that captured skin textures are in accurate alignment across scans, facilitating the detection of changing lesions~\cite{zhao2021detection}.
We adopted the trained network LoopReg~\cite{bhatnagar2020loopreg} on the Faust dataset~\cite{bogo2014faust} (which contains around 100 scans of undressed people) for correspondence prediction of vertices across scans of the same subject. Such a model does not require instance specific annotations, supports self-supervision and does not require multiple initialisation in comparison with baseline models such as 3D-CODED~\cite{groueix20183d}.
Finally, with the vertices of the registered meshes, we solve the matching problem based on 3D geodesic distance between pairs of lesions, where the corresponding lesion will be located at the more proximate anatomical location of the target mesh.
In this experimental phase, newly-appearing and disappearing skin lesions were not possible to evaluate.  
Fig.~\ref{fig:skin_tracking} shows qualitative results of the longitudinal testing.



3D whole body imaging is notably important in the analytical assessment of individual characteristics by evaluating individual lesions relative to their overall skin lesion ecosystem. Such technology is useful to track lesions of patients with many skin lesions, where diagnostic biopsy of every lesion is unfeasible.
Our low-cost imaging system provides a promising tool to capture whole body images, detect and document lesions. This can potentially save lots of time of skin cancer specialists. 3D imaging can facilitate long-term surveillance by monitoring the entire skin surface rather than individual lesions.




\section{Conclusion}
\label{conclusion}

In this study, we introduced a 3D whole body imaging system and workflow for multiple view image capturing, 3D human body reconstruction, 3D skin lesion localisation and documentation. A software tool has been designed for data curation and management. We have also developed an algorithm to quickly identify skin lesions in whole body images. Software functions are developed based on open sources such as digiCamControl, Meshroom, Darknet, and QT via PySide2 for camera control, 3D human body reconstruction, lesion detection, and user interface development, respectively.
Future work will involve further evaluation of the workflow against a larger image set of expertly labelled images.


\bibliographystyle{splncs04}
