\section{Introduction}

\iffalse


In this paper, we study the problem of compressing an image into a series of abstract compact interpretable, conception-oriented high-level representations from visual data.
Different from traditional compression methods, the proposed conceptual compression concentrates on highly compact visual information instead of simply reducing signal-level data redundancy.
The concept of conceptual compression was first introduced in~\cite{gregor2016towards}, the progressively detailed representations are less interpretable, analyzable or editable, which limits its capability for downstream applications such as image processing and machine vision tasks. 

Intuitively, different global conceptual aspects of visual components (\textit{e.g.}, structure, texture, semantics) compose the meaningful images.
However, the underlying implication of the conceptual representation remains ambiguous, which lacks feasibility in explicitly preserving concerned information.

\fi

The human visual system (HVS)~\cite{kruger2012deep} perceives visual contents by processing and integrating manifold information into abstract high-level concepts (\textit{e.g.}, structure, texture, semantics), which form the basis for subsequent cognitive process~\cite{zhang2020connecting}. 
From the perspective of machine vision, high-level visual concepts also play a more important role in the practical applications than signal-level pixels.
Existing compression methods, including traditional block-based (\textit{e.g.},~JPEG~\cite{pennebaker1992jpeg} and HEVC~\cite{sullivan2012overview}) and deep learning based methods~\cite{balle2019end,minnen2018joint}, mainly focus on the modeling and removal of signal-level redundancy, while the potential and versatility of accomplishing compression task through decomposing visual data into compact conceptual components still lack further exploration.
Following the insight of Marr~\cite{marr1982computational} and Guo~\textit{et al.}~\cite{guo2007primal}, visual objects usually appear as structures and textures.
As two primal visual components, structure and texture not only play a deterministic role in visual content synthesis, but also are critical visual feature descriptors in various analysis-based tasks. 
In particular, due to the lack of explicit modeling for structure, existing compression methods often introduce severe distortions in decoded images under low bit-rate scenarios, such as ringing/blocking artifacts and blurring edges, which not only degrade human perception, but also hamper the performance of visual analysis tasks. 
Moreover, the encoded bitstreams are less correlated to visual concepts, resulting in the difficulty of utilizing the encoded information for subsequent manipulation tasks, such as image synthesis, shape modification and content recreation.

Deep generative models, such as variational auto-encoders (VAE)~\cite{kingma2014auto} and generative adversarial networks (GANs)~\cite{goodfellow2014generative}, have offered a new approach for conceptualizing images with compact latent representations, where the decoding process is implemented in a generative fashion. 
However, existing researches on conceptual compression~\cite{gregor2016towards}\cite{santurkar2018generative} often attempt to capture image contents with a single latent vector, with different conceptual components entangled together. In consequence, the encoded conceptual representation is less interpretable and editable, limiting
its potential towards downstream image processing and machine vision tasks, such as target-guided content manipulation~\cite{wang2018high}. Apparently, it is desirable to develop visual components disentangled representations for a clearer understanding and more flexible control of image contents.











In this work, we propose a novel conceptual compression framework. In this framework, images are encoded into compact structure and texture representations and decoded in a deep synthesis fashion to achieve better visual reconstruction quality, flexible content manipulation and potential support for various analysis tasks. 
More specifically, we propose to process images by a dual-layered model consisting of two complementary visual features: 1) structure layer represented by edges since edges depict key structure information of images, and 2) texture layer which represents non-structural content including texture, color, luminance. 
As shown in Fig.~\ref{fig:framework}, at the encoder side, the structural maps are extracted first and then compressed to bitstream. 
Without complex texture, structural maps are more sparse and compression-friendly.
Meanwhile, the highly compact deep texture representations are extracted from input images with the variational auto-encoder in the form of low-dimensional latent variables and then scaled, quantized and entropy coded, leading to a bits-saving stream.
At the decoder side, a hierachical fusion GAN (HF-GAN) is proposed to integrate texture layer and structure layer to synthesize images after reconstructing texture representations and structural maps.
Apart from the image compression task, the texture pattern and synthesis paradigm are well jointly learned due to the fact deep generative models tend to capture unique statistical features. 
Thus, compared to encoding image into both a raw data layer and a specific feature layer separately~\cite{akbari2019dsslic,ma2018joint},
the proposed framework realizes the unification of visual features and basis data identity in the cross-modality sense, satisfying the demand of machine and human vision with one unified conceptual bitstream.  

Our methods are evaluated upon diverse images of fashion items~\cite{yu2014fine,zhu2016generative}, faces~\cite{karras2018progressive} and natural scenes, achieving significant perceptual gain against traditional block-based and deep learning-based end-to-end image compression frameworks. 
Furthermore, the versatility of the proposed conceptual coding scheme are verified upon a wide range of image processing tasks, including content manipulation, texture synthesis and face detection.
\begin{figure*}[t]
    \centering
    \vspace{-6mm}
    \includegraphics[width=1.0\linewidth]{./Image/Framework.pdf}
    \vspace{-6mm}
    \caption{\textbf{Overview of proposed conceptual compression framework.} The framework consists of a structure layer and a texture layer. On the encoder side, structural maps and texture representations are extracted from input images using edge detector and texture extractor (VAE) respectively. Both two layers are compressed into bitstreams and reconstructed with specific codecs individually. A proposed generator integrates structure and texture to synthesize target images on the decoder side.}
    \label{fig:framework}
    \vspace{-6mm}
\end{figure*}

Our contributions can be summarized as follows:
\begin{itemize}
    \item We propose a novel compression framework which encodes by abstracting visual data into compact structure and texture representations and decodes by deep synthesis processing to produce cross-modal unification of visual features and basis data.
   
   
    \item We propose to realize conceptual compression via extracting sparse structural maps and deep texture representations, and an HF-GAN is proposed as decoder to reconstruct images of high visual quality from texture and structure.  
   
    \item The superiority of the proposed framework in image compression and vision tasks is justified and thoroughly analyzed via extensive experiments.
   
\end{itemize}

The rest of the paper is organized as follows. The related work and technologies are introduced in Section \ref{related work}. The proposed framework and networks are presented in Section \ref{proposed method}. Section \ref{experients} shows the detailed experimental results and analyses, and Section~\ref{conclusion} concludes the paper and discuss directions for future research.

\section{Related works}
\label{related work}
\subsection{Traditional Image Compression}
Traditional image compression technologies have played a fundamental role in visual communication and image processing, bringing up a series of crafted image codecs, such as JPEG~\cite{pennebaker1992jpeg}, JPEG2000~\cite{rabbani2002jpeg2000} and H.265/HEVC-based BPG~\cite{bellard2015bpg}.
JPEG is the most popular and widely used compression standard, which integrates well-known technologies including block-partition, discrete cosine transform (DCT), quantization  and entropy coding.
JPEG2000~\cite{rabbani2002jpeg2000} applies discrete wavelet transform (DWT) technology instead of DCT, achieving higher compression ratio and allowing various editing or processing applications.
After decades of development, a series of block-based hybrid prediction/transform coding standards (\textit{e.g.}, MPEG-4 AVC/H.264~\cite{wiegand2003overview}, AVS~\cite{gao2014overview} and HEVC~\cite{sullivan2012overview}), are built to significantly improve the coding efficiency by reducing pixel-level redundancy.
Different from JPEG, HEVC utilizes more intra prediction modes from neighboring reconstructed blocks in spatial domain to remove redundancy. 
\begin{figure*}[t]
    \centering
    \vspace{-6mm}
    \includegraphics[width=0.95\linewidth]{./Image/Generator.pdf}
    \vspace{-6mm}
    \caption{\textbf{Architecture of hierachical fusion generator networks.} Generator consists of fully connected layers (FC), Residual blocks (Resblock), RGB transformation module (To RGB), upsampling and cumulatively summing module. Structural maps are resized and concatenated to feature maps as input for corresponding Resblock.}
    \label{fig:networks}
    \vspace{-6mm}
\end{figure*}
\subsection{Deep Learning-based Image Compression}
Recent years have witnessed a surge of interest in learning-based image compression, which benefits from large-scale data, effective network architectures, end-to-end optimization, and other advanced techniques such as generative models and unsupervised learning. 
Typical deep image compression systems usually adopt an end-to-end framework~\cite{ma2019image} to model the pixel distribution and jointly optimize rate-distortion performance with reconstruction tasks, outperforming popular block-based codecs, such as JPEG, JPEG2000 and HEVC. 
Pixel probability modeling~\cite{salimans2017pixelcnn++} and auto-encoder~\cite{balle2019end,minnen2018joint} are two main approaches in deep learning based image coding schemes, as pointed in~\cite{liu2020deep}.
Meanwhile, multiple deep structures, such as CNN~\cite{balle2019end}, RNN~\cite{toderici2015variable} and GAN~\cite{santurkar2018generative,agustsson2019generative}, are utilized to explore efficient compression architectures.
Despite the constant improvement of compression performance, most deep learning-based end-to-end compression algorithms mainly aim to model signal-level correlations, while the visual concepts are less explored.

In addition to rate-distortion optimization, subsequent analysis tasks, such as image retrieval~\cite{zhang2017deep} and semantic analysis~\cite{luo2018deepsic,torfason2018towards}, have also been incorporated into learning-based compression frameworks to encourage analysis-friendly signal representation. 
However, extra task-related networks are adopted in their frameworks for joint image analysis and compression, potentially limiting the application to specific tasks. 
\subsection{Conceptual Compression}
Conceptual compression~\cite{gregor2016towards, chang2019layered} aims to encode images into compact, high-level interpretable representations for reconstruction, allowing a more efficient and analysis-friendly compression architecture. 
Gregor \textit{et al.}~\cite{gregor2016towards} introduce convolutional Deep Recurrent Attentive Writer (DRAW), which extends VAE by using RNNs as encoder and decoder, to transform an image into a series of increasingly detailed representations. 
However, the models in~\cite{gregor2016towards} are only verified on datasets of small resolutions.
Hu \textit{et al.}~\cite{hu2020towards} compress images into compact structure and color representations with generative models, where the representations are characterized with edge maps and reference sample pixels near edges specifically.
Despite that the structure representation can be well analyzed, the support for vision tasks, such as image synthesis and modification, are limited by pixel-level color representations. 
The semantic guidance is used to assist in reconstructing images from compact compressed visual data in~\cite{agustsson2019generative,akbari2019dsslic}, where the main data stream is still signal-oriented though.


\section{Conceptual image compression framework}
\label{proposed method}
Our goal is to explore the potential of performing compression via decomposing visual data into compact visual components and develop a set of feasible examplar scheme. 
In particular, we propose to compress images by extracting the compact representations of two primal complementary visual features, structure and texture.
In the proposed framework, the images ${I}$ are represented with compact texture representations $I^t$ and sparse structural maps $I^s$,~\textit{i.e.}, $I \to I^s + I^t$.
As shown in Fig.~\ref{fig:framework}, images are decomposed into texture layer and structure layer and compressed separately on the encoder side. 
We obtain the sparse structure maps $I^s$ with the edges extractor $\mathit{Enc^s}$ (Section~\ref{structure layer compression}) and low-dimensional texture representations $I^t$ with variational texture auto-encoder $\mathit{{Enc}^t}$ (Section~\ref{texture layer compression}), \textit{i.e.}, $I^s=Enc^s(I)$, $I^t=Enc^t(I)$. 
Both texture representations and sparse structural maps are further compressed to bitstreams for transmission using existing standard codec. 
On the decoder side, a hierarchical fusion generator $Gen$ is designed to reconstruct the high quality images $\hat{I}$ from decoded components, texture $\hat{I}^t$ and structure $\hat{I}^s$,~\textit{i.e.}, $\hat{I}=Gen(\hat{I}^t,\hat{I}^s)$ (Section~\ref{image synthesis}).
To achieve reconstruction of high visual quality and fidelity, we take advantage of both adversarial training and multi-perspective distortion measurement to supervise the reconstruction process.
In particular, we also propose a latent regression loss to better learn the paradigm of texture extraction and synthesis (Section~\ref{loss objectives}).
\subsection{Structure Layer Compression}
\label{structure layer compression}
Considering that edges are one of the most sparse and abstract image representations and can depict the key structure information of images, edge maps are extracted as the structure layer representation via widely used edge detection methods, such as holistically-nested edge detection (HED)~\cite{xie2015holistically} and Canny edge detection~\cite{ding2001canny}.
On account of the sparsity and binarization of structural maps, we employ the Lanczos downsampling algorithm to further reduce data volume of structure layer with 4-scale.
Furthermore, screen content coding~\cite{6783726} is adopted to compress the structure maps into bitstreams since this framework has strong capability of compressing images with abundant sharp edges.

At the decoder side, the process is reversed to recover the structural maps before synthesizing images.
To regain the structural maps of original resolution, we upsample the decoded low resolution structure maps using one of the state-of-the-art super-resolution methods, deep back-projection networks (DBPN)~\cite{haris2018deep}.
The DBPN model is optimized with the mean square error (MSE) in~\cite{haris2018deep} which is not sensitive to the fluctuation of sparse binary data, leading to the significant distortion in reconstructed edges. 
To improve the restoration performance on the datasets which are characterized by sparse and binary edges, the binary cross entropy (BCE) is employed to replace the MSE loss for the super resolution model training,
\begin{equation}
\begin{split}
\mathcal{L}_{BCE}=&-\sum_u \sum_v [I_{(u,v)}^s\log (\hat{I}^s_{(u,v)})\\&+(1-I_{(u,v)}^s)\log(1-\hat{I}^s_{(u,v)})],
\end{split}
\end{equation}
where $I_{(u,v)}^s$ represents the true pixel value of structural map $I^s$ at position $(u,v)$, while $\hat{I}^s_{(u,v)}$ represents the predicted value of upsampled structural map $\hat{I}^s$ at position $(u,v)$.
\subsection{Texture Layer Compression}
\label{texture layer compression}
As shown in Fig.~\ref{fig:framework}, the compact texture representations are extracted with a neural network-based encoder and integrated with the reconstructed structural maps to synthesize target images by an elaborate generator.
The encoder $Enc^t$ and generator $Gen$ are jointly trained in an end-to-end manner.
In this section, we will introduce the process of texture extraction and compression.

The image texture extractor is designed based on variational auto-encoder for conceptual level information extraction~\cite{gregor2016towards}.
In particular, the encoder $Enc^t$ is constructed with several residual blocks~\cite{he2016deep} and convolutional layers to model the input image $I$ into a posterior multivariate Gaussian distribution $q_{\phi}(I^t|I)$. 
Then the representation $I^t$ is generated by sampling from the posterior distribution $q_{\phi}(I^t|I)$ using re-parameterization~\cite{kingma2014auto} method, \textit{i.e.}, $I^t \sim q_{\phi}(I^t|I)$.

Subsequently, the extracted texture discrete representation $I^t$ is further compressed through scalar quantization and entropy coding. We follow the quantization method defined in HEVC~\cite{sullivan2012overview}, where the quantization step $Q_{step}$ is determined by the quantization parameter ($QP$) and we take the factor $s=2^{10}$ empirically:
\begin{equation}
    Q_{step}=\frac{2^\frac{QP-4}{6}}{s}=2^{\frac{QP-4}{6}-10}.
    \label{quantization}
\end{equation}
The quantization is performed by,
\begin{equation}
    I^{qt}=floor\left(\frac{I^t}{Q_{step}}\right),
    \label{floor}
\end{equation}
where representation $I^t$ is quantized to $I^{qt}$ and $floor(\cdot)$ represents the truncation operation for designated binary digit precision.
Finally the quantized texture representations are encoded with Arithmetic codec~\cite{witten1987arithmetic} for transmission. 
The decoded texture representations for image synthesis can be denoted by $\hat{I}^t$, where $\hat{I}^t=I^{qt}*Q_{step}$.
\begin{figure}[t]
    \centering
    \vspace{-4mm}
    \includegraphics[width=1.0\linewidth]{./Image/Resblock.pdf}
    \vspace{-8mm}
    \caption{\textbf{Basic fusion residual block in generator.} Resblock input ($h \times w \times (n+1)$) consists of feature maps ($h \times w \times n$) and a structural map ($h \times w \times 1$). Texture representation ($d \times 1$) is transformed into two groups of affined parameters: $\alpha$ ($(n+1)\times 1$) and $\beta$ ($(n+1) \times 1$). Structural maps and texture representations are fused through parameterized normalization in each subblock.}
    \label{fig:resblock}
    \vspace{-5mm}
\end{figure}

\subsection{Image Synthesis}
\label{image synthesis}
Motivated by the strong capability of the instance normalization in style transfer~\cite{karras2020analyzing}, we adopt the adaptive instance normalization~\cite{huang2017arbitrary} to integrate texture layer and structure layer. 
Specifically, a group of modulation parameters, including mean and variance, are learned from texture representations.
The feature statistics of the feature maps are transferred with learned modulation parameters during adaptive affine transformations, where feature maps contain both structure information and previously generated texture information in forward propagation as shown in Fig.~\ref{fig:resblock}.
The particular channel-wise modulation parameters are learned corresponding to each intermediate feature map. 
In this manner, the structure and texture are fused and integrated gradually through convolutional operation and affine transformation. 
Additionally, inspired by impressive generation capability of progressive growing GAN~\cite{karras2018progressive}, we devise a generator which progressively increases the resolution of synthesized feature maps as shown in Fig.~\ref{fig:networks}.  
The generator consists of a pile of residual blocks as basic units and takes advantage of skip connections and hierarchical fusion. 
Each residual block includes three fully convolutional layers followed by adaptive instance normalization operation. The structural maps are concatenated to the feature maps as new input for each residual block. 
Finally, the target images are obtained by upsampling and summing the contributions of RGB outputs corresponding to different resolutions as~\cite{karras2020analyzing}.
The formulation of the hierarchical fusion process is presented as follows. 

Given the input image $I \subseteq \mathbb{R}^{H\times W \times 3}$, we can obtain the structure map $I^s \subseteq \mathbb{R}^{H\times W \times 3}$ and the texture representation $I^t \subseteq \mathbb{R}^{d \times 1} $, where $H,W$ represent the size of input image and $d$ denotes the dimension of texture representation. 
Let the initial block of the generator function $\mathcal{G}_0$ be defined as $\mathcal{G}_0:I^s \mapsto A_0$ such that $A_0 \subseteq \mathbb{R}^{2\times 2 \times c_0}$ and $c_0$ denotes the number of feature channels. 
We define the intermediate output of the $i$-th residual block as $A_i$, and $i \in \mathbb{N}$. The structure map $I^s$ is resized to $I^s_i$ which has the same size as $A_i$. Thus, the input of each residual block is defined as:
\begin{equation}
    \bar{A}_i=[A_i;I^s_i],
\end{equation}
where [;] is a channel-wise concatenation operation. Let $\mathcal{G}_i$ be a generic function which acts as the basic generator block: $\mathcal{G}_i: \bar{A}_{i-1} \mapsto A_i$, where $A_i \in \mathbb{R}^{2^{i+1}\times 2^{i+1}\times c_i}$ and $c_i$ is the number of channels in the intermediate activations of the $i$-th generator block. Then $A_k$ can be obtained by the general formula composed with a sequence of $\mathcal{G}$ functions, where $k \in \mathbb{N}$:
\begin{equation}
\begin{aligned}
    A_k = & \mathcal{G}_k(\cdot,I^s_k) \circ \mathcal{G}_{k-1}(\cdot,I^s_{k-1}) \circ \cdots \mathcal{G}_i(\cdot,I^s_i)\\
    & \circ \cdots \mathcal{G}_1(\cdot,I^s_1) \circ \mathcal{G}_0(I^s).
\end{aligned}
\end{equation}

We employ skip connections and multi-scale fusion to improve reconstruction quality of the generator. $up^{(2)}$ is defined as the upsampling function by 2-scale and $conv^{(3\times 3)}$ simply serves as a $3 \times3 $ convolution function which converts the output feature map $A_i$ of each block into RGB images: $conv^{(3\times 3)}_i:A_i \mapsto B_i$, where $B_i\subseteq \mathbb{R}^{2^{i+1}\times 2^{i+1}\times 3}$. Hence, the RGB output of initial convolutional layer $x_0$ is:
\begin{equation}
    x_0= conv^{(3\times 3)}_0(A_0)=conv^{(3\times 3)}_0(\mathcal{G}_0(I^s)).
\end{equation}
The RGB output of the $i$-th intermediate block can be computed with the recurrence formula as following:
\begin{equation}
    x_i=conv^{(3\times 3)}_i(A_i)+up^{(2)}(x_{i-1}), i\geq1.
\end{equation}
Therefore, the final synthesized image from the proposed generator consisting of $7$ blocks is $x_7$.

\begin{figure*}[t]
    \centering
    \vspace{-11mm}
   
    \includegraphics[width=0.9\linewidth]{./Image/Loss_v2.pdf}
    \vspace{-4mm}
    \caption{\textbf{Loss functions of end-to-end VAE-HFGAN.} During training, various losses are adopted: for adversarial training, a discriminator is adopted to discriminate real labels and fake labels; to reduce the distortion of reconstructed images, we apply self reconstruction $L_1$ loss, structure similarity loss $SSIM$ and perceptual $L_1$ loss which measure the distance between features encoded from \textit{VGG} networks; to align texture representations with a prior Gaussian distribution, a $KL$ loss is incorporated; to enforce the connection between paired texture representations and images, the latent regression $L_1$ loss is employed to measure the difference between texture representations extracted from original image and reconstructed image. }
    \label{fig:loss}
    \vspace{-5mm}
\end{figure*}
In the basic residual block of the generator shown in Fig.~\ref{fig:resblock}, there are three convolutional layers whose scale and bias are modulated by three groups of parameters learned from texture representations by three fully connected layers respectively. 
The feature maps at the site of $j$-th convolutional layer of $i$-th basic block is denoted by $A_{ij}$. Receieving the texture representation $I^t$, a fully connected function $f_{ij}:I^t \mapsto {\alpha_{ij},\beta_{ij}}$ first maps $I^t$ into affine parameters, where $\alpha_{ij},\beta_{ij} \subseteq \mathbb{R}^{1\times 1 \times c_i}$. Thus, we adaptively normalize the previous activations $A_{ij}$ with the learnable texture affine parameters:
\begin{equation}
    AdaIN(A_{ij},I^t)=\alpha_{ij} \frac{A_{ij}-\mu(A_{ij})}{\sigma(A_{ij})}+\beta_{ij},
\end{equation}
where the normalized feature maps are scaled with $\alpha_{ij}$, and shifted with $\beta_{ij}$ in the channel-wise manner, and $\mu(A_{ij})$ and $\sigma(A_{ij})$ are the means and standard deviations of the activations in $A_{ij}$.
By transferring the feature statistics from texture representations and concatenating the structural maps with intermediate generated content, structure and texture are effectively integrated to reconstruct images of which texture are progressively enriched.
Moreover, during the training of reconstruction task, the mapping and synthesis paradigm between deep latent space and spatially-aware texture content are jointly well learned and stored as deep prior in the designed generative model, allowing the flexible examplar-based image content manipulation in the compressed domain.
   
   
\subsection{Loss Objectives}
\label{loss objectives}
The proposed texture encoder and the image generator are jointly trained in an end-to-end manner with a multi-scale discriminator. 
The image compression and reconstruction tasks are mainly optimized on three type of losses as shown in Fig.~\ref{fig:loss}: \textit{\textbf{reconstruction loss}} which aims to improve reconstruction visual quality and fidelity, \textit{\textbf{prior divergence}} which provides a prior distribution instruction for deep texture representation, and the proposed \textit{\textbf{latent regression loss}} which constraints the mapping between deep latent space and synthsis texture content.

Compared to typical learned lossy compression methods~\cite{balle2019end,minnen2018joint} which only optimize image reconstruction quality with pixel-wise similarity metric, we introduce diverse distortion metrics to supervise reconstruction task from low level pixel-wise loss to high level perceptual loss and adversarial loss. The~\textbf{reconstruction loss} is introduced as follows:
\begin{itemize}
    \item Self-reconstruction loss: the pixel-wise loss $\mathcal{L}{^{rec}_{1}}$ is imposed to force the visual components to complete reconstruct the original images: 
    \begin{equation}
    \mathcal{L}{^{rec}_{1}} = \mathbb{E}_{I\sim p(I), {I^t} \sim p(I^t)}\left\|{I-Gen(\hat{I}^t,\hat{I}^s))} \right \|_1.
    \end{equation}
    \item Structure similarity loss: the SSIM~\cite{wang2004image} loss is incorporated to supervise the optimization process of improving structural fidelity: 
    \begin{equation}
    \mathcal{L}_{SSIM}=\mathbf{SSIM}(I,Gen(\hat{I}^t, \hat{I}^s)).
    \end{equation}
    \item Perceptual loss: we use the perceptual loss $\mathcal{L}_{vgg}$ in~\cite{johnson2016perceptual} to encourage the perceptual fidelity through deep feature matching from pre-trained VGG-16 netoworks~\cite{simonyan2014very}.
   
   
   
   
   
   
    \item Adversarial loss: we apply the discriminator $Dis$ and follow the variant of conditional adversarial training scheme~\cite{mao2017least} to encourage the visual realism of reconstructed images:
    \begin{multline}
         \mathcal{L}{^{GAN}_{G}} = \frac{1}{2}\mathbb{E}_{I\sim p(I)}||1-Dis(I,I^s)||_2 \\ +
         \frac{1}{2}\mathbb{E}_{I\sim p(I), I^t \sim p(I^t))}||Dis(Gen(\hat{I}^t, \hat{I}^{s}),I^s)||_2,
    \end{multline}
    \begin{multline}
        \mathcal{L}^{GAN}_{D} = -\mathbb{E}_{I\sim p(I), I^t \sim p(I^t))} \\ || Dis(Gen(\hat{I}^t, \hat{I}^{s}),I^s)||_2.
    \end{multline}
\end{itemize}

Additionally, to obtain meaningful texture representations by stochastic sampling and benefit entropy coding process, we introduce the \textbf{prior divergence} to enforce the distribution of extracted texture representation to be close to the prior Gaussian distribution,~\textit{i.e.} $p_{\theta}(I^t)\sim\mathcal{N}(0,\mathbf{I})$ and $\mathbf{I}$ denotes the identity matrix:
\begin{equation}
\mathcal{L}_{KL} = \mathbb{E}_{I\sim p(I)}[\mathcal{D}_{KL}(q_{\phi}(I^t|I)\|p_{\theta}(I^t))].
\end{equation}

Furthermore, we propose a \textbf{latent regression loss} to further constrain the bidirectional mapping between the learned texture representation ${I^t}$ and the synthesis texture content $\hat{I}$.
More specifically, the texture encoder $Enc^t$ is utilized to analyze and extract the texture representation $\hat{I}^t_{est}$ of the generated image $\hat{I}$.
Given the fact that the reconstructed images $\hat{I}$ are trained to maintain texture fidelity, the texture representations which are extracted from original images and reconstructed images should be encouraged to be consistent,~\textit{i.e.}~$I^t = Enc^t(I) \approx \hat{I}^t_{est} = Enc^t(\hat{I})$.
Thus, the latent regression loss $\mathcal{L}{^{latent}_1}$ is proposed to minimize the $L_1$ distance between two texture representations as follows,
\begin{equation}
    \mathcal{L}{^{latent}_1}=E{_{I^t \sim p(I^t),\hat{I}^t_{est} \sim p({\hat{I}^t_{est})}}} \|{I^t-\hat{I}^t_{est}} \|_1.
\label{con:latent}
\end{equation}
The unique association between specific representation and corresponding texture are further reinforced by the latent regression loss, which not only helps learn a continuous latent space for texture expression in the data-driven manner, but also benefits the learning of synthesis paradigm. 

Above all, the final training objective is shown as following:
\begin{multline}
\label{loss_G}
   
    \mathcal{L}_{G,D,E}= \lambda_{GAN} \mathcal{L}{^{GAN}_{G}}+\lambda_{ rec}\mathcal{L}_{1}^{rec}+
    \lambda_{vgg}\mathcal{L}_{vgg}
    \\+\lambda_{SSIM}\mathcal{L}_{SSIM}+ +\lambda_{KL}\mathcal{L}{_{KL}} +\lambda_{ latent}\mathcal{L}_{1}^{latent},
\end{multline}
where the hyper-parameters $\lambda_{name}$ weights each loss respectively.
\begin{figure*}[t]
    \centering
    \vspace{-6mm}
    \includegraphics[width=0.9\linewidth]{./Image/qualitative_results.pdf}
    \caption{\textbf{Qualitative comparisons with baselines under similar low bit-rate.} The \textbf{\textit{bpp}} of each image is reported under corresponding image. The results show that the proposed method can reconstruct high visual quality images and outperform other methods in regard to visual fidelity and aesthetic sensibility.} 
    \label{fig:qualitative}
    \vspace{-6mm}
\end{figure*}
\section{Experiments}
\label{experients}
\subsection{Implementation details}
We implement the proposed model\footnote{For reproducible research, the source codes of our method will be made public when this paper is accepted.} using PyTorch and trained on two NVIDIA Tesla V100 GPUs. 
For training, we use the Adam optimizer with exponential decay rates $(\beta_1,\beta_2)=(0.5,0.999)$. 
The batch size is set to 16 and the learning rate is set to 0.0002.
The training procedure follows the Least Squares GANs (LSGANs)~\cite{mao2017least}.
We adopt the following hyper-parameters in all experiments for the training: $\lambda_{GAN}=1.0, \lambda_{rec}=10.0, \lambda_{SSIM}=0.25, \lambda_{vgg}=0.2, \lambda_{latent}=1.0, \lambda_{KL}=0.01$. 
As for the size of texture representations, we find that the capacity and capability of texture representations are growing as size increases. Thus, factored by texture complexity of specific datasets, the best sizes vary for different datasets. 
In our experiments, the dimension of texture representations is empirically set to $d = 64$ across all datasets for comparison.  
\subsection{Datasets}
We conduct experiments on several datasets including edges2shoes~\cite{yu2014fine}, edges2handbags~\cite{zhu2016generative}, CelebA-HQ~\cite{karras2018progressive}, and the multiple seasons dataset. All images are resized to $256\times 256$ in the experiments.

\textbf{Edges2shoes and edges2handbags.}
We combine these two datasets for training due to their high content similarity.
It contains $188,392$ paired training images and $400$ images for testing.
The images of edges are utilized for providing structural information.

\textbf{CelebA-HQ.} It includes $30000$ high-quality face images~\cite{karras2018progressive}.
We split $29800$ as training set and $200$ as testing set.
For structural maps, we employ facial landmark detection to obtain contours in the facial region, and the Canny edge detector~\cite{ding2001canny} to obtain structural edges in the background region.   

\textbf{Multiple seasons dataset. }
We collect $8000$ images from the Yosemite dataset~\cite{zhu2017unpaired} and the alps seasons dataset~\cite{anoosheh2018combogan}.
The dataset contains four seasons of images, and $400$ images are split for testing.
We use the Canny edge detector incorporated with the Gaussian blur algorithm to acquire the corresponding structural maps. 



\begin{figure*}[t]
    \centering
   
    \includegraphics[width=1.0\linewidth]{./Image/User_study.pdf}
    \vspace{-6mm}
    \caption{ \textbf{Fidelity and aesthetics preference results.} A user study is conducted, in which participants are asked to choose results which better match original images (Fidelity) and have better perceptual quality (Aesthetics) through pairwise comparisons. The numbers in histogram indicate the percentage of preference for corresponding comparison pair. The results show that the proposed method obtains the best preference ratio in each comparison pair for both fidelity and aesthetics.}
    \label{fig:user_study}
    \vspace{-4mm}
\end{figure*}
\begin{table*}[t]
    \centering
    \caption{Quantitative results on edges2shoes$\&$handbags, CelebA-HQ and multiple seasons datasets. $\downarrow$ means lower score is better.}
    \begin{tabular}{lcccccc}
    \toprule
          Datasets & \multicolumn{2}{c}{Edges2shoes$\&$handbags} & \multicolumn{2}{c}{CelebA-HQ} & \multicolumn{2}{c}{Multiple seasons} \\
         \hline
         & LPIPS$\downarrow$ & DISTS$\downarrow$ & LPIPS$\downarrow$ & DISTS$\downarrow$ & LPIPS$\downarrow$ & DISTS$\downarrow$ \\
         \hline
         JPEG & 0.274 & 0.326 & 0.492 & 0.400 & 0.517 & 0.392 \\
         JPEG2000 & 0.309 & 0.265 & 0.289 & 0.241 & 0.532 & 0.331 \\
         BPG & \textbf{0.110} & 0.169 & 0.180 & 0.187 & 0.371 & 0.261 \\
         Minnen \textit{et al.}~\cite{minnen2018joint} & 0.121 & 0.199 & 0.179 & 0.177 & 0.363 & 0.265 \\
       
         Proposed method & 0.118 & \textbf{0.163} & \textbf{0.164} & \textbf{0.149} & \textbf{0.354} & \textbf{0.215} \\ 
         \bottomrule
    \end{tabular}
    \label{tab:quatitative_results}
    \vspace{-4mm}
\end{table*}
\subsection{Compression Performance Comparison}
In this subsection, we conduct extensive qualitative and quantitative experiments to compare the compression performance with traditional and learning-based approaches. 
To compress structure and texture representation into bitstreams with high efficiency, we adopt different strategies for structure layer and texture layer compression.
In particular, the reference software of screen content coding~\cite{6783726} with $QP = 35$ is employed to compress the structural maps. 
Moreover, texture representations are quantized with $QP=51$ (Eq.~\ref{quantization}) and truncated with $16$ bits range (Eq.~\ref{floor}).
Then, the quantized results are encoded with lossless Arithmetic codec~\cite{witten1987arithmetic}.
Note that quantization loss does not cause visible degradation of reconstruction results according to experimental results.  

\subsubsection{Baselines}
We compare the proposed method with traditional compression standards and learning-based compression methods on the test datasets.
For traditional compression framework, we choose widely-used compression standards JEPG and JPEG2000, and current state-of-the-art HEVC-based image compression codec BPG for comparison.
For deep learning-based image compression systems adopting end-to-end optimization, the method in Minnen \textit{et al.}~\cite{minnen2018joint} is used for comparison. 
The specific settings are detailed as follows:
\begin{itemize}
    \item \textbf{JPEG}: we use JPEG Encoder of Matlab with quality factor $QF = 1$, which obtains maximum compression ratio of JPEG.
    \item \textbf{JPEG2000}: the images are compressed by OpenJPEG platform with quality parameter aligned with the compression ratio of proposed method, ranging from $200$ to $500$.
    \item \textbf{BPG}: we use standard bpg codec\footnote{https://bellard.org/bpg} with $QP$ ranging from $40$ to $51$ for compression ratio alignment.
    \item \textbf{Minnen \textit{et al.}~\cite{minnen2018joint}}: The deep compression network is trained exactly following the procedure in~\cite{minnen2018joint} with hyper prior. We set $\lambda=0.0025$ for obtaining average 0.16 bits per pixel (bpp), and $\lambda=0.005$ for average 0.2 bpp. 
   
\end{itemize}


\subsubsection{Qualitative Evaluations}
For the evaluation of visual reconstruction quality, we mainly focus on the global structural fidelity and aesthetic sensibility. 
Making comparisons at the same bitrate is difficult since most compression methods cannot generate a specified bitrate.
As such, we select comparison images with sizes that are as close as possible to our encoding bpp.
We present qualitative comparisons in Fig.~\ref{fig:qualitative}.
Overall, the results demonstrate that our method can produce reconstructed images of high visual quality under extreme low compression bitrate and outperform the baselines in regard to visual fidelity and aesthetic sensibility. 

In particular, the decoded images of JPEG and JPEG 2000 show serious structure and color distortion and blocking artifacts under extreme low bit-rate scenarios resulting from the limitation of block-wise processing.
In the mean time, BPG and Minnen \textit{et al.}~\cite{minnen2018joint} provide similar visual quality and blurring texture.
The results of BPG show more distortion and degradation in visual quality than which of ours, \textit{e.g.}, the face skin looks unnatural and partial key edges are missing in the second row. 
Meanwhile, the structure and texture are over smoothed and blurred in Minnen \textit{et al.}~\cite{minnen2018joint}, hence losing significant structural details and lessening sense of reality compared to our method, \textit{e.g.}, the teeth in the third row of Fig.~\ref{fig:qualitative}. 
Above all, compared to other compression frameworks, our proposed conceptual compression shows significant advantages in synthesizing sharp, realistic images while preserving high visual fidelity.
\begin{figure*}[t]
    \centering
    \vspace{-2mm}
    \includegraphics[width=1.0\linewidth]{./Image/texture_transfer.pdf}
    \caption{\textbf{Texture synthesis results.} With all accessible images serving as a texture source library, the proposed method is able to synthesize different texture by replacing the texture representation with ones encoded from images of desired texture styles, and generate creative and attractive images.}
    \label{fig:texture_transfer}
    \vspace{-4mm}
\end{figure*}
\subsubsection{Quantitative Evaluations}
Due to the ultimate arbiter of lossy compression remaining human evaluation, we choose LPIPS~\cite{zhang2018unreasonable}, DISTS~\cite{ding2020image} and user preference as perceptual quality metric to evaluate the fidelity quantitatively, which highly correlate with human quality judgements instead of only assessing signal fidelity.
Lower score of LPIPS and DISTS indicates higher visual fidelity of reconstructed images.
The average results of LPIPS and DISTS on each dataset from different comparison methods are shown in Table~\ref{tab:quatitative_results}. 
It should be noted that the bitrate of comparison images is strictly close or higher than ours.
It can be clearly seen that the proposed method almost excels all other comparison methods on DISTS and LPIPS in all scenes under similar bit-rate (except for one score of LPIPS on edges2shoes and edges2handbags testset). 

A user study of pairwise comparison is conducted to evaluate fidelity and aesthetics.
Given image pairs, users are asked to choose the one matching original image better (fidelity) in the first part of the survey, and the one of better visual quality (aesthetic) in the second part.
16 cases from test data of three datasets are selected to show for comparison.
A total of 46 subjects participate in the user study and a total of 2944 selections are tallied. 
The preference ratio in comparison between proposed method and each compared method is calculated for evaluation.
User preference results are shown in Fig.~\ref{fig:user_study}. 
The proposed conceptual compression method obtains the best preference ratio in each comparison pair for both fidelity and aesthetics.
The quantitative results fully verify the superiority of our method in obtaining better visual quality and maintaining higher structure and texture fidelity.
\subsection{Application in Image Manipulation}
In our conceptual compression framework, images are explicitly disentangled to structure domain and texture domain, and reconstructed through integrating texture and structure with generator.
The structure and texture representations act as both raw frame data and inter-operable and manipulable visual features.
During the data-driven training, a continuous bidirectional mapping between deep latent representations and spatially-aware texture is well learned, allowing generator to synthesize corresponding content from any representation in texture latent space.
Benefited from the disentanglement of visual components and learned synthesis paradigm, the generator is able to progressively render the texture following the instruction of any given structural maps. 
Thus, besides high efficiency compression, the proposed framework can be also applied in image manipulation tasks via editing structure and texture representations. 

In particular, we can synthesize new texture in the way of changing the texture representations while constraining specific structural maps as image content.
On the other hand, we can also modify the structural maps while maintaining the image texture to satisfy particular needs for image manipulation.
The powerful capability of manipulating images flexibly in compression domain demonstrates the versatility of our conceptual compression framework in supporting both compression tasks and efficient content manipulation in the compressed domain compared to normal compression methods.
\subsubsection{Texture Synthesis}
The results of the texture synthesis results are shown in Fig.~\ref{fig:texture_transfer}. 
Attributed to analyzing and compressing images into compact disentangled structure and texture representations, the proposed method is able to process images in two independent layer before decoding.
Due to the unique association between specific representation and global texture of image instance, the implication of texture representation can be confirmed and reflected in the source images and synthesized images.
Thus, with all accessible images serving as a texture source library, the proposed method can synthesize different texture by replacing the texture representation with ones encoded from images of desired texture styles, and construct creative and attractive images.
The texture synthesis results in Fig.~\ref{fig:texture_transfer} show that our model has strong capacity of effectively capturing image global texture distribution, integrating representation layers without relying on encoder, and synthesizing pleasing target images, which suggest a great potential of proposed method in support of joint image compression and vision tasks. 
\begin{figure}[t]
    \centering
    \vspace{-2mm}
    \includegraphics[width=1.0\linewidth]{./Image/shape_transfer.pdf}
    \caption{\textbf{Image synthesis results via modifying structural maps.} With the structural maps edited in optional ways, \textit{e.g.}, user interactive edition, image morphing algorithms, the texture layer can be flexibly rendered to fit the layout and shape of updated structural maps.}
    \label{fig:shape_transfer}
    \vspace{-4mm}
\end{figure}
\subsubsection{Structure Modification}
We also demonstrate example results of applying the proposed method to structure modification in Fig.~\ref{fig:shape_transfer}.
Under the layered framework, the shape of images can be modified through editing the structural maps in optional ways, \textit{e.g.}, user interactive edition, image morphing algorithms.
The texture layer can be flexibly rendered fitting the layout and shape of updated structural maps and the results show that original texture can perfectly fit new shapes in synthesized images.
Structure modification is very useful in visual communication and image-based retrieval applications.
Moreover, operating machine vision tasks in the compressed domain is shown to be more efficient than operating it after decoding under low bit-rate scenarios~\cite{ma2018joint}.
Our experimental results also prove the potential advantage of the proposed method in performing high efficiency coding and machine vision tasks jointly.
\subsection{Advantage in Image Analysis}
To further verify the advantage of our framework in image analysis task, we also perform the facial landmark detection task on the decoded images.
Facial landmark detection~\cite{kazemi2014one} is carried out on the original images in CelebA-HQ testset and the detection results are served as ground truth.
For comparison, we perform landmark detection on the decoded images from JPEG and the proposed method respectively.
Then the normalized root mean squared error (NRMSE) between the detection results on compressed images and original images is calculated as quantitative metric.
Fig.~\ref{fig:RMSE} illustrates the cumulative error distribution of our method and JPEG with quality factor $\left \{ 2,3,4 \right \}$, where about $97\%$ of the test images reconstructed by proposed method have minor errors less than $0.4$.
It should be noted that the average bit-rate of our method and JPEG under $QF=1$ are $0.099$ bpp and $0.237$ bpp respectively.
Our method can achieve $58.1\%$ bits saving and $56.5\%$ accuracy improving of landmark detection task compared to JPEG under $QF=1$.
Above all, the face landmark detection results demonstrate robustness and accuracy of the proposed image compression method in image analysis tasks. 

In essence, in the proposed implementation scheme, the structural maps already contain key facial edges which act like dense landmarks, allowing straightforward content analysis without decoding.
Moreover, the concrete form of the structure representation can be further adjusted towards specific vision applications, leading to a unified framework for connecting machine vision and human perception through inter-operable conceptual representations.
\begin{figure}[t]
    \centering
    \vspace{-4mm}
    \includegraphics[width=1.0\linewidth]{./Image/RMSE.png}
    \caption{\textbf{Cumulative error distribution of facial landmark detection with JPEG under quality factor ${2,3,4}$ and proposed method.} It can be seen from the figure that the decoded test images from the proposed method ($97\%$), JPEG with quality factor 4 ($88\%$), JPEG with quality factor 3 ($65\%$), JPEG with quality factor 2 ($49\%$) achieve errors of less than 0.4 respectively.}
   
    \label{fig:RMSE}
    \vspace{-4mm}
\end{figure}
\section{Conclusions and Future Works}
\label{conclusion}
This paper proposes a novel conceptual compression framework for efficient, interpretable and versatile visual data representation, leading to high efficiency compression, better visual reconstruction quality, increased flexibility in content manipulation and potential support for various vision tasks.
In particular, the proposed framework decomposes the image contents into structural and textural layers, and performs compression in two layers respectively.
To reconstruct the original image from the compressed layered features, a hierarchical fusion GAN is proposed to integrate texture layer and structure layer in a disentangled fashion.
Qualitative and qualitative results show that the proposed method can reconstruct high visual quality images with better structural fidelity and aesthetic sensibility.
The advantages of our conceptual compression framework are also verified in compression, content manipulation and  analysis tasks through extensive experiments.

Despite the current achievements, the conceptual compression will further benefit from the enhanced rate-distortion optimization schemes and better layered decomposition of image contents. Furthermore, extending the proposed conceptual image compression framework to video domain is another promising direction, where effective spatio-temporal representation learning plays a crucial part in a wide range of video analysis tasks, such as action recognition, tracking, summarization, and editing. 
\iffalse
\begin{itemize}
    \item \textit{Joint Rate-Distortion Optimization.} In this work, the trade-off between reconstruction quality and bitrate isn't explicitly optimized during training. Joint rate-distortion optimization is promising to further broaden the application of the proposed method.
    \item \textit{Separable Conceptual Compression for Video Compression.} Although training deep models on video data is more difficult than image data due to temporal correlation, it is meaning and necessary to extend separable conceptual compression to video compression for various applications.
    \item \textit{Multiple Layered Combination.} This paper implements the structure-texture separable conceptual compression as an example to prove the feasibility of separable conceptual compression. Multiple layered combination can play a significant role in the future study.
\end{itemize}
\fi
\bibliographystyle{IEEEbib}
