\section*{Acknowledgments}
\else
 
  \section*{Acknowledgment}
\fi


The authors would like to thank...



\section{\textbf{Introduction}}
\label{sec:intro}}
\IEEEPARstart{a}spect-based sentiment analysis (ABSA)~\cite{thet2010aspect, bing2012sentiment} has two important subtasks, i.e. aspect term extraction and aspect-level sentiment classification, which aim to extract opinionated aspect terms and determine the sentiment polarities of the aspect terms, respectively. 
Recently, End2end ABSA receives a lot of attention. It employs
a unified tagging schema to transduce the two subtasks into a sequence tagging problem. As shown in Fig.~\ref{fig:intro}, given a review ``\textit{The keyboard is good but the mouse is not very responsive.}'', the expected extracted aspect-sentiment pairs for End2end ABSA should be \{\textit{``keyboard'': ``positve''},\textit{``mouse'': ``negative''}\}. For the convenience of description, the ABSA mentioned below means End2End ABSA. Although 
the previous methods \cite{wang2018towards, li2019unified, luo2019doer} achieve good performance, they heavily rely on in-domain labeled data which are 
non-trivial to obtain for ABSA in new domains \cite{gong2020unified}. The lack of labeled data becomes the main obstacle for in-domain ABSA due to the high cost of manual annotation.

\begin{figure}[t]

\centering

\includegraphics[width=0.45\textwidth]{figures/intro1-cropped.pdf}

\caption{An illustration of the input and output for the ABSA task and its two fundamental subtasks including ATE and aspect-level sentiment classification.
The input ``Review'' refers to the review example in the top box. The terms highlighted in red indicate aspect terms in the review.}

\label{fig:intro}

\end{figure}

Domain adaptation \cite{DBLP:journals/tkde/PanY10} is a popular solution to address the data deficiency problem for ABSA, which aims to generalize the model trained on labeled data of source domain to a different target domain. Previous domain adaptation ABSA methods are mainly based on structure correspondence learning (SCL) \cite{blitzer2007biographies}.
They construct pivot-based auxiliary task in which a pivot classifier is trained to predict categories of pivot features such as dependency relations \cite{li2012cross, ding2017recurrent, wang2018recursive, gong2020unified} or aspect categories \cite{lekhtman2021dilbert}.
The auxiliary task is performed on data of both source domain and target domain to align feature representations of both domains.
\begin{figure}[t]

\centering

\includegraphics[width=0.5\textwidth]{figures/limitation1-cropped.pdf}

\caption{A comparison of previous methods and SDAM. ``Source domain'' and ``Target domain'' indicate
review texts from source domain and target domain, respectively.
``Pseudo instance'' indicates pseudo training instance.
 ``Predicted pivot feature'' in (a) indicates a category of pivot features, e.g., \textit{O}$\rightarrow$\textit{amod}$\rightarrow$\textit{A}. Blue lines represent the flow of aspect terms, with the flow of other tokens omitted.
Black dotted lines in (b) represent the generation of pseudo training instance, where \textit{``food''} in pseudo aspect term set is retrieved by syntactic structure similarity, then the aspect term \textit{``keyboard''} in source domain is replaced by \textit{``food''} to generate a pseudo training instance. } 

\label{fig:lim}

\end{figure}
However, there are two limitations in previous studies. (1) They cannot effectively transfer the learnt knowledge of sentiment prediction
from source domain to target domain. As shown in Fig.~\ref{fig:lim}(a), the previous methods only align
the feature representation of \textit{``keyboard''} and \textit{``food''} in the pivot-based auxiliary task according to the corresponding structure such as similar syntactic dependency relation \textit{O}$\rightarrow$\textit{amod}$\rightarrow$\textit{A}, but cannot recognize that they have different sentiment polarities, i.e., \textit{``keyboard''} is positive while \textit{``food''} is negative. As a result in the inference stage, the previous methods give a wrong prediction result for the aspect term \textit{``food''}.
(2) The results of sentiment prediction
may be inconsistent for different words in a multi-gram aspect term.
For example, given the review sentence \textit{``the battery life is excellent 6-7 hours without charging.''} in the laptop domain, the previous models trained on restaurant domain may not be able to predict consistent sentiment for the words in the aspect term \textit{``battery life''}.
The methods predict the sentiment polarities for the words
\textit{``battery''} and \textit{``life''} as positive and negative, respectively.
More examples can be found in Table~\ref{tab:case} in Section~\ref{sec:ana}.

In this paper,
to overcome aforementioned limitations, we propose a \textbf{S}yntax-guided \textbf{D}omain \textbf{A}daptation \textbf{M}odel for ABSA, namely SDAM\xspace. SDAM\xspace mainly includes three modules:
a syntax-based pseudo instance generation
module, a syntax-based masked language model module (SMLM), and a span-based joint aspect term extraction and sentiment analysis module. Specifically, to address the first limitation, we propose to construct pseudo training instances for learning the sentiment polarities of aspect terms for target domain. For the example
shown in Fig.~\ref{fig:lim}(b), SDAM\xspace first retrieves the candidate word \textit{``food''} in target domain based on its syntactic structure similarity with the aspect term \textit{``keyboard''} in source domain, and then generates a pseudo training instance by replacing the aspect term. Supervised learning method is employed to learn the relations about sentiment prediction between both domains.
To further capture domain-invariant features, we propose a novel variant of BERT mask language model (MLM) \cite{kenton2019bert} based on syntactic structure similarity, named SMLM, in which we modify the token masking strategy to render the pre-trained language model focus more on aspect terms.
For addressing the second limitation, the span-based joint aspect term extraction and sentiment analysis module is proposed to consider multi-gram aspect terms as a whole for sentiment polarity prediction, instead of performing sentiment analysis for each word separately.

Experimental results on five benchmark datasets show that SDAM\xspace improves at least 13.28\% on average compared to the-state-of-art baselines in Micro-F1 for cross-domain ABSA. We further carry out ablation studies to quantitatively measure the effectiveness of each module in our proposed SDAM\xspace.

The main contribution of this paper can be summarized as follows:

\begin{itemize}

\item We propose a novel syntax-guided domain adaptation model for more accurate aspect-based sentiment analysis, which consists of three modules: syntax-based pseudo instance generation module, syntax-based masked language model module and span-based joint aspect term and sentiment analysis module.

\item Extensive experiments have been conducted to verify the validity of the proposed model.




\end{itemize}

The rest of this paper is organized as follows. We present the background and related works of ABSA, domain adaptation and their crossover in Section \ref{sec:rel}. The technical details of our SDAM are presented in Section \ref{sec:met}. Then we present experimental settings in Section \ref{sec:exp}. We present experimental analysis include main results, parameter study, case study in Section \ref{sec:ana}.

\section{\textbf{Related Work}}
\label{sec:rel}


ABSA and domain adaptation are both popular research areas, there has not been much research on the intersection of them though.
We first describe in-domain ABSA, and then introduce
domain adaptation. Finally, we describe the prior research
at the intersection of both areas.

\subsection{Aspect-based Sentiment Analysis}
Apart from sentence-level sentiment analysis, ABSA also has been widely studied since it was proposed in 2004 year \cite{hu2004mininga, hu2004miningb}. 
The task consists of two subtasks including ATE and aspect-level sentiment classification.
ATE aims at
extracting opinionated entities from reviews. Early ATE studies mainly focus on applying feature engineering techniques to obtain linguistic-driven features. The release of SemEval datasets \cite{pontiki-etal-2014-semeval, pontiki2015semeval, pontiki2016semeval}, MAMS datasets \cite{jiang2019challenge} and the rise of deep learning have promoted
ABSA research.
Most deep learning-based studies utilize aspect boundary tags \{B,I,O\} to define ATE as a sequence labeling
task. Jakob and Gurevych \cite{jakob2010extracting} first apply Conditional Random Field (CRF) to extract aspect terms in the supervised way. Liu et al. \cite{liu2015fine} train recurrent neural networks (RNNs) with word embeddings and
outperform CRF models in the task. Wang et al. \cite{wang2016recursive} integrate RNNs and CRF into a unified framework for explicit aspect and opinion term co-extraction. Shu et al. \cite{shu2017lifelong} incorporate CRF into lifelong learning to utilize knowledge learnt from previous domains. Recent studies incorporate pre-trained language models into ATE and improve the performance further. Xu et al. \cite{xu2019bert} post-train pre-trained language model BERT on domain-specific data to enhance the representation, which shows that continuous pre-training on domain-specific unlabeled corpus is effective for ATE.

The aspect-level sentiment classification task aims at
determining the sentiment polarities of aspect terms in reviews. Various neural models have been proposed for this task in recent years. They focus on capturing the interaction between the aspect term and context
by utilizing neural architectures such as RNNs \cite{tang2016effective}, CNNs \cite{huangparameterized, li2018transformation}, attention-based networks \cite{wang2016attention}, memory networks \cite{tang2016aspect, chen2017recurrent} and graph neural networks \cite{xue2018aspect, wang2020relational}.

Rather than solving the two subtasks separately, many studies propose to tackle them together. However, simply merging the two subtasks in a pipeline
manner will lead to an error-propagation problem \cite{ma2018joint}. Several recent studies \cite{wang2018towards, li2019unified, luo2019doer} have made successful attempts to utilize a unified tagging scheme which combines the aspect boundary tags \{B,I,O\} and sentiment polarities \{POS,NEU,NEG\} to
formulate ABSA as a sequence labeling problem. Other studies utilize multi-task learning to jointly extract aspect terms and predict
sentiment polarities. Besides, span-based methods \cite{hu2019open, DBLP:conf/aaai/MaoSYC21} are proposed to tackle the sentiment inconsistency problem of in-domain ABSA.

\subsection{Domain Adaptation}
  Deep learning has produced state-of-the-art results in a lot of NLP tasks. While such approaches for supervised
learning have performed well, they heavily rely on labeled training data which are scare for some tasks or domains.
Unsupervised domain adaptation can handle situations where a neural network is trained on labeled data from a source domain and unlabeled data from a different target domain with the goal of performing well at test time on
the target domain \cite{wilson2020survey}. 

Unsupervised domain adaptation approaches include domain-invariant feature learning, instance re-weighting, and self-training. A feature representation is domain-invariant if the features follow the same distribution regardless of whether the input data are from the source or target domain \cite{zhao2019learning}. In this case, models that trained on labeled source domain may generalize well to the target domain since the feature of target domain matches the distribution. A variety of methods have been proposed to align the distribution. Minimizing the divergence that measures the distance between the distributions is the first to be proposed. The measures of distance mainly include maximum mean discrepancy (MMD) \cite{rozantsev2018beyond}, correlation alignment (CORAL) \cite{sun2016return} and their variants \cite{long2015learning, zhang2018unsupervised}. Adversarial learning also has been utilized to align the distribution. The fundamental idea is to make the domain classifier perform poorly when training the feature extractor. For example, DANN \cite{ganin2015unsupervised} maximally confuses the domain classifier by negating the gradient from the domain classifier with a gradient reversal layer. Besides, structure correspondence learning (SCL)  \cite{blitzer2006domain} is propose to learn domain-invariant feature representation from unlabeled data of both the source and target domains. It splits features into pivots features and non-pivot features. Pivot features are those who meet the following two criteria: (a) they appear frequently in both domains; and (b) they are highly correlated to the task labels. Non-pivot features are those features that do not meet at least one of the above criteria. SCL uses pivot features to learn a mapping from the original feature spaces of both domains to a shared feature space.
Self-training is another widely studied solution for domain adaptation which can directly learn concepts from target domain in a fully-automatic manner without any human intervention \cite{liu2012unsupervised}. The core idea is to generate pseudo-labels of unlabeled data, and jointly train the model with source labels and target pseudo-labels \cite{inoue2018cross, zou2019confidence}. 



\subsection{Domain Adaptation for ABSA}
Although domain adaptation and ABSA are both hot areas of study, there is not much crossover between them. Most domain adaptation ABSA methods are based on structure correspondence learning. Early studies \cite{jakob2010extracting, li2012cross, ding2017recurrent} use pre-defined dependency link rules as pivot features and use CRF or RNN to construct auxiliary task to predict categories of pivot features. Lekhtman et al. \cite{lekhtman2021dilbert} use aspect category information as pivot features and propose a combination of category-based masked language modeling task and an auxiliary task: category prediction. Their method achieve the state-of-art performance in cross-domain ATE. The rule-based methods heavily rely on prior knowledge and have poor extensibility when adapting to new domains, though. Wang et al. \cite{wang2018recursive} automatically generate dependency relations as the pivot features and incorporate an auto-encoder into the auxiliary task to group the relations into different clusters. Gong et al. \cite{gong2020unified} construct two kinds of auxiliary tasks using different
pivot features: POS tags and dependency relation. Besides they incorporate instance re-weighting to achieve instance domain adaptation. Their methods achieve the state-of-art performance in cross-domain ABSA. However, all these SCL-based methods only align feature representation of aspect term in different domain in pivot-based auxiliary task, and ignore the sentiment polarity. In addition, they may face
sentiment inconsistency issue in multi-gram aspect terms as discussed in section \ref{sec:intro}. In this work, we proposed a syntax-guide domain adaptation model to resolve the limitations.


\section{\textbf{Problem Formulation}}
\label{sec:pre}
User-item interactions can be represented by a bipartite graph ${\mathcal{G}_1=\{(u,r',i)|u\in\mathcal{U},i\in\mathcal{I}}\}$, where $r'$ generalizes all user behaviours, e,g., \textit{click, purchase}, as one relation type between user $u$ and item $i$. 
${\mathcal{U}}$ and ${\mathcal{I}}$ are the sets of users and items, respectively.
We use $y_{ui}=1$ to indicate the user $u$ has interacted with the item $i$, otherwise $y_{ui}=0$. 
We have the knowledge graph ${\mathcal{G}_2=\{(h,r,t)|h,t\in\mathcal{E},r\in\mathcal{R}\}}$ in which $\mathcal{E}$ and $\mathcal{R}$ are the collections of entities and relations.
Each knowledge triple $(h,r,t)$ denotes that there is a relation $r$ connecting from head entity $h$ to tail entity $t$. 
Similar to~\cite{2019_KGAT,2020_ckan}, each item can be matched with an entity in the knowledge graph (i.e., $\mathcal{I}\subseteq\mathcal{E}$). For ease of interpretation, we unify the user-item bipartite graph $\mathcal{G}_{1}$ and knowledge graph $\mathcal{G}_{2}$ into the \textit{collaborative knowledge graph} (CKG)  $\mathcal{G}=(\mathcal{E}^\prime, \mathcal{R}^\prime)$, where $\mathcal{E}^\prime=\mathcal{E} \cup \mathcal{U}$ and $\mathcal{R}^\prime=\mathcal{R}\cup\{r'\}$. 



\textbf{Notations.} We use bold lowercase, bold uppercase, and calligraphy letters for vectors, matrices, and sets, respectively. 
Non-bold ones are used to denote graph nodes or scalars.
We list the key notations
in Table \ref{tab:Symbol}.



\textbf{Task Formulation.}
Given the CKG $\mathcal{G}$, we aim to train a recommender model $\mathcal{F}(\cdot, \cdot|\Theta,\mathcal{G})$, with the capability of estimating  probability $\hat{y}_{ui}$ that user $u$ will engage with item $i$, i.e., $\hat{y}_{ui}=\mathcal{F}(u,i|\Theta,\mathcal{G})$, where $\Theta$ is the parameter set in $\mathcal{F}$.


\begin{table}[tbp]
\caption{{Symbols and Description.}}
  \label{tab:Symbol}
  \begin{center}
  \begin{tabular}{c|m{6.0cm}}
    \hline
    \textbf{Notation} & \textbf{Description} \\
    \hline
    \hline
    $\mathcal{U, I}$ & The sets of users, items. \\
    \hline
    $\mathcal{E, R}$ & The sets of entities, relations. \\
    \hline
    $r'$ & Generalized user behaviour type. \\
    \hline
    $(h,r,t)$ & The knowledge triple. \\
    \hline
    $\mathcal{G}_1, \mathcal{G}_2$ & The user-item bipartite graph and item-side knowledge graph. \\
    \hline
    $\mathcal{G}=(\mathcal{E}^\prime, \mathcal{R}^\prime)$ & The collaborative knowledge graph contains $\mathcal{G}_1$ and $\mathcal{G}_2$, with $\mathcal{E}^\prime=\mathcal{E}\cup\mathcal{U}$,  $\mathcal{R}^\prime=\mathcal{R}\cup\{r'\}$. \\
    \hline
    $o$ & The uniform placeholder for user symbol $u$ and item symbol $i$. \\
    \hline
    $\emb{{v}}_{o} \in \mathbb{R}^d$ & The embedding of $o$.\\
    \hline
    $\emb{{v}}_{o}^{k} \in \mathbb{R}^d$ & The knowledge-aware embedding of $o$ at hop $k$.\\
   
   
    \hline
    $\emb{{v}}_h, \emb{{v}}_t, \emb{{v}}_r \in \mathbb{R}^d$ & The embedding of head entity $h$, tail entity $t$, and relation $r$.\\
   
   
    \hline
    $\mathcal{E}_o^{k}$ & The $k$-hop entity-neighbor set of $o$. \\
    \hline
    $\mathcal{T}_o^{k}$ & The sampled $k$-hop triple-neighbor set of $o$. \\
   
   
    \hline
    $l$ & The neighbor sample size in interactive information encoding. \\
    \hline
    $K$ & The depth of knowledge extraction. \\
    \hline
    $\mathcal{S}$ & The initial seed pool. \\
    \hline
    $\{\emb{{t}}_{x}\}_{x=1}^{|\mathcal{S}|}$ & The embedding table. \\
    \hline
    $\emb{t}_o^{*}\in \mathbb{R}^{2d}$ & The feature referencing embedding of $o$. \\
    \hline
    $\emb{z}_{o} \in \{0,1\}^{|\mathcal{S}|}$ & The indicator for seed selection of $o$. \\
   
   
    \hline
    $\emb{v}_o^{+}\in \mathbb{R}^{2d}$ & The gated aggregation embedding of $o$. \\
   
   
   
   
   
    \hline

  \end{tabular}
  \end{center}
\end{table}














\section{\textbf{Methodology}}
\label{sec:met}

\subsection{Problem Statement}
Instead of formulating the cross-domain ABSA as a sequence labeling problem which may cause sentiment inconsistency problem, we propose to use the span-based labeling scheme \cite{hu2019open} as follows: given an input sentence
$X = \{x_1, x_2, ..., x_n\}$ with length $n$, and aspect term list
$A = \{a_1, a_2, ..., a_m\}$ where the number of aspect terms is
$m$ and each aspect term $a_i$ is annotated with its start position, its end position, and its sentiment polarity.
The goal is to find all aspect terms from the sentence as well as predict their polarities. In the cross-domain setting, labeled data are only available in source domain. Given a set of labeled data from source domain $D^S = \{(X^S_i, A^S_i)\}^{N_S}_{i=1}$ and a set of unlabeled data from target domain $D^U = \{(X^U_i)\}^{N_U}_{i=1}$, our goal is to predict
aspect terms and corresponding sentiment polarities for unseen target test dataset in target domain $D^T = \{(X^T_i)\}^{N_T}_{i=1}$.



\subsection{Syntactic Structure Similarity}
Although aspect terms may differ from domains, their syntactic roles are generally the same \cite{chen2021bridge}. We propose to capture aspect terms in target domain based on the
syntactic structure similarity to aspect terms in source domain. Part of speech (POS) and syntactic dependency relations are chosen to measure the syntactic structure similarity between tokens. For the example in Fig.~\ref{fig:ss}, the aspect terms ``\textit{food}'' and ``\textit{screen}'' have the same POS tag ``\textit{NN}'' and dependency relations ``\{\textit{det}, \textit{nsubj}\}'', indicating that
these words are similar in syntax. Follow Chen's setting \cite{chen2021bridge}, to measure the similarity, for each word $x_i$, we use one-hot vector $b^{pos}_i \in R^{N_{pos}}$ and multi-hot vector $b^{dep}_i \in R^{N_{dep}}$ to represent its POS tag and syntactic dependency relations, respectively, where $N_{pos}$/$N_{dep}$ indicate the size of POS tag/syntactic dependency relation set. 
To compute $b^{dep}_i$, we merge all relations involving the token $x_i$. In order to get the average syntactic structure representation of aspect terms $\bar{a} = (b^{pos}, b^{dep})$, we propose to aggregate all aspect terms in source domain:
${b^{pos}} = \sum_{i}^{A}\frac{b^{pos}_i}{N_A}$ and ${b^{dep}} = \sum_{i}^{A}\frac{b^{dep}_i}{N_A}$, where $A$ is the set of aspect terms and $N_A$ is the size of $A$. The syntactic structure similarity between a token $x_i$ and $\bar{a}$ is defined as: 
\begin{equation}\label{equ:1}
S_{sim}(x_i, \bar{a}) = \cos(b^{pos}_i, b^{pos}) \times cos(b^{dep}_i, b^{dep})
\end{equation}
where $\cos(,)$ is the cosine similarity.

\begin{figure}[t]

\centering

\includegraphics[width=0.45\textwidth]{figures/DT.pdf}

\caption{Dependency tree of reviews from laptop and restaurant domain. The brown ovals
indicate that aspect terms from the two domains have the same dependency relation ``\{\textit{det}, \textit{nsubj}\}'', and the brown boxes indicate that aspect terms from the two domains have the same POS tag ``\textit{NN}''.}

\label{fig:ss}

\end{figure}

\subsection{Syntax-based Pseudo Instance Generation}
\begin{figure*}[t]

\centering

\includegraphics[width=0.95\textwidth]{figures/sst.pdf}

\caption{Overall process of syntax-based pseudo instance generation. For each instance in source training dataset, we create one pseudo training instance. The red circled words ``LCD screen'' indicate a labeled aspect term in the instance of source domain.}

\label{fig:sst}

\end{figure*}

In order to exploit the sentiment information in source domain for facilitating the prediction in target domain,
we propose
to build pseudo training instances based syntactic structure similarities. In this way, we can apply supervised learning in pseudo training instances to learn corresponding sentiment polarities of aspect terms for target domain. When constructing pseudo training datasets, we first construct one-gram pseudo aspect term set $\mathcal{A}_1$ and N-gram pseudo aspect term set $\mathcal{A}_N$ from target domain. The elements in $\mathcal{A}_1$ and $\mathcal{A}_N$ are similar to aspect terms of source domain in terms of syntactic structure. Then we replace the aspect terms in source domain with the pseudo aspect terms in set $\mathcal{A}_1$ and $\mathcal{A}_N$ to create pseudo training instances. The overall process is shown in Fig.~\ref{fig:sst}, with the algorithm illustrated in Algorithm \ref{alg:Framwork}.


Concretely, words and phrases which present higher similarities to the average aspect term representation $\bar{a}$ are selected as candidates for composing the one-gram pseudo aspect term set $\mathcal{A}_1$ and N-gram pseudo aspect term set $\mathcal{A}_N$, respectively (Line 2$\sim$16). All the selected terms are ranked according to the term frequencies, and the most frequent ones compose the final pseudo aspect term sets Lines (17$\sim$18).
Then each aspect term in
source domain is randomly replaced by one pseudo aspect term
in $\mathcal{A}_1$ or $\mathcal{A}_N$ (Lines 19$\sim$29). For each original instance, we create one pseudo instance. In pseudo training dataset, the original sentiment information of the review text
is preserved, so that the sentiment knowledge can be exploited for the prediction in target domain.

\begin{algorithm}[htb]
  \caption{Syntax-based pseudo instances generation}
  \label{alg:Framwork}
  \begin{algorithmic}[1]
    \Require
        
    \Statex  The set of labeled source domain: $D^S$;
    \Statex  The set of unlabeled target domain: $D^U$;
    \Statex  One-gram syntax-similarity threshold: $\sigma_1$;
    \Statex  N-gram syntax-similarity threshold: $\sigma_N$;
    \Statex  Maximum size of $\mathcal{A}_1$: $\beta_1$;
    \Statex  Maximum size of $\mathcal{A}_N$: $\beta_N$;
    \Statex  Average syntactic structure representation of aspect terms: $\bar{a} = (b^{pos}, b^{dep})$
    \Ensure
      The pseudo training set for target domain: $D^{ST}$
      
    \State Initiate $D^{ST} = \{\}$; $\mathcal{A}_1 = \{\}$; $\mathcal{A}_N = \{\}$

   
   
   
   
   

   
   
   
   
    \For {instance $X$ in $D^U$} 
        \State initiate $pre\_x_i$ = NULL
        \For{$x_i$ in $X$}
           
            \State $S_{sim}(x_i, \bar{a}) = \cos(b^{pos}_i, b^{pos}) * \cos(b^{dep}_i, b^{dep})$
            \If {$S_{sim}(pre\_x_i, \bar{a}) > \sigma_N$ and $S_{sim}(x_i, \bar{a}) > \sigma_N$}
                \State $\mathcal{A}_N[-1] = \mathcal{A}_N[-1] + {x_i}$
            
            \ElsIf {$S_{sim}(x_i, \bar{a}) > \sigma_N$}
                \State $\mathcal{A}_N = \mathcal{A}_N + {x_i}$
                
            \EndIf
            
            \If {$S_{sim}(x_i, \bar{a}) > \sigma_1$}
                \State $\mathcal{A}_1 = \mathcal{A}_1 + {x_i}$
            \EndIf
            \State $pre\_x_i$ = $x_i$
        \EndFor
    \EndFor
    \State $\mathcal{A}_1 = most\_common(\mathcal{A}_1, \beta_1)$ 
    \State $\mathcal{A}_N = most\_common(\mathcal{A}_N, \beta_N)$ \textcolor{black}{\Comment{filter the elements occurring most frequently in the pseudo aspect term set}}
    \For {instance $(X, A)$ in $D^S$} 
   
   
            \For {$a$ in $A$}
                \If{$a$ is one-gram}
                    \State replace $a$ by random($\mathcal{A}_1$)
                \EndIf
                \If{$a$ is N-gram}
                    \State replace $a$ by random($\mathcal{A}_N$)
                \EndIf
            \EndFor
            
        $D^{ST} = D^{ST} + \{(X, A)\}$
    \EndFor
    \State
    \Return $D^{ST}$;
  \end{algorithmic}
\end{algorithm}

\subsection{Syntax-based Masked Language Model}
\begin{figure}[t]

\centering

\includegraphics[width=0.45\textwidth]{figures/SMLM.PDF}

\caption{Overall architecture of syntax-based mask language model (SMLM).}

\label{fig:smlm}

\end{figure}



We present a variant of BERT MLM pre-training model: syntax-based mask language model (SMLM), with the architecture
shown in Fig.~\ref{fig:smlm}. 
For standard MLM pre-training model, all the input tokens have the same probability to be chosen for prediction. Differently,
SMLM harnesses the
syntactic structure similarities between tokens and average aspect term of source domain to choose tokens that are more likely to be aspect terms in the unlabeled corpora for prediction. 

We perform SMLM pre-training model
on massive unlabeled corpora from both
source domain and target domain to learn domain-invariant representation.
Specifically, we choose the top $\alpha\%$ of the input tokens based on the similarity $S_{sim}(x_i, \bar{a})$ in Equ.~(\ref{equ:1}), where $\alpha$ is the masking threshold.
Similar to standard MLM, the chosen tokens have 80\% chance of being replaced by the [MASK] token, 10\% chance of being replaced by another token in the vocabulary randomly, and 10\% chance of being unchanged.

Our token masking strategy ensures that the pre-trained language model focuses on words that are likely to be aspect words in one of the domains. For example, given the input sentence ``\textit{The screen is in reasonable size, I really liked it.}'' from the laptop domain, SMLM would choose the word ``\textit{screen}'', which has a high syntactic structure similarity score with the average aspect term, while
original MLM would randomly choose a word to mask. In the process of predicting these aspect-term-likely words, SMLM
could learn task-adaptive domain representation.

\subsection{Span-based Joint Aspect Term and Sentiment Analysis Framework}

\begin{figure}[t]

\centering

\includegraphics[width=0.45\textwidth]{figures/span.pdf}

\caption{Overall architecture of span-based joint aspect term and sentiment analysis framework. $D^S$ and $D^{ST}$ indicate source training dataset and pseudo training dataset, respectively. $P^{start}$ and $P^{end}$ indicate probability distribution of the start position and end position, respectively.}

\label{fig:span}

\end{figure}

Previous ABSA studies \cite{gong2020unified, li2019transferable} formulate cross-domain ABSA as a sequence labeling problem by using unified tagging scheme
which combines the aspect boundary tags \{B,I,O\} and sentiment polarities \{POS,NEU,NEG\}.

However, there are two weaknesses of the unified tagging scheme. The first
is the huge search space
due to the compositionality of labels. Second, the
sentiment inconsistency problem may occur
in the cross-domain setting. For example, a model trained on restaurant domain may not be able to predict consistent sentiment for aspect term \textit{``battery life''} in sentence \textit{``the battery life is excellent 6-7 hours without charging.''} of laptop domain. The predicted result
for \textit{``battery''} may be positive
and for \textit{``life''} may be negative.
Hence, we propose
span-based joint aspect term and sentiment analysis framework in cross-domain ABSA. The overall illustration of the proposed framework is shown in Fig.~\ref{fig:span}. We firstly apply SMLM to pre-train BERT to get representation extractor backbone SMLM-BERT, then use SMLM-BERT to map word non-contextualized embeddings into contextualized embeddings. The next is to use LSTM, CRF and a filtering algorithm to extract aspect terms from sentences. Finally we predict sentiment polarities for all aspect terms based on their summarized representation.

Concretely, For the SMLM-BERT encoder, the input sequence $E$ is generated by concatenating a [CLS] token, the original word sequence $X$, and a [SEP] token. Each token $x_i$ in $X$ is converted into an input vector $e_i$ by summing the token, segment, and position embeddings. Assume $BERT_{SMLM}$(·) is the the SMLM-BERT encoder. The hidden representation of the input sequence can be obtained:
\begin{equation}\label{equ:2}
H = BERT_{SMLM} (E)
\end{equation}
Then we detect aspect terms by predicting their start and end positions in the sentence. We use LSTM, CRF and softmax to obtain probability distribution of the start position and end position as below:


\begin{gather}
H^{start} = CRF(LSTM_{start}(H))\\
H^{end} = CRF(LSTM_{end}(H))\\
P^{start} = softmax(FFNN_{start}(H^{start}))\\
P^{end} = softmax(FFNN_{end}(H^{end}))
\end{gather}

During training stage, instead of predicting the combination of labels of
aspect term boundary \{B,I,O\} and sentiment polarities \{POS,NEU,NEG\}, we propose to predict
the start position and end position for each aspect term, i.e., the term span. For each token $x_i$ in sentence $X$, we label its
$y_i^{start}$ which represents whether the token is the start of an aspect term or not and
$y_i^{end}$ indicating
whether the token is the end of an aspect term or not. We apply cross entropy loss between probability of aspect term's position $p_i^{start}$, $p_i^{start}$ and their ground truth $y_i^{start}$, $y_i^{start}$ to optimize the aspect term extraction module:
\begin{equation}\label{equ:4}
L_{ate} = - \sum_{D^S+D^{ST}}( \sum_{i=1}^{n+2} {y_i^{start} log(p_i^{start})} + \sum_{i=1}^{n+2} {y_i^{end} log(p_i^{end})})
\end{equation}

As shown in Fig \ref{fig:span}, the ATE module only outputs the token's probabilities of being start and end positions of aspect terms, respectively, and is not able to infer the number of aspect terms in the review text. To determine the aspect terms, we further propose an aspect term filtering algorithm based on the predicted probabilities, with details shown in Algorithm \ref{alg:span}.

Specifically, we first create candidate aspect term set $R$, corresponding ranking score set $U$ and output aspect term set $O$ (Line 1). Term span whose sum of start and end probability is not less than the minimum score threshold $\gamma$ will be added into candidate aspect term set $R$.
We compute a ranking score $u$, which indicates the likelihood of being an aspect term, for each term span in $R$ (Line 2$\sim$9). The ranking score $u$ is calculated as below.
\begin{equation}\label{equ:u}
    u = p_i^{start} + p_j^{end} - 0.1 \times (j - i + 1)
\end{equation}
where $i$ and $j$ indicates the start index and end index of the term span, respectively. The design is based on our intuition that
the longer the term span, the less likely it would be
an aspect term.
Then we filter the candidate aspect term set $R$ according to the computed ranking score $u$ (Line 11$\sim$19). The term span $r_{max}$ with
the highest score $u_{max}$ will be added into the output set $O$ (Line 13). Then we remove any span $r_k$ overlapping
with $r_{max}$ from $R$ in each round of the cycle (Line 14$\sim$18).
For example, once the term span \textit{``
LCD screen''} is considered an aspect term, the word \textit{``screen''} will
not be considered an aspect term. The design is based on that
the higher the ranking score, the more likely its corresponding term span would be an
aspect term. 
Besides, the term spans with intersections cannot be aspect terms at the same time
according to the labeling rule of the datasets.




\begin{algorithm}[htb]
  \caption{Aspect terms filtering}
  \label{alg:span}
  \begin{algorithmic}[1]
    \Require
    
    \Statex  The probability of start position: $P^{start}$;
    \Statex  The probability of end position: $P^{end}$;
    \Statex  Minimum score threshold: $\gamma$;
    \Statex  Maximum number of aspect terms: $K$
      
    \Ensure Aspect term set: $O$
    
    \State Initiate candidate aspect term set $R$, ranking score set $U$, $O$ = \{\}, \{\}, \{\}
    
    \For {$i=1$ to $n+2$} 
        \For {$j=1$ to $n+2$} 
            \If {$i$ $<$ $j$ and  $p_i^{start} + p_j^{end} >= \gamma$}
                \State $u = p_i^{start} + p_j^{end} - 0.1 \times (j - i + 1)$
                \State $r = (i, j)$
                \State $R = R + \{r\}, U = U + \{u\}$
            \EndIf
        \EndFor
    \EndFor
    
    
    \While{$R \not= \{\}$ and $size(O) < K$}  
        \State $max = argmax(U)$  \textcolor{black}{\Comment{ $max$ is the subscript of the largest element in $U$
    }}
        \State $O = O + \{r_{max}\}; R = R - \{r_{max}\}; U = U - \{u_{max}\}$
        \For{$r_k \in R$}
            \If{$overlap(r_{max}, r_k) \not= 0$}                
                \State $R = R - \{r_k\}; U = U - \{u_k\}$
            \EndIf
        \EndFor
    \EndWhile
    \State
\Return $O$
  \end{algorithmic}
\end{algorithm}
After extracting the aspect terms, we aggregate their representations and use feed-forward neural networks to predict sentiment polarity for each aspect term:

\begin{gather}
    v = \frac{1}{j-i+1} \sum_{t=i}^{j} h_{t}\\
    p^{pol} = softmax(tanh(FFNN(v)))
\end{gather}

\noindent where $h_t \in H$ is the token representation learnt by SMLM-BERT, and $p^{pol}$ is the predicted sentiment polarity probability. We also apply cross entropy loss to optimize the sentiment classification module:
\begin{equation}\label{equ:6}
L_{sc} = - \sum_{i=1}^{k} {y_i^{pol} log(p_i^{pol})},
\end{equation}
where $k$ is is the number of sentiment classes and $y^{pol}$ is the one-hot labeled true sentiment polarities of the aspect terms. We finally summarize the two losses for optimizing SADM:
\begin{equation}\label{equ:7}
L = L_{ate} + L_{sc}
\end{equation}


\section{\textbf{Model Analysis}}


\section{\textbf{EXPERIMENTS}}
\label{sec:exp}

\subsection{Data \& Experiment Setup}
\textbf{Datasets:}  We adopt
unlabeled corpora from the Amazon laptop reviews\footnote{http://jmcauley.ucsd.edu/data/amazon/links.html} and the Yelp restaurant reviews\footnote{https://www.yelp.com/dataset/challenge} to perform SMLM pre-training model. The labeled data from the laptop domain are
taken from SemEval-2014 ABSA \cite{pontiki-etal-2014-semeval}. Following Lekhtman's setting in \cite{lekhtman2021dilbert}, for the labeled data from restaurant domain, we combine the SemEval 2014, 2015, 2016 ABSA \cite{pontiki-etal-2014-semeval, pontiki2015semeval, pontiki2016semeval} restaurant datasets and remove the duplicated instances. And for more challenging setup than SemEval dataset, we consider MAMS dataset \cite{jiang2019challenge}, in which each sentence contains at least two aspects with different sentiment polarities. Although we just pre-train BERT in unlabeled corpus from laptop and restaurant domain, we also verify the validity of our model in additional domains: data of device domain is from Toprak et al. \cite{toprak2010sentence} and data of service domain is from Hu and Liu \cite{hu2004mining}. Detailed statistics are shown in Table \ref{tab:data}.
\begin{table}[t]
    \caption{Statistics of the benchmark datasets.}
    \centering
    \begin{tabular}{c|c|c|c|r}
    \toprule
     Dataset & Domain & Total & Training & Testing   \\
    \midrule
      L & Laptop & 384,5 & 304,5 & 800 \\
      R & Restaurant & 603,5 & 387,7 & 215,8 \\
      M & MAMS & 529,7 & 429,7 & 100,0  \\
      D & Device & 383,6 & 255,7 & 127,9  \\
      S & Service & 223,9 & 149,2 & 747\\
      \bottomrule
    \end{tabular}
    \label{tab:data}
\end{table} \\

\begin{table*}[t]
    \caption{Main results for cross-domain ABSA on twelve source-target pairs.}
    \centering
        \resizebox{\textwidth}{!}
{
    \begin{tabular}{c|c|c|c|c|c|c|c|c|c|c|c|c|c|c}
    \toprule
     Approach    & M$\rightarrow$L & L$\rightarrow$M & R$\rightarrow$L & L$\rightarrow$R & S$\rightarrow$R & D$\rightarrow$R & R$\rightarrow$S & L$\rightarrow$S & D$\rightarrow$S & S$\rightarrow$L & R$\rightarrow$D & S$\rightarrow$D & AVG(1$\sim$12). & AVG(3$\sim$12). \\
    \midrule
      Hier-Joint  & - & - & 20.72 & 33.54 & 46.39 & 42.96 & 27.18 & 25.22 & 29.28 & 33.02 & 34.81 & 35.00 & - & 32.81 \\
      RNSCN & - & - & 26.63 & 35.65 & 33.21 & 34.60 & 20.04 & 16.59 & 20.03 & 18.87 & 33.26 & 22.00 & - & 26.09 \\
      AD-SAL & - & - & 34.13 & 43.04 & 41.03 & 41.01 & 28.01 & 27.20 & 26.62 & 27.04 & 35.44 & 33.56 & - & 34.68  \\
      BERT-Base & 32.21 & 22.70 & 31.44 & 40.38 & 44.66 & 40.32 & 19.48 & 25.78  & 30.31 & 30.47 & 27.55 & 33.96 & 31.61 & 32.44  \\
      BERT-Cross & 34.60 & 25.75 & 39.72 & 45.4 & 51.34 & 42.62 & 24.44 & 23.28 & 28.18 & 35.04 & 33.22 & 33.22 & 34.73 & 35.65 \\
      BERT-Base-DANN & 24.33 & 24.63 & 30.41 & 41.63  & 45.84 & 34.68 & 21.60 & 25.10 & 18.62 & 31.92 & 34.41 & 23.97 & 29.76 & 30.79  \\
      BERT-Cross-DANN & 33.51 & 28.67 & 38.83 & 47.39  & 50.31 & 42.20 & 28.35 & 26.69 & 28.77 & 34.29 & 33.42 & 37.14 & 35.80 & 36.74  \\
      BERT-Base-UDA & 27.19 & 27.25 & 33.68 & 45.46  & 47.09 & 42.68 & 33.12 & 27.89 & 28.03 & 34.77 & 34.93 & 32.10 & 34.52 & 35.98 \\
      BERT-Cross-UDA & 32.47 & 33.03 & 43.95 & 49.52  & 53.97 & 51.84 & 30.67 & 27.78 & 34.41 & 35.76 & 40.35 & 38.05 & 39.32 & 40.63 \\
      \midrule
      SDAM & \textbf{46.65} & \textbf{44.75} & \textbf{54.55} & \textbf{63.11} & \textbf{58.65} & \textbf{60.98} & \textbf{45.61} & \textbf{45.32} & \textbf{55.19} & \textbf{46.69} & \textbf{51.63} & \textbf{58.01} & \textbf{52.60} & \textbf{53.97} \\
      \bottomrule
    \end{tabular}
    }
    \label{tab:four2}
\end{table*}

\begin{table}[t]
    \caption{Main results for cross-domain ATE on four source-target pairs.}
    \centering
    \begin{tabular}{c|c|c|c|c|c}
    \toprule
     Approach    & M$\rightarrow$L & L$\rightarrow$M & R$\rightarrow$L & L$\rightarrow$R & AVG. \\
    \midrule
      Hier-Joint  & - & - & 34.11 & 48.61 & 41.36 \\
      RNSCN  & - & - & 45.01 & 56.12 & 50.37 \\
      AD-SAL & - & - & 45.01 & 56.12 & 50.37 \\
      BERT-Cross & 35.30 & 29.82 & 45.89 & 39.32 & 37.48 \\
      BERT-Base-DANN & 34.51 & 35.92 & 36.79 & 48.34 & 38.89  \\
      BERT-Cross-DANN & 35.72 & 41.28 & 46.90 & 52.40 & 44.08  \\
      BERT-Base-UDA & 36.52 & 39.59 & 48.32 & 49.52 & 43.50  \\
      BERT-Cross-UDA & 41.29 & 45.62 & 53.51 & 56.12 & 49.14  \\
      Combridge & 39.98 & 47.42 & 53.32 & 66.20 & 51.73  \\
      DILBERT & 43.72 & 58.96 & 56.07 & 61.04 & 54.95  \\

      \midrule
      SDAM & \textbf{48.11} & \textbf{65.38} & \textbf{56.26} & \textbf{69.26} & \textbf{58.45} \\
      \bottomrule
    \end{tabular}
    \label{tab:four1}
\end{table}

\begin{table}[t]
    \caption{Ablation study of our SDAM approach.}
    \centering
    \resizebox{0.5\textwidth}{!}{
    \begin{tabular}{c|c|c|c|c|c}
    \toprule
     Approach & M$\rightarrow$L & L$\rightarrow$M & R$\rightarrow$L & L$\rightarrow$R & AVG. \\
    \midrule
      Without pseudo instances  & 43.02 & 38.58 & 50.47 & 59.24 & 47.83  \\
      Without SMLM              & 42.81 & 38.74 & 51.34 & 56.27 & 47.29 \\
      Without span-based framework & 32.51 & 38.77 & 42.29 & 56.05 & 42.41  \\
      SDAM                      & \textbf{46.65} & \textbf{44.75} & \textbf{54.55} & \textbf{63.11} & \textbf{52.27} \\
      \bottomrule
    \end{tabular}
    }
    \label{tab:ablation}
\end{table}

\begin{table*}[t]
    \caption{Influence of syntax-similarity threshold $\gamma$ and size of pseudo aspect term set $\beta$}
    \centering
    \begin{tabular}{c|c|c|c|c|>{\columncolor{pink!60}}c|>{\columncolor{pink!60}}c|>{\columncolor{pink!60}}c|c}
    \toprule
     \diagbox{threshold}{size}    & 50 & 100 & 150 & 200 & 250 & 300 & 350 & 400 \\
    \midrule
      0.20 & 61.08 & 61.79 & 61.47 & 60.79 & 62.62 & 61.69 & 62.84 & 62.80 \\
      0.25 & 61.55 & 63.27 & 62.29 & 61.55 & 62.49 & 62.02 & 63.00 & 62.03 \\
      \rowcolor{green!20}
      0.30 & 61.91 & 62.22 & 62.62 & 62.97 & \cellcolor{purple}63.23 & \cellcolor{purple}63.11 & \cellcolor{purple}64.07 & 63.37\\
      \rowcolor{green!20}
      0.35 & 62.69 & 60.96 & 62.28 & 61.98 & \cellcolor{purple}62.46 & \cellcolor{purple}63.08 & \cellcolor{purple}62.11 & 62.27 \\
      \rowcolor{green!20}
      0.40 & 62.06 & 61.61 & 61.60 & 63.45 & \cellcolor{purple}61.91 & \cellcolor{purple}62.40 & \cellcolor{purple}63.66 &  60.98 \\
      \rowcolor{green!20}
      0.45 & 61.70 & 62.95 & 62.34 & 64.67 & \cellcolor{purple}61.97 & \cellcolor{purple}61.83 & \cellcolor{purple}62.97 & 63.17 \\
      \rowcolor{green!20}
      0.50 & 61.95 & 61.62 & 62.45 & 62.86 & \cellcolor{purple}63.33 &  \cellcolor{purple}63.71 & \cellcolor{purple}62.51 & 61.80 \\
      0.55 & 60.47 & 61.56 & 62.83 & 59.70 & 61.95 &  62.86 & 62.05 & 62.73\\
      0.60 & 60.41 & 60.70 & 60.65 & 61.52 & 62.17 &  61.12 & 59.85 & 61.70  \\
      0.65 & 58.23 & 60.50 & 60.99 & 59.79 & 60.83 &  61.74 & 58.83 & 59.83\\
      0.70 & 60.83 & 59.88 & 61.89 & 58.77 & 58.66 & 61.38  & 59.27 & 59.16 \\
    \bottomrule
    \end{tabular}
    \label{tab:parameter}
\end{table*}
\textbf{Settings:} 
Considering that our model pre-trains
on laptop domain and restaurant domain, for the sake of fairness, we first conduct experiments on 4 source-target transfer pairs such as L$\rightarrow$R using laptop, restaurant, MAMS domains in Table~\ref{tab:data}.
Following the setup in \cite{lekhtman2021dilbert}, we remove R$\rightarrow$M and M$\rightarrow$R, as R and M are similar. Then to verify the validity of our model in additional domains, we conduct experiments on 8 additional source-target transfer pairs using laptop, restaurant, device and service domains. Following the setup in \cite{gong2020unified}, we remove L$\rightarrow$D and D$\rightarrow$L since L and D are similar. The DILBERT \cite{lekhtman2021dilbert} and Combridge \cite{chen2021bridge} are proposed to tackle cross-domain aspect term extraction task, so we just compare with them on cross-domain aspect term extraction.
For each transfer pair, the training data consist of labeled training data in source domain and pseudo training data for target domain. Meanwhile, we employ the labeled test data of the source domain as validation set and the testing data of the target domain as evaluation set. 

\textbf{Implementation Details:} We use Spacy\footnote{https://spacy.io} for dependency parsing. There are 51 classes of POS tags and 47 classes of denpendency relations in total in the five datasets. 
For implementing the SMLM pre-training modue, we
employ the BERT-Base-Uncased model of Hugging-Face Transformers package \cite{wolf2020transformers}. We fine-tune all BERT layers and
mask full words instead of sub-words to reduce the influence of the tokenizer. We set the masking threshold $\alpha = 15$ and pre-train for 2 epochs using Adamw optimizer with learning rate at 3e-5, adam epsilon at 1e-8 and batch size at 16. When constructing pseudo training data for the target domain, we set both the one-gram syntax-similarity threshold $\sigma_1$ and N-gram syntax-similarity threshold $\sigma_N$ at 0.3. The sizes of one-gram pseudo aspect term set $\mathcal{A}_1$ and N-gram pseudo aspect term set $\mathcal{A}_N$ are both 350. For filtering aspect terms, minimum score threshold $\gamma$ is manually tuned
on each dataset and maximum number of aspect terms $K$ is set at 10. For facilitating training on cross-domain ABSA, we use adamw optimizer with learning rate at 2e-5, adam epsilon at 1e-8 and batch size at 8.

\textbf{Evaluation Metric:}  Following
previous studies \cite{lekhtman2021dilbert, chen2021bridge, gong2020unified}, we evaluate the models with Micro-F1 and only exact match could be counted as correct. All experiments are repeated 9 times and we present the average results.




\subsection{Baselines}

We compare our model with several outstanding models as follows:\\

\begin{itemize}

\item \textbf{DILBERT \cite{lekhtman2021dilbert}}: A Transformer-based model that performs category-based masked language modeling task and category proxy prediction task. It incorporates
aspect category information from external domain and achieves the state-of-the-art performance in cross-domain aspect term extraction. It focuses on cross-domain aspect term extraction.

\item \textbf{Combridge \cite{chen2021bridge}}: A bridge-based convolution neural network that incorporates syntactic structures and cross-semantic information, combined with an adversarial component. It focuses on cross-domain aspect term extraction.

\item \textbf{BERT-Cross \cite{xu2019bert}}: BERT-Cross post-trains BERT-Base on a combination of Yelp and Amazon corpus. 


\item \textbf{BERT-Base-UDA and BERT-Cross-UDA \cite{gong2020unified}}: UDA is a Transformer-based neural network that performs fine-tuning with syntactic-driven auxiliary tasks and a modified attention mechanism. BERT-Base-UDA and BERT-Cross-UDA are designed differently in the initialization.
BERT-Base-UDA is initialized by the BERT-Base model while BERT-Cross-UDA is initialized by BERT-Cross.

\item \textbf{BERT-Base-DANN and BERT-Cross-DANN \cite{ganin2016domain}}: DANN is a domain-adversarial training neural network. Similarly, BERT-Base-DANN and BERT-Cross-DANN are initialized by BERT-Base and BERT-Cross respectively.

\item \textbf{AD-SAL \cite{li2019transferable}}: A recursive neural network which can automatically capture aspect-opinion latent relations to achieve token-level adversarial adaptation. 

\item \textbf{RNSCN \cite{wang2018recursive}}: A recursive neural structural correspondence network that incorporates syntactic structures. 

\item \textbf{Hier-Joint \cite{ding2017recurrent}}: A recurrent neural network with manually designed rule-based auxiliary tasks.


\end{itemize}



\section{\textbf{Experimental analysis}}
\label{sec:ana}

\subsection{Main Results}
Table \ref{tab:four2} shows the comparison results for cross-domain ABSA on laptop, restaurant, MAMS, service and device domain based on Micro-F1. As can be seen, the proposed SDAM\xspace achieves the best performance in terms of the exact-match F1 metric. Specifically, SDAM\xspace significantly boosts the average performance of BERT-Cross-UDA from 39.32\% to 52.60\%. The improvement is very large on all transfer pairs.
We summarize
the following observations, which
demonstrate the effectiveness of SDAM for cross-domain ABSA.

(1) \textbf{BERT is beneficial
for cross-domain ABSA.} The BERT-Base baselines (i.e., UDA) generally perform better than non-BERT-based baselines (i.e., Hier-Joint). For example, BERT-Base-UDA outperforms AD-SAL, RNSCN and hier-Joint on most transfer pairs, with the average increase at 1.30\% with respect to AD-SAL's 34.68\%.

(2) \textbf{Adversarial training presents limited effect on the performance of cross-domain ABSA.} As can be seen, compared with BERT-Base whose average performance is 31.61\%, the average performance of BERT-Base-DANN is only 29.76\%. Besides, BERT-Cross-DANN only improves the average performance of BERT-Cross at 1.08\%. The results indicate that the adversarial training, i.e., DANN, shows limited contribution to the performance of cross-domain ABSA.

(3) \textbf{SDAM achieves
the most effective performance for cross-domain ABSA.}
SDAM\xspace significantly boosts the average performance of the state-of-the-art baseline BERT-Cross-UDA from 39.74\% to 52.60\%.
Besides, BERT-Cross-UDA performs poorly on more challenging transfer pairs such as
M$\rightarrow$L and L$\rightarrow$M (i.e., column 1$\sim$2), only achieving 32.47\% and 33.03\%, respectively. On the contrary, our model exhibits promising
transferability on the more challenging transfer pairs, achieving 46.65\% and 44.74\%, respectively. 




\subsection{Results for ATE}
We also report the comparison results for cross-domain ATE on laptop, restaurant and MAMS domain based on Micro-F1. As shown in Table \ref{tab:four1}, each model shows a slight increase in performance over cross-domain ABSA, which is reasonable as ATE is a subtask of ABSA. SDAM boosts the average performance of the state-of-the-art ATE domain adaptation approach DILBERT from 54.95\% to 58.45\%. Specifically, on the more challenging transfer pairs L$\rightarrow$M and L$\rightarrow$R, SDAM improve by 6.42\% and 8.22\% from DILBERT, respectively.


\subsection{Ablation Study}


Since our SDAM includes three components: syntax-based mask language model, syntax-based pseudo instances generation and span-based joint aspect
term and sentiment analysis framework, we further conduct experiments over different variants of SDAM in Table \ref{tab:ablation} to show the effect of each component. For the model without span-based framework, it utilizes
a unified tagging scheme which combines the aspect boundary tags \{B,I,O\} and sentiment polarities \{POS,NEU,NEG\} to formulate ABSA as a sequence labeling problem. As can be seen, each component plays a considerable role in cross-domain ABSA. Pseudo training instances and SMLM bring
similar improvement (4.44\% and 4.98\%, respectively) to the performance of the overall model in laptop, restaurant and MAMS domain. The improvement brought by span-based joint aspect term and sentiment analysis framework is the most obvious
in M$\rightarrow$L and R$\rightarrow$L transfer pairs. The reason is that there are more multi-gram aspect terms in the laptop domain. For example, the percentages of multi-gram aspect terms in laptop, rest and mams test datasets are 37.04\%, 25.64\% and 17.82\%, respectively.

\subsection{Parameter Study}
\begin{figure*}[t]
\centering
\includegraphics[width=13cm]{figures/smlm_result.pdf}
\caption{Top five prediction for a given masked token with respect to BERT-Cross and SMLM. BERT-Cross and SMLM are both pre-trained on unlabeled corpora from laptop and restaurant domain, and not fine-tuned on downstream tasks. Both example (1) and (2) are from the training dataset of restaurant domain, and example (3) is from the training dataset of laptop domain.}
\label{fig:smlm_result}
\end{figure*}
We study
the influence of two important parameters, i.e., syntax-similarity threshold $\sigma$ and size of pseudo aspect term set $\beta$ on the performance of SDAM\xspace, with results shown in Table \ref{tab:parameter}. For simplifying the parameter analysis, we set the parameters for both one-gram and N-gram to be the same.
As can be seen, when we fix the size of pseudo aspect term,
the performance increases first and then decreases as the syntax-similarity threshold increases. The performance of SDAM\xspace reaches its maximum when the threshold is set between 0.3 and 0.5. We speculate a smaller threshold
will introduce more noisy
terms, while a larger threshold 
tends to filter out meaningful terms. When we fix the threshold $\alpha$, the performance generally reaches its maximum when the size is between 250 and 350. During experimental evaluation, we define threshold to be 0.3 and size to be 350, respectively.

\subsection{Capacity of SMLM}
To study the ability of learning task-adaptive representation of the proposed SMLM model, we pre-train SMLM model and BERT-Cross model on unlabeled corpora from the Amazon laptop reviews and the Yelp restaurant reviews, and test them on sentences form these domains. Fig.~\ref{fig:smlm_result} shows the comparison of top five prediction of BERT-Cross and SMLM.
Example (1) and (2) are from unlabeled Yelp restaurant reviews and (3) is from unlabeled Amazon laptop reviews. For the example (1) \textit{``The MASK came out and it looked like it had been sitting in the back a while.''}, BERT-Cross only ranks
the relevant word in the second order; on the contrary, SMLM ranks the
relevant words which are actually aspect terms in top five.
For the example (2) \textit{``Do yourself a favor and engage with the wait MASK''}, BERT-Cross does not predict any relevant word, but SMLM successfully ranks
the mask word \textit{``staff''} in the first order. For the example (3) \textit{``Pre-installed software is fine, Hard MASK1 MASK2 is more than enough''}, BERT-Cross predicts the mask words \textit{``disk memory''} in the fifth order, while SMLM predicts the mask words in the first order.

It is obvious that prediction of SMLM is more similar to the mask token than BERT-Cross, when the mask token is an aspect term. This also illustrates
that our token masking strategy can facilitate BERT model to learn task-adaptive representation for cross-domain ABSA. 


\subsection{Case Study}

\begin{table*}[h]
    \caption{Case study for transfer pair L$\rightarrow$R. We use brackets to indicate the boundary and subscript \{POS,NEG,NEU\} to indicate the sentiment polarity of the aspect term. \textit{NONE} indicates that the model does not output any aspect term. The marker \XSolidBrush  indicates a wrong prediction.}
    \centering
    \resizebox{\textwidth}{!}
{
    
    \begin{tabular}{p{6cm}|c|c|c|c}
    \toprule
        \multirow{2}{*}{Input:(Target domain R)} &\multicolumn{2}{c|}{BERT-Cross-UDA} &\multicolumn{2}{c}{SDAM}  \\ \cline{2-5}
         & \makecell{ATE} & \makecell{ABSA} & \makecell{ATE} & \makecell{ABSA}  \\
    \midrule
      1.Straight-forward, no surprises, very decent [\textcolor{red}{\textit{Japanese food}}]$_{_{POS}}$. & \textit{NONE} & \textit{NONE} & \makecell{Japanese food} & \makecell{[Japanese food]$_{_{POS}}$} \\
    \midrule
     2.While there's a decent [\textcolor{red}{\textit{menu}}]$_{_{POS}}$, it shouldn't & menu & [menu]$_{_{POS}}$ & menu &  [menu]$_{_{POS}}$\\take ten minutes to get your [\textcolor{red}{\textit{drinks}}]$_{_{NEU}}$ and 45 & & & drink & [drink]$_{_{NEU}}$ \\for a [\textcolor{red}{\textit{dessert pizza}}]$_{_{NEU}}$. & & & dessert pizza & [dessert pizza]$_{_{NEU}}$\\
    \midrule
    3.Anytime and every time I find myself in the & portions & [portions]$_{_{POS}}$ & sushi & [sushi]$_{_{POS}}$\\ 
    neighborhood I will go to Sushi Rose for & price & [price]$_{_{POS}}$ & portions & [portions]$_{_{POS}}$ \\
    fresh [\textcolor{red}{\textit{sushi}}]$_{_{POS}}$ and great [\textcolor{red}{\textit{portions}}]$_{_{POS}}$ all at a & & & price & [price]$_{_{POS}}$ \\
    reasonable [\textcolor{red}{\textit{price}}]$_{_{POS}}$. & & & &\\
    \midrule
    4.Definitely try the [\textcolor{red}{\textit{calamari}}]$_{_{POS}}$, any [\textcolor{red}{\textit{pasta}}]$_{_{POS}}$, &calamari &[cal$_{_{POS}}$ama$_{_{NEU}}$ri$_{_{POS}}$] \XSolidBrush &calamari &[calamari]$_{_{POS}}$\\
    or even the [\textcolor{red}{\textit{sliced steak entree}}]$_{_{POS}}$. &pasta &[pasta]$_{_{NEU}}$ \XSolidBrush&pasta &[pasta]$_{_{POS}}$\\
     & sliced steak entree &[sliced$_{_{NEU}}$ steak$_{_{NEU}}$ \XSolidBrush& sliced steak entree &[sliced steak entree]$_{_{POS}}$\\
     & & en$_{_{POS}}$tree$_{_{NEU}}$]& &\\
    \midrule
   5.I've had the [\textcolor{red}{\textit{jellyfish}}]$_{_{POS}}$, [\textcolor{red}{\textit{horse mackerel}}]$_{_{POS}}$, &jellyfish &[jelly$_{_{POS}}$fish$_{_{NEU}}$] \XSolidBrush&jellyfish &[jellyfish]$_{_{POS}}$\\
   the [\textcolor{red}{\textit{blue fin tuna}}]$_{_{POS}}$ and the [\textcolor{red}{\textit{sake ikura roll}}]$_{_{POS}}$ &erel \XSolidBrush&[erel]$_{_{NEU}}$ \XSolidBrush&horse mackerel &[horse mackerel]$_{_{POS}}$\\
   among others, and they were all good. &sake \XSolidBrush&[sake]$_{_{POS}}$ \XSolidBrush&blue fin tuna &[blue fin tuna]$_{_{POS}}$\\
    &ura roll \XSolidBrush&[uro$_{_{NEU}}$ roll$_{_{POS}}$] \XSolidBrush&sake ikura roll &[sake ikura roll]$_{_{POS}}$\\
    \bottomrule
    \end{tabular}
}
    \label{tab:case}
\end{table*}

We further perform case study to demonstrate the effectiveness of the proposed SDAM\xspace. Table \ref{tab:case} illustrates the results of SDAM\xspace and BERT-Cross-UDA for five examples of transfer pair L$\rightarrow$R. We choose BERT-Cross-UDA for analysis since it is the best baseline model in cross-domain ABSA (as shown in Table \ref{tab:four1}).
For Example 1 \textit{``Straight-forward, no surprises, very decent Japanese food''}, BERT-Cross-UDA does not recognize the aspect term \textit{``Japanese food''}. For Example 2 \textit{``While there’s a decent menu, it shouldn’t take ten minutes to get your drink and 45 for a dessert pizza''}, BERT-Cross-UDA does not recognize the aspect terms \textit{``drink''} and \textit{``dessert pizza''}. For Example 3 \textit{``Anytime and every time I find myself in the neighborhood I will go to Sushi Rose for fresh sushi and great portions all at a reasonable price}, BERT-Cross-UDA does not recognize the aspect term \textit{``sushi''}.
The above examples show that the baseline model
tends to fail when predicting
multi-word aspect term or multiple aspect terms in a sentence, compared to SDAM. 

Example 4 and 5 show the sentiment inconsistency problem of the baseline model.
Due to the tokenization mechanism of BERT model, aspect terms may be sliced into multiple subwords. For example, the \textit{``Calamari''} in example 4 will be sliced into \textit{``cal''}, \textit{``ama''} and \textit{``ri''}. BERT-Cross-UDA predicts inconsistent
sentiment polarities for these subwords, e.g., the sentiment polarities of \textit{``cal''} and \textit{``ri''} are predicted to be positive but the sentiment polarity of \textit{``ama''} is predicted to be neutral.
When the aspect terms have more than one word, 
the baseline model also fails to predict consistent sentiment polarity for the words in one aspect term.
For instance, the aspect term \textit{``sliced steak entree''} in example 4 can be sliced into three words, i.e., \textit{``sliced''}, \textit{``steak''}, and \textit{``entree''}. The words \textit{``sliced''} and \textit{``steak''} are predicted to be neutral, but the subword \textit{``en''} of \textit{``entree''} is predicted to be positive. For example 5 \textit{``I've had the jellyfish, horse mackerel, the blue fin tuna and the sake ikura roll among others, and they were all good''}, 
the baseline model predicts wrong sentiment polarities for all the aspect terms.
The above examples show that the proposed SDAM\xspace can more accurately predict the sentiment polarities of aspect terms.
\section{\textbf{Conclusion}}
\label{sec:con}
In this paper, we propose a novel domain adaptation method for aspect-based sentiment analysis. We propose a variant of BERT MLM pre-training model deliberately designed for domain adaptation of ABSA based on syntactic structure similarity cross domains. To further bridge the cross-domain gap, we propose to
explicitly construct pseudo training instances based on the syntactic structure. We adopt an effective span-based ABSA framework to tackle the sentiment inconsistency issue for the aspect terms with subwords. Extensive experiments on five benchmark datasets demonstrate the superiority of our approach over existing methods in cross-domain ABSA.

\section{Proof of the First Zonklar Equation}
Appendix one text goes here.

\section{}
Appendix two text goes here.

\section{Introduction}\label{sec:introduction}}
\else
\section{Introduction}
\label{sec:introduction}
\fi




\IEEEPARstart{T}{his} demo file is intended to serve as a ``starter file''
for IEEE Computer Society journal papers produced under \LaTeX\ using
IEEEtran.cls version 1.8b and later.
I wish you the best of success.

\hfill mds
 
\hfill August 26, 2015

\subsection{Subsection Heading Here}
Subsection text here.


\subsubsection{Subsubsection Heading Here}
Subsubsection text here.













\section{Conclusion}
The conclusion goes here.








\appendices
\section{Proof of the First Zonklar Equation}
Appendix one text goes here.

\section{}
Appendix two text goes here.


\ifCLASSOPTIONcompsoc
 
  \section*{Acknowledgments}
\else
 
  \section*{Acknowledgment}
\fi


The authors would like to thank...


\ifCLASSOPTIONcaptionsoff
  \newpage
\fi






\section{Introduction}
This demo file is intended to serve as a ``starter file''
for IEEE Computer Society conference papers produced under \LaTeX\ using
IEEEtran.cls version 1.8b and later.
I wish you the best of success.

\hfill mds
 
\hfill August 26, 2015

\subsection{Subsection Heading Here}
Subsection text here.


\subsubsection{Subsubsection Heading Here}
Subsubsection text here.













\section{Conclusion}
The conclusion goes here.







\ifCLASSOPTIONcompsoc
 
  \section*{Acknowledgments}
\else
 
  \section*{Acknowledgment}
\fi


The authors would like to thank...








\section{Introduction}\label{sec:introduction}}




\IEEEPARstart{T}{his} demo file is intended to serve as a ``starter file''
for IEEE Computer Society journal papers produced under \LaTeX\ using
IEEEtran.cls version 1.8b and later.
I wish you the best of success.

\hfill mds
 
\hfill August 26, 2015

\subsection{Subsection Heading Here}
Subsection text here.


\subsubsection{Subsubsection Heading Here}
Subsubsection text here.













\section{Conclusion}
The conclusion goes here.








\appendices
\section{Proof of the First Zonklar Equation}
Appendix one text goes here.

\section{}
Appendix two text goes here.


\ifCLASSOPTIONcompsoc
 
  \section*{Acknowledgments}
\else
 
  \section*{Acknowledgment}
\fi


The authors would like to thank...


\ifCLASSOPTIONcaptionsoff
  \newpage
\fi






\section{Introduction}\label{sec:introduction}}

\input{2_intro}
\input{3_related}
\input{5_method}
\input{6_ana}
\input{7_exp}


\input{8_conclusion}


\ifCLASSOPTIONcaptionsoff
  \newpage
\fi
\bibliographystyle{IEEEtran}
