\section{Introduction}

Knowledge graphs (KGs) \citep{bollacker2008freebase,carlson2010toward, suchanek2007yago} are a type of knowledge base, which store multi-relational factual knowledge in the form of triplets. Each triplet specifies the relation between a head entity and a tail entity. KGs conveniently capture rich structured knowledge about many different types of entities (objects, events, and even abstract concepts) and thus facilitate numerous knowledge-intensive applications such as information retrieval~\citep{xiong2017explicit}, recommender systems~\citep{wang2019kgat}, and question answering \citep{west2014knowledge}. To better advance these downstream applications, the adopted KGs are expected to be as comprehensive as possible to provide all kinds of required knowledge. However, existing large-scale KGs are known to be far from complete with large portions of triplets missing~\citep{bollacker2008freebase,carlson2010toward}. Imputing these missing triplets is of great importance. Furthermore, new knowledge (triplets) is constantly emerging even between existing entities, which also calls for dedicated efforts to predict these new triplets and complete KGs~\citep{garcia2018learning,jin2019recurrent}. Therefore, \emph{knowledge graph completion} (KGC) has been attracting increasing attention and becoming one of the most essential problems in KG research~\citep{lin2015learning,yu2021knowledge}. A crucial step towards better KGC performance is to learn low-dimensional continuous embeddings for both entities and relations~\citep{BordesUGWY13,YangYHGD14a}. 

Recently, due to the intrinsic graph-structure of KGs, graph neural networks (GNNs)
have been adopted to learning more powerful embeddings for their entities and relations, and thus facilitate the KGC tasks. However, GNNs are typically designed for homogeneous graphs, with single node and relation types. They often update node features through an information aggregation process over the graph, where each node collects and transforms features from its neighbors. When adapting GNNs for KGs, dedicated efforts are often devoted to developing more sophisticated aggregation process that are customized for better capturing multi-relational information~\citep{VashishthSNT20,schlichtkrull2018modeling,ye2019vectorized}. The improvement brought by GNN-based models is thus naturally attributed to these enhanced information aggregation processes.
Therefore, current research on developing better GNNs for KGs is still largely focused on advancing information aggregation processes. 


\textbf{Present Work.} In this work, we find that, surprisingly, the information aggregation process in the GNN-based models is not the most critical reason for  reported performance improvements for KGC. 
Specifically, we replaced the information aggregation process in several state-of-the-art KGC-focused GNN models such as RGCN~\citep{schlichtkrull2018modeling},  CompGCN~\citep{VashishthSNT20} and KBGAT~\citep{nathani2019learning}  with simple Multiple Layer Perceptrons (MLPs) and achieved comparable performance to their corresponding GNN-based models, across a variety of datasets and implementations. We carefully scrutinized  these three GNN-based models and discovered that they also differ from each other in other key components such as scoring functions and loss functions. Thus, we suspect that the discrepancy in the scoring and loss functions in these models might be responsible for improved performance. To better study how these different components contribute to the model, we conducted comprehensive experiments to demonstrate the effectiveness of each component. Our results indicate that the scoring and loss functions indeed have stronger influence while the information aggregation process make almost no contributions. Based on our findings, we further provide ensemble models built upon various MLP-based models, which are able to achieve better performance than GNN-based KGC models; these implications are powerful in practice, owing to scalability advantages of MLPs over GNNs \citep{zhang2021graphless}.
The significance of this work is summarized as follows:
\begin{compactitem}[\textbullet]

 \item  We reveal that the aggregation process in existing GNN-based models are not effective for KGC. This calls for more careful investigations into these processes in future development.
 
 \item  We find that simpler MLP-based models can achieve competitive performance as these GNN-based models.  Advantageously, given their simplicity and scalability, such MLP-based models can be more easily applied to large-scale graphs, and thus have great potential for practical use. Our work also suggests the future value of incorporating MLP-based model baselines in KGC research. 

 
 
 
    \item  We highlight the importance of the scoring and loss functions for KGC on knowledge graphs. To further facilitate the KGC task, it is critical to devote dedicated efforts to developing better scoring and loss functions.  
   
   
   

    
   
\end{compactitem}
















\section{Related Work}

In this section, we briefly review some related works, including the knowledge graph, Graph Neural Networks (GNN), and GNNs applied to knowledge graphs. 


\noindent{\bf Knowledge Graphs.} Knowledge graphs can be treated as multi-relational graphs, where nodes correspond to entities and edges represents the relations in knowledge graphs. Knowledge graphs play a pivotal role in a variety of applications, such as recommendation systems \citep{wang2019kgat, wang2019multi}, question answering  \citep{west2014knowledge, huang2019knowledge}, information retrieval \citep{xiong2017explicit}, and natural language processing \citep{qiu2019machine}, etc.
Some representative knowledge bases (KBs) include Freebase \citep{bollacker2008freebase}, NELL \citep{carlson2010toward}, and YAGO \citep{suchanek2007yago}. However, existing KGs are usually far from complete; thus, considerable studies~\citep{BordesUGWY13,YangYHGD14a, Kazemi018} have been performed for the knowledge graph completion.



\noindent{\bf Graph Neural Networks.}
Graph neural networks (GNNs), which generalize deep learning techniques for graph structured data, have been widely applied to many graph-related tasks such as recommender system~\cite{fan2019graph,wang2019kgat,ying2018graph}, computer vision~\cite{yan2018spatial,wang2019graph}, and natural language processing~\cite{marcheggiani2017encoding,song2018exploring}. GNN models generally follow a message passing mechanism, which updates node embeddings by iteratively aggregating and transforming the information from their neighboring nodes. Some of the representative GNN models include GCN~\citep{KipfW17}, GAT \citep{velivckovic2018graph} and GraphSAGE  \citep{hamilton2017inductive}. A comprehensive review of GNNs and their applications can be found in~\cite{ma2020deep}.

\noindent{\bf GNNs on Knowledge Graphs.}
To extend GNNs for tackling knowledge graphs, recent efforts have been made to deal with the multi-relational edges in KGs~\citep{yu2021knowledge, VashishthSNT20,schlichtkrull2018modeling,shang2019end,nathani2019learning,wang2019kgat}. Typically, this is achieved by designing new message passing operations, which can better capture the relational information during feature transformation and aggregation. Among these GNN-based models, RGCN \citep{schlichtkrull2018modeling} introduces the relation-specific transformation matrices. CompGCN \citep{VashishthSNT20} integrates neighboring information based on entity-relation composition operations. KBGAT~\citep{nathani2019learning} learns the attention coefficients to distinguish the role of entity in various relations. More recently, NBFNet~\citep{zhu2021neural} proposes path-based approach to directly learn pair-wise (edge) representations without learning entity (node) embeddings. Different from most existing works, it does not base on the aggregation mechanism for generating entity embeddings. In this paper, we focus on investigating how the aggregation component in the GNNs affects their performance in the KGC task. Hence, we do not include NBFNet into the comparison.


\section{Preliminaries}
Before moving to main content, we first introduce KGC-related preliminaries, as well as five datasets and three GNN-based models we adopt for our investigation

\subsection{Knowledge graph completion (KGC)}
The task of KGC is to infer missing triplets based on known facts in the KG. In KGC, given triplets missing either the head or tail entity, our goal is to predict these missing entities. Specifically, we denote the triplets with missing head entity (or tail entity) as $(h,r,?)$ (or $(?,r,t)$), where the question mark indicates the entity we aim to predict. Since the head entity prediction and tail entity prediction tasks are symmetric to each other, in the following, we only use the tail entity prediction task as an example for illustration. When conducting the KGC task for a specific triplet $(h,r,?)$, we use all entities in the KG as candidates and try to select the best one as the tail entity. Typically, for each candidate entity $t'$, we evaluate its score for this triplet $(h,r,t')$ with the following function: 
\begin{align}
    s_{h,r}(t') = f(h,r,t'),
\end{align}
where $s_{h,r}(t')$ is the score of $t'$ given the head entity $h$ and the relation $r$, and $f$ is a scoring function to be designed. We choose the entity $t'$ with the largest score as the predicted tail entity. $f(\cdot)$ can be modeled in various ways which will be discussed later. 

\subsubsection{Datasets}
We use five well-known KG datasets, i.e., \textbf{FB15k}~\cite{BordesUGWY13}, \textbf{FB15k-237} \cite{toutanova2015representing,toutanova2015observed}, \textbf{WN18} \cite{schlichtkrull2018modeling}, \textbf{WN18RR} \cite{ettmersMS018} and \textbf{NELL-995}~\citep{XiongHW17} for this study. The detailed descriptions and data statistics can be found in {\bf Appendix~\ref{sec:data}}. Following the settings in previous works~\citep{VashishthSNT20,schlichtkrull2018modeling}, triplets in these datasets are randomly split into a training, validation, and test set, denoted as $\mathcal{D}_{train},\mathcal{D}_{val}, \mathcal{D}_{test}$, respectively. The triplets in the training set are regarded as the known facts, which are used to train models. We manually remove the head/tail entity of the triplets in the validation and test sets for model selection and evaluation. Specifically, for the tail entity prediction task, given a triplet $(h,r,t^*)$, we remove $t^*$ and construct a test sample $(h,r,?)$. The tail entity $t^*$ is regarded as the ground-truth for this sample. 





\subsubsection{Evaluation Metrics.}

When evaluating the performance, we focus on the predicted scores for the ground-truth entity of the triplets in the test set $\mathcal{D}_{test}$. For each triplet $(h,r,?)$ in the test set, we sort all candidate entities $t$ in a non-increasing order according to $s_{h,r}(t)$. Then, we use the rank-based measures to evaluate the quality of the prediction including  Mean Reciprocal Rank (\textbf{MRR}) and \textbf{Hits@N}. In this work, we choose $N\in\{1,3,10\}$. Their detailed definitions can be found in {\bf Appendix~\ref{sec:metric}}.





\begin{figure*}
  \centering
  \includegraphics[width=0.7\linewidth]{figures/general_framework_KG.pdf}
  \caption{{General framework of GNN models for KGC. The framework consists of three main components: aggregation, scoring function and loss function.}}
  \label{fig:framework}
   \vspace{-0.1in}
\end{figure*}

\vspace{-0.05in}
\subsection{GNN-based KGC}\label{sec:gnn-kg}
Various GNN-based models have been utilized for KGC by learning representations for the entities and relations of a KG. The learnt representations are then used as input to a scoring function $f(\cdot)$. Next, we first introduce GNN models specifically designed for learning representations for KG. Then, we introduce scoring functions. Finally, we describe the training process, including loss functions. 

\vspace{-0.05in}
\subsubsection{GNNs for learning KG representations}
\label{sec:aggregation}
KGs can be naturally treated as graphs with triplets being the relational edges. When GNN models are adapted to learn representations for KGs, the information aggregation process in the GNN layers is tailored for handling such relational data (triplets). In this paper, we investigate three representative GNN-based models for KGC, i.e., \textbf{CompGCN} \cite{VashishthSNT20},\textbf{RGCN} \cite{schlichtkrull2018modeling} and \textbf{KBGAT}~\citep{nathani2019learning}, which are most widely adopted.
As in traditional GNN models, these models stack multiple layers to iteratively aggregate information throughout the KG. Each intermediate layer takes the output from the previous layer as the input, and the final output from the last layer serves as the learned embeddings. In addition to entity embeddings, some GNN-based models also learn relation embeddings. For a general triplet $(h,r,t)$, we use ${\bf x}^{(k)}_h$ , ${\bf x}^{(k)}_r$, and ${\bf x}^{(k)}_t$ to denote the head, relation, and tail embeddings obtained after the $k$-th layer. Specifically, the input embeddings of the first layer ${\bf x}^{(0)}_h$ , ${\bf x}^{(0)}_r$ and ${\bf x}^{(0)}_t$ are randomly initialized. RGCN~\cite{schlichtkrull2018modeling} aggregates neighborhood information with the relation-specific transformation matrices. CompGCN~\cite{VashishthSNT20}
defines direction-based transformation matrices and introduces relation embeddings to aggregate the neighborhood information. It introduces the composition operator to combine the embeddings to leverage the entity-relation information. KBGAT~\citep{nathani2019learning} proposes attention-based aggregation process by considering both the entity embedding and relation embedding. More details about the information aggregation process for CompGCN, RGCN and KBGAT can be found in {\bf Appendix~\ref{app:aggreation}}. For GNN-based models with $K$ layers, we use ${\bf x}^{(K)}_h$, ${\bf x}^{(K)}_r$, and ${\bf x}^{(K)}_t$ as the final output embeddings and denote them as ${\bf x}_h$ , ${\bf x}_r$, and ${\bf x}_t$ for the simplicity of notations. Note that RGCN does not involve relation embedding ${\bf x}_r$ in the aggregation component, which will be randomly initialized if required by the scoring function.


\subsubsection{Scoring functions}\label{sec:scoring_function}

After obtaining the final embeddings from the GNN aggregation layers, they are utilized as input to the scoring function $f$. Various scoring functions can be adopted. Two widely used scoring functions are DistMult~\citep{YangYHGD14a} and ConvE~\citep{ettmersMS018}.
The definitions of these scoring functions are as follows.
\begin{align}
   & f^{\text{DistMult}}(h,r,t) = \mathbf{x}_h \mathbf{R}_r\mathbf{x}_t \label{eq:dist}\\
   &     f^{ConvE}(h,r,t) = g(\text{vec}(g([\overline{\mathbf{x}_h}||\overline{\mathbf{x}_r}]\ast \omega))\mathbf{W})
    \mathbf{x}_t \label{eq:conve}
\end{align}
\noindent $\mathbf{R}_r \in \mathbb{R}^{d_k \times d_k}$ in Eq.~\eqref{eq:dist} is a diagonal matrix corresponding to the relation $r$. In Eq.~\eqref{eq:conve}, $\overline{\mathbf{x}_h}$ denotes a 2D-reshaping of $\mathbf{x}_h$, $\omega$ is the convolutional filter, and $\mathbf{W}$ is the learnable matrix. vec($\cdot$) is an operator to reshape a tensor into a vector. $||$ is the concatenation operator. ConvE feeds the stacked 2D-reshaped head entity embedding and relation embedding into convolution layers. It is then reshaped back into a vector that multiplies the tail embedding to generate a score.

Specifically, in~\citep{schlichtkrull2018modeling}, RGCN adopts the DistMult scoring function.  In~\citep{VashishthSNT20}, both scoring functions are investigated for CompGCN and ConvE is shown to be more suitable in most cases. Hence, in this paper, we use ConvE as the default scoring function for CompGCN. For DistMult, there are different ways to define the diagonal matrix ${\bf R}_r$: For example, in RGCN~\cite{schlichtkrull2018modeling}, the diagonal matrix is randomly initialized for each relation $r$, while CompGCN~\citep{VashishthSNT20} define the diagonal matrix by diagonalize the relation embedding ${\bf x}_r$.





\subsubsection{Training GNN-based models for KGC}\label{sec:training}






To train the GNN model, the KGC task is often regarded as a binary classification task to differentiate the true triplets in the training set from the randomly generated ``fake'' triplets.
During training, all triplets in $\mathcal{D}_{train}$ and the corresponding inverse triplets $\mathcal{D}'_{train} = \{(t,r_{in},h)| (h,r,t) \in \mathcal{D}_{train}\}$ are treated as positive samples, where $r_{in}$ is the inverse relation of $r$. The final positive sample set can be denoted as $\mathcal{D}^{*}_{train} = \mathcal{D}_{train} \bigcup \mathcal{D}'_{train}$. The negative samples are generated by corrupting the triplets in $\mathcal{D}^{*}_{train}$. 
Specifically, for a triplet $(e_1,rel,e_2)\in \mathcal{D}^{*}_{train}$, we corrupt it by replacing its tail entities with other entities in the KG. 
More formally, the set of negative samples corresponding to the triplet $(e_1,rel,e_2)$ can be denoted as:
$\mathcal{C}_{(e_1,rel,e_2)} = \{(e_1,rel,e_2')| e_2' \in \mathcal{V}, e_2' \neq e_2\}$ where $\mathcal{V}$ is the set of entities in KG.  
Binary cross-entropy (BCE) is adopted as the loss function, which can be modeled as follows:
\begin{align}
  & \nonumber \mathcal{L} =- \sum_{(e_1,rel,e_2) \in \mathcal{D}^{*}_{train}}\bigg( \log \sigma(f(e_1,rel,e_2))+ \\
  & \sum_{ (e_1,rel,e_2')\in \mathcal{C}_{(e_1,rel,e_2)}} \log (1-\sigma(f(e_1,rel,e_2')))  \bigg).
    \label{Eq:loss}
\end{align}
The loss function in Eq.~\eqref{Eq:loss} is adopted for training the CompGCN model in~\citep{VashishthSNT20}. However, not all negative samples are utilized for training the RGCN model in~\citep{schlichtkrull2018modeling}. Instead, for each positive sample triplet in $\mathcal{D}^{*}_{train}$, they adopt negative sampling to select $10$ such samples from  $\mathcal{C}_{(e_1,rel,e_2)}$, and only these chosen negative samples are utilized for training. Also, for RGCN~\cite{schlichtkrull2018modeling}, any relation ${r}$ and its inverse relation $r_{in}$ share the same diagonal matrix for DistMult in Eq.~\eqref{eq:dist}. 




\begin{figure}[t]
\begin{center}
 \centerline{
{\subfigure[\texttt{FB15K-237}\xspace]
{\includegraphics[width=0.5\linewidth]{figures/primary_findings/com_mlp_fb.pdf} }}

{\subfigure[\texttt{WN18RR}\xspace]
{\includegraphics[width=0.5\linewidth]{figures/primary_findings/com_mlp_wn.pdf}  }}
}
\vspace{-0.15in}
\caption{KGC results of CompGCN and CompGCN-MLP on \texttt{FB15K-237}\xspace and \texttt{WN18RR}\xspace. On both datasets, CompGCN-MLP achieves compare performance as CompGCN. 
}
\vspace{-0.2in}
\label{fig:com_finding}
\end{center}

\end{figure}

 \begin{figure*}[t]
\begin{center}
 \centerline{
{\subfigure[\texttt{FB15K-237}\xspace]
{\includegraphics[width=0.28\linewidth]{figures/primary_findings/rgcn_mlp_fb.pdf} }}

{\subfigure[\texttt{WN18}\xspace]
{\includegraphics[width=0.28\linewidth]{figures/primary_findings/rgcn_mlp_wn.pdf}  }}

{\subfigure[\texttt{FB15K}\xspace]
{\includegraphics[width=0.28\linewidth]{figures/primary_findings/rgcn_mlp_fb15k.pdf}  }}
}
\vspace{-0.1in}
\caption{KGC results of RGCN and RGCN-MLP on \texttt{FB15K}\xspace, \texttt{FB15K-237}\xspace and \texttt{WN18}\xspace. On all three datasets, RGCN-MLP achieves comparable performance as RGCN. }
\vspace{-0.2in}
\label{fig:rgcn_finding}
\end{center}

\end{figure*}

\begin{figure*}[t]
\begin{center}
 \centerline{
{\subfigure[\texttt{FB15K-237}\xspace]
{\includegraphics[width=0.28\linewidth]{figures/primary_findings/kbgat_mlp_fb.pdf} }}

{\subfigure[\texttt{WN18RR}\xspace]
{\includegraphics[width=0.28\linewidth]{figures/primary_findings/kbgat_mlp_wn.pdf}  }}

{\subfigure[\texttt{NELL-995}\xspace]
{\includegraphics[width=0.28\linewidth]{figures/primary_findings/kbgat_mlp_nell.pdf}  }}
}
\vspace{-0.1in}
\caption{KGC results of KBGAT and KBGAT-MLP on  \texttt{FB15K-237}\xspace , \texttt{WN18RR}\xspace and \texttt{NELL-995}\xspace. On all three datasets, KBGAT-MLP achieves comparable performance as KBGAT. }
\vspace{-0.2in}
\label{fig:kbgat_finding}
\end{center}

\end{figure*}

\subsubsection{Major differences between RGCN, CompGCN, and KBGAT}\label{sec:diff}


We demonstrate an overview of GNN-based model frameworks for the KGC task in Figure~\ref{fig:framework}. Specifically, the framework consists of several key components including the GNN aggregation component introduced in Section~\ref{sec:aggregation}, the scoring function introduced in Section~\ref{sec:scoring_function}, and the loss function introduced in Section~\ref{sec:training}. Training can typically be conducted end-to-end. Both RGCN and CompGCN follow this framework with various designs in each component. We provide a more detailed comparison about them later in this section. However, in~\cite{nathani2019learning}, KBGAT adopts a two-stage training process, which separates the training process of the GNN aggregation component (representation learning) and the scoring function. KBGAT achieves strong performance as reported in~\cite{nathani2019learning}, which was later attributed to a test leakage issue as identified in~\cite{sun2020re}. After addressing this test leakage issue, we found that fitting KBGAT into the general framework described in Figure~\ref{fig:framework} leads to much higher performance than training it with the two-stage process (around $10\%$ improvement on \texttt{FB15K-237}\xspace). Hence, in this paper, we conduct analysis for KBGAT by fitting its aggregation component (described in Eq.~\eqref{eq:kbgat} in {\bf Appendix~\ref{app:aggreation}}) into the framework described in Figure~\ref{fig:framework}. Next, we summarize the major differences between RGCN, CompGCN, and KBGAT in terms of the three major components as follows.




\begin{compactitem}[\textbullet]
    \item \textbf{Aggregation.} Their information aggregation processes are different as described in Section~\ref{sec:aggregation} and detailed in Eq.~\eqref{eq:rgcn}, Eq.~\eqref{eq:compgcn} and Eq.~\eqref{eq:kbgat} in {\bf Appendix~\ref{app:aggreation}}.
    \item \textbf{Scoring Function.} They adopt different scoring functions. RGCN adopts the DistMult scoring function while CompGCN  achieves the best performance with ConvE~\citep{VashishthSNT20}. Thus, in this paper, we use ConvE as its default scoring function. For KBGAT, we also adopt ConvE as its default scoring function.
    \item \textbf{Loss Function.} As described in Section~\ref{sec:training}, CompGCN utilizes all entities in the KG as negative samples for training, while RGCN adopts a negative sampling strategy to select only a fraction of entities as true negative samples for training. For KBGAT, we also utilizes all entities to construct negative samples, similar to CompGCN.
\end{compactitem}






 


 
\section{What Matters for GNN-based KGC?}

Recent efforts in adapting GNN models for KG mostly focus on designing more sophisticated aggregation operations to better handle multi-relational edges in KG. These recently proposed GNN-based methods have reported {\it strong performance} on the KGC task. Meanwhile, according to~\cite{schlichtkrull2018modeling,VashishthSNT20, nathani2019learning}, they, i.e., RGCN, CompGCN and KBGAT, achieve different performance. Their strong performance compared to traditional embedding based KGC models and their performance difference are widely believed to be due to the aggregation components~\cite{schlichtkrull2018modeling,VashishthSNT20,nathani2019learning}. However, as summarized in Section~\ref{sec:diff}, they differ from each other in several ways, besides their aggregation designs. Little attention has been paid to understand how each component affects these models. Thus, what truly matters for GNN-based KGC performance is still unclear. To answer this question, we design careful experiments to ablate the choices of these components in RGCN, CompGCN and  KBGAT to understand their roles, across multiple datasets. Since the aggregation component is often regarded as the major contributor, we first investigate: is the aggregation component really helpful? Subsequently, we study the impact of the scoring function and the loss function. 





\begin{table*}[!ht]


\caption{KGC results with various scoring functions. All models behave differently when adopting different scoring functions. }
\begin{adjustbox}{width =0.9 \textwidth}
\begin{tabular}{cc|llll|llll |llll }
\toprule
   & & \multicolumn{4}{c|}{\texttt{FB15K-237}\xspace} & \multicolumn{4}{c|}{\texttt{WN18RR}\xspace} &\multicolumn{4}{c}{\texttt{NELL-995}\xspace}    \\ 
         &   & MRR  & Hits@1  & Hits@3 & Hits@10 & MRR & Hits@1 & Hits@3 & Hits@10  &MRR  & Hits@1  & Hits@3 & Hits@10\\ \midrule

\multirow{2}{*}{CompGCN} &DistMult     & 0.3365 & 0.2466 &0.3694 &0.5149 &   0.4289 & 0.3896 & 0.4391 & 0.5171 &0.3225&	0.2430&	0.3613&	0.4736 \\
& ConvE& 0.3549 &0.2641 & 0.3897 & 0.5364 &0.4721 & 0.4373 &0.4852 &0.5402 &0.3813	&0.3036&	0.4222&	0.5289\\
\midrule
\multirow{2}{*}{RGCN}  &DistMult     & 0.2955 &0.1907 &0.3399 &0.5007   &0.4299 & 0.3855 &0.4500 &0.5077 &0.2781&	0.1988&	0.3135&	0.4298 \\
& ConvE& {0.2956}&{0.2031}& {0.3273}& {0.4785}&0.2892 & 0.1735 & 0.3689 & 0.4884&0.3174&	0.2329 &	0.3534	& 0.4848 \\
\midrule
\multirow{2}{*}{KBGAT}  &DistMult   & 0.3340&	0.2448	& 0.3663&	0.5132 & 0.4210&	0.3866 &	0.4308&	0.4959& 0.3302&	0.2547&	0.3682&	0.4726 \\
& ConvE& 0.3503&	0.2597 &	0.3849&	0.5307&0.4642&	0.4261&	0.4788&	0.5386& 0.3774&	0.2994	&0.4201	&0.5233 \\

\bottomrule
\end{tabular}
\label{tab:inv_score}

\end{adjustbox}
\end{table*}

\begin{table*}[!ht]


\caption{KGC results with various the loss function. The choice of loss function has significant impact on the model performance.
}
\begin{adjustbox}{width =0.9 \textwidth}
\begin{tabular}{cc|llll|llll  |llll}
\toprule
   & & \multicolumn{4}{c|}{\texttt{FB15K-237}\xspace} & \multicolumn{4}{c|}{\texttt{WN18RR}\xspace} &\multicolumn{4}{c}{\texttt{NELL-995}\xspace}    \\ 
         &   & MRR  & Hits@1  & Hits@3 & Hits@10 & MRR & Hits@1 & Hits@3 & Hits@10 & MRR & Hits@1 & Hits@3 & Hits@10  \\ \midrule

\multirow{2}{*}{CompGCN} &with &{0.3145} &   {0.2223} & {0.3478} & {0.4957} & {0.3291}& {0.2439} & {0.3898} & {0.4668} & 0.3199&	0.2375&	0.3570 &	0.4806\\
& w/o & 0.3549 &0.2641 & 0.3897 & 0.5364 &0.4721 & 0.4373 &0.4852 &0.5402 &0.3813	&0.3036&	0.4222&	0.5289 \\
\midrule
\multirow{2}{*}{RGCN}  &with     & 0.2955 &0.1907 &0.3399 &0.5007   &0.4299 & 0.3855 &0.4500 &0.5077 & 0.2781&	0.1988&	0.3135&	0.4298\\
& w/o & {0.3335} & {0.2432} & {0.3666} & {0.5138} & 0.4446 & 0.4089 & 0.4547 & 0.5184 &0.3457&	0.2696&	0.3832&	0.4940\\

\midrule
\multirow{2}{*}{KBGAT}  &with  &0.3014&	0.2102&	0.3324&	0.48103& 0.3011&	0.1864&	0.3784&	0.4978& 0.3260 &	0.2427&	0.3632&	0.4868 \\
& w/o & 0.3503&	0.2597 &	0.3849&	0.5307 &0.4642&	0.4261&	0.4788&	0.5386& 0.3774&	0.2994	&0.4201	&0.5233\\

\bottomrule
\end{tabular}
\end{adjustbox}

\label{tab:inv_loss}
\end{table*}



\subsection{Does Aggregation Really Help KGC?}
\label{sec:finding}






For RGCN~\cite{schlichtkrull2018modeling} and CompGCN~\cite{VashishthSNT20}, we follow the settings in the original papers to reproduce their reported performance. For KBGAT~\citep{nathani2019learning}, we follow the same setting of CompGCN as mentioned in Section~\ref{sec:diff}. Specifically, we run these three models in datasets in their original papers. Namely,
we run RGCN on \texttt{FB15K-237}\xspace, \texttt{WN18}\xspace and \texttt{FB15K}\xspace, CompGCN on \texttt{FB15K-237}\xspace and \texttt{WN18RR}\xspace, and  KBGAT on  \texttt{FB15K-237}\xspace, \texttt{WN18RR}\xspace and \texttt{NELL-995}\xspace. To understand the role of the aggregation component, we  keep the other components untouched and replace their aggregation components with a simple MLP, which has the same number of layers and hidden dimensions with the corresponding GCN-based models; Note that since a GNN layer is effectively an aggregation over the graph combined with a feature transformation~\cite{ma2021unified}, replacing the aggregation component with MLP can also be achieved by replacing the adjacency matrix of the graph with an identity matrix to nullify the effect of message passing. We denote the MLP models corresponding to RGCN, CompGCN and KBGAT  as RGCN-MLP, CompGCN-MLP and KBGAT-MLP, respectively.
We present the part-counterpart results for CompGCN in \figurename~\ref{fig:com_finding}
, RGCN in \figurename~\ref{fig:rgcn_finding} and  KBGAT in \figurename~\ref{fig:kbgat_finding}\footnote[1]{We conduct similar experiment using the setting in the original KBGAT paper. We find that KBGAT and KBGAT-MLP have similar performance on \texttt{FB15K-237}\xspace, \texttt{WN18RR}\xspace and \texttt{NELL-995}\xspace, which is consistent with Observation 1.}.
We summarize the key observation from these figures: 

\begin{finding}\label{find:agg} 
The counterpart MLP-based models (RGCN-MLP, CompGCN-MLP and KBGAT-MLP) achieve comparable performance to their corresponding GNN-based models on all datasets, suggesting that aggregation does not significantly improve model performance.
\label{finding}
\end{finding}

Moreover, comparing RGCN with CompGCN on \texttt{FB15K-237}\xspace, we find that they achieve very different performance, while also noting that  Observation~\ref{find:agg} clarifies that the difference in aggregation components is not the main reason leading to this difference. This naturally raises a question: what are the important contributors? According to Section~\ref{sec:diff}, RGCN and CompGCN also adopt different scoring and loss functions, which provides some leads in answering this question. Correspondingly, we next empirically analyze the impact of the scoring and the loss functions with comprehensive experiments. Note that \texttt{FB15K}\xspace and \texttt{WN18}\xspace suffer from the inverse relation leakage issue~\citep{toutanova2015observed,ettmersMS018}: a large number of test triplets can be obtained from inverting the triplets in the training set. Hence, to prevent these inverse relation leakage from affecting our studies and evaluations, we conduct experiments on three datasets \texttt{NELL-995}\xspace, \texttt{FB15K-237}\xspace and \texttt{WN18RR}\xspace, where \texttt{FB15K-237}\xspace and \texttt{WN18RR}\xspace are the filtered versions of \texttt{FB15K}\xspace and \texttt{WN18}\xspace (the inverse relation leakage issue were addressed). 







\begin{table*}[!ht]


\caption{KGC results with varying number of negative samples in the loss function. In general, only utilizing $10$ negative samples is not enough. Also, for different datasets and methods, the optimal number of negative samples varies.}
\begin{adjustbox}{width =0.9 \textwidth}
\begin{tabular}{cc|llll|llll  |llll }
\toprule
   &  & \multicolumn{4}{c|}{\texttt{FB15K-237}\xspace} & \multicolumn{4}{c}{\texttt{WN18RR}\xspace} &\multicolumn{4}{c}{\texttt{NELL-995}\xspace}    \\ 
         &  \#Negative Samples & MRR  & Hits@1  & Hits@3 & Hits@10 & MRR & Hits@1 & Hits@3 & Hits@10 & MRR & Hits@1 & Hits@3 & Hits@10  \\ \midrule

\multirow{5}{*}{CompGCN} &10 &{0.3145} &   {0.2223} & {0.3478} & {0.4957} & {0.3291}& {0.2439} & {0.3898} & {0.4668}&0.3199&	0.2375&	0.3570 &	0.4806 \\
&50 &{0.3430}&{0.2473} & {0.3809} &{0.5301} &  {0.3997}& {0.3304} & {0.4400} & {0.5164} & 0.3715&	0.2866	&0.4161&	0.5313\\
&200 &{0.3533} &{0.2553} &{0.3918} &{0.5383} &{0.4360} & {0.3919} & {0.4529} & {0.5227} & 0.3916&	0.3102&	0.4362&	0.5429 \\
&0.5$N$& {0.3455} & {0.2532} & {0.3832} & {0.5269} & {0.4402} & {0.4058} & {0.4514} & {0.5085} &0.4072&	0.3337&	0.4442&	0.5474 \\    
& $N$& {0.3420}  &{0.2500}  & {0.3791}  & {0.5224}  &{0.4401} & {0.4070}  & {0.4505} & {0.5080} & 0.4034&	0.3301&	0.4399&	0.5444\\
\midrule
\multirow{5}{*}{RGCN}  &10     & 0.2955 &0.1907 &0.3399 &0.5007   &0.4299 & 0.3855 &0.4500 &0.5077 & 0.2781&	0.1988&	0.3135&	0.4298  \\
&50 &{0.3254} & {0.2252} & {0.3671} &{0.5195} &0.4387 & 0.3955 &0.4563 &0.5181   & 0.2956	&0.2168&	0.3321&	0.4457\\
&200 & {0.3321} &{0.2320} &{0.3761} &{0.5218} &0.4411&0.3991&0.4572&0.5195 & 0.3003 &0.2171&	0.3432	&0.4572 \\    
&0.5$N$ & {0.3327} &{0.2425} &{0.3686} &{0.5092} &0.4444&0.4073 &0.4549 &0.5195 & 0.3364 &	0.2671&	0.3388 &	0.4666\\    
& $N$& {0.3298} & {0.2390} & {0.3654} & {0.5048} & 0.4449 & 0.4063 & 0.4575 & 0.5243 & 0.3367&	0.2688&	0.3698&	0.4638\\

\midrule
\multirow{5}{*}{KBGAT} & 10 & 0.3014&	0.2102&	0.3324 &	0.4810& 0.3011&	0.1864&	0.3784&	0.4978 & 0.3260&	0.2427&	0.3632&	0.4868\\
&50 &0.3358	&0.2416&	0.3726	&0.5193&0.3555&	0.2570&	0.4258&	0.5129& 0.3744&	0.2896&	0.4192&	0.5358\\
&200 & 0.3471&	0.2510&	0.3876&	0.5328&0.3963&	0.3273 &	0.4338&	0.5162&0.3915 &0.3102 &	0.4363&	0.5429 \\
&0.5$N$ &0.3397&	0.2467&	0.37703&	0.5221& 0.4431 &	0.4078&	0.4551&	0.5110& 0.4010&	0.3320&	0.4354&	0.5322 \\
& $N$ & 0.3364&	0.2435&	0.3728	&0.5198& 0.4383&	0.4007&	0.4528&	0.5112& 0.3963&	0.3277&	0.4298	&0.5286  \\
\bottomrule
\end{tabular}
\end{adjustbox}

\label{tab:inv_negativesample}
\end{table*}




\subsection{Scoring Function Impact}
\label{sec:score}









 
Next, we investigate the impact of the scoring function on CompGCN, RGCN 
and KBGAT while fixing their loss function and experimental setting mentioned in Section~\ref{sec:diff}. 
The KGC results on \texttt{FB15K-237}\xspace, \texttt{WN18RR}\xspace and \texttt{NELL-995}\xspace are shown in Table~\ref{tab:inv_score}. 
In the original setting, CompGCN and KBGAT use ConvE as the scoring function while RGCN adopts DistMult. In Table~\ref{tab:inv_score}, we further present the results of CompGCN and KBGAT with DistMult and RGCN with ConvE. Note that we only make changes to the scoring function, while fixing all the other settings. Hence, in Table~\ref{tab:inv_score}, we still use RGCN, CompGCN and KBGAT to differentiate these three models but use DistMult and ConvE to indicate the specific scoring functions adopted. From this table, we have the following observations:
\begin{compactitem}[\textbullet]
    \item In most cases, CompGCN, RGCN and KBGAT behave differently when adopting different scoring functions. For instance, CompGCN and KBGAT generally achieve better performance when adopting ConvE as the scoring function in three datasets. RGCN with DistMult performs similar to that with ConvE on \texttt{FB15K-237}\xspace. However, it dramatically outperforms RGCN with ConvE on \texttt{WN18RR}\xspace and \texttt{NELL-995}\xspace. This indicates that the choice of scoring functions has strong impact on the model performance. Also, the impact is dependent on the specific dataset that the methods are applied to. 
   
    
    
   
   
    \item  Comparing CompGCN (or KBGAT) with RGCN on \texttt{FB15K-237}\xspace, even if the two methods adopt the same scoring function (either DistMult or ConvE), they still achieve quite different performance. On the \texttt{WN18RR}\xspace dataset, the observations are more involved. The two methods achieve similar performance when DistMult is adopted but behave quite differently with ConvE. Overall, these observations indicate that the scoring function is not the only factor impacting the model performance. 
   
    
   
   
   
\end{compactitem}






\subsection{Loss Function Impact}
\label{sec:loss}
In this subsection, we investigate the impact of the loss function on these three methods while fixing the  scoring function and other experimental settings. As introduced in Section~\ref{sec:training}, in the original settings,  CompGCN, RGCN and KBGAT adopt the BCE loss. The major difference in the loss function is that CompGCN and  KBGAT utilize all negative samples while RGCN adopts a sampling strategy to randomly select $10$ negative samples for training. For convenience, we use \emph{w/o sampling} and \emph{with sampling} to denote these two settings and investigate how these two settings affect the model performance. 
\subsubsection{The impact of negative sampling}
To investigate the impact of negative sampling strategy, we also run CompGCN and KBGAT under the \emph{with sampling} setting (i.e., using $10$ negative samples as the original RGCN), and RGCN under the \emph{w/o sampling} setting. The results are shown in Table~\ref{tab:inv_loss}, where we use ``with'' and ``w/o'' to denote these two settings. From Table~\ref{tab:inv_loss}, we observe that RGCN, CompGCN and KBGAT achieve stronger performance under the ``\emph{w/o sampling}'' setting on three datasets. Specifically, the performance of CompGCN dramatically drops by $30.3\%$ from $0.4721$ to $0.3291$  when adopting the sampling strategy, indicating that the sampling strategy significantly impacts model performance. Notably, only using $10$ negative samples proves insufficient. Hence, we further investigate how the number of negative samples affects the model performance in the following subsection.  








\subsubsection{The impact of the number of negative samples}
In this subsection, we investigate how the number of negative samples affects the model performance under the ``\emph{with sampling}'' setting for both methods. Specifically, we run RGCN CompGCN and KBGAT with varied numbers of negative samples. Following the settings of scoring functions as mentioned in Section ~\ref{sec:diff}. , we adopt DistMult for RGCN and ConvE for CompGCN and KBGAT as scoring functions. The results are shown in Table~\ref{tab:inv_negativesample}. Note that in Table~\ref{tab:inv_negativesample}, $N$ denotes the number of entities in a KG. Thus, the value of $N$ is different for three datasets. In general, increasing the number of negative samples from $10$ to a larger number is helpful for all methods. This partially explains why the original RGCN typically under-performs CompGCN and KBGAT. On the other hand, to achieve strong performance, it is not necessary to utilize as many negative samples as possible. For example, on \texttt{FB15K-237}\xspace, CompGCN achieves the best performance when the number of negative samples is set to $200$; this is advantageous, as using all negative samples is more computationally expensive.  In short, carefully selecting the negative sampling rate for each model and dataset is important.












\vspace{-0.1in}
\section{KGC without Aggregation}

It is well known that the aggregation operation is the key bottleneck for GNNs scaling to large graphs \cite{jin2021graph, zhang2021graphless, zhao2021stars}. Observation~\ref{find:agg} suggests that the aggregation may be not helpful for KGC. Thus, in this section, we investigate if we can develop MLP-based methods (without aggregation) for KGC that can achieve comparable or even better performance than existing GNN-based methods. Compared with the GNN-based models, MLP-based methods enjoy the advantage of being more efficient during training and inference, as they do not involve expensive aggregation operations. The scoring and loss functions play an important role in GNN-based methods. Likewise, we next first study the impact of the scoring and loss functions on MLP-based methods. 

\begin{table*}[!ht]
\caption{KGC performance of MLP-based methods with different combinations of scoring and loss functions (including various number of negative samples). Both the scoring and loss functions impact the performance of MLP-based models. Strong performance can be achieved under the \emph{with sampling} setting, which is typically more efficient than \emph{w/o sampling} setting.}
\begin{adjustbox}{width = 0.9\textwidth}
\begin{tabular}{cc|llll|llll |llll }
\toprule
   & & \multicolumn{4}{c|}{\texttt{FB15K-237}\xspace} & \multicolumn{4}{c}{\texttt{WN18RR}\xspace} & \multicolumn{4}{c}{\texttt{NELL-995}\xspace}    \\ 
         & \#Negative Samples  & MRR  & Hits@1  & Hits@3 & Hits@10 & MRR & Hits@1 & Hits@3 & Hits@10  & MRR  & Hits@1  & Hits@3 & Hits@10    \\ \midrule

\multirow{6}{*}{DistMult} &5& 0.2448&0.1690&0.2660 & 0.3934 &0.4064 &0.3716 & 0.4258 &0.4632 & 0.2400&	0.1846&	0.2628 &	0.3453 \\
&10 &{0.2909} &   {0.1914} & {0.3271} & {0.4889} & \textbf{0.4399}& {0.3949} & \textbf{0.4572} & \textbf{0.5186} & 0.2753&	0.1995&	0.3094&	0.4198\\
&50 &{0.3131}&{0.2119} & {0.3542} &{0.5113} &  {0.4248}& {0.3827} & {0.4386} & {0.5106}  & 0.2745 &	0.1936&	0.3167&	0.4271\\
&200 &{0.3250} &{0.2230} &\textbf{0.3706} & \textbf{0.5197} & {0.4147}& {0.3756} & {0.4258} & {0.4961} & 0.2908 &	0.2106 &	0.3322 &	0.4386 \\
&0.5$N$& {0.3284} & {0.2342} & {0.3682} & {0.5074} & {0.4135}& {0.3796} & {0.4245} & {0.4848} & { 0.3153}&	0.2386	&0.3549 &	0.4528\\    
& $N$& {0.3269}  &{0.2349}  & {0.3646}  & {0.5039}  & {0.4138}& {0.3835} & {0.4221} & {0.4772} & 0.3099	&0.2335&	0.3479&	0.4508 \\
&w/o & \textbf{0.3341} & \textbf{0.2455 }& {0.3661} & {0.5105} &0.4334& \textbf{0.3990} & 0.4458 &0.5068 & {\bf 0.3275}&	{\bf 0.2499}&	{ \bf 0.3653} &	{\bf 0.4771} \\
\midrule
\multirow{7}{*}{ConvE} &10 & {0.3033} & {0.2107} & {0.3361} & {0.4857} & 0.3550 & 0.2756 & 0.4049 & 0.4925 & 0.3157&	0.2349&	0.3504&	0.4724\\
&50& {0.3404} & 0.2447 & 0.3787 & 0.5261 &0.4205 & 0.3626 &0.4502 &0.5239 & 0.3702&	0.2859&	0.4140&	0.5307 \\
&200 & {0.3503} & {0.2552} & {0.3896} & {0.5342} & 0.4419 & 0.3994 & 0.4569 & 0.5257 & 0.3886 &	0.3075 &	0.4331 &	{ \bf 0.5396} \\
&500 & 0.3525 &0.2571 & \textbf{0.3923} &0.5364 & 0.4448 &0.4062 & 0.4568 & 0.5231 & 0.3932 &	0.3164	&{ \bf 0.4354}&	0.5356 \\
&0.5$N$ & {0.3432} &{0.2503} & {0.3811} & {0.5244} & {0.4535} & {0.4175}& {0.4637} & {0.5259} & {\bf 0.3999}&{\bf	0.3329}&	0.4331 &	0.5290\\
&$N$ & {0.3395} & {0.2476} & {0.3766} &{0.5189} & {0.4541} & {0.4183} &{0.4647} & {0.5243} & 0.3972&	0.3297 &	0.4311&	0.5254 \\
&w/o &\textbf{0.3548} & \textbf{0.2637}& {0.3892} & \textbf{0.5367} & \textbf{0.4732} & \textbf{0.4370} & \textbf{0.4880 }&  \textbf{0.5441} & 0.3809 &	0.3043&	0.4211&	0.5247\\
\bottomrule
\end{tabular}
\end{adjustbox}
\label{tab:inv_mlp}
\end{table*}

\begin{table*}[!ht]
\caption{KGC performance of the ensembled MLP-based methods. MLP-ensemble outperforms the GNN-based models.}
\begin{adjustbox}{width =0.9 \textwidth}
\begin{tabular}{c|llll|llll |llll  }
\toprule
    & \multicolumn{4}{c|}{\texttt{FB15K-237}\xspace} & \multicolumn{4}{c}{\texttt{WN18RR}\xspace} & \multicolumn{4}{c}{\texttt{NELL-995}\xspace}    \\ 
            & MRR  & Hits@1  & Hits@3 & Hits@10 & MRR & Hits@1 & Hits@3 & Hits@10  & MRR  & Hits@1  & Hits@3 & Hits@10 \\ \midrule
CompGCN     &    0.3549  &    0.2641    &   0.3897    &    0.5364    &   0.4721 &   0.4373    & 0.4852   &   0.5402  & 0.3813	&0.3036&	0.4222&	0.5289    \\
RGCN     &    0.2955  &    0.1907    &   0.3399    &    0.5007 & 0.4299 & 0.3855 &0.4500 & 0.5077 & 0.2781&	0.1988&	0.3135&	0.4298\\
 KBGAT &0.3503&	0.2597 &	0.3849&	0.5307 &0.4642&	0.4261&	0.4788&	0.5386& 0.3774&	0.2994	&0.4201	&0.5233\\
 \midrule
MLP-best & 0.3548 & 0.2637& 0.3892 & {0.5367} & {0.4732} & 0.4370 & {0.4880 }& {0.5441}& {0.3999}&{0.3329}&	0.4331 &	0.5290\\
MLP-ensemble     &   \textbf{0.3690}& \textbf{0.2745} & \textbf{0.4079} & \textbf{0.5535} & \textbf{0.4767} & \textbf{0.4386 }& \textbf{0.4887} & \textbf{0.5535}  & {\bf 0.4172}&	{\bf 0.3474}	& {\bf 0.4514}&	{\bf 0.5519} \\        
\bottomrule
\end{tabular}
\end{adjustbox}
\label{tab:mlp_ensemble}
\end{table*}

\subsection{Impact of scoring  and loss functions on MLP-based models}
\label{sec:mlp}
We investigate the performance of MLP-based models with different combinations of scoring and loss functions. Specifically, we adopt DistMult and ConvE as scoring functions. For each scoring function, we try both the \emph{with sampling} and \emph{w/o sampling} settings for the loss function. Furthermore, for the \emph{with sampling} setting, we vary the number of negative samples. The results of MLP-based models with different combinations are shown in Table~\ref{tab:inv_mlp}, which begets the following observations.  
\begin{compactitem}[\textbullet]
\item The results from Table~\ref{tab:inv_mlp} further confirm that the aggregation component could be unnecessary for KGC.
The MLP-based models can achieve comparable or even stronger performance than GNN models.
\item Similarly, the scoring and the loss functions play a crucial role in the KGC performance for MLP-based methods. Meanwhile, their impact varies depending on the data. For example, it is not always necessary to adopt the \emph{w/o} setting for strong model performance: On the \texttt{FB15K-237}\xspace dataset, when adopting ConvE for scoring, the MLP-based model achieves comparable performance with $500$ negative samples; on \texttt{WN18RR}\xspace, when adopting DistMult for scoring, the model achieves best performance with $10$ negative samples; on \texttt{NELL-995}\xspace, when adopting ConvE for scoring, it achieves the best performance with $0.5N$ negative samples.
\end{compactitem}

These observations indicate that the importance of the choices of the scoring and loss functions which are dependent on the given data. Hence, in the next subsection, we study a simple ensembling strategy to combine different MLP-based models, in efforts to to obtain a strong model which can generally perform well for various datasets with limited complexity without aggregation. 







\subsection{Ensembling MLPs}\label{sec:ensemble}
In this subsection, we investigate if we can ensemble various MLP-based models to generate stronger models, which can achieve better performance. According to Section~\ref{sec:mlp}, the performance of MLP-based methods is affected by the scoring function and the loss function (especially the negative sampling strategy). These models with various combinations of scoring function and loss functions can potentially capture important information from different perspectives. Therefore, an ensemble of these models could provide an opportunity to combine the information from various models to achieve better performance. Hence, we select some MLP-based models that exhibit relatively good performance on the validation set and ensemble them for the final prediction. Next, we briefly describe the ensemble process. These selected models are individually trained, and then assembled together for the inference process. Specifically, during the inference stage, to calculate the final score for a specific triplet $(h,r,t)$, we utilize each selected model to predict a score for this triplet individually and then add these scores to obtain the final score for this triplet. The final scores are then utilized for prediction. In this work, our focus is  to show the potential of ensembling instead of designing the best ensembling strategies. Thus, we only use the above straightforward strategy. However, more sophisticated strategies can be adopted. For example, currently we use equal weights to combine predictions from different methods and we can automatically search these weights via autoML techniques~\cite{jin2021automated}. We leave this to future work.  



We briefly introduce the MLP-based models we utilized for constructing the ensemble model for these three datasets as follows. 
For the \texttt{FB15K-237}\xspace dataset, we ensemble the following models: DistMult + \emph{w/o sampling}; DistMult + \emph{with sampling} (two different settings with the number of negative samples as $0.5N$ and $N$, respectively); ConvE + \emph{w/o sampling}; ConvE + \emph{with sampling} (five different settings with the number of negative samples as $50, 200, 500,$ $0.5N$ and $N$, respectively). 
For the \texttt{WN18RR}\xspace dataset, we ensemble the following models: DistMult + \emph{w/o sampling}; ConvE+ \emph{w/o sampling}; ConvE+ \emph{with sampling} (one setting with the number of negative samples as $N$). 
For the \texttt{NELL-995}\xspace dataset, we ensemble the following models: ConvE+ \emph{w/o sampling};
ConvE+\emph{with sampling} (five settings with the number of negative samples as $50, 200, 500,0.5N$ and $N$).

The results of these ensemble methods are shown in Table~\ref{tab:mlp_ensemble}, where we use MLP-ensemble to generally denote the ensemble model for the three datasets. Note that MLP-best in the table denotes the best performance from individual MLP-based methods from Table~\ref{tab:inv_mlp}. From the table, we can clearly observe that MLP-best can achieve comparable or even slightly better performance than GNN-based methods. Furthermore, the MLP-ensemble can obtain better performance than both the best individual MLP-based methods and the GNN-based models, especially on \texttt{FB15K-237}\xspace and \texttt{NELL-995}\xspace. These observations further support that the aggregation component is not necessary. They also indicate that these scoring and loss functions are potentially complementary to each other, and as a result, even the simple ensemble method can produce better performance.  


\section{Discussion}
\noindent\textbf{Key Findings}

\begin{compactitem}[\textbullet]
\item The aggregation component in GNN-based methods does not significantly contribute to KGC performance, and MLP-based methods without aggregation can achieve comparable performance to GNN-based models. 
\item The scoring and the loss function design (i.e. using negative sampling choice and count) play a much more crucial role for both GNN-based and MLP-based methods. 
\item The impact of the scoring and the loss functions varies significantly across datasets. 
\item The scoring and the loss functions are complementary, and a simple strategy to combine them in MLP-based methods can produce better KGC performance. 
\end{compactitem}

\noindent\textbf{Practical Implications}

\begin{compactitem}[\textbullet]
\item MLP-based models do not involve the complex aggregation process and thus they are considerably more efficient than the GNN-based models \cite{zhang2021graphless}. Hence, such models are more scalable and can be applied to large-scale KGC applications for practical impact.
\item The simplicity and scalability of the MLP-based models allow the possibilities for building ensemble models, which have been demonstrated to be effective in Section~\ref{sec:ensemble}.
\item The adoption of MLP-based models enables us to more conveniently apply existing techniques to advance KGC. For instance, Neural Architecture Search (NAS) algorithms~\cite{zoph2016neural} can be adopted to automatically search better model architectures, since NAS research for MLPs is much more extensive than for GNNs.
\end{compactitem}

\noindent\textbf{Implications for Future Research}

\begin{compactitem}[\textbullet]
\item Investigating better designs of scoring and loss functions are (currently) stronger levers to improving KGC. Further dedicated efforts are required for developing suitable aggregation operations in GNN-based models for KGC.
\item MLP-based models should be adopted as default baselines for future KGC studies.  This aligns with several others which suggest the underratedness of MLPs for vision-based problems \cite{liu2021pay, tolstikhin2021mlp}.
\item The impact of scoring and loss functions on KGC is varied across datasets and they are complementary; thus designing better strategies to combine them is promising.
\item KGC is a type of link prediction. There are various GNN-based link prediction applications such as recommendations ~\cite{ying2018graph,fan2019graph,wang2019kgat}. Similarly, strong performance has been reported for GNN-based methods for these applications. This work motivates a more pressing need to understand the role of the aggregation component in these applications.
\end{compactitem}

\section{Conlusion}
In this paper, we surprisingly find that the MLP-based models are able to achieve competitive performance compared with three GNN-based models (i.e., CompGCN, RGCN and KBGAT) across a variety of datasets. It suggests that the aggregation operation in these  models is not the key component to achieve strong performance. To explore which components potentially contribute to the model performance, we conduct extensive experiments on other key components such as scoring function and loss function. We found both of them play crucial roles, and their impact varies significantly across datasets. Based on these findings, we further propose ensemble methods built upon MLP-based models, which are able to achieve better performance than GNN-based models. In the future, we would like to investigate more sophisticated strategies to ensemble MLP-based methods. Also, we would like to further explore the aggregation operation and propose more suitable ones for KGC. 
\section{Dataset}
\label{sec:data}
We use five well-known KG datasets -- Table \ref{table:datasets} details their statistics:
\begin{compactitem}[\textbullet]
    \item \textbf{FB15k}~\cite{BordesUGWY13} is a subset of the Freebase database~\cite{bollacker2008freebase} containing general facts. It is constructed by selecting a subset of entities that are both in the Wikilinks database\footnote{https://code.google.com/archive/p/wiki-links/} and Freebase. 
   
    \item \textbf{FB15k-237} \cite{toutanova2015representing,toutanova2015observed} is a subset of the FB15k which removes the inverse relations from \texttt{FB15K}\xspace to prevent direct inference.
    \item \textbf{WN18} \cite{schlichtkrull2018modeling} is subset of the WordNet database \cite{fellbaum2010wordnet} which contains lexical relations between words.
    \item \textbf{WN18RR} \cite{ettmersMS018} is a subset of the WN18. WN18 contains triplets in the test set that are generated by inverting triplets from the training set. WN18RR is constructed by removing these triplets to avoid inverse relation test leakage.
    \item \textbf{NELL-995}~\citep{XiongHW17} is constructed from the 995-th iteration of the NELL system~\citep{carlson2010toward} which constantly extracts facts from  the web.
\end{compactitem} 


\begin{table}[]
\centering
\footnotesize
 \caption{Data statistics for four datasets.}
\begin{tabular}{lccccc}
\toprule
   Datasets& Entities & Relations & Train edges &Val. edges  &Test edges  \\ 
   \midrule
    FB15k-237  & 14,541& 237  &272,115 & 17,535&20,466 \\
     WN18RR &40,943 &11&86,835 & 3,034 &3,134 \\
     WN18& 40,943&18&141,442 & 5,000 & 5,000 \\
     FB15k & 14,951 & 1,345 &483,142 & 50,000 & 59, 071 \\
     NELL-995 &75.492 &200  & 126,176 & 13,912 & 14,125\\
  \bottomrule
\end{tabular}
 \label{table:datasets}
\end{table}


\section{Evaluation Metrics}
\label{sec:metric}
We use the rank-based measures to evaluate the quality of the prediction including  Mean Reciprocal Rank (\textbf{MRR}) and \textbf{Hits@N}. Their detailed definitions are introduced below: 
\begin{compactitem}[\textbullet]
    \item Mean Reciprocal Rank (\textbf{MRR}) is the mean of the reciprocal predicted rank for the ground-truth entity over all triplets in the test set. A higher MRR indicates better performance. 
    \item \textbf{Hits@N} calculates the proportion of the groundtruth tail entities with a rank smaller or equal to $N$ over all triplets in the test set. Similar to MRR, a higher Hits@N indicates better performance.
\end{compactitem}
 These metrics are indicative, but they can be flawed when a tuple (i.e., $(h,r)$ or $(r,t)$) has multiple ground-truth entities which appear in either the training, validation or test sets. Following the filtered setting in previous works \cite{BordesUGWY13, schlichtkrull2018modeling, VashishthSNT20}, we remove the misleading entities  when ranking and report the filtered results. 

\section{Information Aggregation in GNN-based KGC}
\label{app:aggreation}

For a general triplet $(h,r,t)$, we use ${\bf x}^{(k)}_h$ , ${\bf x}^{(k)}_r$, and ${\bf x}^{(k)}_t$ to denote the head, relation, and tail embeddings obtained after the $k$-th layer. Specifically, the input embeddings of the first layer ${\bf x}^{(0)}_h$ , ${\bf x}^{(0)}_r$ and ${\bf x}^{(0)}_t$ are randomly initialized. Next, we describe the information aggregation process in the $(k+1)$-th layer for the studied three GNN-based models, i.e., CompGCN, RGCN and KBGAT. 

\begin{compactitem}[\textbullet]
\item \textbf{RGCN}~\cite{schlichtkrull2018modeling} aggregates neighborhood information with the relation-specific transformation matrices:
\begin{equation}
\label{eq:rgcn}
    \mathbf{x}_h^{(k+1)} = g \left( \sum_{( r,t) \in \mathcal{N}_h}\frac{1}{c_{h,r}} \mathbf{W}_r^{(k)} \mathbf{x}_t^{(k)} + \mathbf{W}_o^{(k)} \mathbf{x}_h^{(k)}  \right)
\end{equation}
where $\mathbf{W}_o^{(k)} \in \mathbb{R}^{d_{k+1} \times d_k}$ and $\mathbf{W}_r^{(k)} \in \mathbb{R}^{d_{k+1} \times d_k}$ are learnable matrices. $\mathbf{W}_r^{(k)}$ corresponds to the relation $r$, $\mathcal{N}_h$ is the set of neighboring tuples $(r,t)$ for entity $h$, $g$ is a non-linear function, and $c_{h,r}$ is a normalization constant that can be either learned or predefined.

\item \textbf{CompGCN}~\cite{VashishthSNT20}
defines direction-based transformation matrices and introduce relation embeddings to aggregate the neighborhood information:
\begin{equation}
\label{eq:compgcn}
    \mathbf{x}_t^{(k+1)} = g \left( \sum_{(h, r) \in \mathcal{N}_t} \mathbf{W}_{\lambda(r)}^{(k)} \: \phi(\mathbf{x}_h^{(k)}, \mathbf{x}_r^{(k)}) \right),
\end{equation}
where $\mathcal{N}_t$ is the set of neighboring entity-relation tuples $(h,r)$ for entity $t$, $\lambda(r)$ denotes the direction of relations: original relation, inverse relation, and self-loop. $\mathbf{W}_{\lambda(r)}^{(k)} \in \mathbb{R}^{d_{k+1} \times d_k}$ is the direction specific learnable weight matrix in the $k$-th layer, and $\phi(\cdot)$ is the composition operator to combine the embeddings to leverage the entity-relation information. The composition operator $\phi(\cdot)$ is defined as the subtraction, multiplication, or cross correlation of the two embeddings \cite{VashishthSNT20}. CompGCN generally achieves best performance when adopting the cross correlation. Hence, in this work, we use the cross correlation as its default composition operation for our investigation. CompGCN updates the relation embedding through linear transformation in each layer, i.e., $\mathbf{x}_r^{(k+1)} = \mathbf{W}_{rel}^{(k)} \mathbf{x}_r^{(k)}$ where $\mathbf{W}_{rel}^{(k)}$ is the learnable weight matrix.

\item \textbf{KBGAT}~\citep{nathani2019learning}  proposes attention-based aggregation process by considering both the entity embedding and relation embedding:
\begin{equation}
\label{eq:kbgat}
    \mathbf{x}_h^{(k+1)} = g  \left( \sum_{(r,t)\in  \mathcal{N}_h} \alpha_{h,r,t}^{(k)} c_{h,r,t}^{(k)}\right)
\end{equation}
where $ c_{h,r,t}^{(k)} = {\bf W}_1^{(k)}[{\bf x}_h^{(k)} || {\bf x}_t^{(k)} || {\bf x}_r]$,  $||$ is the concatenation operation. Note that the relation embedding is randomly initilized and shared by all layers, i.e, ${x_r^{(k)}=x_r}$. The coefficient $\alpha_{h,r,t}^{(k)}$ is the attention score for $(h,r,t)$ in the $k$-th layer, which is formulated as follows:
\begin{equation}
\label{eq:kbgat_attention}
   \alpha_{h,r,t}^{(k)} = \frac{\exp(\text{LeakyReLU}({\bf W}_2^{(k)} c_{h,r,t}^{(k)}))}{\sum_{(r,t')\in \mathcal{N}_h}\exp(\text{LeakyReLU}({\bf W}_2^{(k)} c_{h,r,t'}^{(k)}))}
\end{equation}
where ${\bf W}_1^{(k)} \in \mathbb{R}^{d_{k+1}\times 3d_{k}}$,  ${\bf W}_2^{(k)} \in \mathbb{R}^{1 \times d_{k+1}}$ are two sets of learnable parameters.

\end{compactitem}
For GNN-based models with $K$ layers, we use ${\bf x}^{(K)}_h$, ${\bf x}^{(K)}_r$, and ${\bf x}^{(K)}_t$ as the final embeddings and denote them as ${\bf x}_h$ , ${\bf x}_r$, and ${\bf x}_t$ for the simplicity of notations. Note that RGCN does not involve $x_r$ in the aggregation component, which will be randomly initialized if required by the scoring function.


