
\section{0pt}{14pt plus 2pt minus 2pt}{0pt plus 2pt minus 2pt}


\usepackage{tikz}
\usetikzlibrary{pgfplots.groupplots}
\usepackage{pgfplots}\pgfplotsset{compat=1.17} 

\usepackage[font=small]{caption}
\renewcommand*{\UrlFont}{\footnotesize}

\title{A Unified Model for Reverse Dictionary and Definition Modelling}


\author{Pinzhen Chen \qquad\qquad Zheng Zhao \\
  School of Informatics, University of Edinburgh \\
  \texttt{\{pinzhen.chen, zheng.zhao\}@ed.ac.uk}}

\begin{document}
\maketitle
\begin{abstract}
We train a dual-way neural dictionary to guess words from definitions (reverse dictionary), and produce definitions given words (definition modelling). Our method learns the two tasks simultaneously, and handles unknown words via embeddings. It casts a word or a definition to the same representation space through a shared layer, then generates the other form from there, in a multi-task fashion. The model achieves promising automatic scores without extra resources. Human annotators prefer the proposed model's outputs in both reference-less and reference-based evaluation, which indicates its practicality. Analysis suggests that multiple objectives benefit learning. 

\end{abstract}

\section{Introduction}
A monolingual dictionary is a collection of words paired with their definitions on a large scale. The main use of such a resource is to find a word or a definition having known the other. Formally, the task of generating a textual definition given a word is called \textit{definition modelling}; on the contrary, the task of retrieving a word given a definition is called \textit{reverse dictionary}. Lately, the two tasks are approached using neural networks \citep{hill-etal-2016-learning-understand, noraset-etal-2017-definition}; in turn, the tasks help researchers better understand word sense and embeddings. This can also benefit low-resource languages where high-quality dictionaries are not available \citep{yan-etal-2020-bert}. Moreover, there are practical applications including language education, paraphrasing, semantic search, etc.

While previous work solves one problem at a time, we argue that both tasks can be learned and dealt with concurrently, based on the intuition that a word and its definition share the same meaning. We design a neural model to embed words and definitions into a shared semantic space, and generate them from this space. Besides, the training paradigm also includes reconstruction and embedding similarity tasks. Such a system can be viewed as a neural dictionary that supports two-way indexing and searching. Experiments on established datasets demonstrate the ease and effectiveness of our methods. Our model implementation and evaluation scripts are publicly available.\footnote{\url{https://github.com/PinzhenChen/unifiedRevdicDefmod}}

\section{Related Work}
Although research on the two tasks can be traced back to the early 2000s, recent research has shifted towards neural networks, which we describe here. 

\textbf{Reverse dictionary:} \citet{hill-etal-2016-learning-understand} pioneer the use of RNN and bag-of-words models to convert texts to word vectors, on top of which \citet{morinaga-yamaguchi-2018-improvement} add an extra word category classifier. \citet{pilehvar-2019-importance} integrates super-sense into target embeddings to disambiguate polysemous words. \citet{zhang-etal-2020-multi-channel} design a multi-channel network to predict a word with its features like category, POS tag, morpheme, sememe, etc.

Nonetheless, our work tackles the problem without disambiguation or linguistics resources. The proposed framework learns autoencodings for definitions and words, instead of mapping texts to plain word vectors. From the aspect of autoencoding, \citet{bosc-vincent-2018-auto} trains word embeddings via definition reconstruction.

\textbf{Definition modelling:}\label{sec:2-defmod} \citet{noraset-etal-2017-definition} use RNNs for definition generation, followed by \citet{gadetsky-etal-2018-conditional} who add attention and word context, and \citet{chang-etal-2018-xsense} whose model projects words and contexts to a sparse space and generate from selected dimensions only. \citet{mickus-etal-2019-mark}'s model encodes a context sentence and marks the word of interest, whereas \citet{bevilacqua-etal-2020-generationary}'s defines a flexible span of words. Apart from generating definitions freely, \citet{chang-chen-2019-word} re-formulated the task to definition retrieval from a closed dictionary, given a word with context.


\section{Methodology}
\subsection{A unified model with multi-task training}
In human languages, a word and its definition share the same meaning, despite being represented by different tokens. When approached using a neural method, we hypothesize that a word and its definition can be encoded into a consistent embedding in a semantic space too. This gives rise to our core architecture in the paper: a model that transforms inputs into a shared embedding space that can represent both words and definitions. We then have downstream networks that convert the shared embeddings back to words or definitions. Essentially, a shared representation can be viewed as an autoencoding of the ``meaning'' of a word and its definition. In the learning process, definition modelling and reverse dictionary are jointly trained to aid each other. At inference time, only half of the network needs to be used.

\begin{figure}[htb]
    \includegraphics[width=1.0\linewidth]{codwoe_drawing.pdf}
    \caption{An illustration of our designed model.}
    \label{fig:arch1}
\end{figure}

The proposed architecture with four sub-task workflows are illustrated in Figure~\ref{fig:arch1}. The autoencoding capability is accomplished through a shared linear layer \textit{L}$_{share}$ between the encoder and the decoder networks, the output of which is the encoded words and definitions. We use linear layers \textit{L}$_{in}$ and \textit{L}$_{out}$ to process words W$_{in}$ and W$_{out}$ before and after the shared layer. Likewise, we have definitions D$_{in}$ and D$_{out}$ converted to and from the shared layer, using Transformer blocks \textit{T}$_{in}$ and \textit{T}$_{out}$ \citep{vaswani-etal-2017-attention}. In addition, we encourage the shared layer's representations of the input word W$_{in}$ and definition D$_{in}$ to be as close as possible. The Transformer blocks operate on self-attention but not encoder-decoder attention.

With a word embedding distance $\mathrm{embed\_dist}$ and a token loss $\mathrm{token\_loss}$, canonical reverse dictionary and definition modelling will have losses:

{\small\setlength{\abovedisplayskip}{1ex}
    \begin{align}
        \mathcal{L}_\mathrm{revdic}=&\mathrm{embed\_dist}(\mathrm{W}_\mathit{gold}, \mathit{L}_\mathit{out}(\mathit{L}_\mathit{share}(\mathit{T}_\mathit{in}(\mathrm{D}_\mathit{in})))) \nonumber \\
        \mathcal{L}_\mathrm{defmod}=&\mathrm{token\_loss}(\mathrm{D}_\mathit{gold},  \mathit{T}_\mathit{out}(\mathit{L}_\mathit{share}(\mathit{L}_\mathit{in}(\mathrm{W}_\mathit{in})))) \nonumber
    \end{align}
}%

\noindent Our model also optimizes on the losses from word and definition autoencoding:

{\small\setlength{\abovedisplayskip}{1ex}
    \begin{align}
        \mathcal{L}_\mathrm{wordAE}=&\mathrm{embed\_dist}(\mathrm{W}_\mathit{gold}, \mathit{L}_\mathit{out}(\mathit{L}_\mathit{share}(\mathit{L}_\mathit{in}(\mathrm{W}_\mathit{in})))) \nonumber \\
        \mathcal{L}_\mathrm{defAE}=&\mathrm{token\_loss}(\mathrm{D}_\mathit{gold}, \mathit{T}_\mathit{out}(\mathit{L}_\mathit{share}(\mathit{T}_\mathit{in}(\mathrm{D}_\mathit{in})))) \nonumber
    \end{align}
}%

\noindent The distance between the shared word and definition representations is calculated as:

{\small\setlength{\abovedisplayskip}{1ex}
    \begin{align}
        \mathcal{L}_\mathrm{sim}=\mathrm{embed\_dist}(\mathit{L}_\mathit{share}(\mathit{T}_\mathit{in}(\mathrm{D}_\mathit{in})), \mathit{L}_\mathit{share}(\mathit{L}_\mathit{in}(\mathrm{W}_\mathit{in}))) \nonumber
    \end{align}
}%

\noindent Finally, our training minimizes the overall objective that includes all losses weighted equally:

{\small\setlength{\abovedisplayskip}{1ex}
    \begin{align}
        \mathcal{L} = \mathcal{L}_\mathrm{revdic} + \mathcal{L}_\mathrm{defmod} + \mathcal{L}_\mathrm{wordAE} + \mathcal{L}_\mathrm{defAE} + \mathcal{L}_\mathrm{sim} \nonumber
    \end{align}
}%

\subsection{Word-sense disambiguation}
A word is usually associated with multiple definitions due to the presence of polysemy, sense granularity, etc.
In our practice, the one-to-many word-definition relationship does not harm reverse dictionary, where our model can master mapping different definitions into the same word vector. It is problematic for definition modelling, as telling the exact word sense without context is hard.
Thus, we embed words in their usage context (provided by the data we use) using BERT \citep{devlin-etal-2019-bert}. Since it operates on sub-words, we sum up the sub-word embeddings for each word.

\section{Experiments and Results}
\label{sec:4-exp}

\subsection{Data and evaluation}
\textbf{\textsc{Hill}}: we evaluate reverse dictionary on \citet{hill-etal-2016-learning-understand}'s English data. There are roughly 100k words
and 900k word-definition pairs. Three test sets are present to test a system's memorizing and generalizing capabilities: 500 \textit{seen} pairs from training data, 500 \textit{unseen} pairs, and 200 \textit{human description} pairs. The evaluation metrics are ranking accuracies at 1, 10 and 100, as well as the median and standard deviation of the target words' ranks.\footnote{Previous papers might use ``standard deviation'' and ``rank variance'' interchangeably. We stick to ``standard deviation''.



\begin{table*}[t]
\centering\vspace{-1ex}
\hfill
\begin{subtable}[h]{.66\textwidth}
\small
\centering
\setlength{\tabcolsep}{0.6ex}
\begin{tabular}{|l|cccc|cccc|} 
 \hline
  & \multicolumn{4}{c|}{unseen} & \multicolumn{4}{c|}{human description} \\
 \cline{2-9}
  & \makecell{med.\\rank} & \makecell{acc@\\1/10/100} & \makecell{rank\\std.\textsuperscript{\textdagger}} & \makecell{real\\std.} & \makecell{med.\\rank} & \makecell{acc@\\1/10/100} & \makecell{rank\\std.\textsuperscript{\textdagger}} & \makecell{real\\std.} \\
 \hline
 OneLook.com  & - & - & - & - & 5.5& .33/.54/.76 & 332 & - \\
 bag-of-words & 248 & .03/.13/.39 & 424 & - & 22 &.13/.41/.69 & 308 & - \\
 RNN          & 171 & .03/.15/.42 & 404 & - & 17 & .14/.40/.73 & 274 & - \\
 category infer & 170 & .05/.19/.43 & 420 & - & 16 & .14/.41/.74 & 306 & - \\
 multi-sense  & 276 & .03/.14/.37 & 426 & - & 1000 &.01/.04/.18 & 404 & - \\
 super-sense  & 465 & .02/.11/.31 & 454 & - & 115 &.03/.15/.47 & 396 & - \\
 multi-channel & 54 & .09/.29/.58 & \textbf{358} & - & \textbf{2} & \textbf{.32}/.64/.88 & 203 & - \\
 \hline
 Transformer & 79 & .01/.14/.59 & 473 & 125 & 27 & .05/.23/.87 & 332 & 49 \\
 unified     & \textbf{18} & \textbf{.13}/\textbf{.39}/\textbf{.81} & {386}  & \textbf{93} & 4 & .22/.64/\textbf{.97} & \textbf{183} & \textbf{30} \\
 \ + share embed  & 20 & .08/.36/.77 & 410 & 99 & 4 & .23/\textbf{.65}/\textbf{.97} & \textbf{183} & 32 \\
 \hline
\end{tabular}
\caption{results on \textsc{Hill} with past results from \citeauthor{zhang-etal-2020-multi-channel}'s re-run.\footnotesize\textsuperscript{\textdagger}They force-set a rank larger than 100 to 1000; we follow suit for comparison, but also include the real std.}\vspace{-1ex}
\label{tab:1}
\end{subtable}
\hfill
\begin{subtable}[h]{.3\textwidth}
\small
\centering
\setlength{\tabcolsep}{0.5ex}
\begin{tabular}{|l|c|c|}
\hline
     & \multicolumn{2}{c|}{unseen} \\
\cline{2-3}
    & \textsc{Bleu} & \textsc{Rouge-l} \\
\hline
 RNN       & 1.7           & 15.8 \\
 xSense    & 2.0  & 15.9 \\
\hline
 Transformer  & 2.4 & 17.9\\

 unified       & 2.2 & 18.5 \\
 \ + share embed  & \textbf{3.0} & \textbf{20.2} \\
\hline
\end{tabular}
\caption{results on \textsc{Chang} with past results from \citeauthor{chang-chen-2019-word}'s replicate.}\vspace{-1ex}
\label{tab:2}
\end{subtable}
\hfill
\caption{Test performance of reverse dictionary (\ref{tab:1} on the left) and definition modelling (\ref{tab:2} on the right).}\vspace{-1ex}
\label{tab:3}
\end{table*}

\textbf{\textsc{Chang}}: definition modelling is experimented on \citet{chang-chen-2019-word}'s data from the Oxford English Dictionary.
Each instance is a tuple of a word, its usage (context), and a definition. The data has two splits: \textit{seen} and \textit{unseen}. The \textit{seen} split has a training size of 556k, and a test set comprising 9.3k words with 151.3k context and definitions; the \textit{unseen} split consists of 530k training instances, and the test set is 1k words paired with 16.0k context and definitions.
The performance is measured by corpus \textsc{Bleu} from \textsc{Nltk}, and \textsc{Rouge-l} F1\footnote{\url{https://github.com/pltrdy/rouge}} \citep{papineni-etal-2002-bleu, lin-2004-rouge, nltk2009}.


\subsection{The questionable \textit{seen} test set}
Understandably, a dictionary needs to ``memorize'' word entries, so both \textsc{Hill} and \textsc{Chang} supply a \textit{seen} test drawn from the training data. However, this is impractical in deep learning, for it implicitly encourages overfitting. Moreover, the foremost goal of building a neural dictionary is not to memorize seen words; otherwise a traditional rule-based one suffices. Hence, we omit evaluation on the \textit{seen} tests and request future research to not focus on it.

\subsection{System configurations}
Our baseline is a 4-layer Transformer block: a Transformer encoder for reverse dictionary, and a Transformer decoder for definition modelling. Encoder-decoder attention is not present in any system. Hyperparameters are detailed in Appendix~\ref{sec:appendix-hyper-parameters}. We apply a whitespace tokenizer to split all training definitions into an open vocabulary. We use mean squared error (MSE) as the distance function for embeddings, and cross-entropy for definition tokens (the baseline fails to converge with cosine similarity on embeddings).

Our proposed model essentially joins and trains the above two Transformer baselines. The shared layer has the same size as the input embeddings and a residual connection \citep{he-etal-2016-deep-residual}. As an additional variant, we tie both Transformer blocks' embedding and output layers \citep{press-wolf-2017-using}. This is only possible with our multi-task framework, since a vanilla Transformer block will have either an encoder or decoder embedding layer but not both. The unified model optimizes roughly twice as many parameters as a single-task baseline; in other words, to perform both tasks, our systems have the same size as the baseline models.

For reverse dictionary, we compare with a number of existing works on \textsc{Hill}: {OneLook.com}, bag-of-words, RNN \citep{hill-etal-2016-learning-understand}, category inference \citep{morinaga-yamaguchi-2018-improvement}, multi-sense \citep{kartsaklis-etal-2018-mapping}, super-sense \citep{pilehvar-2019-importance} and multi-channel \cite{zhang-etal-2020-multi-channel}.
Following \citet{zhang-etal-2020-multi-channel}'s treatment, we embed target words with 300d \textit{word2vec} \citep{mikolov-etal-2013-efficient}, but definition tokens are encoded from one-hot to 256d, instead of pretrained embeddings. Such an embedding choice ensures a fair comparison to previous works.

For definition modelling on \textsc{Chang}, words are embedded by 768d \textit{BERT-base-uncased}, while definition tokens are one-hot. We include RNN \citep{noraset-etal-2017-definition} and xSense \citep{chang-etal-2018-xsense} for reference but not \citet{chang-chen-2019-word}'s results
from an oracle retrieval experiment

\subsection{Results}

\textbf{Reverse dictionary} results in Table~\ref{tab:1} show a solid baseline, which our proposed models significantly improve upon. Compared to previous works, we obtain the best ranking and accuracies on \textit{unseen} words. On \textit{human descriptions} our models yield compelling accuracies with the best standard deviation, indicating a consistent performance.


One highlight is that our model attains a superior position without extra linguistic resources, other than a word embedder which is always used in previous research. Consequently, ours can be a concluded as a more generic method for this task.

\textbf{Definition modelling} results are in Table~\ref{tab:2}. On the \textit{unseen} test, our model with tied embeddings achieves state-of-the-art \textsc{Bleu} and \textsc{Rouge-l}. The model without it has performance similar to the baseline. Nonetheless, the single-digit \textsc{Bleu} hints that the quality of the generation is overall poor.


\section{Analysis and Discussions}
\subsection{Shared embeddings and the vocabulary}\label{sec:vocabulary}
For definition modelling, a shared embedding and output layer brings significant improvement to our proposed approach, but in reverse dictionary, our models arrive at desirable results without it. This is reasonable as well-trained embedding and output layers particularly benefit language generation. It further indicates that our multi-task approach is useful, whereby all embedding and output layers share the same weights, in the Transformer sub-models for the two tasks.

We have used an open vocabulary, which has weaknesses like being oversized and vulnerable to unknown tokens. Therefore, we add a model with a 25k unigram SentencePiece vocabulary \citep{kudo-richardson-2018-sentencepiece} to definition modelling. All other configurations remain the same as the best-performing model. \textsc{Bleu} and \textsc{Rouge-L} drop to 2.5 and 18.7, implying that an open vocabulary is not an issue in our earlier experiments


\subsection{Human evaluation on definitions}
For definition modelling, we notice that the low \textsc{Bleu} may not be indicative. As a further investigation, we conduct both reference-less and reference-based human evaluation, on the Transformer baseline and the best-performing unified model. In a \textit{reference-less} evaluation, a human sees a source word, and picks the preferred definition output, whereas in a \textit{reference-based} setting, a human sees the reference definition instead. In each setting, test instances are sampled, then the models' outputs are presented in a shuffled order to evaluators. Two annotators, in total, evaluated 80 test instances for each setting.
We record the number of times each model is favoured over the other in Table~\ref{tab:human_eval}.

\begin{table}[bht]
\small
\centering
\setlength{\tabcolsep}{0.65ex}\vspace{-1ex}
\begin{tabular}{|l|c|c|}
\hline
    & reference-less & reference-based \\
\hline
 Transformer      & 25 (31\%) & 32 (40\%) \\
 unified + share embed  & \textbf{50 (63\%)} & \textbf{42 (53\%)}  \\
\hline
\end{tabular}
\caption{Chances a model's output is preferred by human evaluators. Columns do not add up to 80 (100\%), because we do not count cases where both models output the same.}
\label{tab:human_eval}
\vspace{-1ex}
\end{table}

Regardless of the evaluation settings, the human evaluators favour our model's outputs over the baseline's. Specifically in the reference-less evaluation, which resembles a real-life application of definition generation, our proposed model wins notably.

\subsection{Ablation studies on the objectives}
Our models are trained with five objectives from five tasks: definition modelling and reverse dictionary, two reconstruction tasks and a shared embedding similarity task. We design an ablation study to understand how multi-task learning contributes to performance. We designate our unified model on \textsc{Hill}'s reverse dictionary with shared embeddings a ``5-task'' model. From there, we exclude word and embedding reconstruction by disabling respective losses to form a ``3-task'' model. Further, we build a single-task model by removing definition modelling and embedding similarity losses. We then run similar experiments for definition modelling. We plot the statistics during training in Figure~\ref{fig:ablation_losses}: embedding MSE against epochs for reverse dictionary, and generation cross-entropy against epochs for definition modelling. The curve plotting stops when validation does not improve.

\input{ablation_plots.tex}

As Figure~\ref{fig:ablation_losses}a shows, the single-task \textsc{Hill} model does not converge, probably because in reverse dictionary the Transformer block is far away from the output end, and only receives small gradients from just one loss. The 3-task and 5-task models display similar losses, but the 3-task loss curve is smoother. In Figure~\ref{fig:ablation_losses}b for definition modelling, the 3-task model trains the fastest, but 1-task and 5-task models reach better convergence. The analysis implies that training on two tasks is always beneficial, and reconstruction is helpful but not crucial.


\section{Conclusion}
We build a multi-task model for reverse dictionary and definition modelling. The approach records good numbers on public datasets. Also, our method delegates disambiguation to BERT and minimizes dependency on linguistic resources, so it can potentially be made cross-lingual and multilingual.
A limitation is that the evaluation centres on English, without exploring low-resource languages.


\section*{Acknowledgements}
\lettrine[image=true, lines=2, findent=1ex, nindent=0ex, loversize=.15]{eu-logo.png} Pinzhen Chen is supported by funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 825303 (Bergamot). Zheng Zhao is supported by the UKRI Centre for Doctoral Training in Natural Language Processing (UKRI grant EP/S022481/1). This work reflects the view of the authors and not necessarily that of the funders.



