\section{Introduction}
\label{sec:intro}


Keywords are words (or multi-word expressions) that best describe the subject of a document, effectively summarise it and can also be used in several document categorization tasks. In online news portals, keywords help efficient retrieval of articles when needed. Similar keywords characterise articles of similar topics, which can help editors to link related articles, journalists to find similar articles and readers to retrieve articles of interest when browsing the portals. For journalists manually assigning tags (keywords) to articles represents a demanding task, and high-quality automated keyword extraction shows to be one of components in news digitalization process that many media houses seek for.

The task of keyword extraction can generally be tackled in an unsupervised way, i.e., by relying on frequency based statistical measures \citep{campos2018yake} or graph statistics \citep{vskrlj2019rakun}, or with a supervised keyword extraction tool, which requires a training set of sufficient size and from appropriate domain. While supervised methods tend to work better due to their ability to adapt to a specifics of the syntax, semantics, content, genre and keyword assignment regime of a specific text \citep{martinc2020tnt}, their training for some less resource languages is problematic due to scarcity of large manually annotated resources. For this reason, studies about supervised keyword extraction conducted on less resource languages are still very rare. To overcome this research gap, in this paper we focus on supervised keyword extraction on three less resource languages, Croatian, Latvian and Estonian, and one fairly well resourced language (Russian) and conduct experiments on data sets of media partners in the EMBEDDIA project\footnote{http://embeddia.eu/}.

In media house environments, automatic keyword extraction systems are expected to return a diverse list of keyword candidates (of constant length), which is then inspected by a journalist who manually selects appropriate candidates. While the supervised language model based approaches in most cases offer good enough precision for this type of usage as a recommendation system, the recall of these systems is nevertheless problematic. Supervised systems learn how many keywords should be returned for each news article on the gold standard train set, which generally contains only a small amount of manually approved candidates for each news article. For example, among the datasets used in our experiments (see Section \ref{sec:dataset}), the Russian train set contains the most (on average 4.44) present keywords (i.e., keywords which appear in the text of the article and can be used for training of the supervised models) per article, while the Croatian test set contains only 1.32 keywords per article. This means that for Croatian, the model will learn to return around 1.32 keywords for each article, which is not enough.

To solve this problem we show that we can improve the recall of the existing supervised keyword extraction system by:

\begin{itemize}
    \item Proposing an additional TF-IDF tagset matching technique, which finds additional keyword candidates by ranking the words in the news article that have appeared in the predefined keyword set containing words from the gold standard train set. The new hybrid system first checks how many keywords were returned by the supervised approach and if the number is smaller than needed, the list is expanded by the best ranked keywords returned by the TF-IDF based extraction system.
    \item Combining the outputs of several state-of-the-art supervised keyword extraction approaches. 
\end{itemize}

The rest of this work is structured as follows: Section \ref{sec:dataset} describes the datasets on which we evaluate our method. Section \ref{sec:methodology} describes our proposed method with all corresponding steps. The experiment settings are described in Section \ref{sec:media-experiments} and the evaluation of the proposed methods is shown in Section \ref{sec:eval}, followed by the conclusions and the proposed further work in Section \ref{sec:conc}.  
    
\section{Related work}
\label{sec:related-work}

Many different approaches have been developed to tackle the problem of extracting keywords. The early approaches, such as KP-MINER \citep{el2009kp} and RAKE \citep{rose2010automatic} rely on unsupervised techniques which employ frequency based metrics for extraction of keywords from text. Formally, aforementioned approaches search for the words $w$ from vocabulary $\mathcal{V}$ that maximize a given metric $h$ for a given text $t$: 
\begin{equation*}
    \textrm{kw} = \argmax_{w \in \mathcal{V}} h(w,t). 
\end{equation*}
In these approaches, frequency of high relevance. These approaches assume that the more frequent a given word, the more important the meaning this word carries for a given document. Most popular such metrics are the na\"ive frequency - simple counting and the term frequency-inverse document frequency (TF-IDF) \cite{tfidf}. 

Most recent state-of-the-art statistical approaches, such as YAKE \citep{campos2018yake}, also employ frequency based features, but combine them with other features such as casing, position, relatedness to context and dispersion of a specific term in order to derive a final score for each keyword candidate.

Another line of research models this problem by exploiting concepts from graph theory. Approaches, such as TextRank \citep{mihalcea2004textrank}, Single Rank \citep{wan2008single}, TopicRank \citep{bougouin2013topicrank} and Topical PageRank \citep{sterckx2015topical} build a graph $G$, i.e., a mathematical construct described by a set of vertexes $V$ and a set of edges $E$ connecting two vertices. In one of the most recent approaches called RaKUn \cite{vskrlj2019rakun}, a directed graph is constructed from text, where vertexes $V$ and two words $w_{i}, w_{i+1}$ are linked if they appear following one another. Keywords are ranked by a shortest path-based metric from graph theory - the load centrality.
   
The task of keyword extraction can also be tackled in a supervised way. One of the first supervised approaches was an algorithm named KEA \citet{witten2005kea}, which uses only TF-IDF and the term's position in the text as features for term identification. More recent neural approaches to keyword detection consider the problem as a sequence-to-sequence generation task \cite{meng2017deep} and employ a generative model for keyword prediction with a recurrent encoder-decoder framework and an attention mechanism capable of detecting keywords in the input text sequence whilst also potentially finding keywords that do not appear in the text. 

Finally, the newest branch of models consider keyword extraction as a sequence labelling task and tackle keyword detection with transformers. \citet{sahrawat2019keyphrase} fed contextual embeddings generated by several transformer models (BERT \citep{devlin2018bert}, RoBERTa \citep{liu2019roberta}, GPT-2 \citep{radford2019gpt2}, etc.) into two types of neural architectures, a bidirectional Long short-term memory network (BiLSTM) and a BiLSTM network with an additional Conditional random fields layer (BiLSTM-CRF). 

Another state-of-the-art transformer based approach is TNT-KID (Transformer-based Neural Tagger for Keyword Identification) \cite{martinc2020tnt}, which does not rely on pretrained language models such as BERT, but rather allows the user to train their own language model on the appropriate domain. The study shows that smaller unlabelled domain specific corpora can be successfully used for unsupervised pretraining, which makes the proposed approach easily transferable to low-resource languages. It also proposes several modifications to the transformer architecture in order to adapt it for a novel task and improve performance of the model. 

\section{Data description}
\label{sec:dataset}


We conducted experiments on datasets containing news in four languages; Latvian, Estonian, Russian and Croatian. Latvian, Estonian and Russian datasets contain news from the Ekspress Group, specifically from Estonian Ekspress Meedia (news in Estonian and Russian) and from Latvian Delfi (news in Latvian and Russian). The dataset statistics are presented in Table \ref{tab:media-dataset}. 
The media-houses provided news articles from 2015 up to the 2019. We divided them into training and test sets. For the training set, we used the articles from 2018, while for the test set the articles from 2019 were used. In our study, we also use tagsets of keywords. Tagset corresponds either to a collection of keywords maintained by editors of a media house (see e.g. Estonian tagset), or an automatically constructed tagset from assigned keywords from articles available in a training set. The type of tagset and the number of unique tags for each language are listed in Table \ref{tab:media-tagset}.


\begin{table}[h]
    \centering
    \caption{Distribution of tags provided per language. The media houses provided tagsets for Estonian and Russian, while the tags for Latvian and Croatian were considered from the train dataset.}
        \begin{tabular}{| c | c | c |}
        \hline
        Dataset & Unique tags & Type of tags\\ \hline
        Croatian & 25599 & Constructed  \\ 
        Estonian & 52068 & Provided \\ 
        Russian & 5899 &  Provided  \\  
        Latvian & 4015 & Constructed  \\ \hline

    \end{tabular}
    \label{tab:media-tagset}
\end{table}












\begin{table*}[h]
    \centering
    \caption{Media partners' datasets used for empirical evaluation of keyword extraction algorithms. 
    \resizebox{\linewidth}{!}{
        \begin{tabular}{| c | c | c | c | c | c | c | c | c | c | c | c | c |}
        \cline{4-13}
         \multicolumn{3}{c}{}& \multicolumn{5}{|c|}{ Avg. Train} & \multicolumn{5}{|c|}{Avg. Test} \\ \hline

         Dataset & Total docs &  Total kw. & Total docs   & doc len. &  kw. & \% present kw. & present kw. & Total docs & doc len & kw. & \% present kw. & present kw.   \\ \hline
        Croatian & 52756 & 26896 & 47479 & 420.32 & 3.10 & 0.47 & 1.32 & 5277 &  464.14 & 3.28 & 0.55 & 1.62   \\ 
        Estonian & 18497 & 59242 & 10750 & 395.24 & 3.81 &  0.65 & 2.77 & 7747 & 411.59 & 4.09 & 0.69 & 3.12  \\ 
        Russian & 25306 & 5953 & 13831  & 392.82 & 5.66 & 0.76 & 4.44 & 11475 & 335.93 &  5.43 & 0.79 & 4.33 \\  
        Latvian & 24774 &  4036 & 13133 & 378.03 & 3.23 & 0.53 & 1.69 & 11641 & 460.15 & 3.19 & 0.55 & 1.71   \\ \hline
    \end{tabular}}
    \label{tab:media-dataset}
\end{table*}





\section{Methodology}
\label{sec:methodology}

The recent supervised neural methods are very precise, but, as was already mentioned in Section \ref{sec:intro}, in same cases they do not return a sufficient number of keywords. This is due to the fact that the methods are trained on the training data with a low number of gold standard keywords (as it can be seen from Table \ref{tab:media-dataset}). To meet the media partners' needs, we designed a method, which complements state-of-the-art neural methods (the TNT-KID method \cite{martinc2020tntkid} and the transformer-based method proposed by \cite{sahrawat2019keyphrase}, which are both described in Section \ref{sec:related-work}) by a tagset matching approach, returning constant number of keywords ($k$=10). 

\subsection{TF-IDF(tm) tagset matching}

In our approach, we first take the keywords returned by a neural keyword extraction method and next complement the returned keyword list by adding the missing keywords to achieve the set goal of $k$ keywords. The added keywords are selected by taking the top-ranked candidates from the TF-IDF tagset matching extraction conducted on the preprocessed news articles and keywords. 

\subsubsection{Preprocessing}

First, we concatenate the body and the title of the article. After that we lowercase the text and remove stopwords. Finally, the text is tokenized and lemmatized with the Lemmagen3 lemmatizer\citep{jurvsic2010lemmagen}, which supports lemmatization for all the languages except Latvian. For Latvian we use the LatvianStemmer \footnote{\url{https://github.com/rihardsk/LatvianStemmer}}. For the stopword removal we used the \textit{Stopwords-ISO} \footnote{\url{https://github.com/stopwords-iso}} Python library which contained stopwords for all four languages. The final cleaned textual input consists of the concatenation of all of the preprocessed words from the document. We apply the same preprocessing procedure on the predetermined tagsets for each language. The preprocessing procedure is visualized in Figure \ref{fig:prep}.


\begin{figure}[h]
\resizebox{\linewidth}{!}{\includegraphics{graphics/fig_prep.png}}
\caption{Preprocessing pipeline used for the document normalization and cleaning.} 
\label{fig:prep}
\end{figure}
 
\subsubsection{TF-ID weighting scheme}
\label{sec:tf-idf}

The TF-IDF weighting scheme \cite{tfidf} assigns each word its weight $w$ based on the frequency of the word in the document (term frequency) and the number of documents the word appears in (inverse document frequency). More specifically, TF-IDF is calculated with the following equation:
   
\begin{equation*}
     \begin{aligned}
         TF-IDF\textsubscript{i} = tf\textsubscript{i,j} \cdot \log_e(\frac{|D|}{df_{i}})
     \end{aligned}
\end{equation*}

The formula has two main components:
\begin{itemize}
 \item  \textit{Term-frequency} (tf) that counts the number of appearances of a word in the document (in the equation above, $tf\textsubscript{i,j}$ denotes  the number of occurrences of the word $i$ in the document $j$)
 \item  \textit{Inverse-document-frequency} (idf) ensures that words appearing in more documents are assigned lower weights (in the formula above $df_{i}$ is the number of documents containing word $i$ and $|D|$ denotes the number of documents).
\end{itemize}

The assumption is that words with a higher TF-IDF value are more likely to be keywords. 


\subsection{Tagset matching keyword expansion}

For a given neural keyword extraction method \textit{N}, and for each document \textit{d}, we select $l$ best ranked keywords according to the TF-IDF(tm), which appear in the keyword tagset for each specific dataset. Here, \textit{l} corresponds to \textit{k} - \textit{m}, where $k=10$ and \textit{m} corresponds to the number of keywords returned by a neural method.

Since some of the keywords in the tagsets provided by the media partners were variations of the same root word (i.e., keywords are not lemmatized), we created a mapping from a root word (i.e., a word lemma or a stem) to a list of possible variations in the keyword dataset. For example, a word \textit{'riigieksam'} (\textit{'exam'}) appearing in the article, could be mapped to three tags in the tagset by the Estonian media house with the same root form \textit{'riigieksam'}: \textit{'riigieksamid', 'riigieksamide'} and \textit{'riigieksam'}.

We tested several strategies for mapping the occurrence of a word in the news article to a specific tag in the tagset. For each lemma that mapped to multiple tags, we tested returning a random tag, a tag with minimal length and a tag of maximal length. In the final version, we opted to return the tag with the minimal length, since this tag corresponded to the lemma of the word most often selected.

\section{Experimental settings}
\label{sec:media-experiments}

We conducted experiments on the datasets described in Section \ref{sec:dataset}. We evaluate the following methods and combinations of methods: 

\begin{itemize}
\item \textbf{TF-IDF(tm):} Here, we employ the preprocessing and TF-IDF-based weighting of keywords described in Section \ref{sec:methodology} and select the top-ranked keywords that are present in the tagset. 

\item \textbf{TNT-KID} \cite{martinc2020tntkid}: For each dataset, we first pretrain the model with an autoregressive language model objective. After that, the model is fine-tuned on the same train set for the keyword extraction task. We use the same hyperparameters as proposed in the original study (i.e., sequence length of 256, embedding size of 512 and batch size 8) and employ the same preprocessing (i.e., we lowercase and tokenize the text with a Sentencepiece \citep{kudo2018sentencepiece} byte-pair encoding tokenization scheme).

\item \textbf{BERT + BiLSTM-CRF}\citep{sahrawat2019keyphrase}: We employ an uncased multilingual BERT\footnote{More specifically, we use the 'bert-base-multilingual-uncased' implementation of BERT from the Transformers library (\url{https://github.com/huggingface/transformers}).} model with an embedding size of 768 and 12 attention heads, with an additional BiLSTM-CRF token classification head, same as in \citep{sahrawat2019keyphrase}.

\item \textbf{TNT-KID \& BERT + BiLSTM-CRF}: We extracted keywords with both of the methods and complemented the TNT-KID extracted keywords with the BERT + BiLSTM-CRF extracted keyword in order to retrieve more keywords. Duplicates (i.e., keywords extracted by both methods) are removed. 

\item \textbf{TNT-KID \& TF-IDF}: If the keyword set extracted by TNT-KID contains less than 10 keywords, it is expanded with keywords retrieved with the proposed TF-IDF(tm) approach, i.e., best ranked keywords according to TF-IDF, which do not appear in the keyword set extracted by TNT-KID.

\item \textbf{BERT + BiLSTM-CRF  \& TF-IDF}: If the keyword set extracted by BERT + BiLSTM-CRF contains less than 10 keywords, it is expanded with keywords retrieved with the proposed TF-IDF(tm) approach, i.e., best ranked keywords according to TF-IDF, which do not appear in the keyword set extracted by BERT + BiLSTM-CRF.

\item \textbf{TNT-KID \& BERT + BiLSTM-CRF \& TF-IDF:} the keyword set extracted with the TNT-KID is complemented by keywords extracted with BERT + BiLSTM-CRF (duplicates are removed). If after the expansion the keyword set still contains less than 10 keywords, it is expanded again, this time with keywords retrieved by the TF-IDF(tm) approach.

\end{itemize} 

For TNT-KID, which is the only model that requires language model pretraining, language models were trained on train sets in Table \ref{tab:media-dataset} for up to ten epochs. Next, TNT-KID and BERT + BiLSTM-CRF  were fine-tuned on the training datasets, which were randomly split into 80 percent of documents used for training and 20 percent of documents used for validation. The documents containing more than 256 tokens are truncated, while the documents containing less than 256 tokens are padded with a special $<\textrm{pad}>$ token at the end. We fine-tuned each model for a maximum of 10 epochs and after each epoch the trained model was tested on the documents chosen for validation. The model that showed the best performance on this set of validation documents (in terms of F@10 score) was used for keyword detection on the test set. 


\section{Evaluation}
    \label{sec:eval}

For evaluation, we employ precision, recall and F1 score. While F1@10 and recall@10 are the most relevant metrics for the media partners, we also report precision@10, precision@5, recall@5 and F1@5. Only keywords which appear in a text (present keywords) were used as a gold standard, since we only evaluate approaches for keyword tagging that are not capable of finding keywords which do not appear in the text. Lowercasing and lemmatization (stemming in the case of Latvian) are performed on both the gold standard and the extracted keywords (keyphrases) during the evaluation. The results of the evaluation of the best-performing models on all four languages is listed in Table \ref{tab:kw_media_eval}.

\begin{table*}[h]
    \centering
    \caption{Results on the EMBEDDIA media partner datasets.}
    \resizebox{\linewidth}{!}{
    \begin{tabular}{| c | c | c | c | c | c | c |}
        \hline
        Model & P@5 & R@5 & F1@5 & P@10 & R@10 & F1@10 \\ \hline
        \multicolumn{7}{|c|} {\textbf{Croatian}}\\\hline
        TF-IDF & 0.3430 &  0.4955 & 0.4054 & 0.3364 & 0.4987 &  0.4018\\
        TNT-KID & 0.3364 & 0.4925 & 0.3998 & 0.3273 & 0.5089 & 0.3984    \\
        BERT + BiLSTM-CRF &  \textbf{0.4737} & 0.4580 & \textbf{0.4657} &  \textbf{0.4733} &  0.4607 & \textbf{0.4669} \\ 

        TNT-KID \& TF-IDF(tm) & 0.2835 & 0.5664 & 0.3779 & 0.2594 & 0.6224 & 0.3662\\ 
        BERT + BiLSTM-CRF \& TF-IDF(tm) & 0.3003 & 0.5569 & 0.3901 & 0.2782 & 0.5732 & 0.3746 \\ 
        TNT-KID \& BERT + BiLSTM-CRF &  0.2961 & 0.5354 & 0.3813 & 0.2778 & 0.5778 & 0.3752 \\ 

        TNT-KID \& BERT + BiLSTM-CRF \& TF-IDF(tm) &  0.2653 & \textbf{0.5685} &  0.3618 & 0.2295 & \textbf{0.6549} & 0.3399\\ \hline

    
           \multicolumn{7}{|c|} {\textbf{Estonian}}\\\hline
        TF-IDF & 0.0716 &  0.1488 & 0.0966 & 0.0496 & 0.1950 & 0.0790 \\
        TNT-KID & \textbf{0.5194} & 0.5676 & \textbf{0.5424} & \textbf{0.5098} & 0.5942 & \textbf{0.5942}    \\
        BERT + BiLSTM-CRF &  0.5118 & 0.4617 & 0.4855 & 0.5078 & 0.4775 & 0.4922 \\ 

        TNT-KID \& TF-IDF(tm) & 0.3463 & 0.5997 & 0.4391 &  0.1978 & 0.6541 & 0.3037\\ 
        BERT + BiLSTM-CRF \& TF-IDF(tm) & 0.3175 & 0.4978 &  0.3877 & 0.1789 & 0.5381 & 0.2686\\ 
        TNT-KID \& BERT + BiLSTM-CRF &  0.4421 & 0.6014 & 0.5096 & 0.4028 & 0.6438 & 0.4956 \\ 

        TNT-KID \& BERT + BiLSTM-CRF \& TF-IDF(tm) &  0.3588 & \textbf{0.6206} & 0.4547 & 0.2107 & \textbf{0.6912} & 0.3230 \\ \hline
      

        \multicolumn{7}{|c|} {\textbf{Russian}}\\\hline
        TF-IDF & 0.1764 &  0.2314 & 0.2002 & 0.1663 & 0.3350 &  0.2223\\
        TNT-KID & \textbf{0.7108} & 0.6007 & \textbf{0.6512} & \textbf{0.7038} & 0.6250 & \textbf{0.6621}    \\
        BERT + BiLSTM-CRF &  0.6901 & 0.5467 & 0.5467 & 0.6849 & 0.5643 & 0.6187 \\ 

        TNT-KID \& TF-IDF(tm) & 0.4519 & 0.6293 & 0.5261 & 0.2981 & 0.6946 & 0.4172\\ 
        BERT + BiLSTM-CRF \& TF-IDF(tm) & 0.4157 & 0.5728 & 0.4818 & 0.2753 & 0.6378 & 0.3846 \\ 
        TNT-KID \& BERT + BiLSTM-CRF &  0.6226 & 0.6375 & 0.6300 & 0.5877 & 0.6707 & 0.6265 \\ 

        TNT-KID \& BERT + BiLSTM-CRF \& TF-IDF(tm) &  0.4622 & \textbf{0.6527} &  0.5412 & 0.2965 & \textbf{0.7213} & 0.4203\\ \hline

        \multicolumn{7}{|c|} {\textbf{Latvian}}\\\hline
        TF-IDF & 0.2258 & 0.5035 & 0.3118 & 0.1708 & 0.5965 & 0.2655 \\
        TNT-KID &  0.6089 & 0.6887 & \textbf{0.6464} & 0.6054 & 0.6960 & \textbf{0.6476} \\
        BERT + BiLSTM-CRF & \textbf{0.6215} & 0.6214 & 0.6214 & \textbf{0.6204} & 0.6243 & 0.6223   \\ 

        TNT-KID \& TF-IDF(tm) & 0.3402 & 0.7934 & 0.4762 & 0.2253 & 0.8653 & 0.3575\\ 
        BERT + BiLSTM-CRF \& TF-IDF(tm) & 0.2985 & 0.6957 & 0.4178 & 0.1889 & 0.7427 & 0.3012 \\ 
        TNT-KID \& BERT + BiLSTM-CRF &  0.4545 & 0.7189 & 0.5569 & 0.4341 & 0.7297 & 0.5443 \\ 

        TNT-KID \& BERT + BiLSTM-CRF \& TF-IDF(tm) &  0.3318 & \textbf{0.7852} & 0.4666 & 0.2124 & \textbf{0.8672} & 0.3414\\ \hline

    \end{tabular}}
    \label{tab:kw_media_eval}
\end{table*}

Results suggest, that neural based approaches, TNT-KID and BERT+BiLSTM-CRF offer comparable performance on all datasets but nevertheless achieve different results for different languages. TNT-KID outperforms BERT-BiLSTM-CRF model according to all the evaluation metrics on the Estonian and Russian news dataset. It also outperforms all other methods in terms of precision and F1 score. On the other hand, BERT+BiLSTM-CRF preforms better on the Croatian dataset in terms of precision and F1-score. On Latvian TNT-KID achieves top results in terms of F1, while BERT+BiLSTM-CRF offers better precision. The proposed TF-IDF tagset matching method performs poorly in most of the cases. The exception is the Croatian dataset, where it outperforms TNT-KID according to all criteria but R@10. Most likely, this is connected to the distribution of articles in the Croatian train and test set, where we have 47{,}479 train articles and 5{,}277 test articles and a train to test ratio of 8.62. In other three languages, the ratio of train to test articles is 1.13 for Latvian, 1.21 for Russian and 1.39 for Estonian. 

Even though the TF-IDF tagset matching method performs poorly on its own, we can nevertheless drastically improve the recall of both neural systems, if we expand the keyword tag sets returned by the neural methods with the TF-IDF ranked keywords. The improvement is substantial and consistent for all datasets, but it nevertheless comes at the expanse of the lower precision and F1 score. This is not surprising, since the final expanded keyword set always returns 10 keywords, i.e., much more than the average number of present gold standard keywords in the media partner datasets (see Table \ref{tab:media-dataset}), which badly affects the precision of the approach. 

Combining keywords returned by TNT-KID and  BERT + BiLSTM-CRF also consistently improves recall, but again at the expanse of lower precision and F1 score. Overall, for all four languages, the best performing method in terms of recall is the TNT-KID \& BERT + BiLSTM-CRF \& TF-IDF(tm).


\section{Conclusion and future work}
\label{sec:conc}

In this work we tested two state-of-the-art neural approaches for keyword extraction, TNT-KID \cite{martinc2020tnt} and BERT BiLSTM-CRF \cite{sahrawat2019keyphrase}, on three less resourced European languages, Estonian, Latvian, Croatian, as well as on Russian. We also proposed a tagset based keyword expansion approach, which drastically improves the recall of the method, making it more suitable for the application in the media house environment. 

Our study is one of the very few studies where supervised keyword extraction models were employed on several less resource languages. The results suggest that these models perform well on languages other than English and could also be successfully leveraged for keyword extraction on morphologically rich languages. 

The focus of the study was whether we can improve the recall of the supervised models, in order to make them more useful as recommendation systems in the media house environment. Our method manages to increase the number of retrieved keywords, which drastically improves the recall for all languages. For example, by combing all neural methods and the TF-IDF based apporach, we improve on the recall@10 achieved by the best performing neural model, TNT-KID, by 14.6 percentage points for Croatian , 9.70 percentage points for Estonian, 9.63 percentage points for Russian and 17.12 percentage points for Latvian. The resulting method nevertheless offers lower precision, which we will try to improve in the future work.

In the future we also plan to perform a qualitative evaluation of our methods by journalists from the media houses. Next, we plan to explore how adding background knowledge from knowledge bases - lexical (e.g. Wordnet\cite{wordnet}) or factual (e.g. WikiData\cite{wikidata}) would benefit the aforementioned methods. The assumption is that with the linkage of the text representation and the background knowledge we would achieve a more representative understanding of the articles and the concepts appearing in them, hoping for a more successful keyword extraction. 

In traditional machine-learning setting a common practice of combining different classifier outputs to a single output is referred  to as stacking. We propose a further research in the scope of combination of various keyword extraction models to be conducted. One way to incorporate this would be to add the notion of the positional encoding, since some of the keywords in the news-media domain often can be found in the beginning of the article and the TF-IDF(tm) does not account this while applying the weighting on the matched terms.    




\section{Acknowledgements}
Acknowledgements will be added for the final, non-anonymized, version of the article.
\bibliographystyle{acl_natbib}
