\section{Introduction}

Biomedical literature is a rich source of biomedical knowledge that is constantly expanding. For example, MEDLINE/PubMed\footnote{\url{https://pubmed.ncbi.nlm.nih.gov/}}, which already gathers about 30 million articles, is continuously growing at a rate that exceeds one million articles annually, during the last five years\footnote{\url{https://www.nlm.nih.gov/bsd/medline_pubmed_production_stats.html}}. 
Precise and comprehensive identification of relevant articles for specific topics in such a rich collection is a real challenge for biomedical researchers.
In this direction, the National Library of Medicine (NLM) indexes citations in the MEDLINE database with topic descriptors from the Medical Subject Headings (MeSH) thesaurus\footnote{\url{https://meshb.nlm.nih.gov/}}. 
This semantic indexing process allows MEDLINE/PubMed to offer advanced semantic search strategies, that can retrieve citations relevant to a topic of interest, addressing issues such as synonymy and polysemy of biomedical concepts.

MeSH provides a comprehensive set of descriptors for more than thirty thousand topics of biomedical interest, such as diseases and chemicals.
The descriptors are hierarchically organized so that most are subcategories of at least one broader descriptor.
For example, the descriptor for ``Epilepsy, Reflex''\footnote{\url{https://meshb-prev.nlm.nih.gov/record/ui?ui=D020195}} is a subcategory of the broader descriptor ``Epilepsy''\footnote{\url{https://meshb-prev.nlm.nih.gov/record/ui?ui=D004827}} as shown in Figure~\ref{fig:MeSH_Example}.
Some of these descriptors cover several related but distinct subordinate concepts, which are usually narrower\footnote{In some cases subordinate concepts can be related or even broader to the main topic. } than the main topic of the descriptor.
For example, the ``Epilepsy, Reflex'' descriptor contains the synonymous term ``Reflex Epilepsy'' as \textit{preferred concept}, but includes also additional terms for narrower concepts such as ``Reflex Epilepsy, Audiogenic'' and ``Tactile Reflex Epilepsy'', that are not MeSH descriptors themselves.

\begin{figure}[!t]
\centerline{\includegraphics[width=0.85\textwidth]{figures/MeSH_example.png}}
\caption{ The internal structure of the MeSH descriptor ``Epilepsy, Reflex'' and its position in the hierarchy under the broader descriptor ``Epilepsy''. }\label{fig:MeSH_Example}
\end{figure}

Such narrower concepts of the coarse-grained descriptor are not distinguishable by the semantic indexing process, which is done at the level of descriptors. 
Therefore, semantic search cannot be used at this level of detail. 
The goal of this work is to enable such fine-grained semantic search, through  the automated refinement of semantic indexing from the descriptor level to the level of each narrower concept. 
For example, we would like to distinguish which of the ``Epilepsy, Reflex'' articles are for ``Audiogenic Epilepsy'' and which for ``Tactile Reflex Epilepsy'' or other subcategories of ``Epilepsy, Reflex''.

Fine-grained indexing at the level of concepts is beyond the current practice in MEDLINE/PubMed and, if enforced, it would almost double the number of distinct topics used for indexing.
Doing this additional work manually is not practical, given the growing volumes of biomedical literature.
Furthermore, state-of-the-art systems that provide automated topic suggestions 
focus on known MeSH topics and usually require several labelled training  examples for any new topic that is to be covered~\cite{Nentidis2022}. Therefore, obtaining reliable topic suggestions for new fine-grained topics at the level of concepts is an open research issue. 
The lack of labelled data for the development and evaluation of methods at this level of detail is a major challenge.

In this work, we propose \textit{Deep Beyond MeSH} ($DBM$), a new method for the refinement of coarse-grained MeSH annotations based on deep pre-trained language models. 
To overcome the lack of labelled data for supervised training, DBM relies on weak supervision. The latter is based on the recognition of biomedical concepts in the abstract of articles (\textit{concept occurrence}).  
In the past we have experimented with this idea~\cite{nentidis2020beyond}, using small ground truth datasets, developed for two specific MeSH descriptors. Those experiments suggested that \textit{concept occurrence} can provide a strong basis for fine-grained semantic indexing. 
However, the approach needed to be tested on larger datasets of ground truth data for several MeSH descriptors. 
In this direction, we devised a retrospective scenario, based on the evolution of MeSH indexing in MEDLINE/PubMed, and introduce here \textit{Retrospective Beyond MeSH} ($RetroBM$), a new method to derive large-scale datasets for individual concepts that got promoted over time to new fine-grained MeSH descriptors. 

In brief, the main questions driving the work presented in this paper are the following:
\begin{itemize}
    \item Can the extension of MeSH towards fine-grained labels that took place in the past be exploited for the retrospective evaluation of automated fine-grained indexing approaches on a large scale?
    \item Is \textit{concept occurrence} a good heuristic for fine-grained concept-level semantic indexing on a large scale? 
    Could the \textit{concept occurrence} be enhanced with other heuristics?
    \item Is it possible to train DL models for fine-grained semantic indexing with concepts for which no ground-truth training data are available, exploiting the weak supervision provided by the above heuristics? 
    Could these models exceed the predictive performance of the heuristics themselves?
\end{itemize}

The rest of this paper is organized as follows.
In section \ref{sec:Background}, we provide some background information on the semantic indexing of biomedical literature, relevant work on fine-grained classification without labelled data, and previous efforts towards more fine-grained biomedical semantic indexing.
In section \ref{sec:Methods}, we introduce our new methods: $RetroBM$ for the automated development of datasets for fine-grained semantic indexing of biomedical literature and $DBM$ for weakly-supervised training of DL models for the automated refinement of coarse-grained labels.
In section \ref{sec:Experiments}, we describe the experimental setup for the evaluation of the predictive performance of the $DBM$ models on $RetroBM$ data, and present the results of these experiments.
Finally, in section \ref{sec:Conclusion} we summarise the presented work and discuss its contribution towards addressing the motivating research questions presented above.  

\section{Background and related work}
\label{sec:Background}

\subsection{Semantic indexing of biomedical literature}

Semantic indexing of biomedical literature in MEDLINE/PubMed has been done  for several decades by human expert indexers of NLM. 
Since 2002 some automated topic suggestions are offered to human experts by the \textit{Medical Text Indexer} (\textit{MTI}) Tool\footnote{\url{https://lhncbc.nlm.nih.gov/ii/tools/MTI.html}}, in order to speed up the annotation process and keep up with the increasing volume of the literature. 
\textit{MTI} is structured into distinct components that provide MeSH topic recommendations. These components are based on complementary approaches, namely \textit{concept occurrence} in the article text, the topics of similar articles, and machine learning models trained on previously indexed articles.
The recommendations are finally ranked and filtered by other components, including one based on the Learning To Rank (LTR) approach\cite{Mork2017}.

In parallel, a variety of alternative approaches has been proposed for the automated semantic indexing of biomedical literature, and  since 2012 a dedicated shared task has been organized for this purpose, in the context of the BioASQ challenge~\cite{tsatsaronis2015overview}. 
The state-of-the-art approaches in this task, are based on DL architectures. 
For example, \textit{BERTMeSH} \cite{you2021bertmesh} extends the label-tree-based DL method of \textit{AttentionXML}~\cite{You2018} with representations based on the pre-trained transformer model \textit{BioBERT}~\cite{lee2019biobert}.
In the same direction, new methods proposed by the NLM team \cite{Rae2021clef} rely on recommendations from a Convolutional Neural Network (CNN) which are ranked by another pre-trained transformer model (\textit{PubMedBERT})~\cite{Gu2022}.   

The availability of millions of manually indexed articles as training examples allowed the gradual development of advanced machine-learning models that achieve satisfactory results~\cite{Nentidis2022}.
As a result, since 2018, NLM introduced fully automated indexing for a subset of the MEDLINE articles, using an improved version of \textit{MTI}. Furthermore, they announced the scale-up of this policy to all MEDLINE articles in 2022\footnote{\url{https://www.nlm.nih.gov/pubs/techbull/nd21/nd21_medline_2022.html}}. 
Despite the significant progress in the field of semantic indexing of biomedical literature, almost all state-of-the-art methods focus on coarse MeSH descriptors. Fine-grained indexing at the concept level remains beyond reach, partly due to the lack of adequate data for training and evaluation of supervised machine learning approaches.    

\subsection{Fine-grained classification without labelled data}

Classification into fine-grained sub-classes has been long studied in the field of image classification and a series of workshops is also organized on Fine-Grained Visual Categorization (FGVC)\footnote{https://sites.google.com/view/fgvc10} since 2011.
These workshops cover a variety of relevant tasks, such as differentiation between similar plant or animal species.
Annotations at the fine-grained level are often difficult to obtain and as a result researchers have investigated learning with limited fine-grained supervision and exploiting the hierarchical relationship between concepts. 
For instance, Taherkhani \textit{et al.}~\cite{Taherkhani2019}, adopt a few-shot approach, where some of the instances do have fine-grained annotations, and rely on multi-task learning so that training for the coarse-grained task improves the fine-grained one, through shared model parameters of a CNN. 
Hsieh \textit{et al.}~\cite{Hsieh2019, Hsieh2021} adopt a few-shot approach as well, extending fine-grained image annotation from multi-class to multi-label classification, where a single image may have multiple fine-grained annotations, containing, for example, both a cat and an apple.
Based on the hierarchical relation between labels of different granularity, their method assigns fine-grained pseudo-labels to unlabeled images, which are used for model training. 
Another few-shot method that relies on pseudo-labeling for FGVC, was proposed by Yang \textit{et al.}~\cite{Yang2020}. In particular, they perform clustering of coarse-grained-class instances into pseudo-fine-grained classes and learn fine-grained features, which are then used for the actual task of FGVC.   
On the other hand, Sohoni \textit{et al.}~\cite{Sohoni2020} use clustering into fine-grained concepts to improve coarse-grained classification per se.

Similar problems have also been studied in the context of fine-grained named entity recognition. For example, a text span referring to a person can be classified into more fine-grained entity classes, such as actor, athlete, or politician. 
Training instances for such fine-grained sub-classes are hard to obtain and  proposed methods rely mostly on distant supervision \cite{Ling2012, Gillick2014, Yogatama2015, Zhang2020} or zero-shot learning \cite{Ma2016, Zhou2017, Obeidat2019, Zhang2021}. 
Hierarchical label relations have been used either to define the classification problem as a problem of differentiating between sibling sub-classes of a coarse-grained parent class \cite{Zhou2017} or by more elaborate approaches that allow information sharing among related labels \cite{Yogatama2015, Zhang2021}. 

In the field of document classification, which is the one closest to our work, some attention has also been drawn to fine-grained classification without ground-truth data, mainly based on weakly-supervised or zero-shot methods.
Most of these approaches are multi-task, in the sense that they aim to improve coarse-grained and fine-grained classification together. 
For instance, \textit{Snorkel MeTaL} \cite{Ratner2018, Ratner2019} exploits fine-grained and coarse-grained weak labels together combining them on the basis of a hierarchical label model. 
That method combines weak labels from multiple sources, which is an interesting research topic, as it requires modeling their coverage and disagreements~\cite{Ratner2016, Bach2017, Ratner2020}. 
However, in the fine-grained biomedical semantic indexing task studied here, the sources of supervision are limited. 

However, 
there is also a lot of work on the use of a single source of weak supervision \cite{Song2015, Li2016, Zha2019, Chu2020}, in many cases taking into account the label hierarchy as well.
Song and Roth \cite{Song2014}, for example, 
deal with hierarchical multi-label text classification,
using only a label hierarchy and some unlabeled documents.
Their approach encodes both labels and documents in a common semantic space.
Measuring label-document semantic similarity in that common space, they develop a dataless classifier and improve it through bootstrapping on unlabeled documents.
Label-document similarity is also used in the \textit{taxoClass} framework~\cite{Shen2021}, which employs a pre-trained textual entailment model, based on \textit{RoBERTa} \cite{Liu2019}. 
Using the label-document semantic similarity and the hierarchy of labels, a set of the most similar ``core'' labels is identified for each document and a taxonomy-enhanced text classifier is trained on these weak labels. 
Finally, multi-label self-training is also employed to improve the classifier.

In contrast to the broader problem of hierarchical text classification, coarse-grained class labels are already available in the biomedical indexing problem that we study.
Recently, Mekala \textit{et al.} \cite{Mekala2021} focused on this special case of fine-grained indexing, where coarse-grained labels are available, proposing the \textit{Coarse2Fine} ($C2F$) method.
$C2F$ relies on the literal occurrence of fine-grained labels to create an initial weakly labeled dataset. These weak fine-grained labels are used together with the ground-truth coarse-grained ones to train a label-conditioned GPT2 \cite{AlecRadford2020} model that generates relevant documents for any given label.
In that method, a hierarchy-aware regularization term is added to the loss function, in order to exploit the hierarchical relations between classes.
This ensures that documents with a literal occurrence of a fine-grained label will have a higher probability compared to documents that include the coarse-grained parent.
The trained GPT2 model is used to produce new weakly labeled documents for each fine-grained label, which are, in turn, used to train \textit{BERT} classifiers \cite{Devlin2019}. 
Finally, the classifiers are improved through the use of unlabeled documents.

Simpler approaches use the available ground-truth coarse-grained labels, by reducing the hierarchical classification problem into several local sub-problems of classifying fine-grained sibling subcategories.
For instance, \textit{wiki2cat} \cite{Jin2021} exploits the graph of Wikipedia categories, as an external knowledge resource, to weakly label Wikipedia articles for 332 fine-grained categories of the Interactive Advertising Bureau (IAB) taxonomy\footnote{\url{https://www.iab.com/guidelines/taxonomy/}}.
In particular, the IAB categories are first mapped to Wikipedia categories, based on the literal match of their names, and then the Wikipedia category graph is used to obtain weakly labeled articles. 
Using these data, weakly-supervised models are trained to classify into fine-grained IAB sibling subcategories under the same parent IAB category.
Our weakly labelling approach relies on \textit{concept occurrence} which is kin to the literal occurrence of label names, using biomedical vocabularies, as external knowledge resources. 


The more general task of indexing documents with new descriptors is also challenging, due to the lack of labelled data for these descriptors.  
For this task, Mylonas \textit{et al.} introduced \textit{ZSLbioSentMax} \cite{Mylonas2020}, a zero-shot semantic indexing approach that relies on the maximum semantic similarity of new descriptor names with sentences in the text of biomedical articles. In particular, they measure the cosine similarity of corresponding dense representations (embeddings), produced by \textit{BioBERT} \cite{lee2019biobert}. 
A weakly-supervised approach, called \textit{WeakMeSH} \cite{Mylonas2021} has also been proposed for the same task. 
In that work, weak labels were obtained by considering both the hierarchical relationship of new labels with existing descriptors and their semantic similarity with sentences in the text of biomedical articles. 
Using these data, weakly-supervised classifiers were trained, achieving promising results in experiments with 62 new MeSH descriptors, most of which cover novel MeSH concepts.
In this paper, we consider the hierarchical relationship between new and existing descriptors as well, but we focus on subordinate MeSH concepts that get promoted into fine-grained subcategories. 
Our method relies on deep pre-trained models as well, however, as we deal with semantically similar fine-grained labels that cover a single concept, we investigate \textit{concept occurrence} as the primary source of weak supervision.    

\subsection{Towards fine-grained semantic indexing of biomedical literature}

Early work towards fine-grained indexing of biomedical literature focused on enriching the main MeSH descriptor-based labels with MeSH qualifiers\footnote{\url{https://www.nlm.nih.gov/mesh/qualifiers\_scopenotes.html}} and MeSH Supplementary Concept Records (SCRs)\footnote{\url{https://www.nlm.nih.gov/mesh/intro\_record\_types.html.}} \cite{Aronson2004}.
MeSH qualifiers are auxiliary categories, such as ``genetics'' or ``diagnosis'', that may accompany a descriptor to specify aspects of it in an article (e.g. ``Lung Cancer/genetics''). 
SCRs, on the other hand, are supplementary entities for specific substances, rare diseases, and organisms that are not explicitly covered by any MeSH descriptor. 
In this direction, Neveol \textit{et al.} \cite{Neveol2007_amia, Neveol2007_psb}, 
proposed various approaches for indexing with ``descriptor/qualifier'' pairs, including ``dictionary-based'' ones, that rely on the literal occurrence of label terms, as well as ``rule-based'' ones, that rely on  manually-developed patterns.  
Recently, Rae \textit{et al.} \cite{Rae2020} proposed a CNN-based method  that achieved improved results on the same task. 

Although qualifiers and SCRs provide additional information to MeSH descriptors, they do not address the coarseness of the descriptors.
To the best of our knowledge, the importance of fine-grained semantic indexing of biomedical literature at the level of MeSH concepts was first noted in \cite{Darmoni2012}. 
The authors of that work used simply 
the literal occurrence of corresponding MeSH concept terms in the title or abstract of the articles.
Despite the strong assumption of literal occurrence, that work concluded that indexing at this level of detail is useful for the precise retrieval of relevant documents.
In this direction, the indexing policies for librarians of the \textit{CISMeF} catalogue\footnote{\url{http://www.chu-rouen.fr/cismef/}} for French medical resources were updated to support indexing at the level of MeSH concepts. 

More recently, we proposed the \textit{Beyond MeSH} method \cite{Nentidis2019_CBMS, nentidis2020beyond} for the automated refinement of MeSH topic annotations at the level of MeSH concepts. 
In that work, we first introduced the concept-occurrence heuristic for the development of weakly labeled training datasets. This heuristic extends simpler dictionary-based ones
using the \textit{MetaMap} tool \cite{Aronson2010}. 
We experimented with different feature sets and common machine-learning algorithms
on small datasets for two specific use cases, comparing the results of trained classifiers against the use of \textit{concept occurrence} itself. Those initial experiments showed that the concept-occurrence heuristic is a strong baseline and only some trained classifiers managed to improve upon it.
In particular, logistic regression models with L2 regularization provided higher recall, but the selection of hyper-parameters, such as the level of L2 regularization, without ground-truth validation data poses a major challenge.  
Building upon this earlier work, in this paper we perform an extensive evaluation of the promising concept-occurrence heuristic in a large-scale scenario for several MeSH descriptors, while also assessing the performance of DL approaches for the first time.

In order to form the large-scale experimental scenario, we look retrospectively into the evolution of MeSH descriptors, focusing on MeSH concepts, which over time become descriptors themselves. 

\section{Material and methods}
\label{sec:Methods}

\begin{figure}[!h]
\begin{center}
\includegraphics[width=1\textwidth]{figures/Schema_method_plus3.png}
\end{center}
\caption{The \textit{Retrospective Beyond MeSH} ($RetroBM$) method for the development of large-scale ground truth datasets for fine-grained semantic indexing and the \textit{Deep Beyond MeSH} ($DBM$) method for weakly-supervised semantic label refinement. $DBM$ is structured into three stages: the development of weakly-supervised datasets for some concepts of interest, the enhancement of the initial weak supervision, and the development of weakly-supervised deep learning models. The datasets of $RetroBM$ are finally used for the large-scale evaluation of $DBM$ models. }
\label{fig:DeepBeyondMeSH_block}      
\end{figure}
In this section, we introduce a) the \textit{Retrospective Beyond MeSH} ($RetroBM$) method for the development of large-scale ground truth datasets for fine-grained semantic indexing and b) the \textit{Deep Beyond MeSH} ($DBM$) method for refining existing coarse-grained biomedical semantic labels into fine-grained ones at the level of concepts.
In the context of $RetroBM$, a scenario is introduced that allows the automated development of ground-truth datasets for specific narrower concepts of coarse-grained descriptors. 
These narrower concepts are the ones that have become descriptors themselves over time. Hence, by examining retrospectively the evolution of MeSH, we can identify a number of such concepts to serve as use cases and generate automatically ground-truth data.
$DBM$, on the other hand, is structured into three distinct stages, namely the development of weakly-labeled datasets, where $RetroBM$ is used for the ground-truth ones, the enhancement of weak supervision, and the development of fine-grained indexing models. 
Figure~\ref{fig:DeepBeyondMeSH_block} illustrates the two processes, showing also how the results of $RetroBM$ are used by $DBM$.
In particular, the weakly-labeled datasets are used for the development of weakly-supervised DL models that classify articles into the narrow concepts. Weak supervision may or may not be enhanced with the use of dictionary-based approaches at the second stage of the process. 
The implementation of these methods in Python and Java is openly available in GitHub\footnote{\url{https://github.com/tasosnent/DBM}}. In the following subsections, we will examine in more detail each step of the process.

\subsection{Retrospective dataset development}
\label{ssec:retro_data}

The main challenge in dataset development for fine-grained biomedical semantic indexing is the lack of ground-truth annotations at this level of detail for the development and testing of predictive models.
Although methods that rely on weak supervision can sidestep the need for ground-truth labels as regards model development, some ground-truth data are still needed for the proper evaluation of model predictions.
Requiring domain experts to manually annotate the test sets does not scale up when moving to larger numbers of articles and use cases. 
Therefore, in this work we introduce \textit{Retrospective Beyond MeSH} ($RetroBM$), a retrospective approach based on the evolution of MeSH indexing already done in MEDLINE/PubMed, to develop fine-grained semantic indexing datasets. 

\begin{figure}[!t]
\centerline{\includegraphics[width=0.82\textwidth]{figures/MeSH_subdivision_Example.png}}
\caption{ The promotion of ``Niemann-Pick Disease, Type A'', a narrower subordinate concept of the coarse-grained MeSH descriptor ``Niemann-Pick Diseases'', to a new dedicated descriptor in 2007.  }\label{fig:MeSH_subdivision_Example}
\end{figure}

\subsubsection{Formulation of a retrospective scenario}
\label{sssec:retro_data_scenario}
Although indexing of articles with subordinate MeSH concepts is beyond the curation process of MEDLINE/PubMed, there are cases where a subordinate concept \textit{c} of a coarse-grained descriptor \textit{d} can be promoted to a new MeSH descriptor \textit{d\textsubscript{c}}, during the annual update of the MeSH thesaurus. 
For example, ``Niemann-Pick Disease, Type A''\footnote{\url{https://meshb.nlm.nih.gov/record/ui?ui=D052536}} used to be a narrower subordinate concept of the ``Niemann-Pick Diseases'' descriptor until 2007, when it was promoted to a MeSH descriptor itself (Fig. \ref{fig:MeSH_subdivision_Example}). 
In such cases, if \textit{c} is the only MeSH concept of the new MeSH descriptor \textit{d\textsubscript{c}},
indexing with \textit{d\textsubscript{c}} is equivalent to indexing with \textit{c}. 
Therefore, articles semantically indexed with a version of MeSH that includes \textit{d\textsubscript{c}}, in practice, have ground-truth annotations for fine-grained indexing with the MeSH concept \textit{c}.  
In the example of Figure~\ref{fig:MeSH_subdivision_Example}, articles for any type of Niemann-Pick disease used to be annotated with the ``Niemann-Pick Diseases'' descriptor until 2006. However, since 2007, articles for ``Niemann-Pick Disease, Type A'' are annotated with a distinct and dedicated descriptor and can be used for the evaluation of methods for fine-grained indexing with this concept.    

\begin{figure}[!t]
\includegraphics[width=1\textwidth]{figures/retro.png}
\caption{Retrospective datasets for the task of refining coarse-grained annotations for a descriptor \textit{d} into a narrower subordinate concept \textit{c}. The development of the datasets is based on the promotion of \textit{c} to a new fine-grained descriptor \textit{d\textsubscript{c}} in year \textit{y}, during the annual update of the MeSH thesaurus.}
\label{fig:retro_data}      
\end{figure}

For this reason, we looked retrospectively for such promoted concept cases to generate automatically ground-truth data for fine-grained indexing methods. 
In particular, we focus on the year \textit{y}, when the promotion of \textit{c} to \textit{d\textsubscript{c}} took place, as shown in Figure~\ref{fig:retro_data}.
All the articles annotated with \textit{d}\footnote{As articles annotated with \textit{d} we consider both articles directly annotated with the descriptor \textit{d} and articles annotated with any descriptor narrower to \textit{d} in the MeSH hierarchy.} prior to year \textit{y} are gathered for the development of a weakly-supervised development dataset for indexing with \textit{c}.
The fact that \textit{c} used to be a subordinate concept of \textit{d} in this period, ensures that any article relevant to \textit{c} will be included in this set of articles annotated with \textit{d}. 
To generate the weak labels we use the heuristic introduced in \textit{Beyond MeSH}~\cite{Nentidis2019_CBMS,nentidis2020beyond}, which relies on the occurrence of concept \textit{c} in the title or abstract of the article, as identified by the \textit{MetaMap} tool \cite{Aronson2010}. 

On the other hand, for any article annotated with \textit{d} from year \textit{y} onward, manual annotations for the narrower descriptor \textit{d\textsubscript{c}} are also available, indicating relevance to concept \textit{c}. 
These more recent articles for \textit{d} can be used for the evaluation of the refined semantic indexers.  
The fact that \textit{d\textsubscript{c}} is a narrower descriptor of \textit{d} in the MeSH hierarchy, ensures that any article relevant to \textit{c} will be included in the set of articles annotated with \textit{d}.
Therefore, articles not indexed under \textit{d} are beyond the scope of the fine-grained indexers, as the aim is to refine coarse-grained annotations already provided by the current indexing practice.

Although the proposed approach relies on a retrospective use of annotated documents, this schema is fully aligned with the prospective use of automated refinement approaches for coarse-grained MeSH annotations.
We can develop weakly-supervised datasets for model development with articles annotated for any coarse-grained descriptor \textit{d\textsubscript{new}} having a narrower subordinate concept \textit{c\textsubscript{new}}, that has not actually been promoted to a descriptor in MeSH yet. 
The trained models can be used to provide automated fine-grained indexing with \textit{c\textsubscript{new}}, without it being a descriptor. However, the accuracy of this annotation process will need to be verified manually.

\subsubsection{Identification of adequate use cases}
\label{ssec:use_case_selection}
Concept promotion events during the evolution of MeSH need to be identified, in order to scale up dataset development, as described in subsection~\ref{sssec:retro_data_scenario}.
However, not all new MeSH descriptors come from the promotion of narrower subordinate concepts as in the example of Figure~\ref{fig:MeSH_subdivision_Example}.
To identify which of the new descriptors satisfy our criteria, we adopt a framework that we introduced recently~\cite{Nentidis2021}
to study the evolution of MeSH descriptors, based on their semantic provenance. 
That framework introduced a typology of ways in which a new descriptor can be introduced.
Out of these types, the one named \textit{subdivision (1.2)}\footnote{We refer to \textit{subdivision} cases with provenance code 1.2 as introduced in Nentidis \textit{et al.} \cite{Nentidis2021}. This corresponds to new descriptors from a subordinate-concept promotion (provenance category 1) that end up as subcategories of their previous host (provenance type 2).} represents subordinate concepts \textit{c} promoted to subcategories \textit{d\textsubscript{c}} of the coarse-gained descriptor \textit{d} that used to host them. 
This type of new descriptor satisfies the concept-promotion scenario of subsection~\ref{sssec:retro_data_scenario} and was chosen to select suitable evaluation use cases.


However, there are additional criteria that need to be satisfied, beyond provenance. One such is that the new descriptor \textit{d\textsubscript{c}} is dedicated to the corresponding promoted concept \textit{c}, and does not cover other subordinate concepts as well. This is to ensure that all articles annotated with \textit{d\textsubscript{c}} are actually relevant to \textit{c}.
In addition, such descriptors are expected to be leaves in the MeSH hierarchy. New descriptors that have other narrower descriptors representing subcategories, in effect semantically cover the concepts of their narrower descriptors as well. Such descriptors are not fine-grained, even if they technically include just a single concept, and they are not considered adequate evaluation use cases.   

The last criterion used to select use cases is the volume of available data for training and validation. For this reason, we introduce a threshold \textit{dev\textsubscript{min}} for the number of articles annotated with \textit{d} that can be used for model development and another threshold \textit{test\_positive\textsubscript{min}} for the number of articles annotated with \textit{d\textsubscript{c}}, which can be used to evaluate the models. 
Some coarse-grained descriptors \textit{d} may be also too general, corresponding to too many articles and posing technical challenges in the dataset-development process. For this purpose, an upper limit \textit{dev\textsubscript{max}} to the number of articles annotated with \textit{d} can also be used in order to work with a realistic and manageable numbers of articles.
Finally, as the construction of the development dataset relies on the concept-occurrence heuristic, there is also a need for a sufficient number of articles with weak labels for \textit{c}.
Therefore, an additional threshold \textit{dev\_positive\textsubscript{min}} is adopted for the number of articles annotated with \textit{d}, where also \textit{c} occurs.  

\begin{table}[!t]
\caption{Criteria for retrospective selection of evaluation use cases.}
\begin{center}
\begin{tabular}{L{.18\linewidth}L{0.72\linewidth}}
Criterion 1. & Promotion event of provenance type \textit{subdivision (1.2)}  \\
Criterion 2. & Fine-grained new descriptor \textit{d\textsubscript{c}}  \\
 & 2.1. \textit{d\textsubscript{c}} corresponding to a single concept \textit{c} \\
 & 2.2. \textit{d\textsubscript{c}} leaf in the MeSH hierarchy \\
Criterion 3. &  Adequate number of articles for dataset development \\
 & 3.1. At least \textit{test\_positive\textsubscript{min}} articles annotated with \textit{d\textsubscript{c}} for the test set\\
 & 3.2. At least \textit{dev\textsubscript{min}} articles annotated with \textit{d} for the training dataset\\
 & 3.3. At most \textit{dev\textsubscript{max}} articles annotated with \textit{d} \\
 & 3.4. At least \textit{dev\_positive\textsubscript{min}} articles in the development dataset with \textit{c} occurrence
\end{tabular}
\label{table:criteria}
\end{center}
\end{table}

The above criteria, summarized in Table~\ref{table:criteria}, were incorporated in the automated method ($RetroBM$) for identifying adequate evaluation use cases, given a specific range of years. 
First, the method selects new descriptors belonging to the \textit{subdivision (1.2)} category, based on their conceptual provenance\footnote{https://github.com/tasosnent/MeSH\_Extension} (criterion 1). 
Then, the internal structure and hierarchical position of each selected new descriptor \textit{d\textsubscript{c}} are checked, in order to keep only the fine-grained ones (criterion 2).
Subsequently, the number of relevant articles for descriptors \textit{d} and \textit{d\textsubscript{c}} of each remaining use case is retrieved from a local Lucene\footnote{https://lucene.apache.org/} index of the Annual Baseline Repository\footnote{\url{https://www.nlm.nih.gov/databases/download/pubmed\_medline.html}} of MEDLINE/PubMed, and is checked against the data availability restrictions (3.1, 3.2, 3.3) of criterion 3.
Finally, in order to check the concept-occurrence criterion (3.4)
the SemMedDB\footnote{\url{https://lhncbc.nlm.nih.gov/ii/tools/SemRep\_SemMedDB\_SKR/dbinfo.html}} database is used, providing occurrences of biomedical concepts in MEDLINE/PubMed abstracts, as extracted automatically with \textit{MetaMap} \cite{Aronson2010}. 

\subsubsection{Multi-label dataset generation}
\label{ssec:dataset_generation}

Several use cases that satisfy the criteria of Table~\ref{table:criteria}, may correspond to a single year \textit{y} and several subordinate concepts may correspond to the same coarse-grained descriptor \textit{d}.
The development and testing datasets for all sibling subordinate concepts that were promoted during the same year consist of the same articles with different labels for each concept.  
Since a single article can be relevant to several distinct coarse-grained and fine-grained descriptors from independent use cases, the classification task is inherently multi-label. 

For this reason, we produce only two files for all use cases in a given year \textit{y} ($D_Cy$), one for the development of weakly-supervised models ($WS\_dev_y$) and one for testing (\textit{test\textsubscript{y}}). This is done by gathering all the corresponding articles together and assigning multi-label annotations accordingly.
However, for refinement into each concept \textit{c} only the articles that are relevant to the corresponding coarse-grained descriptor \textit{d} must be considered. 
We refer to these articles as the \textit{valid} documents for annotation with the concept \textit{c}.
For the use case of ``Niemann-Pick Disease, Type A'' for instance (Fig.~\ref{fig:MeSH_subdivision_Example}), out of all the articles included in the test file for 2007 (\textit{test\textsubscript{2007}}), only the articles annotated as relevant to ``Niemann-Pick Disease'' will be considered as \textit{valid} for ``Niemann-Pick Disease, Type A'' refinement. 
The open-source implementation of this dataset development process in Java is available in GitHub\footnote{\url{https://github.com/ThomasChatzopoulos/MeSH\_retrospective\_dataset}}.

\subsection{Weak supervision enhancement}
\label{ssec:WS_study}
Previous work suggests that \textit{concept occurrence} is a strong heuristic for fine-grained semantic indexing, but simpler dictionary-based approaches can also be quite strong in some cases~\cite{nentidis2020beyond}.
Ιn this work, we consider a range of dictionary-based variants in order to investigate whether they could enhance the quality of the weak labels.
In particular, adopting the terminology introduced in data programming~\cite{Ratner2016}, we refer to these dictionary-based approaches and the concept-occurrence heuristic as labeling functions.
Even though the dictionary-based labeling functions essentially rely on the same initial information that is used also by \textit{concept occurrence}, they may still result in different labels for the same article. 

\subsubsection{Alternative sources for weak supervision}
\label{sssec:LFs}
Eight dictionary-based labeling functions are introduced as potential alternative sources of weak supervision for the fine-grained semantic indexing task.  
Each labeling function (LF) assigns a concept label to an article if any dictionary element associated with the concept literally occurs in the title or abstract of the article. 
In the first LF (\textit{name exact}) only the name of a concept is matched (e.g.``Niemann-Pick Disease, Type A'') and in the second (\textit{synonyms exact}) the synonyms of it available in the MeSH thesaurus are used (e.g.``Classical Niemann-Pick Disease'', ``Sphingomyelin Lipidosis'' etc).

Then, six variants of these two labeling functions are introduced, 
in order to improve recall (see Table~\ref{table:Label_functions}). 
In particular, the dictionary elements are first lower-cased (\textit{name lowercase}, \textit{synonyms lowercase}), then the punctuation marks are removed (\textit{name no punct}, \textit{synonyms no punct}), and finally, phrases are split into single-token elements (\textit{name tokens}, \textit{synonyms tokens}).
For applying the latter six labeling functions, the title and the abstract of the article are also lower-cased, but no further modifications are applied to them. 

\begin{table}[!t]
\caption{The dictionary elements considered by each dictionary-based labeling function for assigning labels regarding the new fine-grained descriptor ``Niemann-Pick Disease, Type A''.}
\begin{center}
\small
\begin{tabular}{L{0.25\linewidth}L{0.095\linewidth}L{0.55\linewidth}}
Labeling function   & Abbr. & Dictionary elements                        \\
\hline
\textit{name exact}        & $NE$ &``Niemann-Pick Disease, Type A''                        \\
\textit{synonyms exact}    & $SE$ &``Classical Niemann-Pick Disease'', ``sphingomyelin lipidosis'', ...\\
\textit{name lowercase}    & $NL$ &``niemann-pick disease, type a''                                  \\
\textit{synonyms lowercase} & $SL$ &``classical niemann-pick disease'', ``sphingomyelin lipidosis'', ...\\
\textit{name no punct}     & $NNP$ &``niemann pick disease type a''                                  \\
\textit{synonyms no punct} & $SNP$ &``classical niemann pick disease'', ``sphingomyelin lipidosis'' ...\\
\textit{name tokens}       & $NT$ &``niemann'', ``pick'', ``disease'',  ``type'', ``a'' \\
\textit{synonyms tokens}    & $ST$ &``classical'', ``niemann'', ``pick'', ``disease'', ``sphingomyelin'', ``lipidosis'', ...
\end{tabular}
\label{table:Label_functions}
\end{center}
\end{table}

\subsubsection{Combination of weak supervision sources}
\label{sssec:ensembles}

In order to combine the predictions of different labeling functions into a single prediction we examine three alternative approaches. 
Namely, two voting classifiers based on \textit{majority voting} ($MV$) and \textit{at-least-one} voting ($ALO$)~\cite{kambhatla2006minority}, as well as a state-of-the-art probabilistic \textit{label modeling} ($LM$) approach for combining noisy heuristics, introduced in the context of Snorkel~\cite{Ratner2017}. 

The $MV$ schema assigns a label to an article if most of the labeling functions (voters) assign this label to this article. This way, we expect the precision of the ensemble to be favored, as labels predicted by only a few labeling functions are disregarded as untrustworthy.
In the $ALO$ schema, on the other hand, a label is assigned to an article if any of the labeling functions assign this label to this article. This is a special case of the \textit{at-least-N} schema, proposed by Kambhatla~\cite{kambhatla2006minority}, where $N=1$.
In this approach, even if a single labeling function predicts a label, this will be included in the prediction of the ensemble, favoring its recall.

Finally, the Snorkel $LM$ approach considers the true label for an article as a latent variable in a probabilistic generative model, which is estimated based on a weighted combination of the labeling functions.
In particular, statistical dependencies between the
labeling functions are considered during model development, to avoid the ``double counting'' problem for correlated labeling functions~\cite{Ratner2016, Ratner2017}.
Eventually, a label is assigned to an article if the estimated probability of the label is greater than 0.5.     


As all the labeling functions are variations of the same dictionary-based approaches, we expect a high degree of correlation, which may harm the accuracy of the ensembles.
For this reason, as part of the weak-supervision-enhancement stage, the optimal subset of labeling functions is sought for. 
In this direction, all 502 combinations of two or more labeling functions out of the nine available are considered in the validation experiments\footnote{Αt least three labeling functions are required for \textit{Snorkel} $LM$, therefore combinations of two labeling functions are only considered for $MV$ and $ALO$.}. 


\subsection{Model development}
\label{ssec:model_dev}
The final part of the \textit{Deep Beyond MeSH} method is the development of weakly-supervised Deep Learning (DL) models for the task of refining existing coarse-grained MeSH annotations to the level of concepts. 
In this direction, the method builds upon a state-of-the-art approach in text classification, that is fine-tuning a pre-trained BERT model, with one additional task-specific output layer~\cite{Devlin2019}. 
This approach has achieved very good results in a range of biomedical text classification tasks, transferring knowledge acquired during the pre-training phase to the task of interest~\cite{Peng2019}. 
In addition, 
experiments with pre-trained \textit{BERT} models for named entity recognition suggest that such methods can have an inherent resilience to label noise~\cite{Tanzer2021}, in accordance with previous experiments on image data~\cite{Hendrycks2019}.
Therefore, in this work, we investigate the adequacy of fine-tuning pre-trained \textit{BERT} models for label refinement, where fine-tunning is based on weak supervision.

\subsubsection{One multi-label model per year}
We have opted to train a single multi-label model for all the fine-grained labels of a single year \textit{y}, as compared to developing a distinct model for each label. 
This way, knowledge from different labels is integrated into shared deep representations for all the use cases of a single year.  
Preliminary experiments with single-label models in a One-Versus-Rest approach confirmed that multi-label ones are at least as good. 
Actually, a common model for multiple years would also be possible but would pose experimental complications as articles included in the test set of one use case may be included in the development set of another.
For this reason, we opt for a simpler scenario, where a distinct multi-label model (\textit{model\_val\textsubscript{y}}) is developed for each year \textit{y}, as depicted in Figure~\ref{fig:BERT_ML}.   
In this architecture, the title and abstract of each article are concatenated, tokenized, and provided as input to the \textit{BERT} model. The embedding of the classification token $CLS$ is then provided as input in a fully-connected output layer that produces a vector of size equal to the number of labels as in~\cite{DBLPSilvaG21}. 

\begin{figure}[!h] \includegraphics[width=1\textwidth]{figures/DBMarchitecture.png}
\caption{The architecture of a multi-label \textit{Deep Beyond MeSH} model for the fine-grained labels introduced in a single year.
}
\label{fig:BERT_ML}      
\end{figure}

\subsubsection{weakly-supervised validation of the number of training epochs}

Preliminary experiments indicated that the number of training epochs can significantly affect the performance of the model. Too few epochs lead the models to under-fit, while training for too many epochs can lead to over-fitting on the weak supervision. Hence, we needed to select a number of epochs that provide good models. For this purpose, 
we split \textit{WS\_dev\textsubscript{y}} into a training (\textit{WS\_train\textsubscript{y}}) and a validation (\textit{WS\_val\textsubscript{y}}) part (90-10\%). 
Then, we trained the model on the \textit{WS\_train\textsubscript{y}} part 
for a sufficient number of epochs 
and checked the loss after each epoch in the unseen \textit{WS\_val\textsubscript{y}} part. 
This way we could identify the model performing best (\textit{best\_ep}) on \textit{WS\_val\textsubscript{y}}, that is the one minimising loss.

Minimising loss could still lead to overfitting on the weak labels. In order to achieve a higher level of generalization, we adopted an early stopping strategy that concludes the fine-tuning process once the training loss on \textit{WS\_val\textsubscript{y}} starts increasing~\cite{prechelt1998early}.
In this case, the $best\_ep$ model is always that of the penultimate epoch of the early stopped fine-tuning process.  
Furthermore, together with the $best\_ep$ model that achieves the best performance on \textit{WS\_val\textsubscript{y}}, we also considered the $prev\_ep$ and $next\_ep$
models that correspond to the epochs just before or after $best\_ep$. 
The motivation for epochs that are close to the best-performing one is that their ``imperfection" may be due to a higher level of generalization. 

\subsubsection{Label imbalance and under-sampling}
Another challenge in training the models is the imbalance of the labels.
That is, the number of positive instances for each label varies among use cases, favoring more popular fine-grained labels. 
For this purpose, we considered extensions of the well-established binary cross entropy (BCE) loss~\cite{Bengio2013} can handle imbalanced training data for multi-label text classification~\cite{Huang2021}. 
In particular, the rebalanced focal loss function (R-BCE-FL) is an extension of BCE loss, first introduced to handle label imbalance in multi-label image classification tasks~\cite{Wu2020} and recently extended to multi-label text classification tasks~\cite{Huang2021}.    
R-BCE-FL modifies BCE in two ways. First, the instances are rebalanced, by oversampling based on the overall frequency of corresponding labels, and second, a focusing parameter increases the weight of more hard-to-classify instances, based on their predicted probability~\cite{Huang2021}. 

Looking separately at each label, there are many more negative instances than positive ones. In extreme cases, this abundance of negative instances  can prevent the models from learning to recognize positive ones, as their overall effect on loss is minimal. 
Previous work has shown that under-sampling can help address this problem~\cite{nentidis2020beyond}. 
Therefore, we experimented also with an under-sampling procedure to remove some negative instances, and achieve an ideal negative-to-positive ratio ($balance\_n$) for each label in the dataset.
As explained in subsection~\ref{ssec:dataset_generation}, not all instances are \textit{valid} for all labels, and this is taken into account in the undersampling of negative instances, i.e. only the \textit{valid} ones for the particular label are used. 
An ideal negative-to-positive ratio is not known beforehand, but a reasonable estimate can be provided through validation. 

A negative instance that is superfluous for a specific label may be positive for another label, or one of the few \textit{valid} negative instances for another.
Therefore, removing random negative instances to strictly enforce the ideal negative-to-positive ratio for a label, may deplete the positive or \textit{valid} negative instances for another. 
For this reason, the undersampling process excludes instances
that are either positive for any of the labels or they are \textit{valid} for labels with a low negative-to-positive ratio. Algorithm~\ref{alg:undersample} explains the undersampling process.


\begin{algorithm}[!t]
\caption{Underasmple negative instances in a $dataset$ towards a negative-to-positive ratio close to $balance\_n$ for all labels.}
\begin{algorithmic}[1]
      \Function{Under-sample}{$dataset$, $balance\_n$}
       
       
        \While{ $\exists$ $negative\_instance$ in $dataset$ that has not been checked \textbf{and} \\ \hskip1.5em $\exists$ label $l$ in $dataset$ with $valid\_negatives > balance\_n * positives$}
            \State Get a random negative instance $i$ from the $dataset$.
            \State $i$ has now been checked.
           
            \If { $\exists$ label $l'$ in $dataset$ with $valid\_negatives \leq balance\_n * positives$  \\ \hskip3.0em \textbf{and} $i$ is valid for $l'$} \Comment{$i$ is a necessary negative for $l'$}
                \State Do nothing. 
            \Else \Comment{$i$ is superfluous}
                \State Remove $i$ from dataset. 
            \EndIf 
        \EndWhile
       
      \EndFunction
\end{algorithmic}
\label{alg:undersample}
\end{algorithm}

\subsubsection{Randomness in the model validation process}
Randomness is introduced in several steps of the model development process~\cite{Dodge2020}, notably in the splitting of training and validation sets, in the under-sampling of negative instances, in weight initialization of the output layer of the pre-trained BERT model, and in stochastic optimization~\cite{Kingma2014}. 
Our preliminary experiments showed that starting with different random seeds can lead to different results. For this reason, each experiment was repeated with different seeds and the predictions were aggregated with a majority voting approach. 
We have also investigated other voting schemes, such as at-least-one~\cite{kambhatla2006minority}, as well as filtering out experiments with low validation performance, but did not obtain better results. Preliminary results also showed that six random seeds are good enough to reach a stable estimate of model performance.

\subsection{Competing machine learning approaches}
\label{ssec:baselines}

For the purposes of comparison, three machine-learning approaches have been included in our experiments.
The first one is a logistic regression ($LR$) model for each fine-grained label, as in \textit{Beyond MeSH}~\cite{nentidis2020beyond}.
For a fair comparison with the method introduced here, we perform the same under-sampling of negative instances, which is repeated for the same random seeds and the results are combined using majority voting.
Both lexical and semantic features are considered, as suggested in~\cite{nentidis2020beyond} and the number of features, as well as the level of regularization, are optimised on validation set. 

The second method is \textit{Coarse2Fine} ($C2F$)~\cite{Mekala2021}, which extends coarse-grained categories into fine-grained ones, as explained in section \ref{sec:Background}. 
The basic difference of $C2F$ to the proposed method is that it does not use semantic information, as does the concept-occurrence heuristic. It relies exclusively on the literal occurrence of the label names. For a fair comparison against $DBM$, we provide $C2F$ with the same weak supervision.

Similarly to \textit{Beyond MeSH}, $C2F$ develops a distinct model for the refinement of each coarse-grained label into corresponding fine-grained models.
However, $C2F$ treats the classification problem as a multi-class problem, where each document belongs to exactly one of the fine-grained sub-categories of a coarse-grained label. Our fine-grained indexing setting is multi-label, with documents allowed to belong to any number of subcategories, even to none of them (unlabelled examples). 
In order to handle the many unlabelled examples, we introduce an additional subcategory for each coarse-grained label X, which we call X\_rest, representing all the documents that belong to X, but not any of the fine-grained subcategories of X. 
The predictions for these X\_rest labels are disregarded when reporting the performance of the $C2F$ models. That is, only the predictions for the fine-grained labels of interest are considered, as in $DBM$ and the other approaches.  

As a core step in the $C2F$ methodology is adding generated instances to handle imbalance, we did not undersample the negative examples. 
We also relied on the original implementation, as provided in the repository of the method\footnote{https://github.com/dheeraj7596/C2F}, and did not experiment with training parameters or different random seeds. 
However, in order to investigate the impact of generating instances as per $C2F$, we also introduce another approach ($DBM^{C2F}$), where we used the supervision produced by the generative model of \textit{Coarse2Fine} to train a single deep model for different use-cases. The $DBM^{C2F}$ approach used the implementation and configurations of $DBM$, including balancing and majority voting over different random seeds.

\subsection{Evaluation measures}
As both precision and recall are important for fine-grained semantic indexing, we adopt the F1-measure ($F1$) as an overall performance measure, as defined in equation~\ref{eq:f1} for a set of m documents. Here, $Y_i$ is the set of ground truth labels for the i-th document and $Z_i$ is the set of the predicted ones.
\begin{equation}
\label{eq:f1}
F_1=\frac{1}{m} \sum_{i=1}^{m} \frac{2|Y_i\cap Z_i|}{|Z_i|+|Y_i|}
\end{equation}
Furthermore, we consider two complementary approaches for averaging the results across different use cases. Namely the label-based macro-averaged F1-measure ($maF1$) and the label-based micro-averaged F1-measure ($miF1$)~\cite{Tsoumakas2009}. The first treats all use cases equally, preventing a few use cases of high prevalence to dominate the final score. The latter takes into account the absolute number of documents in each use case, preventing rare use cases to have too large an impact on the final score.
For completeness, we consider precision and recall as well, both label-based macro-averaged ($maP$, $maR$) and micro-averaged ($miP$, $miR$).

As discussed in subsection~\ref{ssec:dataset_generation}, in the multi-label datasets for fine-grained semantic indexing 
we want to identify articles relevant to a specific fine-grained label, out of all the \textit{valid} articles annotated with its coarse-grained parent.
Therefore, articles that are not \textit{valid} for a specific fine-grained label are not of interest and are not considered in the evaluation of the label.
In practice, given the i-th test document, multi-label models that have been trained on a set of labels $L$, will produce a list of predicted labels $Z_i \subseteq L$. 
To ensure document validity, we remove from $Z_i$ any labels that are not valid for the document. Hence, we get a $Z_i'$.
We refer to this process as \textit{validity filtering} and employ it in the evaluation of all models in this work. 


\section{Experiments and results}
\label{sec:Experiments}

The retrospective method $RetroBM$, described in subsection~\ref{ssec:retro_data}, was applied to several consecutive years (2006-2019) of the evolution of MeSH for the development of respective datasets. These datasets were then used to run experiments with the weak labeling functions, described in subsection~\ref{ssec:WS_study}, and the $DBM$ models for fine-grained semantic indexing, described in subsection~\ref{ssec:model_dev}.
The first set of validation experiments was performed on the dataset developed for fine-grained labels introduced in 2006 (\textit{2006 dataset}). 
These experiments were used to make design choices about the architecture and configurations of the $DBM$ method, such as the specific pre-trained \textit{BERT} model to fine-tune, the $balance\_n$ value, and the specific weak supervision enhancement approach.   
The data introduced in the subsequent years were used in ``evaluation experiments'', which aimed to confirm the choices made on the \textit{2006 dataset}.

\subsection{Retrospective datasets}
\label{ssec:retro_data_2005-2020}
MEDLINE data from fourteen years (2006 to 2019) were processed according to the retrospective scenario of subsection~\ref{ssec:retro_data}, for the development of fourteen respective datasets \textit{WS\_dev\textsubscript{y}} and \textit{test\textsubscript{y}}, adhering to the criteria of Table~\ref{table:criteria}.  
Table~\ref{table:criteria_values} presents the values of the data availability restrictions of criterion 3, in particular\footnote{The remaining criteria 1 and 2 do not require the selection of any specific numerical value.}.
The 2020 version of the MEDLINE/PubMed Annual Baseline Repository was used to retrieve documents and MeSH topic annotations, and version 4.0 of SemMedDB was used for \textit{concept occurrence}.
In total, 88 descriptors were found in the period of these fourteen years as shown in Table~\ref{table:use_cases}\footnote{The full list of descriptors per year is available in the supplementary file S1.}.
In 2006, for instance, 18 fine-grained descriptors were found, and the respective \textit{WS\_dev\textsubscript{2006}} and \textit{test\textsubscript{2006}} datasets consisted of 446,377 and 468,271 articles respectively. As already discussed, for articles in \textit{WS\_dev\textsubscript{2006}} there are only weak labels, whereas for \textit{test\textsubscript{2006}} ground-truth labels are also available.  
Table~\ref{table:use_cases} reveals the uneven distribution of descriptors per year, which is correlated to the similarly uneven distribution of concept promotion events (criterion 1) in MeSH presented in~\cite{Nentidis2021}.  

\begin{table}[!t]
\caption{Data availability restrictions for dataset development with the \textit{Retrospective Beyond MeSH} ($RetroBM$) method.}
\begin{center}
\small
\begin{tabular}{L{0.23\linewidth}L{0.2\linewidth}L{0.094\linewidth}L{0.13\linewidth}L{0.2\linewidth}}
Restriction code & 3.1 & 3.2 & 3.3 & 3.4 \\
Restriction Name & \textit{test\_positive\textsubscript{min}} & 
 \textit{dev\textsubscript{min}}            & 
 \textit{dev\textsubscript{max}}            & 
 \textit{dev\_positive\textsubscript{min}}  \\
Restriction Value & 10& 10 & 1,000,000 & 10 
\end{tabular}
\label{table:criteria_values}
\end{center}
\end{table}

\begin{table}[!t]
\caption{ Retrospective datasets for years 2006 to 2019.
\#$D_Cy$ stands for the number of use cases per year.
For the \textit{WS\_dev\textsubscript{y}} and \textit{test\textsubscript{y}} datasets, we report the number of all articles that are \textit{valid} for any of the use cases (total) and the number of articles with at least one weak label (weak lab.).  
For \textit{test\textsubscript{y}} in particular, the number of articles with at least one ground-truth label is also provided (positive).
}
\begin{center}
\small
\begin{tabular}{M{0.1\linewidth}M{0.07\linewidth}R{0.11\linewidth}R{0.12\linewidth}|R{0.11\linewidth}R{0.12\linewidth}R{0.1\linewidth}}
\multirow{2}{*}{Year} & \multirow{2}{*}{\begin{tabular}[c]{@{}l@{}}\#$D_Cy$\end{tabular}} & \multicolumn{2}{l}{\#Articles \textit{WS\_dev\_y}} & \multicolumn{3}{l}{\#Articles \textit{test\_y}} \\
 &            & total    & weak lab.   & total  & weak lab. & positive \\
\hline
2006  & 18    & 446,377   & 12,262   & 468,271 & 20,542 & 27,118 \\
2007  & 17    & 159,665   & 29,678   & 143,876 & 24,246 & 23,796 \\
2008  & 10    & 176,092   & 8,689    & 117,083 & 4,638  & 5,571  \\
2009  & 6     & 117,706   & 4,545    & 88,309  & 3,852  & 5,814  \\
2010  & 6     & 78,443    & 2,788    & 92,962  & 2,842  & 7,311  \\
2011  & 8     & 848,958   & 5,172    & 526,238 & 6,424  & 8,047  \\
2012  & 6     & 587,895   & 5,030    & 205,894 & 3,313  & 14,860 \\
2013  & 1     & 10,362    & 146     & 5,048   & 148   & 337   \\
2014  & 1     & 1,457     & 222     & 649    & 113   & 120   \\
2015  & 1     & 5,963     & 221     & 934    & 26    & 20    \\
2016  & 3     & 74,133    & 408     & 23,836  & 139   & 409   \\
2017  & 6     & 34,628    & 3,084    & 6,052   & 579   & 838   \\
2018  & 3     & 86,205    & 933     & 12,089  & 467   & 505   \\
2019  & 2     & 96,978    & 7,543    & 3,642   & 283   & 169   
\end{tabular}
\label{table:use_cases}
\end{center}
\end{table}

\subsection{Validation experiments on \textit{2006 dataset}}
\label{ssec:val_experiments}

The 18 fine-grained labels introduced in 2006 (\textit{2006 dataset}) constitute the validation dataset for this work, and were used to inform design choices about the structure of the $DBM$ model and estimate some hyper-parameters.

Table~\ref{table:LF_results} presents the performance of each labeling function on the ground truth data of \textit{test\textsubscript{2006}}.
The results show that \textit{concept occurrence} ($CO$) was clearly the strongest of these heuristics, which is in accordance with previous observations on smaller-scale experiments reported in~\cite{nentidis2020beyond}.
In addition, we observe that most labeling functions achieved better precision than recall, except those based on tokenized terms ($NT$ and $ST$).
A reason for the low precision of the latter can be the low specificity of some too-general tokens that are produced when splitting multi-word terms. Splitting "Niemann-Pick Disease, Type A", for instance, leads to tokens such as "disease", "type", and "a" that are too general (Table~\ref{table:Label_functions}).

\begin{table}[!t]
\caption{Performance of each labeling function ($LF$) on $test_{2006}$ ground truth data. 
The  labeling functions are \textit{concept occurrence} (\textit{CO}),  
\textit{name exact} (\textit{NE}), \textit{synonyms exact} (\textit{SE}),
\textit{name lowercase} (\textit{NL}), \textit{synonyms lowercase} (\textit{SL}),
\textit{name no punct} (\textit{NNP}), \textit{synonyms no punct} (\textit{SNP}),
\textit{name tokens} (\textit{NT}), and \textit{synonyms tokens} (\textit{ST}).
The variance across the 18 labels ($var$) is also reported for the macro-averaged metrics.}
\begin{center}
\small 

\begin{tabular}{lllllll}
LF   			& $maP$\footnotesize{ $\pm var$} & $maR$\footnotesize{ $\pm var$}    & $maF1$\footnotesize{ $\pm var$}	& $miP$   & $miR$   & $miF1$\\
\hline
\textit{CO} 	& 0.715\footnotesize{ $\pm0.03$}& 0.618\footnotesize{ $\pm0.08$}& \textbf{0.634}\footnotesize{  $\pm0.06$}&0.737&0.531&\textbf{0.618}	\\
\textit{NE}     & 0.707\footnotesize{ $\pm0.16$}& 0.082\footnotesize{ $\pm0.04$}& 0.112\footnotesize{ $\pm0.04$}&0.901&0.027&0.052	\\
\textit{SE}   	& 0.656\footnotesize{ $\pm0.18$}& 0.212\footnotesize{ $\pm0.10$}& 0.251\footnotesize{ $\pm0.12$}&0.889&0.046&0.088	\\
\textit{NL}     & 0.649\footnotesize{ $\pm0.08$}& 0.441\footnotesize{ $\pm0.13$}& 0.446\footnotesize{ $\pm0.12$}&0.589&0.203&0.302	\\
\textit{SL} 	& 0.620\footnotesize{ $\pm0.10$}& 0.365\footnotesize{ $\pm0.13$}& 0.408\footnotesize{ $\pm0.11$}&0.709&0.251&0.371	\\
\textit{NNP}    & 0.574\footnotesize{ $\pm0.12$}& 0.440\footnotesize{ $\pm0.14$}& 0.442\footnotesize{ $\pm0.13$}&0.583&0.196&0.293	\\
\textit{SNP}	& 0.597\footnotesize{ $\pm0.12$}& 0.349\footnotesize{ $\pm0.13$}& 0.385\footnotesize{ $\pm0.11$}&0.690&0.206&0.318	\\
\textit{NT}     & 0.322\footnotesize{ $\pm0.07$}& 0.838\footnotesize{ $\pm0.04$}& 0.377\footnotesize{ $\pm0.07$}&0.157&0.807&0.263	\\
\textit{ST}    	& 0.274\footnotesize{ $\pm0.06$}& 0.874\footnotesize{ $\pm0.04$}& 0.338\footnotesize{ $\pm0.07$}&0.081&0.883&0.149	
\end{tabular}   
\end{center}
\label{table:LF_results}
\end{table}

Using the weak supervision provided by $CO$, different models were developed on $dev_{2006}$. These models were developed based on the $DBM$ approach, \textit{Logistic Regression} ($LR$), \textit{Coarse2Fine} ($C2F$), as well as the combined $DBM^{C2F}$ described in subsection~\ref{ssec:baselines}.
Table~\ref{table:model_results_WS_2006} presents the predictive performance of these models on the ground truth data of $test_{2006}$\footnote{Detailed results with scores per descriptor are available in the supplementary file S2.}.
Experiments with $DBM$ models using different $balance\_n$ values ranging from 1 to 100, led to the choice of 10-to-1 as a negative-to-positive instance ratio. Hence, we adopted this value in the remaining experiments.

\begin{table}[!t]
\caption{Model performance on $test_{2006}$ ground truth data. Models developed on $dev_{2006}$ under the weak supervision provided by \textit{concept occurrence} ($CO$).
The variance across the 18 labels ($var$) is also reported for the macro-averaged metrics.
A subscript in $LR$ models indicates the number of features selected.
A subscript in $DBM$ and $DBM^{C2F}$  models indicates the training epoch considered, with the values \textit{p}, \textit{b}, and \textit{n} corresponding to \textit{prev\_ep}, \textit{best\_ep}, and \textit{next\_ep} respectively.
}
\begin{center}
\small 
\begin{tabular}{lllllll}
LF/model   			& $maP$\footnotesize{ $\pm var$} & $maR$\footnotesize{ $\pm var$}    & $maF1$\footnotesize{ $\pm var$}	& $miP$   & $miR$   & $miF1$\\
\hline
$CO$ 		& 0.715\footnotesize{ $\pm0.03$}& 0.618\footnotesize{ $\pm0.08$}& 0.634\footnotesize{ $\pm0.06$}&0.737 &0.531 &0.618\\
$LR_{5}$    & 0.427\footnotesize{ $\pm0.11$}& 0.822\footnotesize{ $\pm0.02$}& 0.474\footnotesize{ $\pm0.10$}&0.295 &0.655 &0.406\\
$LR_{10}$   & 0.358\footnotesize{ $\pm0.10$}& 0.807\footnotesize{ $\pm0.03$}& 0.396\footnotesize{ $\pm0.09$}&0.102 &0.629 &0.176\\
$LR_{100}$  & 0.275\footnotesize{ $\pm0.08$}& 0.809\footnotesize{ $\pm0.03$}& 0.318\footnotesize{ $\pm0.06$}&0.085 &0.620 &0.149\\
$LR_{1000}$ & 0.279\footnotesize{ $\pm0.09$}& 0.798\footnotesize{ $\pm0.04$}& 0.310\footnotesize{ $\pm0.07$}&0.134 &0.644 &0.221\\
$C2F$		& 0.240\footnotesize{ $\pm0.06$}& 0.941\footnotesize{ $\pm0.00$}& 0.332\footnotesize{ $\pm0.08$}&0.079 &0.959 &0.147\\ 
$DBM_p$  	& 0.652\footnotesize{ $\pm0.06$}& 0.663\footnotesize{ $\pm0.07$}& 0.645\footnotesize{ $\pm0.06$}&0.706 &0.572 &0.632\\				
$DBM_b$ 	& 0.657\footnotesize{ $\pm0.06$}& 0.613\footnotesize{ $\pm0.10$}& 0.596\footnotesize{ $\pm0.08$}&0.709 &0.560 &0.626\\ 
$DBM_n$ 	& 0.717\footnotesize{ $\pm0.03$}& 0.666\footnotesize{ $\pm0.08$}& \textbf{0.652}\footnotesize{ $\pm0.06$}&0.713 &0.568 &0.632\\		
$DBM^{C2F}_p$	& 0.736\footnotesize{ $\pm0.03$}& 0.651\footnotesize{ $\pm0.09$}& 0.639\footnotesize{ $\pm0.06$}&0.713 &0.569 &0.632\\ 
$DBM^{C2F}_b$	& 0.739\footnotesize{ $\pm0.03$}& 0.631\footnotesize{ $\pm0.08$}& 0.632\footnotesize{ $\pm0.05$}&0.715 &0.565 &0.631\\
$DBM^{C2F}_n$	& 0.741\footnotesize{ $\pm0.03$}& 0.649\footnotesize{ $\pm0.09$}& 0.645\footnotesize{ $\pm0.06$}&0.716 &0.567 &\textbf{0.633}
\end{tabular}
\end{center}
              
\label{table:model_results_WS_2006}
\end{table}

Beyond the standard \textit{BERT} model~\cite{Devlin2019}, domain-specific variants were also considered in the validation experiments. 
These include both \textit{BERT} models fine-tuned on large biomedical corpora such as PubMed (\textit{BioBERT}~\cite{lee2019biobert}) or pre-trained on such data from scratch (\textit{PubMedBERT}~\cite{Gu2022}). One of the advantages of the latter approach is that the vocabulary considered is domain-specific, allowing it to better capture the semantics of the text. 
The good performance of \textit{PubMedBERT} was confirmed on preliminary fine-tuning experiments on the \textit{2006 dataset}, and it was thus chosen as the basis for all $DBM$ models.       

Regarding the logistic regression models, different numbers of features were considered for feature selection with the F-ANOVA statistic, namely 5, 10, 100, and 1000. 
Both lexical tokens and semantic features were included in the feature sets; whereby semantic features were identified based on \textit{UMLS} concepts, as proposed in~\cite{nentidis2020beyond}. 
The $L2$ regularization level for each $LR$ model was selected via grid search on the weakly-supervised training data ($WS\_dev_{2006}$), considering a range of values from 0.01 to one billion\footnote{Namely these values were considered: 0.01, 0.1, 1, 10, 100, 1K, 10K, 100K, 1M, 1B}.

The results reported in Table~\ref{table:model_results_WS_2006} show that none of the $LR$ models managed to outperform the $CO$ baseline overall.
The best $LR$ model, namely $LR_5$ with five features, managed to outperform $CO$ in $F1$ for about 40\% of the 18 labels of $test_{2006}$\footnote{Detailed results with scores per descriptor are available in the supplementary file S2.}.
This is in accordance with previous results (see~\cite{nentidis2020beyond}) which suggested that $LR$ models may bring improvements upon the results of weak labelling, only in specific cases. However, the identification of these specific configurations (i.e., specific feature set and level of regularization), requires the use of ground truth validation data.

The overall $maF1$ performance of the $C2F$ models was also low, achieving top recall but low precision. 
In particular, $C2F$ did better than the $CO$ baseline for only three out of the 18 fine-grained labels of the dataset.
The low performance of a state-of-the-art method like $C2F$ is an indication that the task and corresponding datasets are very challenging. This is partly due to the high imbalance among the labels and between labeled and ``unlabeled'' documents, that do not belong to any of the labels of interest. 

On the other hand, all $DBM$ models trained on $CO$ weak labels achieved better $miF1$ than $CO$ itself, and most of them did better also in terms of $maF1$ as well.
In particular, $DBM_p$ and $DBM_n$, considering the $prev\_ep$ and $next\_ep$ models respectively, outperformed $CO$ in $maF1$, with $DBM_n$ achieving the top $maF1$ score among all the methods considered.
Additionally, the $DBM^{C2F}_p$ and $DBM^{C2F}_n$ models, also to outperformed $CO$ in terms of $maF1$.
These results suggest that some deep models based on the $DBM$ method can indeed improve upon the weak supervision that they rely on. 
In addition, considering the epoch next to the one achieving the lowest validation loss ($next\_ep$) leads to the best results for both $DBM$ and $DBM^{C2F}$ models in these experiments. 


Next, we experimented with the enhancement of the original weak supervision provided by $CO$, as discussed in subsection~\ref{sssec:ensembles}. 
In particular, for each combination of the available labeling functions, three ensemble approaches were assessed, namely \textit{majority voting} ($MV$), \textit{at-least-one} ($ALO$), and \textit{Snorkel label model} ($LM$).
As shown in Table~\ref{table:ensemble_top_combinations} some of those combinations can lead to further improvements in the classification performance of the models. The table presents the top eight combinations based on the $maF1$ and $miF1$ achieved by any of the three ensemble approaches\footnote{The full table with all the combinations is available in the supplementary file S3.}. 
The highest $maF1$ score achieved on $test_{2006}$ is 0.689. 
This score has been achieved by several distinct labeling-function combinations using either the $Snorkel$ $\textit{label model}$ ($LM$) or the \textit{at-least-one} ($ALO$) approach\footnote{The top $maF1$ score using \textit{majority voting} ($MV$) is 0.679, achieved by a single labeling-function combination ($CO$, $SL$, $NL$, $NT$, $ST$).}. 


\begin{table}[!t]
\caption{
The scores of the 8 labeling-function ($LF$) combinations that achieved the top $maF1$ and $miF1$ scores in $test_{2006}$ (0.689) considering any of the three ensemble approaches: \textit{majority voting} ($MV$), \textit{at-least-one} ($ALO$), and \textit{Snorkel label modeling} ($LM$)
The table is sorted by the variance of $maF1$ across the scores of the three approaches ($var$).
}
\small 
\begin{tabular}{L{0.35\linewidth}L{0.05\linewidth}L{0.05\linewidth}L{0.05\linewidth}L{0.05\linewidth}L{0.05\linewidth}L{0.05\linewidth}L{0.05\linewidth}}
&\multicolumn{4}{c}{$maF1$}   &\multicolumn{3}{c}{$miF1$} \\
$LF$ combination & $MV$ & $ALO$ & $LM$ & $var$ & $MV$ & $ALO$ & $LM$ \\
\hline
$CO$, $NL$, $SL$ 					& 0.586 & \textbf{0.689} & \textbf{0.689} & 0.004 	&	0.410&	\textbf{0.646}&	\textbf{0.646}\\
$CO$, $NL$, $SL$, $NNP$				& 0.540 & \textbf{0.689} & \textbf{0.689} & 0.007 	&	0.359&	\textbf{0.646}&	\textbf{0.646}\\
$CO$, $NL$, $SL$, $NNP$, $SE$		& 0.517 & \textbf{0.689} & 0.670 &  0.009 			&	0.289&	\textbf{0.646}&	0.642\\
$CO$, $NL$, $SL$, $SE$				& 0.480 & \textbf{0.689} & 0.670 &  0.013 			&	0.324&	\textbf{0.646}&	0.642\\
$CO$, $NL$, $SL$, $NE$				& 0.465 & \textbf{0.689} & 0.654 &  0.015 			&	0.313&	\textbf{0.646}&	0.640\\
$CO$, $NL$, $SL$, $NNP$, $SE$, $NE$	& 0.429 & \textbf{0.689} & 0.643 &  0.019 			&	0.240&	\textbf{0.646}&	0.507\\
$CO$, $NL$, $SL$, $NNP$, $NE$		& 0.417 & \textbf{0.689} & 0.670 &  0.023 			&	0.254&	\textbf{0.646}&	0.642\\
$CO$, $NL$, $SL$, $NE$, $SE$		& 0.403 & \textbf{0.689} & 0.654 &  0.024 			&	0.236&	\textbf{0.646}&	0.640
\end{tabular}
\label{table:ensemble_top_combinations}      
\end{table}

Interestingly, none of the top-performing combinations included the low-precision labeling functions $NT$ and $ST$. This helps the $ALO$ approach to achieve high maF1 scores, as it is known to be good at combining precise voters by improving the total recall~\cite{kambhatla2006minority}.
Another important observation is that $CO$, \textit{name lowercase} ($NL$), and \textit{synonyms lowercase} ($SL$) are present in all top-performing combinations, regardless of the ensemble approach. 
In addition, it seems that the combination of only these three labeling functions was sufficient to achieve top performance, as shown by comparing the first and second combinations in Table~\ref{table:ensemble_top_combinations}.
As a result, we have chosen to use the combination of $CO$, $NL$, and $SL$ in further experiments.

Regarding the choice of ensemble approaches, the fact that $ALO$ achieved top results with more labeling-function combinations than other methods\footnote{8 top-performance combinations with $ALO$ against 2 with the \textit{label model}}, indicates that $ALO$ is more robust to the choice of labeling functions in this specific setup. 
This could be due to the fact that in the $ALO$ ensemble, some labeling functions can be completely overshadowed by the ones that have better recall.
For example, in the fifth set ($CO$, $NL$, $SL$, $NE$) presented in Table~\ref{table:ensemble_top_combinations}, $NE$ is overshadowed by $NL$ with the use of $ALO$. This is because every document with a literal occurrence of the exact concept name ($NE$) will also have a literal occurrence when both the document and the concept name are lowercased ($NL$). 
For the $ALO$ approach, which assigns a label to an article if at least one labeling function does, this means that the vote of $NE$ doesn't affect the result. 
As a result, we have opted for the $ALO$ ensemble of $CO$, $NL$, and $SL$ to be used in the \textit{Deep Beyond MeSH} method. 
The detailed performance of the selected labeling functions and the \textit{enhanced supervision} ($ALO_3$) is summarized in Table~\ref{table:WS_enhancement_results}.

\begin{table}[!t]
\caption{The \textit{test\textsubscript{2006}} performance of the \textit{enhanced supervision} ($ALO_3$), the corresponding labeling functions, and the models developed using $ALO_3$.
A prefix ($e$) indicates training under \textit{enhanced supervision}.
The variance across the 18 labels ($var$) is also reported for the macro-averaged metrics.
The subscript in $LR$ models indicates the number of features selected.
The subscript in $DBM$ models indicates the training epoch considered, with the values \textit{p}, \textit{b}, and \textit{n} corresponding to \textit{prev\_ep}, \textit{best\_ep}, and \textit{next\_ep} respectively.
}
\begin{center}
\small 
\begin{tabular}{lllllll}
LF/model   			& $maP$\footnotesize{ $\pm var$} & $maR$\footnotesize{ $\pm var$}    & $maF1$\footnotesize{ $\pm var$}	& $miP$   & $miR$   & $miF1$\\
\hline
$CO$ 				& 0.715\footnotesize{ $\pm0.03$}& 0.618\footnotesize{ $\pm0.08$}& 0.634\footnotesize{ $\pm0.06$} &0.737 &0.531 &0.618\\
$NL$     			& 0.649\footnotesize{ $\pm0.08$}& 0.441\footnotesize{ $\pm0.13$}& 0.446\footnotesize{ $\pm0.12$} &0.589 &0.203 &0.302\\
$SL$ 				& 0.620\footnotesize{ $\pm0.10$}& 0.365\footnotesize{ $\pm0.13$}& 0.408\footnotesize{ $\pm0.11$} &0.709 &0.251 &0.371\\
$MV_3$ 				& 0.752\footnotesize{ $\pm0.02$}& 0.599\footnotesize{ $\pm0.09$}& 0.615\footnotesize{ $\pm0.06$} &0.665 &0.335 &0.446\\
$LM_3$ 				& 0.717\footnotesize{ $\pm0.02$}& 0.698\footnotesize{ $\pm0.05$}& 0.689\footnotesize{ $\pm0.03$} &0.713 &0.591 &0.646\\
$ALO_3$ 			& 0.717\footnotesize{ $\pm0.02$}& 0.698\footnotesize{ $\pm0.05$}& 0.689\footnotesize{ $\pm0.03$} &0.713 &0.591 &0.646\\
$eLR_{5}$        	& 0.419\footnotesize{ $\pm0.09$}& 0.838\footnotesize{ $\pm0.02$}& 0.487\footnotesize{ $\pm0.10$} & 0.299	&0.797	&0.434\\
$eC2F$           	& 0.266\footnotesize{ $\pm0.07$}& 0.949\footnotesize{ $\pm0.00$}& 0.356\footnotesize{ $\pm0.09$} &0.072 &0.969 &0.134\\
$eDBM_{p}$  		& 0.681\footnotesize{ $\pm0.03$}& 0.744\footnotesize{ $\pm0.04$}& 0.700\footnotesize{ $\pm0.03$} &0.689 &0.624 &0.655\\
$eDBM_{b}$ 			& 0.688\footnotesize{ $\pm0.03$}& 0.738\footnotesize{ $\pm0.04$}& \textbf{0.702}\footnotesize{ $\pm0.03$}	&0.690 &0.626 &0.656\\
$eDBM_{n}$  		& 0.689\footnotesize{ $\pm0.03$}& 0.741\footnotesize{ $\pm0.04$}& \textbf{0.702}\footnotesize{ $\pm0.03$}	&0.695 &0.623 &\textbf{0.657}\\
$eDBM^{C2F}_{p}$  	& 0.641\footnotesize{ $\pm0.05$}& 0.716\footnotesize{ $\pm0.06$}& 0.664\footnotesize{ $\pm0.05$} &0.683 &0.623 &0.652\\
$eDBM^{C2F}_{b}$  	& 0.693\footnotesize{ $\pm0.02$}& 0.729\footnotesize{ $\pm0.05$}& 0.692\footnotesize{ $\pm0.03$} &0.692 &0.623 &0.655\\
$eDBM^{C2F}_{n}$  	& 0.700\footnotesize{ $\pm0.02$}& 0.729\footnotesize{ $\pm0.05$}& 0.692\footnotesize{ $\pm0.03$} &0.692 &0.623 &0.655

\end{tabular} 
\end{center}
\label{table:WS_enhancement_results}
\end{table}

These results reveal that the $ALO$ ensemble can exploit information that is already used by the $CO$ heuristic, namely the name and synonyms of the concepts, to achieve scores better than $CO$ alone, and in fact even better than the results of the trained models of Table~\ref{table:model_results_WS_2006}.
Finally, Table~\ref{table:WS_enhancement_results} also presents the performance of models trained on $ALO_3$ weak supervision\footnote{Detailed results with scores per descriptor are available in the supplementary file S2.}.
These results suggest that training on the enhanced $ALO_3$ labels was beneficial for all learning approaches, compared to training on $CO$. 
Again, some $DBM$ models trained on the enhanced $ALO_3$ supervision ($eDBM$) improved upon $ALO_3$ weak labelling itself.

\subsection{Evaluation on new labels introduced after 2006}
\label{ssec:eval_experiments}

In this last part of our experiments, we assessed the $DBM$ method on the data of the 70 new use cases introduced between 2007 and 2019. 
We trained models on the original weak supervision provided by $CO$, as well as on the enhanced weak supervision of $ALO_3$.
The results of these experiments are presented in Table~\ref{table:eval_results}\footnote{Detailed results with scores per descriptor are available in the supplementary file S4.}.

\begin{table}[!t]
\caption{Results on ground truth data for the 70 fine-grained labels introduced after 2006. 
The variance of the $maF1$ score across the 70 labels ($maF1var$) is also reported for each model.
A prefix ($e$) indicates training under \textit{enhanced supervision}.
The variance across the 70 labels ($var$) is also reported for the macro-averaged metrics.
The subscript in $LR$ models indicates the number of features selected.
The subscript in $DBM$ models indicates the training epoch considered, with the values \textit{p}, \textit{b}, and \textit{n} corresponding to \textit{prev\_ep}, \textit{best\_ep}, and \textit{next\_ep} respectively.
}
\begin{center}
\small 
\begin{tabular}{lllllll}
LF/model   			& $maP$\footnotesize{ $\pm var$} & $maR$\footnotesize{ $\pm var$}    & $maF1$\footnotesize{ $\pm var$}	& $miP$   & $miR$   & $miF1$\\
\hline

$CO$		& 0.766\footnotesize{ $\pm0.04$}& 0.590\footnotesize{ $\pm0.08$}& 0.626\footnotesize{ $\pm0.06$}  			& 0.848 &0.554	&0.670\\
$NL$		& 0.647\footnotesize{ $\pm0.13$}& 0.369\footnotesize{ $\pm0.13$}& 0.408\footnotesize{ $\pm0.12$}  			& 0.880 &0.217	&0.348\\
$SL$		& 0.567\footnotesize{ $\pm0.15$}& 0.253\footnotesize{ $\pm0.09$}& 0.287\footnotesize{ $\pm0.09$}  			& 0.809 &0.125	&0.217\\
$LR_{5}$	& 0.420\footnotesize{ $\pm0.11$}& 0.807\footnotesize{ $\pm0.04$}& 0.458\footnotesize{ $\pm0.09$}  			& 0.204	&0.783	&0.324 \\
$C2F$		& 0.206\footnotesize{ $\pm0.07$}& 0.946\footnotesize{ $\pm0.01$}& 0.276\footnotesize{ $\pm0.09$}  			& 0.074 &0.981	&0.138\\
$DBM_n$		& 0.718\footnotesize{ $\pm0.05$}& 0.647\footnotesize{ $\pm0.08$}& 0.641\footnotesize{ $\pm0.06$}  			& 0.826 &0.573	&0.677\\
$DBM^{C2F}_{n}$  	& 0.712\footnotesize{ $\pm0.05$}& 0.626\footnotesize{ $\pm0.00$}& 0.625\footnotesize{ $\pm0.06$} &0.831 &0.568 &0.675\\
$MV_3$  	& 0.782\footnotesize{ $\pm0.05$}& 0.481\footnotesize{ $\pm0.09$}& 0.542\footnotesize{ $\pm0.08$}  			& 0.873 &0.281	&0.462\\
$LM_3$  	& 0.756\footnotesize{ $\pm0.05$}& 0.652\footnotesize{ $\pm0.08$}& 0.661\footnotesize{ $\pm0.06$}  			& 0.842    &0.569	&0.679\\
$ALO_3$ 	& 0.764\footnotesize{ $\pm0.03$}& 0.668\footnotesize{ $\pm0.07$}& \textbf{0.677}\footnotesize{ $\pm0.05$}  	& 0.839 &0.583	&0.688\\
$eLR_5$ 	& 0.403\footnotesize{ $\pm0.10$}& 0.825\footnotesize{ $\pm0.03$}& 0.454\footnotesize{ $\pm0.09$} & 0.196	&0.797	&0.315\\
$eC2F$		& 0.203\footnotesize{ $\pm0.07$}& 0.959\footnotesize{ $\pm0.01$}& 0.272\footnotesize{ $\pm0.09$}  			& 0.076 &0.983	&0.141\\
$eDBM_n$ 	& 0.725\footnotesize{ $\pm0.04$}& 0.695\footnotesize{ $\pm0.06$}& 0.672\footnotesize{ $\pm0.04$}  			& 0.839 &0.621	&\textbf{0.714}\\
$eDBM^{C2F}_{n}$  	& 0.726\footnotesize{ $\pm0.05$}& 0.679\footnotesize{ $\pm0.07$}& 0.656\footnotesize{ $\pm0.05$} &0.803 &0.588 &0.679

\end{tabular}
\end{center}
\label{table:eval_results}
\end{table}

These results confirm that $CO$ is a strong heuristic for the task of fine-grained semantic indexing, outperforming again dictionary-based heuristics.
In addition, the weak labelling enhancement approach ($ALO_3$) achieved an overall $maF1$ about 5pp higher than $CO$ alone, which is a statistically significant difference\footnote{Wilcoxon signed-rank test, $H_1: F1\:CO < F1\:ALO_3$, p-value = 0.000001.}. 
The performance of $LM_3$ was not significantly lower than that of $ALO_3$, as the two approaches achieved practically the same $F1$ score for most of the 70 labels. However, $ALO_3$ achieved $F1$ performance equal to or better than that of $LM_3$ for all 70 labels.

Moving to the trained models, the weakly-supervised $DBM_n$ model under the supervision of $CO$ alone led to a limited overall performance gain of about 1.5pp in terms of $maF1$ over $CO$.
The performance of the $eDBM_n$ model, using the enhanced supervision of $ALO_3$, was higher, achieving about 4.6pp improvement in $maF1$, which is statistically significant\footnote{Wilcoxon signed-rank test, $H_1: F1\:CO < F1\:eDBM_n$, p-value = 0.0065.}. 
However, even $eDBM_n$ achieves only comparable performance to $ALO_3$ on its own. In fact, $ALO_3$ has a slightly higher $maF1$, without statistical significance\footnote{Wilcoxon signed-rank test, $H_1: F1\:ALO_3 \neq F1\:eDBM_n$, p-value = 0.088.}.
Nevertheless, $eDBM_n$ achieved the best $miF1$ score, with an overall performance gain of about 2.6pp over $ALO_3$ on the new unseen labels of the evaluation experiments.

\section{Discussion and conclusions}
\label{sec:Conclusion}
In this work, we investigate the weakly-supervised refinement of coarse-grained MeSH subject annotations into fine-grained ones, at the level of domain concepts. Additionally, we propose an automated approach to generating large-scale ground truth datasets for fine-grained semantic indexing, based on MEDLINE articles. 
Hence, the main contributions of the paper are: a) a new method (\textit{Retrospective Beyond MeSH}) for the automated development of ground-truth datasets for the large-scale evaluation of fine-grained semantic-indexing predictions; b) a voting approach for enhancing the original weak supervision provided by \textit{concept occurrence}, c) a new method (\textit{Deep Beyond MeSH}) for developing weakly-supervised deep learning models for refined indexing, and d) the validation and evaluation of the proposed enhanced weak supervision (b) and weakly-supervised models (c) on the new large-scale retrospective datasets (a). 

Using the retrospective dataset-development method, which exploits the extension of MeSH towards more fine-grained labels, we developed ground-truth test sets for 88 distinct fine-grained labels. These datasets consist of more than two million articles with weak supervision for the development and more than one million articles with ground-truth supervision for testing. 
To the best of our knowledge, this is the first large-scale set of datasets for fine-grained semantic indexing of biomedical literature.

Using these new datasets, we first confirmed the strength of \textit{concept occurrence} as a heuristic for fine-grained semantic indexing of biomedical literature. 
Additionally, we showed that this heuristic can be significantly enhanced when combined in an \textit{at-least-one} ensemble with the dictionary-based heuristics \textit{name lowercase} and \textit{synonyms lowercase}. The latter two heuristics examine the literal occurrence of the name and synonyms of the concept in the title and abstract of an article.
This conclusion is particularly inserting, considering that these dictionary-based heuristics rely on the same concept terms already exploited by \textit{concept occurrence} itself. 

We then moved on to train various machine learning models using the weak labels as supervision.
We have experimented with two existing methods, \textit{logistic regression} and \textit{Coarse2Fine}, the parameters of which we have optimised on validation data. Our experimentation concluded that the methods cannot improve upon the results of the weak labeling heuristics themselves. 


For this reason, we introduced a new method to train deep learning (DL) models, using the weak labels.
This method builds upon the state-of-the-art approach of fine-tuning pre-trained models and making design choices for the specific problem. 
In particular, early stopping seems to be useful, especially when considering the epoch next to the one with the lowest loss on $WS\_val_y$. 
In addition, in order to deal with label imbalance, we use the R-BCE-FL loss and under-sampling. 
Furthermore, we observed that majority voting among models developed with different random seeds can also improve the overall performance, limiting the effect of randomness.
Finally, DL models trained with the weak labels, using the proposed method ($DBM$) achieved competitive indexing performance on new labels.

Moreover, when the enhanced ensemble-based weak supervision ($ALO_3$) is used, the DL models improve significantly, reaching micro F1 levels above the very competitive weak labeling itself ($ALO_3$).
Nevertheless, no trained model managed to outperform the overall macro F1 performance of $ALO_3$ itself on the new unseen labels of the evaluation experiments. 
The strongest DL models ($DBM$) achieved only a comparable performance.

Based on these results, our future plans focus mainly on the improvement of the promising new method for DL model development.
In this direction, we plan to use additional label information, such as hierarchical relationships from MeSH, more fine-grained text representations based on contrastive learning approaches, and probabilistic labels in training instead of binary ones.    
In addition, we plan to investigate approaches for automated filtering of erroneous weak labels assigned by the heuristics. 

\section{Acknowledgment}
This research work was supported by the Hellenic Foundation for Research and Innovation (HFRI) under the HFRI Ph.D. Fellowship grant (Fellowship Number: 697).
The data resources considered in this work, including  MeSH, MEDLINE/PubMed, and SemMedDB, were accessed courtesy of the U.S. National Library of Medicine.
We are grateful to Georgios Katsimpras for the useful discussions on Deep Learning approaches.







\section{Introduction}

Biomedical literature is a rich source of biomedical knowledge that is constantly expanding. For example, MEDLINE/PubMed\footnote{\url{https://pubmed.ncbi.nlm.nih.gov/}}, which already gathers about 30 million articles, is continuously growing at a rate that exceeds one million articles annually, during the last five years\footnote{\url{https://www.nlm.nih.gov/bsd/medline_pubmed_production_stats.html}}. 
Precise and comprehensive identification of relevant articles for specific topics in such a rich collection is a real challenge for biomedical researchers.
In this direction, the National Library of Medicine (NLM) indexes citations in the MEDLINE database with topic descriptors from the Medical Subject Headings (MeSH) thesaurus\footnote{\url{https://meshb.nlm.nih.gov/}}. 
This semantic indexing process allows MEDLINE/PubMed to offer advanced semantic search strategies, that can retrieve citations relevant to a topic of interest, addressing issues such as synonymy and polysemy of biomedical concepts.

MeSH provides a comprehensive set of descriptors for more than thirty thousand topics of biomedical interest, such as diseases and chemicals.
The descriptors are hierarchically organized so that most are subcategories of at least one broader descriptor.
For example, the descriptor for ``Epilepsy, Reflex''\footnote{\url{https://meshb-prev.nlm.nih.gov/record/ui?ui=D020195}} is a subcategory of the broader descriptor ``Epilepsy''\footnote{\url{https://meshb-prev.nlm.nih.gov/record/ui?ui=D004827}} as shown in Figure~\ref{fig:MeSH_Example}.
Some of these descriptors cover several related but distinct subordinate concepts, which are usually narrower\footnote{In some cases subordinate concepts can be related or even broader to the main topic. } than the main topic of the descriptor.
For example, the ``Epilepsy, Reflex'' descriptor contains the synonymous term ``Reflex Epilepsy'' as \textit{preferred concept}, but includes also additional terms for narrower concepts such as ``Reflex Epilepsy, Audiogenic'' and ``Tactile Reflex Epilepsy'', that are not MeSH descriptors themselves.

\begin{figure}[!t]
\centerline{\includegraphics[width=0.85\textwidth]{figures/MeSH_example.png}}
\caption{ The internal structure of the MeSH descriptor ``Epilepsy, Reflex'' and its position in the hierarchy under the broader descriptor ``Epilepsy''. }\label{fig:MeSH_Example}
\end{figure}

Such narrower concepts of the coarse-grained descriptor are not distinguishable by the semantic indexing process, which is done at the level of descriptors. 
Therefore, semantic search cannot be used at this level of detail. 
The goal of this work is to enable such fine-grained semantic search, through  the automated refinement of semantic indexing from the descriptor level to the level of each narrower concept. 
For example, we would like to distinguish which of the ``Epilepsy, Reflex'' articles are for ``Audiogenic Epilepsy'' and which for ``Tactile Reflex Epilepsy'' or other subcategories of ``Epilepsy, Reflex''.

Fine-grained indexing at the level of concepts is beyond the current practice in MEDLINE/PubMed and, if enforced, it would almost double the number of distinct topics used for indexing.
Doing this additional work manually is not practical, given the growing volumes of biomedical literature.
Furthermore, state-of-the-art systems that provide automated topic suggestions 
focus on known MeSH topics and usually require several labelled training  examples for any new topic that is to be covered~\cite{Nentidis2022}. Therefore, obtaining reliable topic suggestions for new fine-grained topics at the level of concepts is an open research issue. 
The lack of labelled data for the development and evaluation of methods at this level of detail is a major challenge.

In this work, we propose \textit{Deep Beyond MeSH} ($DBM$), a new method for the refinement of coarse-grained MeSH annotations based on deep pre-trained language models. 
To overcome the lack of labelled data for supervised training, DBM relies on weak supervision. The latter is based on the recognition of biomedical concepts in the abstract of articles (\textit{concept occurrence}).  
In the past we have experimented with this idea~\cite{nentidis2020beyond}, using small ground truth datasets, developed for two specific MeSH descriptors. Those experiments suggested that \textit{concept occurrence} can provide a strong basis for fine-grained semantic indexing. 
However, the approach needed to be tested on larger datasets of ground truth data for several MeSH descriptors. 
In this direction, we devised a retrospective scenario, based on the evolution of MeSH indexing in MEDLINE/PubMed, and introduce here \textit{Retrospective Beyond MeSH} ($RetroBM$), a new method to derive large-scale datasets for individual concepts that got promoted over time to new fine-grained MeSH descriptors. 

In brief, the main questions driving the work presented in this paper are the following:
\begin{itemize}
    \item Can the extension of MeSH towards fine-grained labels that took place in the past be exploited for the retrospective evaluation of automated fine-grained indexing approaches on a large scale?
    \item Is \textit{concept occurrence} a good heuristic for fine-grained concept-level semantic indexing on a large scale? 
    Could the \textit{concept occurrence} be enhanced with other heuristics?
    \item Is it possible to train DL models for fine-grained semantic indexing with concepts for which no ground-truth training data are available, exploiting the weak supervision provided by the above heuristics? 
    Could these models exceed the predictive performance of the heuristics themselves?
\end{itemize}

The rest of this paper is organized as follows.
In section \ref{sec:Background}, we provide some background information on the semantic indexing of biomedical literature, relevant work on fine-grained classification without labelled data, and previous efforts towards more fine-grained biomedical semantic indexing.
In section \ref{sec:Methods}, we introduce our new methods: $RetroBM$ for the automated development of datasets for fine-grained semantic indexing of biomedical literature and $DBM$ for weakly-supervised training of DL models for the automated refinement of coarse-grained labels.
In section \ref{sec:Experiments}, we describe the experimental setup for the evaluation of the predictive performance of the $DBM$ models on $RetroBM$ data, and present the results of these experiments.
Finally, in section \ref{sec:Conclusion} we summarise the presented work and discuss its contribution towards addressing the motivating research questions presented above.  

\section{Background and related work}
\label{sec:Background}

\subsection{Semantic indexing of biomedical literature}

Semantic indexing of biomedical literature in MEDLINE/PubMed has been done  for several decades by human expert indexers of NLM. 
Since 2002 some automated topic suggestions are offered to human experts by the \textit{Medical Text Indexer} (\textit{MTI}) Tool\footnote{\url{https://lhncbc.nlm.nih.gov/ii/tools/MTI.html}}, in order to speed up the annotation process and keep up with the increasing volume of the literature. 
\textit{MTI} is structured into distinct components that provide MeSH topic recommendations. These components are based on complementary approaches, namely \textit{concept occurrence} in the article text, the topics of similar articles, and machine learning models trained on previously indexed articles.
The recommendations are finally ranked and filtered by other components, including one based on the Learning To Rank (LTR) approach\cite{Mork2017}.

In parallel, a variety of alternative approaches has been proposed for the automated semantic indexing of biomedical literature, and  since 2012 a dedicated shared task has been organized for this purpose, in the context of the BioASQ challenge~\cite{tsatsaronis2015overview}. 
The state-of-the-art approaches in this task, are based on DL architectures. 
For example, \textit{BERTMeSH} \cite{you2021bertmesh} extends the label-tree-based DL method of \textit{AttentionXML}~\cite{You2018} with representations based on the pre-trained transformer model \textit{BioBERT}~\cite{lee2019biobert}.
In the same direction, new methods proposed by the NLM team \cite{Rae2021clef} rely on recommendations from a Convolutional Neural Network (CNN) which are ranked by another pre-trained transformer model (\textit{PubMedBERT})~\cite{Gu2022}.   

The availability of millions of manually indexed articles as training examples allowed the gradual development of advanced machine-learning models that achieve satisfactory results~\cite{Nentidis2022}.
As a result, since 2018, NLM introduced fully automated indexing for a subset of the MEDLINE articles, using an improved version of \textit{MTI}. Furthermore, they announced the scale-up of this policy to all MEDLINE articles in 2022\footnote{\url{https://www.nlm.nih.gov/pubs/techbull/nd21/nd21_medline_2022.html}}. 
Despite the significant progress in the field of semantic indexing of biomedical literature, almost all state-of-the-art methods focus on coarse MeSH descriptors. Fine-grained indexing at the concept level remains beyond reach, partly due to the lack of adequate data for training and evaluation of supervised machine learning approaches.    

\subsection{Fine-grained classification without labelled data}

Classification into fine-grained sub-classes has been long studied in the field of image classification and a series of workshops is also organized on Fine-Grained Visual Categorization (FGVC)\footnote{https://sites.google.com/view/fgvc10} since 2011.
These workshops cover a variety of relevant tasks, such as differentiation between similar plant or animal species.
Annotations at the fine-grained level are often difficult to obtain and as a result researchers have investigated learning with limited fine-grained supervision and exploiting the hierarchical relationship between concepts. 
For instance, Taherkhani \textit{et al.}~\cite{Taherkhani2019}, adopt a few-shot approach, where some of the instances do have fine-grained annotations, and rely on multi-task learning so that training for the coarse-grained task improves the fine-grained one, through shared model parameters of a CNN. 
Hsieh \textit{et al.}~\cite{Hsieh2019, Hsieh2021} adopt a few-shot approach as well, extending fine-grained image annotation from multi-class to multi-label classification, where a single image may have multiple fine-grained annotations, containing, for example, both a cat and an apple.
Based on the hierarchical relation between labels of different granularity, their method assigns fine-grained pseudo-labels to unlabeled images, which are used for model training. 
Another few-shot method that relies on pseudo-labeling for FGVC, was proposed by Yang \textit{et al.}~\cite{Yang2020}. In particular, they perform clustering of coarse-grained-class instances into pseudo-fine-grained classes and learn fine-grained features, which are then used for the actual task of FGVC.   
On the other hand, Sohoni \textit{et al.}~\cite{Sohoni2020} use clustering into fine-grained concepts to improve coarse-grained classification per se.

Similar problems have also been studied in the context of fine-grained named entity recognition. For example, a text span referring to a person can be classified into more fine-grained entity classes, such as actor, athlete, or politician. 
Training instances for such fine-grained sub-classes are hard to obtain and  proposed methods rely mostly on distant supervision \cite{Ling2012, Gillick2014, Yogatama2015, Zhang2020} or zero-shot learning \cite{Ma2016, Zhou2017, Obeidat2019, Zhang2021}. 
Hierarchical label relations have been used either to define the classification problem as a problem of differentiating between sibling sub-classes of a coarse-grained parent class \cite{Zhou2017} or by more elaborate approaches that allow information sharing among related labels \cite{Yogatama2015, Zhang2021}. 

In the field of document classification, which is the one closest to our work, some attention has also been drawn to fine-grained classification without ground-truth data, mainly based on weakly-supervised or zero-shot methods.
Most of these approaches are multi-task, in the sense that they aim to improve coarse-grained and fine-grained classification together. 
For instance, \textit{Snorkel MeTaL} \cite{Ratner2018, Ratner2019} exploits fine-grained and coarse-grained weak labels together combining them on the basis of a hierarchical label model. 
That method combines weak labels from multiple sources, which is an interesting research topic, as it requires modeling their coverage and disagreements~\cite{Ratner2016, Bach2017, Ratner2020}. 
However, in the fine-grained biomedical semantic indexing task studied here, the sources of supervision are limited. 

However, 
there is also a lot of work on the use of a single source of weak supervision \cite{Song2015, Li2016, Zha2019, Chu2020}, in many cases taking into account the label hierarchy as well.
Song and Roth \cite{Song2014}, for example, 
deal with hierarchical multi-label text classification,
using only a label hierarchy and some unlabeled documents.
Their approach encodes both labels and documents in a common semantic space.
Measuring label-document semantic similarity in that common space, they develop a dataless classifier and improve it through bootstrapping on unlabeled documents.
Label-document similarity is also used in the \textit{taxoClass} framework~\cite{Shen2021}, which employs a pre-trained textual entailment model, based on \textit{RoBERTa} \cite{Liu2019}. 
Using the label-document semantic similarity and the hierarchy of labels, a set of the most similar ``core'' labels is identified for each document and a taxonomy-enhanced text classifier is trained on these weak labels. 
Finally, multi-label self-training is also employed to improve the classifier.

In contrast to the broader problem of hierarchical text classification, coarse-grained class labels are already available in the biomedical indexing problem that we study.
Recently, Mekala \textit{et al.} \cite{Mekala2021} focused on this special case of fine-grained indexing, where coarse-grained labels are available, proposing the \textit{Coarse2Fine} ($C2F$) method.
$C2F$ relies on the literal occurrence of fine-grained labels to create an initial weakly labeled dataset. These weak fine-grained labels are used together with the ground-truth coarse-grained ones to train a label-conditioned GPT2 \cite{AlecRadford2020} model that generates relevant documents for any given label.
In that method, a hierarchy-aware regularization term is added to the loss function, in order to exploit the hierarchical relations between classes.
This ensures that documents with a literal occurrence of a fine-grained label will have a higher probability compared to documents that include the coarse-grained parent.
The trained GPT2 model is used to produce new weakly labeled documents for each fine-grained label, which are, in turn, used to train \textit{BERT} classifiers \cite{Devlin2019}. 
Finally, the classifiers are improved through the use of unlabeled documents.

Simpler approaches use the available ground-truth coarse-grained labels, by reducing the hierarchical classification problem into several local sub-problems of classifying fine-grained sibling subcategories.
For instance, \textit{wiki2cat} \cite{Jin2021} exploits the graph of Wikipedia categories, as an external knowledge resource, to weakly label Wikipedia articles for 332 fine-grained categories of the Interactive Advertising Bureau (IAB) taxonomy\footnote{\url{https://www.iab.com/guidelines/taxonomy/}}.
In particular, the IAB categories are first mapped to Wikipedia categories, based on the literal match of their names, and then the Wikipedia category graph is used to obtain weakly labeled articles. 
Using these data, weakly-supervised models are trained to classify into fine-grained IAB sibling subcategories under the same parent IAB category.
Our weakly labelling approach relies on \textit{concept occurrence} which is kin to the literal occurrence of label names, using biomedical vocabularies, as external knowledge resources. 


The more general task of indexing documents with new descriptors is also challenging, due to the lack of labelled data for these descriptors.  
For this task, Mylonas \textit{et al.} introduced \textit{ZSLbioSentMax} \cite{Mylonas2020}, a zero-shot semantic indexing approach that relies on the maximum semantic similarity of new descriptor names with sentences in the text of biomedical articles. In particular, they measure the cosine similarity of corresponding dense representations (embeddings), produced by \textit{BioBERT} \cite{lee2019biobert}. 
A weakly-supervised approach, called \textit{WeakMeSH} \cite{Mylonas2021} has also been proposed for the same task. 
In that work, weak labels were obtained by considering both the hierarchical relationship of new labels with existing descriptors and their semantic similarity with sentences in the text of biomedical articles. 
Using these data, weakly-supervised classifiers were trained, achieving promising results in experiments with 62 new MeSH descriptors, most of which cover novel MeSH concepts.
In this paper, we consider the hierarchical relationship between new and existing descriptors as well, but we focus on subordinate MeSH concepts that get promoted into fine-grained subcategories. 
Our method relies on deep pre-trained models as well, however, as we deal with semantically similar fine-grained labels that cover a single concept, we investigate \textit{concept occurrence} as the primary source of weak supervision.    

\subsection{Towards fine-grained semantic indexing of biomedical literature}

Early work towards fine-grained indexing of biomedical literature focused on enriching the main MeSH descriptor-based labels with MeSH qualifiers\footnote{\url{https://www.nlm.nih.gov/mesh/qualifiers\_scopenotes.html}} and MeSH Supplementary Concept Records (SCRs)\footnote{\url{https://www.nlm.nih.gov/mesh/intro\_record\_types.html.}} \cite{Aronson2004}.
MeSH qualifiers are auxiliary categories, such as ``genetics'' or ``diagnosis'', that may accompany a descriptor to specify aspects of it in an article (e.g. ``Lung Cancer/genetics''). 
SCRs, on the other hand, are supplementary entities for specific substances, rare diseases, and organisms that are not explicitly covered by any MeSH descriptor. 
In this direction, Neveol \textit{et al.} \cite{Neveol2007_amia, Neveol2007_psb}, 
proposed various approaches for indexing with ``descriptor/qualifier'' pairs, including ``dictionary-based'' ones, that rely on the literal occurrence of label terms, as well as ``rule-based'' ones, that rely on  manually-developed patterns.  
Recently, Rae \textit{et al.} \cite{Rae2020} proposed a CNN-based method  that achieved improved results on the same task. 

Although qualifiers and SCRs provide additional information to MeSH descriptors, they do not address the coarseness of the descriptors.
To the best of our knowledge, the importance of fine-grained semantic indexing of biomedical literature at the level of MeSH concepts was first noted in \cite{Darmoni2012}. 
The authors of that work used simply 
the literal occurrence of corresponding MeSH concept terms in the title or abstract of the articles.
Despite the strong assumption of literal occurrence, that work concluded that indexing at this level of detail is useful for the precise retrieval of relevant documents.
In this direction, the indexing policies for librarians of the \textit{CISMeF} catalogue\footnote{\url{http://www.chu-rouen.fr/cismef/}} for French medical resources were updated to support indexing at the level of MeSH concepts. 

More recently, we proposed the \textit{Beyond MeSH} method \cite{Nentidis2019_CBMS, nentidis2020beyond} for the automated refinement of MeSH topic annotations at the level of MeSH concepts. 
In that work, we first introduced the concept-occurrence heuristic for the development of weakly labeled training datasets. This heuristic extends simpler dictionary-based ones
using the \textit{MetaMap} tool \cite{Aronson2010}. 
We experimented with different feature sets and common machine-learning algorithms
on small datasets for two specific use cases, comparing the results of trained classifiers against the use of \textit{concept occurrence} itself. Those initial experiments showed that the concept-occurrence heuristic is a strong baseline and only some trained classifiers managed to improve upon it.
In particular, logistic regression models with L2 regularization provided higher recall, but the selection of hyper-parameters, such as the level of L2 regularization, without ground-truth validation data poses a major challenge.  
Building upon this earlier work, in this paper we perform an extensive evaluation of the promising concept-occurrence heuristic in a large-scale scenario for several MeSH descriptors, while also assessing the performance of DL approaches for the first time.

In order to form the large-scale experimental scenario, we look retrospectively into the evolution of MeSH descriptors, focusing on MeSH concepts, which over time become descriptors themselves. 

\section{Material and methods}
\label{sec:Methods}

\begin{figure}[!h]
\begin{center}
\includegraphics[width=1\textwidth]{figures/Schema_method_plus3.png}
\end{center}
\caption{The \textit{Retrospective Beyond MeSH} ($RetroBM$) method for the development of large-scale ground truth datasets for fine-grained semantic indexing and the \textit{Deep Beyond MeSH} ($DBM$) method for weakly-supervised semantic label refinement. $DBM$ is structured into three stages: the development of weakly-supervised datasets for some concepts of interest, the enhancement of the initial weak supervision, and the development of weakly-supervised deep learning models. The datasets of $RetroBM$ are finally used for the large-scale evaluation of $DBM$ models. }
\label{fig:DeepBeyondMeSH_block}      
\end{figure}
In this section, we introduce a) the \textit{Retrospective Beyond MeSH} ($RetroBM$) method for the development of large-scale ground truth datasets for fine-grained semantic indexing and b) the \textit{Deep Beyond MeSH} ($DBM$) method for refining existing coarse-grained biomedical semantic labels into fine-grained ones at the level of concepts.
In the context of $RetroBM$, a scenario is introduced that allows the automated development of ground-truth datasets for specific narrower concepts of coarse-grained descriptors. 
These narrower concepts are the ones that have become descriptors themselves over time. Hence, by examining retrospectively the evolution of MeSH, we can identify a number of such concepts to serve as use cases and generate automatically ground-truth data.
$DBM$, on the other hand, is structured into three distinct stages, namely the development of weakly-labeled datasets, where $RetroBM$ is used for the ground-truth ones, the enhancement of weak supervision, and the development of fine-grained indexing models. 
Figure~\ref{fig:DeepBeyondMeSH_block} illustrates the two processes, showing also how the results of $RetroBM$ are used by $DBM$.
In particular, the weakly-labeled datasets are used for the development of weakly-supervised DL models that classify articles into the narrow concepts. Weak supervision may or may not be enhanced with the use of dictionary-based approaches at the second stage of the process. 
The implementation of these methods in Python and Java is openly available in GitHub\footnote{\url{https://github.com/tasosnent/DBM}}. In the following subsections, we will examine in more detail each step of the process.

\subsection{Retrospective dataset development}
\label{ssec:retro_data}

The main challenge in dataset development for fine-grained biomedical semantic indexing is the lack of ground-truth annotations at this level of detail for the development and testing of predictive models.
Although methods that rely on weak supervision can sidestep the need for ground-truth labels as regards model development, some ground-truth data are still needed for the proper evaluation of model predictions.
Requiring domain experts to manually annotate the test sets does not scale up when moving to larger numbers of articles and use cases. 
Therefore, in this work we introduce \textit{Retrospective Beyond MeSH} ($RetroBM$), a retrospective approach based on the evolution of MeSH indexing already done in MEDLINE/PubMed, to develop fine-grained semantic indexing datasets. 

\begin{figure}[!t]
\centerline{\includegraphics[width=0.82\textwidth]{figures/MeSH_subdivision_Example.png}}
\caption{ The promotion of ``Niemann-Pick Disease, Type A'', a narrower subordinate concept of the coarse-grained MeSH descriptor ``Niemann-Pick Diseases'', to a new dedicated descriptor in 2007.  }\label{fig:MeSH_subdivision_Example}
\end{figure}

\subsubsection{Formulation of a retrospective scenario}
\label{sssec:retro_data_scenario}
Although indexing of articles with subordinate MeSH concepts is beyond the curation process of MEDLINE/PubMed, there are cases where a subordinate concept \textit{c} of a coarse-grained descriptor \textit{d} can be promoted to a new MeSH descriptor \textit{d\textsubscript{c}}, during the annual update of the MeSH thesaurus. 
For example, ``Niemann-Pick Disease, Type A''\footnote{\url{https://meshb.nlm.nih.gov/record/ui?ui=D052536}} used to be a narrower subordinate concept of the ``Niemann-Pick Diseases'' descriptor until 2007, when it was promoted to a MeSH descriptor itself (Fig. \ref{fig:MeSH_subdivision_Example}). 
In such cases, if \textit{c} is the only MeSH concept of the new MeSH descriptor \textit{d\textsubscript{c}},
indexing with \textit{d\textsubscript{c}} is equivalent to indexing with \textit{c}. 
Therefore, articles semantically indexed with a version of MeSH that includes \textit{d\textsubscript{c}}, in practice, have ground-truth annotations for fine-grained indexing with the MeSH concept \textit{c}.  
In the example of Figure~\ref{fig:MeSH_subdivision_Example}, articles for any type of Niemann-Pick disease used to be annotated with the ``Niemann-Pick Diseases'' descriptor until 2006. However, since 2007, articles for ``Niemann-Pick Disease, Type A'' are annotated with a distinct and dedicated descriptor and can be used for the evaluation of methods for fine-grained indexing with this concept.    

\begin{figure}[!t]
\includegraphics[width=1\textwidth]{figures/retro.png}
\caption{Retrospective datasets for the task of refining coarse-grained annotations for a descriptor \textit{d} into a narrower subordinate concept \textit{c}. The development of the datasets is based on the promotion of \textit{c} to a new fine-grained descriptor \textit{d\textsubscript{c}} in year \textit{y}, during the annual update of the MeSH thesaurus.}
\label{fig:retro_data}      
\end{figure}

For this reason, we looked retrospectively for such promoted concept cases to generate automatically ground-truth data for fine-grained indexing methods. 
In particular, we focus on the year \textit{y}, when the promotion of \textit{c} to \textit{d\textsubscript{c}} took place, as shown in Figure~\ref{fig:retro_data}.
All the articles annotated with \textit{d}\footnote{As articles annotated with \textit{d} we consider both articles directly annotated with the descriptor \textit{d} and articles annotated with any descriptor narrower to \textit{d} in the MeSH hierarchy.} prior to year \textit{y} are gathered for the development of a weakly-supervised development dataset for indexing with \textit{c}.
The fact that \textit{c} used to be a subordinate concept of \textit{d} in this period, ensures that any article relevant to \textit{c} will be included in this set of articles annotated with \textit{d}. 
To generate the weak labels we use the heuristic introduced in \textit{Beyond MeSH}~\cite{Nentidis2019_CBMS,nentidis2020beyond}, which relies on the occurrence of concept \textit{c} in the title or abstract of the article, as identified by the \textit{MetaMap} tool \cite{Aronson2010}. 

On the other hand, for any article annotated with \textit{d} from year \textit{y} onward, manual annotations for the narrower descriptor \textit{d\textsubscript{c}} are also available, indicating relevance to concept \textit{c}. 
These more recent articles for \textit{d} can be used for the evaluation of the refined semantic indexers.  
The fact that \textit{d\textsubscript{c}} is a narrower descriptor of \textit{d} in the MeSH hierarchy, ensures that any article relevant to \textit{c} will be included in the set of articles annotated with \textit{d}.
Therefore, articles not indexed under \textit{d} are beyond the scope of the fine-grained indexers, as the aim is to refine coarse-grained annotations already provided by the current indexing practice.

Although the proposed approach relies on a retrospective use of annotated documents, this schema is fully aligned with the prospective use of automated refinement approaches for coarse-grained MeSH annotations.
We can develop weakly-supervised datasets for model development with articles annotated for any coarse-grained descriptor \textit{d\textsubscript{new}} having a narrower subordinate concept \textit{c\textsubscript{new}}, that has not actually been promoted to a descriptor in MeSH yet. 
The trained models can be used to provide automated fine-grained indexing with \textit{c\textsubscript{new}}, without it being a descriptor. However, the accuracy of this annotation process will need to be verified manually.

\subsubsection{Identification of adequate use cases}
\label{ssec:use_case_selection}
Concept promotion events during the evolution of MeSH need to be identified, in order to scale up dataset development, as described in subsection~\ref{sssec:retro_data_scenario}.
However, not all new MeSH descriptors come from the promotion of narrower subordinate concepts as in the example of Figure~\ref{fig:MeSH_subdivision_Example}.
To identify which of the new descriptors satisfy our criteria, we adopt a framework that we introduced recently~\cite{Nentidis2021}
to study the evolution of MeSH descriptors, based on their semantic provenance. 
That framework introduced a typology of ways in which a new descriptor can be introduced.
Out of these types, the one named \textit{subdivision (1.2)}\footnote{We refer to \textit{subdivision} cases with provenance code 1.2 as introduced in Nentidis \textit{et al.} \cite{Nentidis2021}. This corresponds to new descriptors from a subordinate-concept promotion (provenance category 1) that end up as subcategories of their previous host (provenance type 2).} represents subordinate concepts \textit{c} promoted to subcategories \textit{d\textsubscript{c}} of the coarse-gained descriptor \textit{d} that used to host them. 
This type of new descriptor satisfies the concept-promotion scenario of subsection~\ref{sssec:retro_data_scenario} and was chosen to select suitable evaluation use cases.


However, there are additional criteria that need to be satisfied, beyond provenance. One such is that the new descriptor \textit{d\textsubscript{c}} is dedicated to the corresponding promoted concept \textit{c}, and does not cover other subordinate concepts as well. This is to ensure that all articles annotated with \textit{d\textsubscript{c}} are actually relevant to \textit{c}.
In addition, such descriptors are expected to be leaves in the MeSH hierarchy. New descriptors that have other narrower descriptors representing subcategories, in effect semantically cover the concepts of their narrower descriptors as well. Such descriptors are not fine-grained, even if they technically include just a single concept, and they are not considered adequate evaluation use cases.   

The last criterion used to select use cases is the volume of available data for training and validation. For this reason, we introduce a threshold \textit{dev\textsubscript{min}} for the number of articles annotated with \textit{d} that can be used for model development and another threshold \textit{test\_positive\textsubscript{min}} for the number of articles annotated with \textit{d\textsubscript{c}}, which can be used to evaluate the models. 
Some coarse-grained descriptors \textit{d} may be also too general, corresponding to too many articles and posing technical challenges in the dataset-development process. For this purpose, an upper limit \textit{dev\textsubscript{max}} to the number of articles annotated with \textit{d} can also be used in order to work with a realistic and manageable numbers of articles.
Finally, as the construction of the development dataset relies on the concept-occurrence heuristic, there is also a need for a sufficient number of articles with weak labels for \textit{c}.
Therefore, an additional threshold \textit{dev\_positive\textsubscript{min}} is adopted for the number of articles annotated with \textit{d}, where also \textit{c} occurs.  

\begin{table}[!t]
\caption{Criteria for retrospective selection of evaluation use cases.}
\begin{center}
\begin{tabular}{L{.18\linewidth}L{0.72\linewidth}}
Criterion 1. & Promotion event of provenance type \textit{subdivision (1.2)}  \\
Criterion 2. & Fine-grained new descriptor \textit{d\textsubscript{c}}  \\
 & 2.1. \textit{d\textsubscript{c}} corresponding to a single concept \textit{c} \\
 & 2.2. \textit{d\textsubscript{c}} leaf in the MeSH hierarchy \\
Criterion 3. &  Adequate number of articles for dataset development \\
 & 3.1. At least \textit{test\_positive\textsubscript{min}} articles annotated with \textit{d\textsubscript{c}} for the test set\\
 & 3.2. At least \textit{dev\textsubscript{min}} articles annotated with \textit{d} for the training dataset\\
 & 3.3. At most \textit{dev\textsubscript{max}} articles annotated with \textit{d} \\
 & 3.4. At least \textit{dev\_positive\textsubscript{min}} articles in the development dataset with \textit{c} occurrence
\end{tabular}
\label{table:criteria}
\end{center}
\end{table}

The above criteria, summarized in Table~\ref{table:criteria}, were incorporated in the automated method ($RetroBM$) for identifying adequate evaluation use cases, given a specific range of years. 
First, the method selects new descriptors belonging to the \textit{subdivision (1.2)} category, based on their conceptual provenance\footnote{https://github.com/tasosnent/MeSH\_Extension} (criterion 1). 
Then, the internal structure and hierarchical position of each selected new descriptor \textit{d\textsubscript{c}} are checked, in order to keep only the fine-grained ones (criterion 2).
Subsequently, the number of relevant articles for descriptors \textit{d} and \textit{d\textsubscript{c}} of each remaining use case is retrieved from a local Lucene\footnote{https://lucene.apache.org/} index of the Annual Baseline Repository\footnote{\url{https://www.nlm.nih.gov/databases/download/pubmed\_medline.html}} of MEDLINE/PubMed, and is checked against the data availability restrictions (3.1, 3.2, 3.3) of criterion 3.
Finally, in order to check the concept-occurrence criterion (3.4)
the SemMedDB\footnote{\url{https://lhncbc.nlm.nih.gov/ii/tools/SemRep\_SemMedDB\_SKR/dbinfo.html}} database is used, providing occurrences of biomedical concepts in MEDLINE/PubMed abstracts, as extracted automatically with \textit{MetaMap} \cite{Aronson2010}. 

\subsubsection{Multi-label dataset generation}
\label{ssec:dataset_generation}

Several use cases that satisfy the criteria of Table~\ref{table:criteria}, may correspond to a single year \textit{y} and several subordinate concepts may correspond to the same coarse-grained descriptor \textit{d}.
The development and testing datasets for all sibling subordinate concepts that were promoted during the same year consist of the same articles with different labels for each concept.  
Since a single article can be relevant to several distinct coarse-grained and fine-grained descriptors from independent use cases, the classification task is inherently multi-label. 

For this reason, we produce only two files for all use cases in a given year \textit{y} ($D_Cy$), one for the development of weakly-supervised models ($WS\_dev_y$) and one for testing (\textit{test\textsubscript{y}}). This is done by gathering all the corresponding articles together and assigning multi-label annotations accordingly.
However, for refinement into each concept \textit{c} only the articles that are relevant to the corresponding coarse-grained descriptor \textit{d} must be considered. 
We refer to these articles as the \textit{valid} documents for annotation with the concept \textit{c}.
For the use case of ``Niemann-Pick Disease, Type A'' for instance (Fig.~\ref{fig:MeSH_subdivision_Example}), out of all the articles included in the test file for 2007 (\textit{test\textsubscript{2007}}), only the articles annotated as relevant to ``Niemann-Pick Disease'' will be considered as \textit{valid} for ``Niemann-Pick Disease, Type A'' refinement. 
The open-source implementation of this dataset development process in Java is available in GitHub\footnote{\url{https://github.com/ThomasChatzopoulos/MeSH\_retrospective\_dataset}}.

\subsection{Weak supervision enhancement}
\label{ssec:WS_study}
Previous work suggests that \textit{concept occurrence} is a strong heuristic for fine-grained semantic indexing, but simpler dictionary-based approaches can also be quite strong in some cases~\cite{nentidis2020beyond}.
Ιn this work, we consider a range of dictionary-based variants in order to investigate whether they could enhance the quality of the weak labels.
In particular, adopting the terminology introduced in data programming~\cite{Ratner2016}, we refer to these dictionary-based approaches and the concept-occurrence heuristic as labeling functions.
Even though the dictionary-based labeling functions essentially rely on the same initial information that is used also by \textit{concept occurrence}, they may still result in different labels for the same article. 

\subsubsection{Alternative sources for weak supervision}
\label{sssec:LFs}
Eight dictionary-based labeling functions are introduced as potential alternative sources of weak supervision for the fine-grained semantic indexing task.  
Each labeling function (LF) assigns a concept label to an article if any dictionary element associated with the concept literally occurs in the title or abstract of the article. 
In the first LF (\textit{name exact}) only the name of a concept is matched (e.g.``Niemann-Pick Disease, Type A'') and in the second (\textit{synonyms exact}) the synonyms of it available in the MeSH thesaurus are used (e.g.``Classical Niemann-Pick Disease'', ``Sphingomyelin Lipidosis'' etc).

Then, six variants of these two labeling functions are introduced, 
in order to improve recall (see Table~\ref{table:Label_functions}). 
In particular, the dictionary elements are first lower-cased (\textit{name lowercase}, \textit{synonyms lowercase}), then the punctuation marks are removed (\textit{name no punct}, \textit{synonyms no punct}), and finally, phrases are split into single-token elements (\textit{name tokens}, \textit{synonyms tokens}).
For applying the latter six labeling functions, the title and the abstract of the article are also lower-cased, but no further modifications are applied to them. 

\begin{table}[!t]
\caption{The dictionary elements considered by each dictionary-based labeling function for assigning labels regarding the new fine-grained descriptor ``Niemann-Pick Disease, Type A''.}
\begin{center}
\small
\begin{tabular}{L{0.25\linewidth}L{0.095\linewidth}L{0.55\linewidth}}
Labeling function   & Abbr. & Dictionary elements                        \\
\hline
\textit{name exact}        & $NE$ &``Niemann-Pick Disease, Type A''                        \\
\textit{synonyms exact}    & $SE$ &``Classical Niemann-Pick Disease'', ``sphingomyelin lipidosis'', ...\\
\textit{name lowercase}    & $NL$ &``niemann-pick disease, type a''                                  \\
\textit{synonyms lowercase} & $SL$ &``classical niemann-pick disease'', ``sphingomyelin lipidosis'', ...\\
\textit{name no punct}     & $NNP$ &``niemann pick disease type a''                                  \\
\textit{synonyms no punct} & $SNP$ &``classical niemann pick disease'', ``sphingomyelin lipidosis'' ...\\
\textit{name tokens}       & $NT$ &``niemann'', ``pick'', ``disease'',  ``type'', ``a'' \\
\textit{synonyms tokens}    & $ST$ &``classical'', ``niemann'', ``pick'', ``disease'', ``sphingomyelin'', ``lipidosis'', ...
\end{tabular}
\label{table:Label_functions}
\end{center}
\end{table}

\subsubsection{Combination of weak supervision sources}
\label{sssec:ensembles}

In order to combine the predictions of different labeling functions into a single prediction we examine three alternative approaches. 
Namely, two voting classifiers based on \textit{majority voting} ($MV$) and \textit{at-least-one} voting ($ALO$)~\cite{kambhatla2006minority}, as well as a state-of-the-art probabilistic \textit{label modeling} ($LM$) approach for combining noisy heuristics, introduced in the context of Snorkel~\cite{Ratner2017}. 

The $MV$ schema assigns a label to an article if most of the labeling functions (voters) assign this label to this article. This way, we expect the precision of the ensemble to be favored, as labels predicted by only a few labeling functions are disregarded as untrustworthy.
In the $ALO$ schema, on the other hand, a label is assigned to an article if any of the labeling functions assign this label to this article. This is a special case of the \textit{at-least-N} schema, proposed by Kambhatla~\cite{kambhatla2006minority}, where $N=1$.
In this approach, even if a single labeling function predicts a label, this will be included in the prediction of the ensemble, favoring its recall.

Finally, the Snorkel $LM$ approach considers the true label for an article as a latent variable in a probabilistic generative model, which is estimated based on a weighted combination of the labeling functions.
In particular, statistical dependencies between the
labeling functions are considered during model development, to avoid the ``double counting'' problem for correlated labeling functions~\cite{Ratner2016, Ratner2017}.
Eventually, a label is assigned to an article if the estimated probability of the label is greater than 0.5.     


As all the labeling functions are variations of the same dictionary-based approaches, we expect a high degree of correlation, which may harm the accuracy of the ensembles.
For this reason, as part of the weak-supervision-enhancement stage, the optimal subset of labeling functions is sought for. 
In this direction, all 502 combinations of two or more labeling functions out of the nine available are considered in the validation experiments\footnote{Αt least three labeling functions are required for \textit{Snorkel} $LM$, therefore combinations of two labeling functions are only considered for $MV$ and $ALO$.}. 


\subsection{Model development}
\label{ssec:model_dev}
The final part of the \textit{Deep Beyond MeSH} method is the development of weakly-supervised Deep Learning (DL) models for the task of refining existing coarse-grained MeSH annotations to the level of concepts. 
In this direction, the method builds upon a state-of-the-art approach in text classification, that is fine-tuning a pre-trained BERT model, with one additional task-specific output layer~\cite{Devlin2019}. 
This approach has achieved very good results in a range of biomedical text classification tasks, transferring knowledge acquired during the pre-training phase to the task of interest~\cite{Peng2019}. 
In addition, 
experiments with pre-trained \textit{BERT} models for named entity recognition suggest that such methods can have an inherent resilience to label noise~\cite{Tanzer2021}, in accordance with previous experiments on image data~\cite{Hendrycks2019}.
Therefore, in this work, we investigate the adequacy of fine-tuning pre-trained \textit{BERT} models for label refinement, where fine-tunning is based on weak supervision.

\subsubsection{One multi-label model per year}
We have opted to train a single multi-label model for all the fine-grained labels of a single year \textit{y}, as compared to developing a distinct model for each label. 
This way, knowledge from different labels is integrated into shared deep representations for all the use cases of a single year.  
Preliminary experiments with single-label models in a One-Versus-Rest approach confirmed that multi-label ones are at least as good. 
Actually, a common model for multiple years would also be possible but would pose experimental complications as articles included in the test set of one use case may be included in the development set of another.
For this reason, we opt for a simpler scenario, where a distinct multi-label model (\textit{model\_val\textsubscript{y}}) is developed for each year \textit{y}, as depicted in Figure~\ref{fig:BERT_ML}.   
In this architecture, the title and abstract of each article are concatenated, tokenized, and provided as input to the \textit{BERT} model. The embedding of the classification token $CLS$ is then provided as input in a fully-connected output layer that produces a vector of size equal to the number of labels as in~\cite{DBLPSilvaG21}. 

\begin{figure}[!h] \includegraphics[width=1\textwidth]{figures/DBMarchitecture.png}
\caption{The architecture of a multi-label \textit{Deep Beyond MeSH} model for the fine-grained labels introduced in a single year.
}
\label{fig:BERT_ML}      
\end{figure}

\subsubsection{weakly-supervised validation of the number of training epochs}

Preliminary experiments indicated that the number of training epochs can significantly affect the performance of the model. Too few epochs lead the models to under-fit, while training for too many epochs can lead to over-fitting on the weak supervision. Hence, we needed to select a number of epochs that provide good models. For this purpose, 
we split \textit{WS\_dev\textsubscript{y}} into a training (\textit{WS\_train\textsubscript{y}}) and a validation (\textit{WS\_val\textsubscript{y}}) part (90-10\%). 
Then, we trained the model on the \textit{WS\_train\textsubscript{y}} part 
for a sufficient number of epochs 
and checked the loss after each epoch in the unseen \textit{WS\_val\textsubscript{y}} part. 
This way we could identify the model performing best (\textit{best\_ep}) on \textit{WS\_val\textsubscript{y}}, that is the one minimising loss.

Minimising loss could still lead to overfitting on the weak labels. In order to achieve a higher level of generalization, we adopted an early stopping strategy that concludes the fine-tuning process once the training loss on \textit{WS\_val\textsubscript{y}} starts increasing~\cite{prechelt1998early}.
In this case, the $best\_ep$ model is always that of the penultimate epoch of the early stopped fine-tuning process.  
Furthermore, together with the $best\_ep$ model that achieves the best performance on \textit{WS\_val\textsubscript{y}}, we also considered the $prev\_ep$ and $next\_ep$
models that correspond to the epochs just before or after $best\_ep$. 
The motivation for epochs that are close to the best-performing one is that their ``imperfection" may be due to a higher level of generalization. 

\subsubsection{Label imbalance and under-sampling}
Another challenge in training the models is the imbalance of the labels.
That is, the number of positive instances for each label varies among use cases, favoring more popular fine-grained labels. 
For this purpose, we considered extensions of the well-established binary cross entropy (BCE) loss~\cite{Bengio2013} can handle imbalanced training data for multi-label text classification~\cite{Huang2021}. 
In particular, the rebalanced focal loss function (R-BCE-FL) is an extension of BCE loss, first introduced to handle label imbalance in multi-label image classification tasks~\cite{Wu2020} and recently extended to multi-label text classification tasks~\cite{Huang2021}.    
R-BCE-FL modifies BCE in two ways. First, the instances are rebalanced, by oversampling based on the overall frequency of corresponding labels, and second, a focusing parameter increases the weight of more hard-to-classify instances, based on their predicted probability~\cite{Huang2021}. 

Looking separately at each label, there are many more negative instances than positive ones. In extreme cases, this abundance of negative instances  can prevent the models from learning to recognize positive ones, as their overall effect on loss is minimal. 
Previous work has shown that under-sampling can help address this problem~\cite{nentidis2020beyond}. 
Therefore, we experimented also with an under-sampling procedure to remove some negative instances, and achieve an ideal negative-to-positive ratio ($balance\_n$) for each label in the dataset.
As explained in subsection~\ref{ssec:dataset_generation}, not all instances are \textit{valid} for all labels, and this is taken into account in the undersampling of negative instances, i.e. only the \textit{valid} ones for the particular label are used. 
An ideal negative-to-positive ratio is not known beforehand, but a reasonable estimate can be provided through validation. 

A negative instance that is superfluous for a specific label may be positive for another label, or one of the few \textit{valid} negative instances for another.
Therefore, removing random negative instances to strictly enforce the ideal negative-to-positive ratio for a label, may deplete the positive or \textit{valid} negative instances for another. 
For this reason, the undersampling process excludes instances
that are either positive for any of the labels or they are \textit{valid} for labels with a low negative-to-positive ratio. Algorithm~\ref{alg:undersample} explains the undersampling process.


\begin{algorithm}[!t]
\caption{Underasmple negative instances in a $dataset$ towards a negative-to-positive ratio close to $balance\_n$ for all labels.}
\begin{algorithmic}[1]
      \Function{Under-sample}{$dataset$, $balance\_n$}
       
       
        \While{ $\exists$ $negative\_instance$ in $dataset$ that has not been checked \textbf{and} \\ \hskip1.5em $\exists$ label $l$ in $dataset$ with $valid\_negatives > balance\_n * positives$}
            \State Get a random negative instance $i$ from the $dataset$.
            \State $i$ has now been checked.
           
            \If { $\exists$ label $l'$ in $dataset$ with $valid\_negatives \leq balance\_n * positives$  \\ \hskip3.0em \textbf{and} $i$ is valid for $l'$} \Comment{$i$ is a necessary negative for $l'$}
                \State Do nothing. 
            \Else \Comment{$i$ is superfluous}
                \State Remove $i$ from dataset. 
            \EndIf 
        \EndWhile
       
      \EndFunction
\end{algorithmic}
\label{alg:undersample}
\end{algorithm}

\subsubsection{Randomness in the model validation process}
Randomness is introduced in several steps of the model development process~\cite{Dodge2020}, notably in the splitting of training and validation sets, in the under-sampling of negative instances, in weight initialization of the output layer of the pre-trained BERT model, and in stochastic optimization~\cite{Kingma2014}. 
Our preliminary experiments showed that starting with different random seeds can lead to different results. For this reason, each experiment was repeated with different seeds and the predictions were aggregated with a majority voting approach. 
We have also investigated other voting schemes, such as at-least-one~\cite{kambhatla2006minority}, as well as filtering out experiments with low validation performance, but did not obtain better results. Preliminary results also showed that six random seeds are good enough to reach a stable estimate of model performance.

\subsection{Competing machine learning approaches}
\label{ssec:baselines}

For the purposes of comparison, three machine-learning approaches have been included in our experiments.
The first one is a logistic regression ($LR$) model for each fine-grained label, as in \textit{Beyond MeSH}~\cite{nentidis2020beyond}.
For a fair comparison with the method introduced here, we perform the same under-sampling of negative instances, which is repeated for the same random seeds and the results are combined using majority voting.
Both lexical and semantic features are considered, as suggested in~\cite{nentidis2020beyond} and the number of features, as well as the level of regularization, are optimised on validation set. 

The second method is \textit{Coarse2Fine} ($C2F$)~\cite{Mekala2021}, which extends coarse-grained categories into fine-grained ones, as explained in section \ref{sec:Background}. 
The basic difference of $C2F$ to the proposed method is that it does not use semantic information, as does the concept-occurrence heuristic. It relies exclusively on the literal occurrence of the label names. For a fair comparison against $DBM$, we provide $C2F$ with the same weak supervision.

Similarly to \textit{Beyond MeSH}, $C2F$ develops a distinct model for the refinement of each coarse-grained label into corresponding fine-grained models.
However, $C2F$ treats the classification problem as a multi-class problem, where each document belongs to exactly one of the fine-grained sub-categories of a coarse-grained label. Our fine-grained indexing setting is multi-label, with documents allowed to belong to any number of subcategories, even to none of them (unlabelled examples). 
In order to handle the many unlabelled examples, we introduce an additional subcategory for each coarse-grained label X, which we call X\_rest, representing all the documents that belong to X, but not any of the fine-grained subcategories of X. 
The predictions for these X\_rest labels are disregarded when reporting the performance of the $C2F$ models. That is, only the predictions for the fine-grained labels of interest are considered, as in $DBM$ and the other approaches.  

As a core step in the $C2F$ methodology is adding generated instances to handle imbalance, we did not undersample the negative examples. 
We also relied on the original implementation, as provided in the repository of the method\footnote{https://github.com/dheeraj7596/C2F}, and did not experiment with training parameters or different random seeds. 
However, in order to investigate the impact of generating instances as per $C2F$, we also introduce another approach ($DBM^{C2F}$), where we used the supervision produced by the generative model of \textit{Coarse2Fine} to train a single deep model for different use-cases. The $DBM^{C2F}$ approach used the implementation and configurations of $DBM$, including balancing and majority voting over different random seeds.

\subsection{Evaluation measures}
As both precision and recall are important for fine-grained semantic indexing, we adopt the F1-measure ($F1$) as an overall performance measure, as defined in equation~\ref{eq:f1} for a set of m documents. Here, $Y_i$ is the set of ground truth labels for the i-th document and $Z_i$ is the set of the predicted ones.
\begin{equation}
\label{eq:f1}
F_1=\frac{1}{m} \sum_{i=1}^{m} \frac{2|Y_i\cap Z_i|}{|Z_i|+|Y_i|}
\end{equation}
Furthermore, we consider two complementary approaches for averaging the results across different use cases. Namely the label-based macro-averaged F1-measure ($maF1$) and the label-based micro-averaged F1-measure ($miF1$)~\cite{Tsoumakas2009}. The first treats all use cases equally, preventing a few use cases of high prevalence to dominate the final score. The latter takes into account the absolute number of documents in each use case, preventing rare use cases to have too large an impact on the final score.
For completeness, we consider precision and recall as well, both label-based macro-averaged ($maP$, $maR$) and micro-averaged ($miP$, $miR$).

As discussed in subsection~\ref{ssec:dataset_generation}, in the multi-label datasets for fine-grained semantic indexing 
we want to identify articles relevant to a specific fine-grained label, out of all the \textit{valid} articles annotated with its coarse-grained parent.
Therefore, articles that are not \textit{valid} for a specific fine-grained label are not of interest and are not considered in the evaluation of the label.
In practice, given the i-th test document, multi-label models that have been trained on a set of labels $L$, will produce a list of predicted labels $Z_i \subseteq L$. 
To ensure document validity, we remove from $Z_i$ any labels that are not valid for the document. Hence, we get a $Z_i'$.
We refer to this process as \textit{validity filtering} and employ it in the evaluation of all models in this work. 


\section{Experiments and results}
\label{sec:Experiments}

The retrospective method $RetroBM$, described in subsection~\ref{ssec:retro_data}, was applied to several consecutive years (2006-2019) of the evolution of MeSH for the development of respective datasets. These datasets were then used to run experiments with the weak labeling functions, described in subsection~\ref{ssec:WS_study}, and the $DBM$ models for fine-grained semantic indexing, described in subsection~\ref{ssec:model_dev}.
The first set of validation experiments was performed on the dataset developed for fine-grained labels introduced in 2006 (\textit{2006 dataset}). 
These experiments were used to make design choices about the architecture and configurations of the $DBM$ method, such as the specific pre-trained \textit{BERT} model to fine-tune, the $balance\_n$ value, and the specific weak supervision enhancement approach.   
The data introduced in the subsequent years were used in ``evaluation experiments'', which aimed to confirm the choices made on the \textit{2006 dataset}.

\subsection{Retrospective datasets}
\label{ssec:retro_data_2005-2020}
MEDLINE data from fourteen years (2006 to 2019) were processed according to the retrospective scenario of subsection~\ref{ssec:retro_data}, for the development of fourteen respective datasets \textit{WS\_dev\textsubscript{y}} and \textit{test\textsubscript{y}}, adhering to the criteria of Table~\ref{table:criteria}.  
Table~\ref{table:criteria_values} presents the values of the data availability restrictions of criterion 3, in particular\footnote{The remaining criteria 1 and 2 do not require the selection of any specific numerical value.}.
The 2020 version of the MEDLINE/PubMed Annual Baseline Repository was used to retrieve documents and MeSH topic annotations, and version 4.0 of SemMedDB was used for \textit{concept occurrence}.
In total, 88 descriptors were found in the period of these fourteen years as shown in Table~\ref{table:use_cases}\footnote{The full list of descriptors per year is available in the supplementary file S1.}.
In 2006, for instance, 18 fine-grained descriptors were found, and the respective \textit{WS\_dev\textsubscript{2006}} and \textit{test\textsubscript{2006}} datasets consisted of 446,377 and 468,271 articles respectively. As already discussed, for articles in \textit{WS\_dev\textsubscript{2006}} there are only weak labels, whereas for \textit{test\textsubscript{2006}} ground-truth labels are also available.  
Table~\ref{table:use_cases} reveals the uneven distribution of descriptors per year, which is correlated to the similarly uneven distribution of concept promotion events (criterion 1) in MeSH presented in~\cite{Nentidis2021}.  

\begin{table}[!t]
\caption{Data availability restrictions for dataset development with the \textit{Retrospective Beyond MeSH} ($RetroBM$) method.}
\begin{center}
\small
\begin{tabular}{L{0.23\linewidth}L{0.2\linewidth}L{0.094\linewidth}L{0.13\linewidth}L{0.2\linewidth}}
Restriction code & 3.1 & 3.2 & 3.3 & 3.4 \\
Restriction Name & \textit{test\_positive\textsubscript{min}} & 
 \textit{dev\textsubscript{min}}            & 
 \textit{dev\textsubscript{max}}            & 
 \textit{dev\_positive\textsubscript{min}}  \\
Restriction Value & 10& 10 & 1,000,000 & 10 
\end{tabular}
\label{table:criteria_values}
\end{center}
\end{table}

\begin{table}[!t]
\caption{ Retrospective datasets for years 2006 to 2019.
\#$D_Cy$ stands for the number of use cases per year.
For the \textit{WS\_dev\textsubscript{y}} and \textit{test\textsubscript{y}} datasets, we report the number of all articles that are \textit{valid} for any of the use cases (total) and the number of articles with at least one weak label (weak lab.).  
For \textit{test\textsubscript{y}} in particular, the number of articles with at least one ground-truth label is also provided (positive).
}
\begin{center}
\small
\begin{tabular}{M{0.1\linewidth}M{0.07\linewidth}R{0.11\linewidth}R{0.12\linewidth}|R{0.11\linewidth}R{0.12\linewidth}R{0.1\linewidth}}
\multirow{2}{*}{Year} & \multirow{2}{*}{\begin{tabular}[c]{@{}l@{}}\#$D_Cy$\end{tabular}} & \multicolumn{2}{l}{\#Articles \textit{WS\_dev\_y}} & \multicolumn{3}{l}{\#Articles \textit{test\_y}} \\
 &            & total    & weak lab.   & total  & weak lab. & positive \\
\hline
2006  & 18    & 446,377   & 12,262   & 468,271 & 20,542 & 27,118 \\
2007  & 17    & 159,665   & 29,678   & 143,876 & 24,246 & 23,796 \\
2008  & 10    & 176,092   & 8,689    & 117,083 & 4,638  & 5,571  \\
2009  & 6     & 117,706   & 4,545    & 88,309  & 3,852  & 5,814  \\
2010  & 6     & 78,443    & 2,788    & 92,962  & 2,842  & 7,311  \\
2011  & 8     & 848,958   & 5,172    & 526,238 & 6,424  & 8,047  \\
2012  & 6     & 587,895   & 5,030    & 205,894 & 3,313  & 14,860 \\
2013  & 1     & 10,362    & 146     & 5,048   & 148   & 337   \\
2014  & 1     & 1,457     & 222     & 649    & 113   & 120   \\
2015  & 1     & 5,963     & 221     & 934    & 26    & 20    \\
2016  & 3     & 74,133    & 408     & 23,836  & 139   & 409   \\
2017  & 6     & 34,628    & 3,084    & 6,052   & 579   & 838   \\
2018  & 3     & 86,205    & 933     & 12,089  & 467   & 505   \\
2019  & 2     & 96,978    & 7,543    & 3,642   & 283   & 169   
\end{tabular}
\label{table:use_cases}
\end{center}
\end{table}

\subsection{Validation experiments on \textit{2006 dataset}}
\label{ssec:val_experiments}

The 18 fine-grained labels introduced in 2006 (\textit{2006 dataset}) constitute the validation dataset for this work, and were used to inform design choices about the structure of the $DBM$ model and estimate some hyper-parameters.

Table~\ref{table:LF_results} presents the performance of each labeling function on the ground truth data of \textit{test\textsubscript{2006}}.
The results show that \textit{concept occurrence} ($CO$) was clearly the strongest of these heuristics, which is in accordance with previous observations on smaller-scale experiments reported in~\cite{nentidis2020beyond}.
In addition, we observe that most labeling functions achieved better precision than recall, except those based on tokenized terms ($NT$ and $ST$).
A reason for the low precision of the latter can be the low specificity of some too-general tokens that are produced when splitting multi-word terms. Splitting "Niemann-Pick Disease, Type A", for instance, leads to tokens such as "disease", "type", and "a" that are too general (Table~\ref{table:Label_functions}).

\begin{table}[!t]
\caption{Performance of each labeling function ($LF$) on $test_{2006}$ ground truth data. 
The  labeling functions are \textit{concept occurrence} (\textit{CO}),  
\textit{name exact} (\textit{NE}), \textit{synonyms exact} (\textit{SE}),
\textit{name lowercase} (\textit{NL}), \textit{synonyms lowercase} (\textit{SL}),
\textit{name no punct} (\textit{NNP}), \textit{synonyms no punct} (\textit{SNP}),
\textit{name tokens} (\textit{NT}), and \textit{synonyms tokens} (\textit{ST}).
The variance across the 18 labels ($var$) is also reported for the macro-averaged metrics.}
\begin{center}
\small 

\begin{tabular}{lllllll}
LF   			& $maP$\footnotesize{ $\pm var$} & $maR$\footnotesize{ $\pm var$}    & $maF1$\footnotesize{ $\pm var$}	& $miP$   & $miR$   & $miF1$\\
\hline
\textit{CO} 	& 0.715\footnotesize{ $\pm0.03$}& 0.618\footnotesize{ $\pm0.08$}& \textbf{0.634}\footnotesize{  $\pm0.06$}&0.737&0.531&\textbf{0.618}	\\
\textit{NE}     & 0.707\footnotesize{ $\pm0.16$}& 0.082\footnotesize{ $\pm0.04$}& 0.112\footnotesize{ $\pm0.04$}&0.901&0.027&0.052	\\
\textit{SE}   	& 0.656\footnotesize{ $\pm0.18$}& 0.212\footnotesize{ $\pm0.10$}& 0.251\footnotesize{ $\pm0.12$}&0.889&0.046&0.088	\\
\textit{NL}     & 0.649\footnotesize{ $\pm0.08$}& 0.441\footnotesize{ $\pm0.13$}& 0.446\footnotesize{ $\pm0.12$}&0.589&0.203&0.302	\\
\textit{SL} 	& 0.620\footnotesize{ $\pm0.10$}& 0.365\footnotesize{ $\pm0.13$}& 0.408\footnotesize{ $\pm0.11$}&0.709&0.251&0.371	\\
\textit{NNP}    & 0.574\footnotesize{ $\pm0.12$}& 0.440\footnotesize{ $\pm0.14$}& 0.442\footnotesize{ $\pm0.13$}&0.583&0.196&0.293	\\
\textit{SNP}	& 0.597\footnotesize{ $\pm0.12$}& 0.349\footnotesize{ $\pm0.13$}& 0.385\footnotesize{ $\pm0.11$}&0.690&0.206&0.318	\\
\textit{NT}     & 0.322\footnotesize{ $\pm0.07$}& 0.838\footnotesize{ $\pm0.04$}& 0.377\footnotesize{ $\pm0.07$}&0.157&0.807&0.263	\\
\textit{ST}    	& 0.274\footnotesize{ $\pm0.06$}& 0.874\footnotesize{ $\pm0.04$}& 0.338\footnotesize{ $\pm0.07$}&0.081&0.883&0.149	
\end{tabular}   
\end{center}
\label{table:LF_results}
\end{table}

Using the weak supervision provided by $CO$, different models were developed on $dev_{2006}$. These models were developed based on the $DBM$ approach, \textit{Logistic Regression} ($LR$), \textit{Coarse2Fine} ($C2F$), as well as the combined $DBM^{C2F}$ described in subsection~\ref{ssec:baselines}.
Table~\ref{table:model_results_WS_2006} presents the predictive performance of these models on the ground truth data of $test_{2006}$\footnote{Detailed results with scores per descriptor are available in the supplementary file S2.}.
Experiments with $DBM$ models using different $balance\_n$ values ranging from 1 to 100, led to the choice of 10-to-1 as a negative-to-positive instance ratio. Hence, we adopted this value in the remaining experiments.

\begin{table}[!t]
\caption{Model performance on $test_{2006}$ ground truth data. Models developed on $dev_{2006}$ under the weak supervision provided by \textit{concept occurrence} ($CO$).
The variance across the 18 labels ($var$) is also reported for the macro-averaged metrics.
A subscript in $LR$ models indicates the number of features selected.
A subscript in $DBM$ and $DBM^{C2F}$  models indicates the training epoch considered, with the values \textit{p}, \textit{b}, and \textit{n} corresponding to \textit{prev\_ep}, \textit{best\_ep}, and \textit{next\_ep} respectively.
}
\begin{center}
\small 
\begin{tabular}{lllllll}
LF/model   			& $maP$\footnotesize{ $\pm var$} & $maR$\footnotesize{ $\pm var$}    & $maF1$\footnotesize{ $\pm var$}	& $miP$   & $miR$   & $miF1$\\
\hline
$CO$ 		& 0.715\footnotesize{ $\pm0.03$}& 0.618\footnotesize{ $\pm0.08$}& 0.634\footnotesize{ $\pm0.06$}&0.737 &0.531 &0.618\\
$LR_{5}$    & 0.427\footnotesize{ $\pm0.11$}& 0.822\footnotesize{ $\pm0.02$}& 0.474\footnotesize{ $\pm0.10$}&0.295 &0.655 &0.406\\
$LR_{10}$   & 0.358\footnotesize{ $\pm0.10$}& 0.807\footnotesize{ $\pm0.03$}& 0.396\footnotesize{ $\pm0.09$}&0.102 &0.629 &0.176\\
$LR_{100}$  & 0.275\footnotesize{ $\pm0.08$}& 0.809\footnotesize{ $\pm0.03$}& 0.318\footnotesize{ $\pm0.06$}&0.085 &0.620 &0.149\\
$LR_{1000}$ & 0.279\footnotesize{ $\pm0.09$}& 0.798\footnotesize{ $\pm0.04$}& 0.310\footnotesize{ $\pm0.07$}&0.134 &0.644 &0.221\\
$C2F$		& 0.240\footnotesize{ $\pm0.06$}& 0.941\footnotesize{ $\pm0.00$}& 0.332\footnotesize{ $\pm0.08$}&0.079 &0.959 &0.147\\ 
$DBM_p$  	& 0.652\footnotesize{ $\pm0.06$}& 0.663\footnotesize{ $\pm0.07$}& 0.645\footnotesize{ $\pm0.06$}&0.706 &0.572 &0.632\\				
$DBM_b$ 	& 0.657\footnotesize{ $\pm0.06$}& 0.613\footnotesize{ $\pm0.10$}& 0.596\footnotesize{ $\pm0.08$}&0.709 &0.560 &0.626\\ 
$DBM_n$ 	& 0.717\footnotesize{ $\pm0.03$}& 0.666\footnotesize{ $\pm0.08$}& \textbf{0.652}\footnotesize{ $\pm0.06$}&0.713 &0.568 &0.632\\		
$DBM^{C2F}_p$	& 0.736\footnotesize{ $\pm0.03$}& 0.651\footnotesize{ $\pm0.09$}& 0.639\footnotesize{ $\pm0.06$}&0.713 &0.569 &0.632\\ 
$DBM^{C2F}_b$	& 0.739\footnotesize{ $\pm0.03$}& 0.631\footnotesize{ $\pm0.08$}& 0.632\footnotesize{ $\pm0.05$}&0.715 &0.565 &0.631\\
$DBM^{C2F}_n$	& 0.741\footnotesize{ $\pm0.03$}& 0.649\footnotesize{ $\pm0.09$}& 0.645\footnotesize{ $\pm0.06$}&0.716 &0.567 &\textbf{0.633}
\end{tabular}
\end{center}
              
\label{table:model_results_WS_2006}
\end{table}

Beyond the standard \textit{BERT} model~\cite{Devlin2019}, domain-specific variants were also considered in the validation experiments. 
These include both \textit{BERT} models fine-tuned on large biomedical corpora such as PubMed (\textit{BioBERT}~\cite{lee2019biobert}) or pre-trained on such data from scratch (\textit{PubMedBERT}~\cite{Gu2022}). One of the advantages of the latter approach is that the vocabulary considered is domain-specific, allowing it to better capture the semantics of the text. 
The good performance of \textit{PubMedBERT} was confirmed on preliminary fine-tuning experiments on the \textit{2006 dataset}, and it was thus chosen as the basis for all $DBM$ models.       

Regarding the logistic regression models, different numbers of features were considered for feature selection with the F-ANOVA statistic, namely 5, 10, 100, and 1000. 
Both lexical tokens and semantic features were included in the feature sets; whereby semantic features were identified based on \textit{UMLS} concepts, as proposed in~\cite{nentidis2020beyond}. 
The $L2$ regularization level for each $LR$ model was selected via grid search on the weakly-supervised training data ($WS\_dev_{2006}$), considering a range of values from 0.01 to one billion\footnote{Namely these values were considered: 0.01, 0.1, 1, 10, 100, 1K, 10K, 100K, 1M, 1B}.

The results reported in Table~\ref{table:model_results_WS_2006} show that none of the $LR$ models managed to outperform the $CO$ baseline overall.
The best $LR$ model, namely $LR_5$ with five features, managed to outperform $CO$ in $F1$ for about 40\% of the 18 labels of $test_{2006}$\footnote{Detailed results with scores per descriptor are available in the supplementary file S2.}.
This is in accordance with previous results (see~\cite{nentidis2020beyond}) which suggested that $LR$ models may bring improvements upon the results of weak labelling, only in specific cases. However, the identification of these specific configurations (i.e., specific feature set and level of regularization), requires the use of ground truth validation data.

The overall $maF1$ performance of the $C2F$ models was also low, achieving top recall but low precision. 
In particular, $C2F$ did better than the $CO$ baseline for only three out of the 18 fine-grained labels of the dataset.
The low performance of a state-of-the-art method like $C2F$ is an indication that the task and corresponding datasets are very challenging. This is partly due to the high imbalance among the labels and between labeled and ``unlabeled'' documents, that do not belong to any of the labels of interest. 

On the other hand, all $DBM$ models trained on $CO$ weak labels achieved better $miF1$ than $CO$ itself, and most of them did better also in terms of $maF1$ as well.
In particular, $DBM_p$ and $DBM_n$, considering the $prev\_ep$ and $next\_ep$ models respectively, outperformed $CO$ in $maF1$, with $DBM_n$ achieving the top $maF1$ score among all the methods considered.
Additionally, the $DBM^{C2F}_p$ and $DBM^{C2F}_n$ models, also to outperformed $CO$ in terms of $maF1$.
These results suggest that some deep models based on the $DBM$ method can indeed improve upon the weak supervision that they rely on. 
In addition, considering the epoch next to the one achieving the lowest validation loss ($next\_ep$) leads to the best results for both $DBM$ and $DBM^{C2F}$ models in these experiments. 


Next, we experimented with the enhancement of the original weak supervision provided by $CO$, as discussed in subsection~\ref{sssec:ensembles}. 
In particular, for each combination of the available labeling functions, three ensemble approaches were assessed, namely \textit{majority voting} ($MV$), \textit{at-least-one} ($ALO$), and \textit{Snorkel label model} ($LM$).
As shown in Table~\ref{table:ensemble_top_combinations} some of those combinations can lead to further improvements in the classification performance of the models. The table presents the top eight combinations based on the $maF1$ and $miF1$ achieved by any of the three ensemble approaches\footnote{The full table with all the combinations is available in the supplementary file S3.}. 
The highest $maF1$ score achieved on $test_{2006}$ is 0.689. 
This score has been achieved by several distinct labeling-function combinations using either the $Snorkel$ $\textit{label model}$ ($LM$) or the \textit{at-least-one} ($ALO$) approach\footnote{The top $maF1$ score using \textit{majority voting} ($MV$) is 0.679, achieved by a single labeling-function combination ($CO$, $SL$, $NL$, $NT$, $ST$).}. 


\begin{table}[!t]
\caption{
The scores of the 8 labeling-function ($LF$) combinations that achieved the top $maF1$ and $miF1$ scores in $test_{2006}$ (0.689) considering any of the three ensemble approaches: \textit{majority voting} ($MV$), \textit{at-least-one} ($ALO$), and \textit{Snorkel label modeling} ($LM$)
The table is sorted by the variance of $maF1$ across the scores of the three approaches ($var$).
}
\small 
\begin{tabular}{L{0.35\linewidth}L{0.05\linewidth}L{0.05\linewidth}L{0.05\linewidth}L{0.05\linewidth}L{0.05\linewidth}L{0.05\linewidth}L{0.05\linewidth}}
&\multicolumn{4}{c}{$maF1$}   &\multicolumn{3}{c}{$miF1$} \\
$LF$ combination & $MV$ & $ALO$ & $LM$ & $var$ & $MV$ & $ALO$ & $LM$ \\
\hline
$CO$, $NL$, $SL$ 					& 0.586 & \textbf{0.689} & \textbf{0.689} & 0.004 	&	0.410&	\textbf{0.646}&	\textbf{0.646}\\
$CO$, $NL$, $SL$, $NNP$				& 0.540 & \textbf{0.689} & \textbf{0.689} & 0.007 	&	0.359&	\textbf{0.646}&	\textbf{0.646}\\
$CO$, $NL$, $SL$, $NNP$, $SE$		& 0.517 & \textbf{0.689} & 0.670 &  0.009 			&	0.289&	\textbf{0.646}&	0.642\\
$CO$, $NL$, $SL$, $SE$				& 0.480 & \textbf{0.689} & 0.670 &  0.013 			&	0.324&	\textbf{0.646}&	0.642\\
$CO$, $NL$, $SL$, $NE$				& 0.465 & \textbf{0.689} & 0.654 &  0.015 			&	0.313&	\textbf{0.646}&	0.640\\
$CO$, $NL$, $SL$, $NNP$, $SE$, $NE$	& 0.429 & \textbf{0.689} & 0.643 &  0.019 			&	0.240&	\textbf{0.646}&	0.507\\
$CO$, $NL$, $SL$, $NNP$, $NE$		& 0.417 & \textbf{0.689} & 0.670 &  0.023 			&	0.254&	\textbf{0.646}&	0.642\\
$CO$, $NL$, $SL$, $NE$, $SE$		& 0.403 & \textbf{0.689} & 0.654 &  0.024 			&	0.236&	\textbf{0.646}&	0.640
\end{tabular}
\label{table:ensemble_top_combinations}      
\end{table}

Interestingly, none of the top-performing combinations included the low-precision labeling functions $NT$ and $ST$. This helps the $ALO$ approach to achieve high maF1 scores, as it is known to be good at combining precise voters by improving the total recall~\cite{kambhatla2006minority}.
Another important observation is that $CO$, \textit{name lowercase} ($NL$), and \textit{synonyms lowercase} ($SL$) are present in all top-performing combinations, regardless of the ensemble approach. 
In addition, it seems that the combination of only these three labeling functions was sufficient to achieve top performance, as shown by comparing the first and second combinations in Table~\ref{table:ensemble_top_combinations}.
As a result, we have chosen to use the combination of $CO$, $NL$, and $SL$ in further experiments.

Regarding the choice of ensemble approaches, the fact that $ALO$ achieved top results with more labeling-function combinations than other methods\footnote{8 top-performance combinations with $ALO$ against 2 with the \textit{label model}}, indicates that $ALO$ is more robust to the choice of labeling functions in this specific setup. 
This could be due to the fact that in the $ALO$ ensemble, some labeling functions can be completely overshadowed by the ones that have better recall.
For example, in the fifth set ($CO$, $NL$, $SL$, $NE$) presented in Table~\ref{table:ensemble_top_combinations}, $NE$ is overshadowed by $NL$ with the use of $ALO$. This is because every document with a literal occurrence of the exact concept name ($NE$) will also have a literal occurrence when both the document and the concept name are lowercased ($NL$). 
For the $ALO$ approach, which assigns a label to an article if at least one labeling function does, this means that the vote of $NE$ doesn't affect the result. 
As a result, we have opted for the $ALO$ ensemble of $CO$, $NL$, and $SL$ to be used in the \textit{Deep Beyond MeSH} method. 
The detailed performance of the selected labeling functions and the \textit{enhanced supervision} ($ALO_3$) is summarized in Table~\ref{table:WS_enhancement_results}.

\begin{table}[!t]
\caption{The \textit{test\textsubscript{2006}} performance of the \textit{enhanced supervision} ($ALO_3$), the corresponding labeling functions, and the models developed using $ALO_3$.
A prefix ($e$) indicates training under \textit{enhanced supervision}.
The variance across the 18 labels ($var$) is also reported for the macro-averaged metrics.
The subscript in $LR$ models indicates the number of features selected.
The subscript in $DBM$ models indicates the training epoch considered, with the values \textit{p}, \textit{b}, and \textit{n} corresponding to \textit{prev\_ep}, \textit{best\_ep}, and \textit{next\_ep} respectively.
}
\begin{center}
\small 
\begin{tabular}{lllllll}
LF/model   			& $maP$\footnotesize{ $\pm var$} & $maR$\footnotesize{ $\pm var$}    & $maF1$\footnotesize{ $\pm var$}	& $miP$   & $miR$   & $miF1$\\
\hline
$CO$ 				& 0.715\footnotesize{ $\pm0.03$}& 0.618\footnotesize{ $\pm0.08$}& 0.634\footnotesize{ $\pm0.06$} &0.737 &0.531 &0.618\\
$NL$     			& 0.649\footnotesize{ $\pm0.08$}& 0.441\footnotesize{ $\pm0.13$}& 0.446\footnotesize{ $\pm0.12$} &0.589 &0.203 &0.302\\
$SL$ 				& 0.620\footnotesize{ $\pm0.10$}& 0.365\footnotesize{ $\pm0.13$}& 0.408\footnotesize{ $\pm0.11$} &0.709 &0.251 &0.371\\
$MV_3$ 				& 0.752\footnotesize{ $\pm0.02$}& 0.599\footnotesize{ $\pm0.09$}& 0.615\footnotesize{ $\pm0.06$} &0.665 &0.335 &0.446\\
$LM_3$ 				& 0.717\footnotesize{ $\pm0.02$}& 0.698\footnotesize{ $\pm0.05$}& 0.689\footnotesize{ $\pm0.03$} &0.713 &0.591 &0.646\\
$ALO_3$ 			& 0.717\footnotesize{ $\pm0.02$}& 0.698\footnotesize{ $\pm0.05$}& 0.689\footnotesize{ $\pm0.03$} &0.713 &0.591 &0.646\\
$eLR_{5}$        	& 0.419\footnotesize{ $\pm0.09$}& 0.838\footnotesize{ $\pm0.02$}& 0.487\footnotesize{ $\pm0.10$} & 0.299	&0.797	&0.434\\
$eC2F$           	& 0.266\footnotesize{ $\pm0.07$}& 0.949\footnotesize{ $\pm0.00$}& 0.356\footnotesize{ $\pm0.09$} &0.072 &0.969 &0.134\\
$eDBM_{p}$  		& 0.681\footnotesize{ $\pm0.03$}& 0.744\footnotesize{ $\pm0.04$}& 0.700\footnotesize{ $\pm0.03$} &0.689 &0.624 &0.655\\
$eDBM_{b}$ 			& 0.688\footnotesize{ $\pm0.03$}& 0.738\footnotesize{ $\pm0.04$}& \textbf{0.702}\footnotesize{ $\pm0.03$}	&0.690 &0.626 &0.656\\
$eDBM_{n}$  		& 0.689\footnotesize{ $\pm0.03$}& 0.741\footnotesize{ $\pm0.04$}& \textbf{0.702}\footnotesize{ $\pm0.03$}	&0.695 &0.623 &\textbf{0.657}\\
$eDBM^{C2F}_{p}$  	& 0.641\footnotesize{ $\pm0.05$}& 0.716\footnotesize{ $\pm0.06$}& 0.664\footnotesize{ $\pm0.05$} &0.683 &0.623 &0.652\\
$eDBM^{C2F}_{b}$  	& 0.693\footnotesize{ $\pm0.02$}& 0.729\footnotesize{ $\pm0.05$}& 0.692\footnotesize{ $\pm0.03$} &0.692 &0.623 &0.655\\
$eDBM^{C2F}_{n}$  	& 0.700\footnotesize{ $\pm0.02$}& 0.729\footnotesize{ $\pm0.05$}& 0.692\footnotesize{ $\pm0.03$} &0.692 &0.623 &0.655

\end{tabular} 
\end{center}
\label{table:WS_enhancement_results}
\end{table}

These results reveal that the $ALO$ ensemble can exploit information that is already used by the $CO$ heuristic, namely the name and synonyms of the concepts, to achieve scores better than $CO$ alone, and in fact even better than the results of the trained models of Table~\ref{table:model_results_WS_2006}.
Finally, Table~\ref{table:WS_enhancement_results} also presents the performance of models trained on $ALO_3$ weak supervision\footnote{Detailed results with scores per descriptor are available in the supplementary file S2.}.
These results suggest that training on the enhanced $ALO_3$ labels was beneficial for all learning approaches, compared to training on $CO$. 
Again, some $DBM$ models trained on the enhanced $ALO_3$ supervision ($eDBM$) improved upon $ALO_3$ weak labelling itself.

\subsection{Evaluation on new labels introduced after 2006}
\label{ssec:eval_experiments}

In this last part of our experiments, we assessed the $DBM$ method on the data of the 70 new use cases introduced between 2007 and 2019. 
We trained models on the original weak supervision provided by $CO$, as well as on the enhanced weak supervision of $ALO_3$.
The results of these experiments are presented in Table~\ref{table:eval_results}\footnote{Detailed results with scores per descriptor are available in the supplementary file S4.}.

\begin{table}[!t]
\caption{Results on ground truth data for the 70 fine-grained labels introduced after 2006. 
The variance of the $maF1$ score across the 70 labels ($maF1var$) is also reported for each model.
A prefix ($e$) indicates training under \textit{enhanced supervision}.
The variance across the 70 labels ($var$) is also reported for the macro-averaged metrics.
The subscript in $LR$ models indicates the number of features selected.
The subscript in $DBM$ models indicates the training epoch considered, with the values \textit{p}, \textit{b}, and \textit{n} corresponding to \textit{prev\_ep}, \textit{best\_ep}, and \textit{next\_ep} respectively.
}
\begin{center}
\small 
\begin{tabular}{lllllll}
LF/model   			& $maP$\footnotesize{ $\pm var$} & $maR$\footnotesize{ $\pm var$}    & $maF1$\footnotesize{ $\pm var$}	& $miP$   & $miR$   & $miF1$\\
\hline

$CO$		& 0.766\footnotesize{ $\pm0.04$}& 0.590\footnotesize{ $\pm0.08$}& 0.626\footnotesize{ $\pm0.06$}  			& 0.848 &0.554	&0.670\\
$NL$		& 0.647\footnotesize{ $\pm0.13$}& 0.369\footnotesize{ $\pm0.13$}& 0.408\footnotesize{ $\pm0.12$}  			& 0.880 &0.217	&0.348\\
$SL$		& 0.567\footnotesize{ $\pm0.15$}& 0.253\footnotesize{ $\pm0.09$}& 0.287\footnotesize{ $\pm0.09$}  			& 0.809 &0.125	&0.217\\
$LR_{5}$	& 0.420\footnotesize{ $\pm0.11$}& 0.807\footnotesize{ $\pm0.04$}& 0.458\footnotesize{ $\pm0.09$}  			& 0.204	&0.783	&0.324 \\
$C2F$		& 0.206\footnotesize{ $\pm0.07$}& 0.946\footnotesize{ $\pm0.01$}& 0.276\footnotesize{ $\pm0.09$}  			& 0.074 &0.981	&0.138\\
$DBM_n$		& 0.718\footnotesize{ $\pm0.05$}& 0.647\footnotesize{ $\pm0.08$}& 0.641\footnotesize{ $\pm0.06$}  			& 0.826 &0.573	&0.677\\
$DBM^{C2F}_{n}$  	& 0.712\footnotesize{ $\pm0.05$}& 0.626\footnotesize{ $\pm0.00$}& 0.625\footnotesize{ $\pm0.06$} &0.831 &0.568 &0.675\\
$MV_3$  	& 0.782\footnotesize{ $\pm0.05$}& 0.481\footnotesize{ $\pm0.09$}& 0.542\footnotesize{ $\pm0.08$}  			& 0.873 &0.281	&0.462\\
$LM_3$  	& 0.756\footnotesize{ $\pm0.05$}& 0.652\footnotesize{ $\pm0.08$}& 0.661\footnotesize{ $\pm0.06$}  			& 0.842    &0.569	&0.679\\
$ALO_3$ 	& 0.764\footnotesize{ $\pm0.03$}& 0.668\footnotesize{ $\pm0.07$}& \textbf{0.677}\footnotesize{ $\pm0.05$}  	& 0.839 &0.583	&0.688\\
$eLR_5$ 	& 0.403\footnotesize{ $\pm0.10$}& 0.825\footnotesize{ $\pm0.03$}& 0.454\footnotesize{ $\pm0.09$} & 0.196	&0.797	&0.315\\
$eC2F$		& 0.203\footnotesize{ $\pm0.07$}& 0.959\footnotesize{ $\pm0.01$}& 0.272\footnotesize{ $\pm0.09$}  			& 0.076 &0.983	&0.141\\
$eDBM_n$ 	& 0.725\footnotesize{ $\pm0.04$}& 0.695\footnotesize{ $\pm0.06$}& 0.672\footnotesize{ $\pm0.04$}  			& 0.839 &0.621	&\textbf{0.714}\\
$eDBM^{C2F}_{n}$  	& 0.726\footnotesize{ $\pm0.05$}& 0.679\footnotesize{ $\pm0.07$}& 0.656\footnotesize{ $\pm0.05$} &0.803 &0.588 &0.679

\end{tabular}
\end{center}
\label{table:eval_results}
\end{table}

These results confirm that $CO$ is a strong heuristic for the task of fine-grained semantic indexing, outperforming again dictionary-based heuristics.
In addition, the weak labelling enhancement approach ($ALO_3$) achieved an overall $maF1$ about 5pp higher than $CO$ alone, which is a statistically significant difference\footnote{Wilcoxon signed-rank test, $H_1: F1\:CO < F1\:ALO_3$, p-value = 0.000001.}. 
The performance of $LM_3$ was not significantly lower than that of $ALO_3$, as the two approaches achieved practically the same $F1$ score for most of the 70 labels. However, $ALO_3$ achieved $F1$ performance equal to or better than that of $LM_3$ for all 70 labels.

Moving to the trained models, the weakly-supervised $DBM_n$ model under the supervision of $CO$ alone led to a limited overall performance gain of about 1.5pp in terms of $maF1$ over $CO$.
The performance of the $eDBM_n$ model, using the enhanced supervision of $ALO_3$, was higher, achieving about 4.6pp improvement in $maF1$, which is statistically significant\footnote{Wilcoxon signed-rank test, $H_1: F1\:CO < F1\:eDBM_n$, p-value = 0.0065.}. 
However, even $eDBM_n$ achieves only comparable performance to $ALO_3$ on its own. In fact, $ALO_3$ has a slightly higher $maF1$, without statistical significance\footnote{Wilcoxon signed-rank test, $H_1: F1\:ALO_3 \neq F1\:eDBM_n$, p-value = 0.088.}.
Nevertheless, $eDBM_n$ achieved the best $miF1$ score, with an overall performance gain of about 2.6pp over $ALO_3$ on the new unseen labels of the evaluation experiments.

\section{Discussion and conclusions}
\label{sec:Conclusion}
In this work, we investigate the weakly-supervised refinement of coarse-grained MeSH subject annotations into fine-grained ones, at the level of domain concepts. Additionally, we propose an automated approach to generating large-scale ground truth datasets for fine-grained semantic indexing, based on MEDLINE articles. 
Hence, the main contributions of the paper are: a) a new method (\textit{Retrospective Beyond MeSH}) for the automated development of ground-truth datasets for the large-scale evaluation of fine-grained semantic-indexing predictions; b) a voting approach for enhancing the original weak supervision provided by \textit{concept occurrence}, c) a new method (\textit{Deep Beyond MeSH}) for developing weakly-supervised deep learning models for refined indexing, and d) the validation and evaluation of the proposed enhanced weak supervision (b) and weakly-supervised models (c) on the new large-scale retrospective datasets (a). 

Using the retrospective dataset-development method, which exploits the extension of MeSH towards more fine-grained labels, we developed ground-truth test sets for 88 distinct fine-grained labels. These datasets consist of more than two million articles with weak supervision for the development and more than one million articles with ground-truth supervision for testing. 
To the best of our knowledge, this is the first large-scale set of datasets for fine-grained semantic indexing of biomedical literature.

Using these new datasets, we first confirmed the strength of \textit{concept occurrence} as a heuristic for fine-grained semantic indexing of biomedical literature. 
Additionally, we showed that this heuristic can be significantly enhanced when combined in an \textit{at-least-one} ensemble with the dictionary-based heuristics \textit{name lowercase} and \textit{synonyms lowercase}. The latter two heuristics examine the literal occurrence of the name and synonyms of the concept in the title and abstract of an article.
This conclusion is particularly inserting, considering that these dictionary-based heuristics rely on the same concept terms already exploited by \textit{concept occurrence} itself. 

We then moved on to train various machine learning models using the weak labels as supervision.
We have experimented with two existing methods, \textit{logistic regression} and \textit{Coarse2Fine}, the parameters of which we have optimised on validation data. Our experimentation concluded that the methods cannot improve upon the results of the weak labeling heuristics themselves. 


For this reason, we introduced a new method to train deep learning (DL) models, using the weak labels.
This method builds upon the state-of-the-art approach of fine-tuning pre-trained models and making design choices for the specific problem. 
In particular, early stopping seems to be useful, especially when considering the epoch next to the one with the lowest loss on $WS\_val_y$. 
In addition, in order to deal with label imbalance, we use the R-BCE-FL loss and under-sampling. 
Furthermore, we observed that majority voting among models developed with different random seeds can also improve the overall performance, limiting the effect of randomness.
Finally, DL models trained with the weak labels, using the proposed method ($DBM$) achieved competitive indexing performance on new labels.

Moreover, when the enhanced ensemble-based weak supervision ($ALO_3$) is used, the DL models improve significantly, reaching micro F1 levels above the very competitive weak labeling itself ($ALO_3$).
Nevertheless, no trained model managed to outperform the overall macro F1 performance of $ALO_3$ itself on the new unseen labels of the evaluation experiments. 
The strongest DL models ($DBM$) achieved only a comparable performance.

Based on these results, our future plans focus mainly on the improvement of the promising new method for DL model development.
In this direction, we plan to use additional label information, such as hierarchical relationships from MeSH, more fine-grained text representations based on contrastive learning approaches, and probabilistic labels in training instead of binary ones.    
In addition, we plan to investigate approaches for automated filtering of erroneous weak labels assigned by the heuristics. 

\section{Acknowledgment}
This research work was supported by the Hellenic Foundation for Research and Innovation (HFRI) under the HFRI Ph.D. Fellowship grant (Fellowship Number: 697).
The data resources considered in this work, including  MeSH, MEDLINE/PubMed, and SemMedDB, were accessed courtesy of the U.S. National Library of Medicine.
We are grateful to Georgios Katsimpras for the useful discussions on Deep Learning approaches.






