\section{Introduction}
\begin{figure*}[!t]
    \centering
    \includegraphics[scale=0.56]{figures/pipeline-white.drawio.pdf}
    \caption{SAINE Workflow and Pipeline.}
    \label{fig:pipeline}
\end{figure*}
A precise classification of publications across and within disciplines is key not only for a fast and comprehensive search to guide researchers to relevant material but also to identify the novelty of research, the standing and significance of scholars as well as their home institutions, and of the relative growth of fields of work.

Machine learning develops into being not only \textit{a} but \textit{the} customary approach to establish such a classification. Clearly, one would expect a search that is geared to identifying a high-quality corpus of keywords to benefit crucially from supervision. Existing classifications of academic output are based on a blend of (supervised) author-chosen and (unsupervised) machine-chosen keyword lists, where the composition of the blend is unknown to the researcher. 

Prevailing systems of keywords for academic publications are lists based on abstracts in a discipline, field, and subfield distilled from
\begin{itemize}
\item unsupervised machine learning (from word or phrase frequencies);
\item supervised learning (mostly from keyword self-reporting by authors);
\item semi-supervised learning (a mixture of the two; e.g., as done by Microsoft Academic Graph (MAG) described in~\citet{sinha2015overview, 10.3389/fdata.2019.00045, wang2020microsoft}).
\end{itemize}

To design an annotation and inference engine that helps us understand the publication space better, \textbf{we should cater to the following needs:} (1) a simple user interface with clear annotation instructions; (2) a reproducible pipeline across various disciplines; (3) good support for inference tailored to downstream tasks (e.g., model retraining) in meta-science studies. 

Among the existing open-source annotation tools, Label Studio \cite{LabelStudio} suits our needs. Note that \citet{gayoso2019review} have reviewed extensively on the annotation tools that facilitate classification tasks. The reviewed tools are either not open-sourced or are domain-specific and hence do not suit our purposes. 

In this system demonstration, we utilize a set of standard open-source software, mainly Label Studio \cite{LabelStudio}, \href{https://mlflow.org/}{MLflow} and \href{https://fastapi.tiangolo.com/}{FastAPI} to configure an annotation and inference engine for scientific publication annotations. In this demonstration, we illustrate the benefit of using supervised learning based on pre-established keyword lists and abstracts, and how annotators can help us better understanding the importance of \textit{supervised} learning in establishing a classification of academic publications. 

This system is built on top of the hitherto largest scale of \textit{multi-class} hierarchical classification studies across all disciplines in both \textit{single-label} and \textit{multi-label} settings (cf.~\citet{rao2023hierarchical}). There, we have built a supervised hierarchical classification system that associates every publication with at least one and potentially several disciplines, fields, and subfields. 

With the annotations above, we conduct a small user study with domain experts using our \textbf{annotation} engine. We then invoke our \textbf{inference} engine to fine-tune the base models in \citet{rao2023hierarchical}. The comparison between the base and fine-tuned models shows that the proposed annotation and inference system is able to benefit the further development of a more accurate classification. 

To summarize, the paper presents a scientific annotation and inference engine called SAINE, which is based on open-source software like Label Studio and MLflow. \textbf{The main contributions of the paper are:}
(1) The demonstration of using SAINE in understanding the space for scholarly publications, particularly in hierarchical discipline classifications.
(2) The result of a user study, which shows that user input collected with the help of SAINE can help better understand the classification process.
(3) The ability of SAINE to benefit the further development of a more accurate classification, demonstrated through the comparison between the base and fine-tuned models.
(4) The potential of SAINE to support downstream meta-science projects and foster greater transparency and understanding of scientific research.


Overall, the paper presents the benefits of supervised learning and the importance of having a simple user interface with clear annotation instructions, reproducible pipelines, and good support for inference in scientific publication annotations. The \href{https://app.heartex.com/user/signup/?token=e2435a2f97449fa1}{live demo website} and \href{https://youtu.be/yToO-G9YQK4}{demonstration video} are also available for those interested in further exploring SAINE. The codebase for development is publically available \href{https://gitlab.ethz.ch/raox/science-clf/-/tree/labelstudio}{under this link} and collocates with the codebase of \citet{rao2023hierarchical}.

In Figure~\ref{fig:pipeline} we illustrate the workflow in SAINE by assigning the roles of "Administrator", "Annotators", "Label Studio", and "MLflow" to each task in the pipeline. The sections are organized as follows. Section~\ref{sec:labelstudio} introduces the functionality of Label Studio and its fit to our annotation needs, as well as our annotation guidelines for experts. Section~\ref{sec:implementation} specifies the annotation design for the field of \textit{Economics} and discusses the annotation results. Section~\ref{sec:inf} discusses the integration of annotation results into the pre-trained base models and fine-tuned ones with MLflow. We then conclude this system demonstration with a discussion of system limitations, ethics, and broader impact statements. 




\section{Annotating Scientific Articles with Label Studio}
\label{sec:labelstudio}
We briefly introduce functionalities of Label Studio and illustrate why Label Studio is a suitable tool to annotate scientific articles. \textbf{The goals of annotations in the present context are three-fold:} 
\textbf{(1) [To judge the appropriateness of an assigned category.]} Experts judge whether a scientific article has a correctly assigned category. If a category does not suit the abstract's content, the annotator will pick a new category from a pre-defined list. 
\textbf{(2) [To evaluate keywords.]} Experts evaluate keywords assigned by MAG and mark missing keywords in the abstract.
\textbf{(3) [To calculate inter-annotator agreement.]} The annotation engine should permit an efficient way of calculating inter-annotator agreement (IAA) scores among annotators.


\subsection{Label Studio}
Label Studio is a powerful and versatile annotation tool that can handle various types of annotation tasks. Here are some of the features of Label Studio that make it a suitable tool for the annotation tasks.

\begin{itemize}
    \item \textbf{[Customizable interface.]} Label Studio allows to design a customized annotation interface. One can create a pre-defined list of categories for the experts to choose from and provide them with clear instructions on how to evaluate the assigned category.
    \item \textbf{[Multiple annotation types.]} Label Studio supports various types of annotation, including text classification, entity recognition, and sequence labeling. Therefore, different types of annotation can be used to evaluate keywords, mark missing keywords, and judge predicted categories.
    \item \textbf{[Collaboration and version control.]} Label Studio enables multiple experts to work on the same project simultaneously, allowing efficient and collaborative annotation. It also includes a version control system that tracks changes to the annotations, facilitating easy comparison and IAA evaluation.
    \item \textbf{[Inter-annotator agreement (IAA).]} Label Studio has built-in tools to calculate IAA scores. These tools can help evaluate the consistency and reliability of chosen annotations.
    \item \textbf{[Integrations with machine-learning models.]} Label Studio also provides integration with various machine learning (ML) models. Although we do not use the integrated ML functions, Label Studio allows us to export the annotation results in JSON, with which we improve the classification models using the annotated data in the inference engine.
\end{itemize}

Overall, Label Studio offers a powerful and customizable annotation platform that can handle relevant annotation tasks, facilitate efficient collaboration among experts, and efficiently compute IAA.

\begin{figure*}[!t]
    \centering
    \includegraphics[scale=0.25]{./figures/labelstudio_admin.png}
    \caption{Administrative Panel of Annotation Tasks in Label Studio.}
    \label{fig:labelstudio_admin}
\end{figure*}

In Figure~\ref{fig:labelstudio_admin} we demonstrate the administrative panel of the project manager. The project manager uses this panel to assign annotation tasks to each registered annotator and can monitor the annotation progress. The manager can also adjust the assigned annotations based on individual progress. The "Filters" and "Order (Annotation results)" tabs make it easy to inspect tasks by annotation progress (e.g., "Annotators", "Agreement", "Completed", "Total annotations per task"). 

\subsection{Annotation Guidelines}
When a publication is annotated, each annotator is provided with the abstract, the keywords offered by MAG, and the assigned category based on the keywords provided by MAG. The categories of a discipline classification (like JEL) are assigned to MAG publications on the basis of the keywords. Therefore, MAG's keywords help us identify potential misalignments and better understand the classifiers we built.

The annotation samples provided in the annotation engine are stratified sampled (ratio: 2e-5) across all classes of the training set introduced by \citet{rao2023hierarchical} for one discipline. Each annotator is required to judge whether a category is correctly assigned to an abstract. If not, the annotator is required to select the suitable one from a predefined list. The annotator is also required to evaluate MAG-generated keywords and make corrections (by removing unqualified keywords/marking suitable keywords from the abstract). Figure~\ref{fig:labelstudio_multipleannotator} shows two annotations of one publication. Label Studio makes it easy to navigate among the annotations generated by various annotators on an identical instance. Note that, as we discussed in \citet{rao2023hierarchical}, our multi-class hierarchical classification system is modularized in both \textit{single-label} and \textit{multi-label} settings. The current annotation engine is equipped with both annotation functionalities. For the sake of system demonstration and user study in Section~\ref{sec:implementation}, we discuss the \textit{single-label} setting. More details on the multi-label setting are provided in Appendix~\ref{app:multi-label}.

\begin{figure*}[!t]
    \centering
    \includegraphics[scale=0.4]{./figures/labelstudio_multipleannotator.png}
    \caption{Publication Annotations by Multiple Annotators.}
    \label{fig:labelstudio_multipleannotator}
\end{figure*}


\section{Implementation: User Study in \textit{Economics}}
\label{sec:implementation}

We now use \textit{Economics} as a discipline to show how we utilize the annotation engine to collect expert annotations.  

\subsection{Annotation Design}
We invited three economist experts from the Chair of Applied Economics at ETH Zurich to join the annotation project by accessing \href{https://app.heartex.com/user/signup/?token=e2435a2f97449fa1}{this link}. The annotation guidelines are stated at \href{https://gitlab.ethz.ch/raox/science-clf/-/blob/labelstudio/labelstudio/annotation_guideline.md}{here}. Of the three experts, one has annotated all provided instances (Annotator 1), one has annotated 10\% of the instances (Annotator 2), and one has annotated a subset of instances with an ex-ante denomination in Urban and Spatial Economics only (Annotator 3). Each annotator was provided a user panel shown in Figure~\ref{fig:labelstudio_annotator} in Appendix~\ref{app:more-function}.

\subsection{Annotation Results in Label Studio}

Altogether, 788 instances of abstracts and keywords from MAG had to be annotated for a \textit{single-label} classification. In Economics, a standardized field and subfield system with keywords exists, and it is called \href{https://www.aeaweb.org/econlit/jelCodes.php?view=jel}{the Journal of Economic Literature (JEL) classification system}. This system is known to all academic economists and serves as a guiding principle to associate an article or a topic with a specific subfield in \textit{Economics}. The subfields in the JEL categories are associated with keywords.

We report the annotation time and IAA scores that are automatically calculated by Label Studio (see \href{https://labelstud.io/guide/stats.html}{the official documentation} for the steps). The final task agreement score is calculated by averaging all IAA scores for each annotation pair. Table~\ref{tab:iaa} illustrates the IAA scores amongst three experts. Annotators 1, 2, and 3 have annotated 788, 181, and 99 instances, respectively. The annotations overlap between the pairs is 4 or 7\% of the overlapping instances (Annotators 2 and 3), 99 or 100\% of the overlapping instances (Annotators 1 and 3), and 181 or 100\% of the overlapping instances (Annotators 1 and 2). The median annotation time of Annotators 1-3 per instance was 17.7s, 29.8s, and 40.9s, respectively. The annotators were entitled to disapprove of the assigned category based on MAG upon suggesting an alternative category. Marking and filling in missing keywords is time consuming, reading the MAG-generated keywords can help to some extent the annotation speed. However, all annotators reported that the MAG-provided keywords could be a source of error for wrongly assigned categories. As discussed among the annotators after they underwent the annotations separately, the category they found the best was for \textit{Mathematical \& Quantitative Methods}, and it was worst for \textit{Macroeconomics} and \textit{Public Economics}. 


\begin{table}[!t]
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{cccc}
\toprule
                     & \textbf{Annotator 1} & \textbf{Annotator 2} & \textbf{Annotator 3} \\
\midrule
\textbf{Annotator 1} &                      & 55\%                 & 55\%                 \\
\textbf{Annotator 2} & 55\%                 &                      & 27\%                 \\
\textbf{Annotator 3} & 55\%                 & 27\%                 &    \\
\bottomrule
\end{tabular}}
\caption{Annotator Agreement Matrix Among Three Expert Annotators.}
\label{tab:iaa}
\end{table}

\begin{figure*}[!t]
    \centering
    \includegraphics[scale=0.3]{figures/inference_3_crop.png}
    \caption{Inference Engine with \textit{MLflow} Integration. API: Batch Inference by Model.}
    \label{fig:inference}
\end{figure*}
 


\section{Inference Engine: Incorporating Annotation Results into the Existing Classification Pipeline}
\label{sec:inf}
We illustrate the pipeline using the discipline \textit{Economics} as discussed in Section \ref{sec:implementation}. 

\subsection{Post-processing of Annotation Results}
We downloaded the annotation results in JSON (\href{https://gitlab.ethz.ch/raox/science-clf/-/blob/labelstudio/labelstudio/labeled_data/labelstudio_econ.json}{here}) of all experts and post-processed them following the protocols below, before feeding them into the pre-trained base models of various neural networks as discussed in \citet{rao2023hierarchical}. In total, we obtained 1,068 partly overlapping annotations (incl.~"Skip", "(Dis)agree", keywords, added categories). The basic statistics on the number of instances of "Agree", "Disagree" and "Not ECON" are 498, 297, and 268, respectively.
   
   
   
    
Here is the post-processing procedure. (1) We removed abstracts that were inadequately classified as belonging in Economics from the sample (206 of 788 instances). Additionally, we deleted 5 instances due to bad annotations. For example, no one labeled this sample ("Skip"), or an annotator chose "Disagree" but did not choose a new category.
(2) For each remaining instance, we counted the percentages of \textit{"Agree"} and \textit{"Disagree"} verdicts relative to the label generated on the basis of MAG keywords. If strictly more experts agreed than disagreed with MAG, the original label was preserved (for 351 of the 577 valid instances).  Otherwise, we took the label suggested by the majority of annotating experts (for 226 of the 577 valid instances). 
(3) In the case of ties, we randomly pick a label from the suggested annotations (for 22 of the 226 category-renewed instances). Following this protocol, we obtained 561 instances with expert-curated labels to fine-tune the base models. 
\begin{table*}[!t]
\centering
\resizebox{0.8\linewidth}{!}{
\begin{tabular}{cccccc}
\toprule
\textbf{Model} & \textbf{Model\_{FT}} & $\Delta$ (\textbf{Model\_{FT}} - \textbf{Model}) & \textbf{Model} = \textbf{Model\_{FT}} & \multicolumn{1}{l}{\textbf{Total}} & \textbf{Model Type}  \\
(1) & (2) & (3) & (4) & (5) & (6) \\
\midrule
54    & 58        & 4                         & 108               & \multirow{4}{*}{190}      & CNN         \\
59    & 69        & 10                        & 148               &                           & \textbf{RNN}         \\
39    & 39        & 0                         & 190               &                           & DNN         \\
31    & 33        & 2                         &  37               &                           & Transformer \\
\bottomrule
\end{tabular}
}
\caption{Results of the User Study. \textit{FT}: Fine-tuned.}
\label{tab:m-vs-mft}
\end{table*}

\subsection{Fine-tuning Pre-trained Base Models}
We used the 561 labels generated by the experts as a fine-tuning set on the base models reported in \citet{rao2023hierarchical} on the discipline of \textit{Economics} (model-1). We compared the inference performances of the base model (\textbf{Model} in Table~\ref{tab:m-vs-mft}) with those of the fine-tuned model (\textbf{Model\_FT} in Table~\ref{tab:m-vs-mft}) on various neural network architectures, Deep Neural Network (DNN), Recurrent Neural Network (RNN), Convolutional Neural Network (CNN), and Transformers. To benchmark the differences in performances between \textbf{Model} and \textbf{Model\_FT}, we created a small test set from the \href{https://www.ssrn.com/index.cfm/en/}{Social Science Research Network (SSRN)}, which is a website that provides a platform for researchers to share and distribute their research papers and other scholarly work in the social sciences and other related fields. We decided to use the \textit{Economics} SSRN publications because they come with human-currated JEL categories, keywords, and abstracts.  

Concretely, we built \href{https://gitlab.ethz.ch/raox/science-clf/-/tree/labelstudio/labelstudio/ssrn_crawler}{a crawler} to download the publication space in \href{https://papers.ssrn.com/sol3/displayjel.cfm}{Economics publications in SSRN}, where all contained research articles in Economics are multi-category-indexed. This means, each publication there is indexed by at least one JEL code and it allows multiple JEL codes per publication. We could easily validate with our multi-label engine in principal, but we focus on \textit{single-label} classifications for this user study. 

To create this test set, we randomly sampled 10 instances from each of 19 JEL field classes, which resulted in a sample of 190 test instances. In the implementation of hierarchical classifications reported in \citet{rao2023hierarchical}, we have used \href{https://mlflow.org/}{MLflow} to track and manage ML experiments, with which we have saved all pre-trained base models. Now, based on them, we could seamlessly integrate model fine-tuning and inference with various models.  
The inference engine API has been implemented using \href{https://fastapi.tiangolo.com/}{FastAPI} with help from \href{https://pydantic-docs.helpmanual.io/}{Pydantic}. We illustrate the batch inference API in Figure~\ref{fig:inference}, with which users can feed the test set into various models (base or fine-tuned) and obtain predictions. In Appendix~\ref{app:inferece} we provide more details about the inference engine. 

\subsection{Benefits of Expert Annotations}
We present the results of user studies in Table~\ref{tab:m-vs-mft}. Specifically, we inspect two types of statistics, the correct predictions of the base and fine-tuned models in Columns (1)-(2), and the identical predictions of the base and fine-tuned models in Column (4). Since each publication is multi-JEL-category-indexed, we count the prediction as "correct" if the indices include the predicted category. Column (1) is the base model trained with the model type specified in Column (6). Column (2) presents the results of the fine-tuned (supervised) model. Column (4) shows that out of a total of 190 test instances, identical predictions were generated by the base and fine-tuned models. We see that fine-tuning with user-generated results has brought benefits to all models except DNN because DNN predicts for all test examples only one class. RNN is the best performer when considering the benefits resulting from expert supervision, because the $\Delta$ in correct predictions has increased the most according to Column (3). Interestingly, fine-tuning a pre-trained Transformer model may not always result in a significant improvement in performance, as we see from a comparison with other base models. However, the current fine-tuning set is too small to draw firm conclusions in this regard.


\section{Conclusions}
In this system demonstration, we utilize a set of standard open-source software (mainly Label Studio \cite{LabelStudio}, \href{https://mlflow.org/}{MLflow} and \href{https://fastapi.tiangolo.com/}{FastAPI}) to configure an annotation and inference engine for scientific publications (SAINE). This system is built on top of hitherto largest multi-class hierarchical classification study across all disciplines in both single-label and multi-label settings (cf.~\citet{rao2023hierarchical}). We illustrate the functionality of the system with a user study in \textit{Economics} and show that the expert inputs into our system can help better understanding the classification process, which benefits the development of a stronger model in the next iteration. We plan to open-source the data and codebase and invite collaborative work in the direction of meta-science. 


\section*{Limitations}


Label Studio has some limitations in incorporating existing ML pipelines into the annotation engine, especially, when using customary code. We will discuss this with the developers at Label Studio and see how we can bring the annotation engine and the ML pipeline closer to each other. 

In terms of annotator selection, at the moment we have to select the experts for each discipline. However, we would wish to rank the annotators by their field expertise and weigh their annotations, accordingly. One future idea is to automatically compute an associative score between a third-party academic product such as Google Scholar and the publication space. The project \href{https://github.com/poloclub/people-map}{PeopleMap} provides interesting techniques to generate researcher profiles based on their research interests and publications taking as input the Google Scholar profile URLs of researchers. At this stage, Label Studio developers suggest that we add a self-declarative questionnaire to each annotator, which can be used as meta-data on annotators when quantifying the annotation confidence score. Due to time constraints, we have not yet added this questionnaire, as the experts in the current user study are selected by our project PI and have strong expertise in \textit{Economics}.

In terms of annotation efforts, given enough time and annotators, we can better benchmark annotation quality in terms of JEL categories and annotator expertise. Annotator 1 has compared the user experience and performance of \href{https://openai.com/blog/chatgpt/}{ChatGPT} in conducting annotations on a random sample of 100 instances. It turns out that ChatGPT captures high-quality keywords and achieves more than 90\% of the IAA score with that annotator. Considering our annotators' feedback that it is time-consuming to extract keywords for humans, it makes sense to use ChatGPT as an annotation-assisting engine for keyword extraction and label assignment. 


\section*{Ethics Statement}
We acknowledge that our system may involve processing potentially sensitive data, and we take data privacy and ethical considerations very seriously. In accordance with ethical guidelines of "ACM Code of Ethics", we have taken steps to protect the privacy of experts who may be affected by our research. We have also made efforts to ensure that our system and its annotations are unbiased and fair. We believe that our work will help foster greater transparency and understanding in scientific research, and we welcome collaboration and feedback from the scientific community to further advance the ethical and responsible use of AI in research.

\section*{Broader Impact Statement}
Our annotation engine and inference engine can further support downstream meta-science projects. We list a few interesting questions we can answer using our pipeline.
\begin{enumerate}
    \item \textbf{[For students.] }Which fields of research are more impactful/growing? 
    \item \textbf{[For policy makers.]} How to design education for cross-/inter-/pluridisciplinary studies? 
    \item \textbf{[For department and tenure committees.]} How to benchmark output and impact levels of an untenured scholar?
    \item \textbf{[For funding institutions.]} How to measure/quantify inter-/pluri-disciplinary standards for institutions such as \href{https://snis.ch/}{SNIS} and \href{https://www.snf.ch/en}{SNSF} which emphasize the interdisciplinarity of research?
    \item \textbf{[For librarians.]} How can one effectively organize bibliographical resources across disciplines and departments in one university?
\end{enumerate}

We plan to add other disciplines covered by \citet{rao2023hierarchical} to our annotation engine. We would also like to incorporate subjective (self-declaration) and objective measurements (e.g., Google Scholar profile integration) into the annotation pipeline. This may help develop confidence scores of one annotation/annotator.

\section*{Acknowledgements}
We thank the colleagues at \href{https://ds3lab.inf.ethz.ch/}{DS3Lab} for providing valuable feedback when prototyping the system design. Without the strong support of our expert group at the \href{https://cae.ethz.ch/}{Chair of Applied Economics}, the user study would not have been possible. We thank Ms.~Piriyakorn Piriyatamwong for her technical support to our project. We appreciate that Label Studio has offered us an academic license for the project, which allows us to invite more experts to contribute in the long run. The user agreement and terms of an academic license are listed \href{https://labelstud.io/academic/}{here}.

