\section{Introduction}

This package is there to make it easier to make annotated equations in \LaTeX, such as in this example:
\begin{LTXexample}[]
\vspace{4em}
\renewcommand{\eqnhighlightheight}{\vphantom{\hat{H}}\mathstrut}
\begin{equation*}
    i \tikzmarknode{hbar}{\mathstrut\hbar} \frac{\partial}{\partial t} \eqnmarkbox[blue]{Psi1}{\Psi(x, t)} = \eqnmark[red]{Hhat}{\hat{H}} \eqnmarkbox[blue]{Psi2}{\Psi(x, t)}
\end{equation*}
\annotate[yshift=3em]{above}{hbar}{$\hbar = \frac{h}{2\pi}$, reduced Planck constant}
\annotate[yshift=1em]{above}{Hhat}{Hamilton operator}
\annotatetwo[yshift=-1em]{below}{Psi1}{Psi2}{Wave function}
\vspace{1em}
\end{LTXexample}
There is still a bit of manual tweaking required (such as adding vertical space before/after the equation), but hopefully this package will already make it a bit more inviting to annotate your equations!

Note that this package relies on TikZ's \texttt{remember picture} option and therefore you have to compile your \LaTeX{} document at least twice to get everything in the right place (or just use \texttt{latexmk}!).

\section{Marking annotation targets within your equation}

\newcommand{\cmdoption}[1]{$\langle$\textit{#1}$\rangle$}

Use
\verb|\eqnmarkbox[|\cmdoption{color}\verb|]{|\cmdoption{node name}\verb|}{|\cmdoption{equation term(s)}\verb|}|
or
\verb|\eqnmark[|\cmdoption{color}\verb|]{|\cmdoption{node name}\verb|}{| \cmdoption{equation term(s)}\verb|}|
to define the target of an annotation within your equation. \verb|\eqnmarkbox| adds background shading, whereas \verb|\eqnmark| changes the text color.
(You can also use
\verb|\tikzmarknode{|\cmdoption{node name}\verb|}{|\cmdoption{equation term(s)}\verb|}|,
but this is likely to end up with the arrow tip too close to the target, so you may want to also pass the \texttt{outer ysep} option, see \cref{sec:outerysep}.)
\begin{LTXexample}[text outside listing,lefthand width=1in]
\begin{equation*}
    \eqnmarkbox[blue]{node1}{e_q^n}
    \eqnmark[red]{node2}{f(x)}
    \tikzmarknode{node3}{kT}
\end{equation*}
\end{LTXexample}
\noindent

\section{Simple annotations}
\label{sec:annotate}

Once you have defined nodes within your equations, you can annotate them using
\verb|\annotate[|\cmdoption{tikz options}\verb|]{|\cmdoption{annotate keys}\verb|}{|\cmdoption{node name[,\dots]}\verb|}{|\cmdoption{annotation text}\verb|}|.
\cmdoption{tikz options} is passed through to the options for the TikZ node defining the annotation; its most important use is to set the \texttt{yshift}.
For \cmdoption{annotate keys}, see \cref{sec:annotate-keys}.
\cmdoption{node name} is the same name you used to mark the node within the equation, e.g.\ using \verb|\eqnmarkbox|.
\cmdoption{annotation text} is the text of the annotation itself.
\begin{LTXexample}[text outside listing,lefthand width=1in]
\begin{equation*}
    \eqnmarkbox[blue]{node1}{e_q^n}
    \eqnmark[red]{node2}{f(x)}
    \tikzmarknode{node3}{kT}
\end{equation*}
\annotate[yshift=1em]{}{node1,node2}{my annotation text}
\end{LTXexample}
\noindent
\label{sec:tikz-options}
You generally need to manually adjust the \texttt{yshift} to move the annotations to an appropriate distance above (or negative values for below) the equation. If you want an annotation below the equation, with negative \texttt{yshift}, remember to also pass the \texttt{below} option (see \cref{sec:annotate-keys}).
(You can also adjust \texttt{xshift} if needed, also positive or negative.)

The annotation picks the same text color as given to \verb|\eqnmarkbox| or \verb|\eqnmark|, but you can also override it using \texttt{color} option.

One annotation can point to multiple targets, and multiple annotations can point to the same target.

\subsection{Annotation options}
\label{sec:annotate-keys}

\cmdoption{annotate keys} can be empty, or contain one or more of:
\begin{itemize}
    \item \texttt{above} (default) or \texttt{below},
    \item \texttt{right} (default) or \texttt{left},
    \item \texttt{label above} (default) or \texttt{label below}.

        Note: currently only works for \verb|\annotatetwo| (\cref{sec:annotatetwo}).
\end{itemize}
\begin{LTXexample}[text outside listing,lefthand width=1in]
\begin{equation*}
    \eqnmarkbox[blue]{node1}{e_q^n}
    \eqnmark[red]{node2}{f(x)}
    \tikzmarknode{node3}{kT}
\end{equation*}
\annotate[yshift=1em]{left}{node1}{my}
\annotate[yshift=-0.5em]{below,left}{node2}{annotation}
\annotate[yshift=-1em]{below}{node3}{text}
\end{LTXexample}
\noindent

\section{Double annotations}
\label{sec:annotatetwo}

\verb|\annotatetwo[|\cmdoption{tikz options}\verb|]{|\cmdoption{annotate keys}\verb|}{|\cmdoption{first node name}\verb|}{|\cmdoption{second node name}\verb|}{|\cmdoption{annotation text}\verb|}|.
\cmdoption{tikz options} and \cmdoption{annotate keys} are as described above in \cref{sec:tikz-options,sec:annotate-keys}. Note that \cmdoption{annotate keys} \texttt{left}/\texttt{right} have no effect in \verb|\annotatetwo|.

\begin{LTXexample}[text outside listing,lefthand width=1in]
\begin{equation*}
    \eqnmarkbox[red]{a1}{a} \eqnmarkbox[blue]{b1}{b} = \eqnmarkbox[green]{b2}{b} \eqnmarkbox{a2}{a}
\end{equation*}
\annotatetwo[yshift=1.5em]{above, label below}{a1}{a2}{var 1}
\annotatetwo[yshift=0.5em]{above}{b1}{b2}{var 2}
\annotatetwo[yshift=-0.5em]{below}{b2}{b1}{var 2}
\end{LTXexample}
\noindent
Color is picked from the first of the two nodes.

\section{Package options}

\subsection{Size of highlight: shrink to content or always full height}

\verb|\eqnhighlightheight| is inserted into every \verb|\eqnhighlight|, \verb|\eqncolor|, \verb|\eqnmark|, and \verb|\eqnmarkbox| and by redefining it you can specify the minimum height for the corresponding box:
\begin{LTXexample}[text outside listing,lefthand width=0.5in]
\renewcommand{\eqnhighlightheight}{} 

\begin{equation*}
    \eqnmarkbox[red]{hbar}{\hbar} \eqnmarkbox[blue]{q}{q}
\end{equation*}
\end{LTXexample}
\noindent
\begin{LTXexample}[text outside listing,lefthand width=0.5in]
\renewcommand{\eqnhighlightheight}{\mathstrut} 

\begin{equation*}
    \eqnmarkbox[red]{hbar}{\hbar} \eqnmarkbox[blue]{q}{q}
\end{equation*}
\end{LTXexample}
\noindent
\verb|\eqnhighlightheight| is used in math mode.


Note that in some cases \verb|\mathstrut| might not be enough, as in the introductory example:
\begin{LTXexample}[text outside listing,lefthand width=0.5in]
\renewcommand{\eqnhighlightheight}{\mathstrut} 

\begin{equation*}
    \eqnmarkbox[red]{Hhat}{\hat{H}} \eqnmarkbox[blue]{Psi}{\Psi}
\end{equation*}
\end{LTXexample}
\noindent
You can create custom 0-width characters using \verb|\vphantom|:
\begin{LTXexample}[text outside listing,lefthand width=0.5in]
\renewcommand{\eqnhighlightheight}{\vphantom{\hat{H}}\mathstrut} 

\begin{equation*}
    \eqnmarkbox[red]{Hhat}{\hat{H}} \eqnmarkbox[blue]{Psi}{\Psi}
\end{equation*}
\end{LTXexample}
\noindent
(It looks more balanced if you still include the \verb|\mathstrut|.)

\subsection{Amount of shading of mark highlight}

\verb|\eqnhighlightshade| defines the percentage of the specified color to take:
\begin{LTXexample}[text outside listing,lefthand width=0.5in]
\renewcommand{\eqnhighlightshade}{17} 

\begin{equation*}
    \eqnmarkbox[red]{hbar}{\hbar} \eqnmarkbox[blue]{q}{q}
\end{equation*}
\end{LTXexample}
\noindent
By redefining this command, you can change the ``alpha'' value of the highlight:
\begin{LTXexample}[text outside listing,lefthand width=0.5in]
\renewcommand{\eqnhighlightshade}{47} 

\begin{equation*}
    \eqnmarkbox[red]{hbar}{\hbar} \eqnmarkbox[blue]{q}{q}
\end{equation*}
\end{LTXexample}
\noindent

\subsection{Default formatting of annotation labels}

\verb|\eqnannotationfont| sets the \texttt{font} field of the TikZ annotation label and can be re-set to change its formatting:
\begin{LTXexample}[text outside listing,lefthand width=0.5in]
\renewcommand{\eqnannotationfont}{\sffamily\footnotesize} 

\begin{equation*}
    \eqnmarkbox[blue]{v}{v}
\end{equation*}
\annotate[yshift=-0.5em]{below}{v}{velocity}
\vspace{1em}
\end{LTXexample}
\noindent
\begin{LTXexample}[text outside listing,lefthand width=0.5in]
\renewcommand{\eqnannotationfont}{\bfseries\small}

\begin{equation*}
    \eqnmarkbox[blue]{v}{v}
\end{equation*}
\annotate[yshift=-0.5em]{below}{v}{velocity}
\vspace{1em}
\end{LTXexample}
\noindent

\verb|\eqnannotationstrut| is defined to be a strut (zero-width height) to
provide minimum distance between the text and the corresponding arrow line. By
default it is \verb|\strut|, which has a similar effect to
\verb|\mathstrut| in \verb|\eqnhighlightheight|.

\begin{LTXexample}[text outside listing,lefthand width=0.5in]
\renewcommand{\eqnannotationstrut}{\strut} 

\begin{equation*}
	\eqnmarkbox[blue]{size}{s}
\end{equation*}
\annotate[yshift=-0.5em]{below}{size}{The size}
\vspace{1em}
\end{LTXexample}
\noindent
\begin{LTXexample}[text outside listing,lefthand width=0.5in]
\renewcommand{\eqnannotationstrut}{}

\begin{equation*}
  \eqnmarkbox[blue]{size}{s}
\end{equation*}
\annotate[yshift=-0.5em]{below}{size}{The size}
\vspace{1em}
\end{LTXexample}


\section{Recommendations, tips \& tricks}

\subsection{Use \texttt{\textbackslash{}colorlet} for consistent, easily changeable colors}


\subsection{Relations such as ``$=$''}
\label{sec:mathrel}

Wrapping a mathematical relation symbol such as {\color{blue}$=$} in, for example, \texttt{\textbackslash{}tikzmarknode}, breaks how \TeX{} determines spacing in equations:
\begin{LTXexample}[text outside listing,lefthand width=0.5in]
\[
a \tikzmarknode{node1}{=} b
\]
\annotate[yshift=-1em]{below}{node1}{equality}
\end{LTXexample}
\noindent
This can be fixed by wrapping the \texttt{\textbackslash{}tikzmarknode} in \texttt{\textbackslash{}mathrel}:
\begin{LTXexample}[text outside listing,lefthand width=0.5in]
\[
a \mathrel{\tikzmarknode{node1}{=}} b
\]
\annotate[yshift=-1em]{below}{node1}{equality}
\end{LTXexample}

\subsection{Extra spacing between \texttt{\textbackslash{}tikzmarknode} and arrow}
\label{sec:outerysep}

If you want more space between arrow tip and annotated term, you can pass the \texttt{outer ysep} option to \texttt{\textbackslash{}tikzmarknode}:
\begin{LTXexample}[text outside listing,lefthand width=0.5in]
\[
a \mathrel{\tikzmarknode[outer ysep=5pt]{node1}{=}} b
\]
\annotate[yshift=-1em]{below}{node1}{equality}
\end{LTXexample}



\section{Known issues}

\begin{itemize}
    \item \texttt{label above}/\texttt{label below} is not implemented for \verb|\annotate|.

    \item Annotations of mathematical relations require some manual patching to get the correct surrounding spacing (see \cref{sec:mathrel}).

\end{itemize}

\end{document}



\section*{Data availability}

The anonymized data are available on Zenodo\footnote{\url{https://doi.org/10.5281/zenodo.7669442}}, and the code can be found in a GitHub repository.\footnote{\url{https://github.com/i6092467/semi-supervised-multiview-cbm}}

\section*{Discussion}\label{sec:discussion}

This work enhanced classical concept bottleneck models \citep{Koh2020}, making them more readily applicable to medical imaging datasets where multiple images or modalities may be observable for each subject. To this end, we proposed a practical architecture based on the hybrid fusion approach \citep{Baltrusaitis2019} that can handle varying numbers of views per data point, partial observability of the concepts from individual images and leverage spatial or temporal ordering. Although aggregating representations using averaging and LSTM has been explored in the prior literature \cite{Havaei2016,Ma2019}, we focused specifically on interpretable models, particularly concept-based classification. To the best of our knowledge, this problem setting has not been discussed by the previous literature despite its relevance to biomedical applications \citep{Wang2020,Qian2021}. 

Another scenario that we studied, similarly pertinent to applications, is when the concept set given to a CBM is insufficient \cite{Yeh2020}, i.e. does not entirely capture the predictive relationship between the covariates and the target. To mitigate limited predictive performance in such cases, our model learns additional representations complementary to the concepts, \mbox{i.e.} de-correlated from the concepts yet helpful in the downstream prediction problem. In practice, this required adjusting the model's architecture, extending the loss function with an adversarial regularization term and adapting the training procedure. 

A few previous works have investigated related limitations of the CBMs and considered alternative model designs. For instance, Sawada \& Nakamura combined CBMs with self-explaining neural networks to learn additional unsupervised concepts \cite{Sawada2022}; however, they did not investigate the disentanglement of the given and learned concepts and the intervenability of their extended bottleneck layer. Yuksekgonul et al. \cite{Yuksekgonul2022} proposed fitting a concept bottleneck \emph{post hoc} for a pretrained backbone and utilized  residual fitting to compensate for an incomplete concept set. Moreover, the authors investigated the global model edition, e.g. to mitigate the classifier's reliance on spurious correlation. By contrast, we assumed the \emph{ante hoc} modeling scenario and focused on the local, i.e. single-data-point, interventions. Another related line of research also studied the problem of unobserved concepts and concept leakage \cite{Marconato2022}. However, Marconato et al. \cite{Marconato2022} adopted a generative representation learning approach whose application to smaller datasets may be challenging in practice. The most closely related is the work by Havasi et al. \cite{Havasi2022}, who extended the standard CBM architecture with a side channel to learn latent concepts and compensate for insufficiency. This work is concurrent with ours but does not treat the problem of multiview learning and does not consider medical imaging data. 

In our experiments, we demonstrated the feasibility of the proposed models and the benefits of the multiview and semi-supervised approach on synthetic and natural image benchmarking data (Figure~\ref{fig:synthetic_res}). Concretely, we observed that, in most settings, MVCBM and SSMVCBM outperformed the vanilla model at the concept and target prediction. Moreover, based on the US data, we designed predictive models for appendicitis, its severity and the management of pediatric patients with abdominal pain  (Tables~\ref{tab:appendicitis_concept_res}--\ref{tab:appendicitis_target_res}). Our results suggest that, for the diagnosis, multiview concept bottlenecks can achieve performance on par with black-box models while allowing medical practitioners to interpret and intervene on the predictions. For management and severity, we observed somewhat inconclusive results with little difference across the single- and multiview classifiers. We attribute the latter to the limited predictive power of the ultrasonographic findings for these targets \cite{Marcinkevics2021}, the diagnostic nature of the chosen concepts and the overall moderate size of the training set. 

Most of the prior work on using ML for appendicitis has focused on tabular datasets with handcrafted features \citep{Hsieh2011,Deleger2013,Reismann2019,Aydin2020,Akmese2020,Stiel2020,Marcinkevics2021,RoigAparicio2021,Xia2022} or more invasive imaging modalities, such as computed tomography \citep{Rajpurkar2020}. This work takes the first step towards the computer-aided diagnosis of appendicitis based on abdominal ultrasound, a noninvasive, accessible, and cheap technique. Moreover, to facilitate replicability and new method comparison and development, we made our anonymized dataset publicly accessible. It includes laboratory, physical exam, clinical and US data from 579 patients (Figure~\ref{fig:data_overview}). In addition, for demonstratory purposes, we deployed the MVCBM model for the diagnosis as an easy- and free-to-use web tool. 

Although appendicitis is a common condition in the pediatric population, diagnosing it and choosing the best therapeutic option is challenging. Early differentiation between simple and complicated, necrotizing appendicitis is crucial for further management steps and prognosis \cite{Reddan2016,Reismann2019,Kiss2021}. The advances in US resolution, especially with the high-frequency quench sonography, support the detection of a normal appendix and the identification of indirect appendicitis signs, such as surrounding tissue inflammation and the reaction of the intestinal bowel wall \cite{Park2011}. ML-based decision support tools may further increase diagnostic accuracy and become instrumental in improving treatment outcomes. The results of the current study are promising since the direct interpretation of the US images by ML models appears to be a realizable goal. Predictive models, such as the ones developed in this study, may assist physicians in interpreting acquired US images and may even enable comparison of the results with the newly conducted US exams to characterize the progress or resolution of the inflammation.

Nevertheless, the current study design, experimental setup and proposed methods have several limitations. The appendicitis dataset represents a moderately-sized and relatively homogeneous patient cohort recruited from a single clinical center over a short time (between 2016 and 2021). Thus, in the future, predictive models need to be validated externally on the data from different US devices, clinical centers and countries. Another limitation is the lack of histologically confirmed diagnoses among the conservatively treated patients. This implies that the model validation and comparison results presented above must be interpreted cautiously since we do not have access to the ground-truth labels for all subjects. The image preprocessing pipeline could be improved further: currently, we discard scale information in the US images, making it impossible to detect the appendix diameter, a relevant sonographic sign of appendicitis \cite{Reddan2016}. From the methodological perspective, we currently have a limited theoretical understanding of the (SS)MVCBMs. In particular, it would be desirable to explore the representations learned by SSMVCBMs and the identifiability of the ground-truth generative factors. Moreover, in the current implementation, it is not trivial to interpret the representations. Thus, additional regularization may be necessary, for example, to render these representations disentangled.  


\section*{Conclusion and outlook}\label{sec:conclusion}

Motivated by the demand for model interpretability in biomedical applications, we investigated the use of concept bottleneck models for predicting the diagnosis, management and severity among pediatric patients with suspected appendicitis based on abdominal ultrasound images. We introduced several enhancements to the conventional concept-based classification approach. Our models can handle multiple views of the object of interest and insufficient concept sets. Overall, our experimental results suggest that 
the proposed methods can achieve competitive performance while offering an alternative to black-box deep learning models and allowing for real-time interaction with the end user.

In future work, we would like to address several limitations outlined above. We plan to validate the predictive models externally on the data from a hospital in another country. Various model design alterations, such as other choices of learnable fusion, further regularization of the learned representations and uncertainty quantification, are also to be considered. Finally, settings beyond the retrospective study are noteworthy. For instance, it would be interesting to explore the use of active learning to decide on the acquisition of US images and concept labels for each subject. From the clinical perspective, developed models should be extended to incorporate clinical and laboratory parameters and consider other conditions, such as COVID-19, during appendicitis. Additionally, more refined definitions of the target variables could be more insightful, e.g. differentiating between subacute and acute appendicitis for the diagnosis and, for the management, predicting the risk of the secondary appendectomy. Adjustments in the model architecture and the acquisition of a larger training dataset will facilitate the incorporation of the color Doppler images in the analysis, potentially making the prediction of the disease severity progression more accurate.
\section*{Extended data}

\setcounter{figure}{0}
\renewcommand{\thefigure}{\arabic{figure}}
\renewcommand{\figurename}{Extended Data Fig.}

\setcounter{table}{0}
\renewcommand{\thetable}{\arabic{table}}
\renewcommand{\tablename}{Extended Data Table}

\input{src/tables/predictive_perf_mvawa.tex}

\input{src/tables/concept_disambiguation.tex}

\begin{figure}[h!]%
\centering
\includegraphics[width=0.9\textwidth]{src/figures/preproc_example.pdf}
\caption{An example of multiple US images acquired from a single patient from the pediatric appendicitis dataset. For this patient, views I and \mbox{II} correspond to longitudinal and transverse sections of the appendix, respectively; view \mbox{III} depicts the reaction in tissue surrounding the appendix. Original images \mbox{(\emph{left})} contain graphical interface elements and expert-made markers, whereas preprocessed images \mbox{(\emph{right})} have been inpainted, cropped, and padded.  }\label{fig:us_example}
\end{figure}

\input{src/tables/concepts_management.tex}

\input{src/tables/concepts_severity.tex}

\begin{landscape}
\begin{figure}[h!]%
\centering
\includegraphics[width=1.4\textwidth]{src/figures/online_tool_schematic.pdf}
\caption{An illustrated use case for the pediatric appendicitis online prediction tool. \mbox{(1) The} user uploads input ultrasound images corresponding to a single patient. \mbox{(2) Optionally,} preprocessing is performed, and the tool displays original and preprocessed US images. \mbox{(3) The} tool displays predicted concept values alongside predicted value histograms obtained from the training data. The user can compare current predictions to those made for labeled training data points. \mbox{(4) The} tool shows the prediction for the target variable, i.e. for the diagnosis. \mbox{(5) The} user may choose to intervene on the concept predictions and, thus, affect the target prediction. In this example, concept predictions that were intervened on are indicated by yellow dotted lines.}\label{fig:tool_gui}
\end{figure}
\end{landscape}

\clearpage

\newpage
\section*{Acknowledgments}

We thank the members of the Medical Data Science group at ETH Zurich for stimulating discussions and feedback. Our gratitude goes to Dr. Johanna Joe and Dr. Markus Ebert from the Ultrasonography Center, KUNO, St. Hedwig Clinic Regensburg, for their help with the data acquisition. We are also sincerely thankful to Marcel Buehring and the IT Service Group of the Department of Computer Science, ETH Zurich, for their assistance with the deployment of the online prediction tool. RM was supported by the SNSF grant \#320038189096, EO was supported by the Hasler Foundation grant \#21050. This preprint was created using the LaPreprint template\footnote{\url{https://github.com/roaldarbol/lapreprint}} by \mbox{Mikkel Roald-Arb\o l \textsuperscript{\orcidlink{0000-0002-9998-0058}}.}

\section*{Author contributions}

All authors made substantial contributions to the conception and design of the study, analysis and interpretation of the data and editing the article. PRW, AP, JZ, MD and CK performed data acquisition and check. PRW performed local study coordination as the responsible investigator. PRW, RM, UK and EO performed literature review. RM, UK and EO conceived the machine learning methodology. DN preprocessed the ultrasound image data. UK and RM  designed and implemented models and experiments. KCC assisted with the data storage and computational infrastructure and designed and implemented the online prediction tool. CK and SW supervised the clinical part of the project. JEV supervised the machine learning part of the project. All authors have contributed to and read the manuscript.

\section*{Competing interests}

The authors declare no competing interests.
\section*{Introduction} \label{sec:intro}


Appendicitis is one of the most frequent causes of abdominal pain resulting in hospital admissions of patients under 18 \citep{Wier2013}. It is often among the main suspicions during the differential diagnosis in cases of unclear abdominal conditions in children and adolescents \cite{Bhangu2015,Morrow2007}. The highest incidence occurs between 10 and 11 years \cite{Rentea2017}. The diagnosis of pediatric appendicitis can be challenging and is based on a combination of clinical, laboratory and imaging parameters \cite{DiSaverio2016}. Although much research has been conducted on this topic, no specific and practically useful biomarkers for the early detection of appendicitis have been identified \cite{Acharya2016,Kiss2021}. Epidemiologically and clinically, there \newgeometry{left=2.5cm,right=2.5cm,top=2cm,bottom=2.5cm} \noindent are two forms of appendicitis: uncomplicated (subacute/exudative, phlegmonous) and complicated (gangrenous, perforated) \cite{Andersson2006,Bhangu2015,Kiss2021}. Management forms include surgery as the standard method \cite{DiSaverio2016,Gorter2016} and conservative therapy, which has been the subject of many studies \cite{Andersson2006,Svensson2012,Svensson2015,Gorter2016,CODA2020}. Spontaneous resolution of uncomplicated appendicitis is possible \cite{Andersson1994,Andersson2006,Bhangu2015,Kiss2021}, and hospital admission with close monitoring is recommended. This can help avoid complications and misdiagnoses \cite{Ohba2016, Gorter2016}.

Typical imaging modalities for suspected pediatric appendicitis include ultrasonography (US), magnetic resonance imaging (MRI), and computed tomography (CT). Due to its wide availability, lack of radiation and improvements in resolution in the last few years, abdominal US has become the primary choice \cite{Park2011}. Repeated ultrasound examinations (brightness-mode and Doppler) in the observation phase improve diagnostic accuracy and help identify disease progression \cite{Dingemann2012,Ohba2016,Gorter2016}. In addition, an abdominal MRI can be indicated as a second option in selected cases, such as obesity, suspected chronic intestinal disease, and atypical clinic presentation, particularly in adolescents \cite{Dingemann2012,Heye2020}.

There is extensive research on leveraging machine learning (ML) models to diagnose and manage patients with suspected appendicitis \citep{Hsieh2011,Deleger2013,Reismann2019,Aydin2020,Akmese2020,Stiel2020,Rajpurkar2020,Marcinkevics2021,RoigAparicio2021,Xia2022}. In brief, most models either utilize simple clinical and laboratory data \citep{Hsieh2011,Aydin2020,Akmese2020,Xia2022}, rely on hand-crafted US annotations \citep{Reismann2019,Stiel2020,Marcinkevics2021,RoigAparicio2021}, or require more expensive and invasive imaging modalities, such as CT \citep{Rajpurkar2020}. Despite having lower sensitivity and specificity than CT, US has been advocated as the preferred imaging modality for diagnosing acute appendicitis due to the absence of ionizing radiation and being real-time \citep{Mostbeck2016}. Although promising and practical, fully automated analysis of abdominal US images in this context remains an under-explored approach.

Ultrasound imaging gives natural rise to multiview and multimodal data \citep{Wang2020,Qian2021}. For instance, the risk of breast cancer may be assessed based on multiview and multimodal US images of lesions, including transversal and longitudinal views of B(rightness)-mode, color Doppler and elastography images. More generally, multiview learning \citep{Xu2013} concerns itself with the data comprising multiple views, essentially feature subsets, of the same source object. Another related field is multimodal learning \citep{Baltrusaitis2019}, which studies models combining, or fusing, multiple heterogeneous modalities, e.g. images and text. Both research directions have experienced renewed interest in the light of contrastive and self-supervised learning \citep{Tian2020,vonKuegelgen2021} and generative modeling \citep{Suzuki2022}.

Another active research direction is interpretable machine learning \citep{DoshiVelez2017,Rudin2019}, with interpretability argued to be an essential model design principle for high-stakes application domains, such as healthcare. One recently re-explored approach is prediction based on high-level and human-understandable concepts \citep{Kumar2009,Lampert2009,Koh2020} or attributes. Most frameworks for concept-based prediction assume that some high-level semantic features are provided as auxiliary supervision at training time. Usually, two models are trained, as, for instance, in concept bottleneck models (CBM) \citep{Koh2020}: \mbox{(i)} one mapping from the explanatory variables to the given concepts and \mbox{(ii)} another predicting the target variable based on the previously predicted concept values. Such concept-based models are deemed interpretable since concepts can be inspected alongside the model's final outputs and perceived as ``explanations''. Moreover, as opposed to classical multitask learning, a human user can readily intervene on and interact with such models at test time by editing concept predictions and affecting downstream output. Beyond the restricted supervised setting above, there have been several efforts at learning semantically meaningful and identifiable representations when the concepts are not given explicitly \citep{Khemakhem2020,Taeb2022}.

This article presents the first effort at leveraging ML to predict diagnosis, management and severity in pediatric patients with suspected appendicitis \emph{directly} from abdominal ultrasonography images, an imaging modality frequently used in daily clinical practice. We specifically focus on interpretable concept-based classification approaches due to their potential acceptance among clinicians and investigate the trade-off between interpretability and predictive performance. Our models were developed on the dataset acquired from an observational cohort of 579 pediatric patients admitted to the surgery department of the tertiary care Children's Hospital St. Hedwig in Regensburg, Germany. Alongside demographic, clinical, scoring and laboratory variables, the collected data include 1709 US images, i.e. multiple images, aka views, per subject. The anonymized dataset and code are publicly available on Zenodo\footnote{\url{https://doi.org/10.5281/zenodo.7669442}} and GitHub\footnote{\url{https://github.com/i6092467/semi-supervised-multiview-cbm}}, respectively. In addition, we deployed one of our best-performing models in a freely accessible educational online prediction tool.\footnote{\url{https://papt.inf.ethz.ch/mvcbm}} 

From the methods perspective, we investigate several extensions of the concept bottleneck models \citep{Koh2020} to improve their scalability to real-world medical imaging data, contributing to the recent works identifying and addressing the limitations of concept-based models \citep{Mahinpei2021,Margeloiu2021,Sawada2022,Marconato2022}. Namely, we extend conventional CBMs to the multiview classification setting (Figure~\ref{fig:workflow}) and investigate semi-supervised representation learning helpful when the set of concepts given is incomplete (in other words, insufficient), i.e. does not capture the entire predictive relationship between the images and labels, making it difficult to achieve high predictive performance. The proposed generalizations of the CBMs are not restricted to the considered use case of pediatric appendicitis and ultrasound and are readily applicable to other multiview and multimodal medical imaging datasets.

\begin{figure}[t]%
\centering
\includegraphics[width=1.0\textwidth]{src/figures/model_summary.pdf}
\caption{Schematic summary of the proposed multiview concept bottleneck model. \mbox{(I) Multiview} ultrasound images are mapped to features using a shared encoder neural network; \mbox{(II) features} are aggregated across the views; \mbox{(III) high-}level human-understandable concepts and representations are predicted based on the aggregated features; \mbox{(IV) using} concepts and representations, the target prediction is made. During training, in addition to the target prediction loss, the model is supervised by the concept prediction loss and penalized by an adversarial regularizer encouraging statistical independence between predicted concepts and representations.}
\label{fig:workflow}
\end{figure}

\section*{Methods}\label{sec:methods}

\subsection*{Data acquisition}

Data from children and adolescents aged 0-18 years, who were admitted with abdominal pain and suspected appendicitis to the Department of Pediatric Surgery and Pediatric Orthopedics at the tertiary Children's Hospital St. Hedwig in Regensburg, Germany, during the six years from January 1, 2016, to December 31, 2021, were initially collected retrospectively from the hospital's database. Only inpatients were included. Subject demographic data, clinical history records, laboratory parameters and ultrasonographic findings were documented (Appendix~\ref{app:appendicitis_data}). For patients treated operatively, surgical and histological parameters were recorded. Ultrasound images from admission and, if available, during initial and, for surgical patients, preoperative clinical course, were retrieved using the software Clinic WinData/E\&L. The images were acquired on Toshiba Xario and Aplio XG machines using Toshiba 6 MHz Convex and 12 MHz Linear transducers.

Furthermore, a prospective follow-up to identify secondary operations and late complications at least six months after discharge was performed for patients treated conservatively with diagnostic criteria for appendicitis (AS and/or PAS $\geq4$ and the appendix diameter in ultrasound $\geq6$ mm, retrospectively classified) and for patients with uncomplicated appendicitis after appendectomy (after obtaining informed consent from parents or legal representatives). Exclusion criteria included ambulant patients, other relevant diagnoses as cause for abdominal pain and relevant comorbidities, e.g. intussusception, pneumonia, neoplasia, chronic intestinal disease, simultaneous appendectomy and antibiotic treatment for a concurrent disease. For the details on patient management, see Supplementary Figure~\ref{fig:app_management_algo} (Appendix~\ref{app:appendicitis_data}). The study was approved by the Ethics Committee of the University of Regensburg (no. 18-1063-101, 18-1063\_1-101 and 18-1063\_2-101) and was performed following applicable guidelines and regulations. 

\subsection*{Data preprocessing}\label{subsec:data-preprocessing}


\paragraph{Ultrasound images}
Before model development and evaluation, B-mode ultrasound images were preprocessed to remove undesired variability. Since the current study is retrospective, we collected ultrasonograms as used in the clinical routine. Hence, the original images feature the elements of the graphical user interface, markers, distance measurements and other annotations. We employed a generative inpainting model DeepFill \citep{Yu2018}, to mask and fill such objects. Subsequently, images were cropped to 400$\times$400 px\textsuperscript{2} dimensions using zero padding when needed. Finally, the contrast-limited histogram equalization (CLAHE) was applied, pixel intensities were scaled to the range between $0$ and $1$ and normalized. Extended Data Figure~\ref{fig:us_example} shows an example of the multiple US views acquired from a single subject from our cohort before and after preprocessing. In addition, all models trained on the appendicitis data leveraged extensive on-the-fly augmentation (Appendix~\ref{app:imp}).

\subsection*{Multiview concept-based classification}\label{subsec:MVCBM}

Throughout the remaining sections, we will assume the following setting and notation. 
Consider a dataset comprising $N$ triples $\left(\left\{\boldsymbol{x}_i^{v}\right\}_{v=1}^{V_i},\, \boldsymbol{c}_i,\,y_i\right)$, for $1\leq i\leq N$, with view sequences $\left\{\boldsymbol{x}_i^{v}\right\}_{v=1}^{V_i}$, concept vectors $\boldsymbol{c}_i\in\mathbb{R}^K$ provided at training time and labels $y_i$. Note that the number of views $V_i\geq1$ may vary across data points $1\leq i\leq N$. We will concentrate on the scenario where all views can be preprocessed and rescaled into the same dimensionality. Nevertheless, our approach can be extended to heterogeneous data types.

Motivated by medical imaging applications, we focus on the data exhibiting characteristics described informally below. \mbox{(i) \emph{Partial}} \emph{observability}: not all concepts are identifiable from all views. \mbox{(ii) \emph{View}} \emph{homogeneity}: most views contain a considerable amount of shared information and are visually similar. \mbox{(iii) \emph{View}} \emph{ordering}: views belonging to the same data point may be loosely ordered, e.g. spatially, temporally, or based on their importance for predicting the label. These properties are inspired by the multiview ultrasound dataset explored in our experiments and support some design choices described below.

\subsection*{Multiview concept bottleneck models}

Below, we introduce an extension of the concept bottleneck models \citep{Koh2020} to the multiview classification setting. Henceforth, we refer to this extension as multiview concept bottleneck model (MVCBM). Figure~\ref{fig:workflow} provides a schematic summary of the architecture. Note that representation learning and adversarial regularization modules are discussed later, in the context of the extended semi-supervised variant of the MVCBM. Equations~\ref{eqn:mvcbm_interim}--\ref{eqn:mvcbm_target} specify a forward pass of the model. In brief, MVCBM consists of the following modules: \mbox{(i) per-view} feature extraction; \mbox{(ii) feature} fusion; \mbox{(iii) concept} prediction and \mbox{(iv) label} prediction. To tackle cases when the given set of concepts is incomplete, for instance, due to the lack of domain knowledge or the cost of acquiring additional annotation, we also introduce a semi-supervised MVCBM (SSMVCBM) that, in addition to utilizing the given concepts, learns an independent representation predictive of the label.

As a summary, for data point $1\leq i\leq N$, a forward pass of the multiview concept bottleneck is given by the following equations:
\begin{subequations}
    \begin{align}
        \textrm{(i) Feature}&\textrm{{ }extraction: }\notag \\
        \boldsymbol{h}^{v}_{i}&=\boldsymbol{h}_{\boldsymbol{\psi}}\left(\boldsymbol{x}_i^{v}\right),\, 1\leq v\leq V_i,\label{eqn:mvcbm_interim}\\
        \textrm{(ii) Feature}&\textrm{{ }fusion: } \notag \\ 
        \boldsymbol{\bar{h}}_i&=\boldsymbol{r}_{\boldsymbol{\xi}}\left(\left\{\boldsymbol{h}^{v}_{i}\right\}_{v=1}^{V_i}\right),\label{eqn:mvcbm_fusion}\\
        \textrm{(iii) Concep}&\textrm{t{ }prediction: } \notag \\ 
        \boldsymbol{\hat{c}}_i&=\boldsymbol{s}_{\boldsymbol{\zeta}}\left(\boldsymbol{\bar{h}}_i\right),\label{eqn:mvcbm_concepts}\\
        \textrm{(iv) Label p}&\textrm{rediction: } \notag \\ 
        \hat{y}_i&=f_{\boldsymbol{\theta}}\left(\boldsymbol{\hat{c}}_i\right),\label{eqn:mvcbm_target}
    \end{align}
    \label{eqn:mvcbm}
\end{subequations}
where Latin letters correspond to functions and variables and Greek letters denote learnable parameters. Observe that parameters $\boldsymbol{\phi}=\left\{\boldsymbol{\psi}, \boldsymbol{\xi}, \boldsymbol{\zeta}\right\}$ define the concept model $\boldsymbol{g}_{\boldsymbol{\phi}}(\cdot)$ mapping a multiview feature sequence to the predicted concept values; whereas $f_{\boldsymbol{\theta}}(\cdot)$ is the target model, linking concepts and labels. Thus, similar to the vanilla concept bottleneck, MVCBM's forward pass can be rewritten as $\hat{y}_i=f_{\boldsymbol{\theta}}\left(\boldsymbol{g}_{\boldsymbol{\phi}}\left(\left\{\boldsymbol{h}^{v}_{i}\right\}_{v=1}^{V_i}\right)\right)$. In the following paragraphs, we detail each of the steps in Equation~\ref{eqn:mvcbm}.

\paragraph{Feature extraction}

Given an ordered view sequence $\left\{\boldsymbol{x}_i^{v}\right\}_{v=1}^{V_i}$, we first encode each view into a lower-dimensional representation (Equation~\ref{eqn:mvcbm_interim}). We use a \emph{shared} encoder neural network, denoted by $\boldsymbol{h}_{\boldsymbol{\psi}}(\cdot)$. Weight sharing is justified by the view homogeneity and could be helpful in smaller datasets with high missingness of views. On the other hand, if the dataset is relatively regular and the differences across views are considerable, especially in the multimodal setting, one could train a dedicated encoder for each view. In practice, it may be prudent to use a pretrained model to initialize $\boldsymbol{h}_{\boldsymbol{\psi}}(\cdot)$, e.g. the use of ResNet and VGG architectures pretrained on natural images is standard for medical imaging applications \citep{Cheplygina2019}. As a result,  we obtain a sequence of view-specific features given by $\boldsymbol{h}^{v}_{i}=\boldsymbol{h}_{\boldsymbol{\psi}}\left(\boldsymbol{x}_i^{v}\right)$ for $1\leq v\leq V_i$ and $1\leq i \leq N$.

\paragraph{Feature fusion}

To handle multiple views, we need to fuse, i.e. aggregate, them within the model (Equation~\ref{eqn:mvcbm_fusion}). MVCBM follows a \emph{hybrid} fusion approach \citep{Baltrusaitis2019}: rather than concatenating views at the input level (\emph{early fusion}) or training an ensemble of view-specific models (\emph{late fusion}); we aggregate intermediate view-specific features $\boldsymbol{h}^{v}_{i}$ from the previous step within a single neural network. Although there are many viable fusion functions, in our context, the fusion must handle varying numbers of views per data point. As a naive approach, we consider arithmetic mean across the views $\boldsymbol{\bar{h}}_i=\frac{1}{V_i}\sum_{v=1}^{V_i}\boldsymbol{h}^{v}_{i}$ \citep{Havaei2016}. More generally,  $\boldsymbol{\bar{h}}_i=\boldsymbol{r}_{\boldsymbol{\xi}}\left(\left\{\boldsymbol{h}^{v}_{i}\right\}_{v=1}^{V_i}\right)$, where $\boldsymbol{\bar{h}}_i$ denotes the fused feature vector and $\boldsymbol{r}_{\boldsymbol{\xi}}(\cdot)$ is the fusion function with parameters $\boldsymbol{\xi}$. Considering partial observability of the concepts and ordering of the views, we, in addition, investigate aggregation via a \emph{learnable} function. Similar to Ma et al. \citep{Ma2019}, who utilize this trick in multiview 3D shape recognition, we combine view-specific representations via a long short-term memory (LSTM) network. In particular, we set the aggregated representation $\boldsymbol{\bar{h}}_i$ to the last hidden state of the view sequence, i.e. at ``time step'' $V_i$. Note that both averaging and LSTM can handle varying numbers of views. Nevertheless, there are other options for $\boldsymbol{r}_{\boldsymbol{\xi}}(\cdot)$, e.g. Hadamard product or weighted average, the investigation of which we leave for future work.

\paragraph{Concept and label prediction}

The last two steps are similar to the vanilla concept bottleneck (Equations~\ref{eqn:mvcbm_concepts}--\ref{eqn:mvcbm_target}). First, we predict concepts $\boldsymbol{\hat{c}}_i$ based on the fused representation $\boldsymbol{\bar{h}}_i$, using a concept encoder network $\boldsymbol{s}_{\boldsymbol{\zeta}}(\cdot)$ parameterized by $\boldsymbol{\zeta}$. Note that the choice of activation functions at the output of $\boldsymbol{s}_{\boldsymbol{\zeta}}(\cdot)$ depends on the type of concepts and should be adapted to whether an individual concept is categorically or continuously valued. The vector $\hat{\boldsymbol{c}}_i$ is then used as an input to the target model $f_{\boldsymbol{\theta}}(\cdot)$, predicting the label $\hat{y}$. The output activation should be chosen based on the downstream task: classification or regression.

\paragraph{Loss function and optimization}

The parameters of vanilla CBMs can be optimized using independent, sequential and joint procedures \citep{Koh2020}. In this work, we focus on the sequential and joint approaches since they offer a more balanced trade-off between predictive performance and intervenability. 

In the \emph{sequential} training, we first optimize the concept model parameters:
\begin{equation}
   \boldsymbol{\hat{\phi}}=\arg\min_{\boldsymbol{\phi}}\sum_{i=1}^N\sum_{k=1}^{K}w_i^t w_i^{c_k}\mathcal{L}^{c_k}(\hat{c}_{i,k}, c_{i,k}),
   \label{eqn:opt_seq_1}
\end{equation}
where $\mathcal{L}^{c_k}(\cdot,\,\cdot)$ is the loss function for the $k$-th concept, e.g. one could use the cross-entropy for categorically valued and squared error for a continuously valued concept, and $c_{i,k}$ refers to the value of the $k$-th concept for the $i$-th data point. Additionally, to address potential imbalances in the concept distributions and sparsity of specific concept-target combinations, we have introduced weights $w_i^{c_k}$ for the $k$-th concept and $w_i^t$ for the target variable of the $i$-th point, s.t. $\sum_{i=1}^N\sum_{k=1}^Kw_{i}^{c_k}=1$ and $\sum_{i=1}^Nw_i^t=1$. In practice, these weights can be set to the normalized inverse counts of samples in the corresponding variable classes, i.e. $w^t_i\propto1/\sum_{j=1}^N\boldsymbol{1}_{\left\{y_j=y_i\right\}}$ and $w^{c_k}_i\propto1/\sum_{j=1}^N\boldsymbol{1}_{\left\{c_{j,k}=c_{i,k}\right\}}$, where $\boldsymbol{1}_{\{\cdot\}}$ is the indicator function. However, other sample weighting schemes are viable.  Next, parameters $\boldsymbol{\hat{\phi}}$ are frozen, and the parameters of the target model $f_{\boldsymbol{\theta}}$ are optimized: 
\begin{equation}
    \boldsymbol{\hat{\theta}}=\arg\min_{\boldsymbol{\theta}}\sum_{i=1}^Nw_i^t\mathcal{L}^t\left(f_{\boldsymbol{\theta}}\left(\boldsymbol{\hat{c}}_i\right), y_i\right),
    \label{eqn:opt_seq_2}
\end{equation}
where $\mathcal{L}^t(\cdot,\,\cdot)$ is the loss function for the target task, and $\boldsymbol{\hat{c}}_i$ are predictions made by the frozen concept model $\boldsymbol{g}_{\boldsymbol{\hat{\phi}}}(\cdot)$.

For the \emph{joint} training, we combine the loss functions from Equations \ref{eqn:opt_seq_1}--\ref{eqn:opt_seq_2} into a single objective: 
\begin{equation}
    \boldsymbol{\hat{\phi}},\,\boldsymbol{\hat{\theta}}=\arg\min_{\boldsymbol{\phi},\,\boldsymbol{\theta}}\Bigg\{\sum_{i=1}^N w^t_i\mathcal{L}^t(\hat{y}_i,y_i)+\alpha\sum_{i=1}^N\sum_{k=1}^{K}w^t_iw_i^{c_k}\mathcal{L}^{c_k}(\hat{c}_{i,k}, c_{i,k})\bigg\}, 
    \label{eqn:opt_joint}
\end{equation}
where $\alpha>0$ controls the trade-off between target and concept predictive performance. Observe that parameters $\boldsymbol{\phi}$ and $\boldsymbol{\theta}$ are optimized simultaneously.

\paragraph{Intervenability}

A salient difference between CBMs and multitask models is that a practitioner utilizing a CBM model can interact with it by intervening on concept predictions, e.g. ``correcting'' the model by setting the predicted values to the ground truth $\hat{c}_{i,k}\defeq c_{i,k}$. In particular, for a data point $1\leq i\leq N$, the updated prediction after the intervention on the concepts from a subset $\mathcal{S}\subseteq\left\{1,...,K\right\}$ is given by
\begin{equation}
    \hat{y}_i^{\mathcal{S}}=f_{\boldsymbol{\hat{\theta}}}\left(\boldsymbol{\hat{c}}_{\left\{1,...,K\right\}\setminus\mathcal{S}},\,\boldsymbol{c}_{\mathcal{S}}\right),
    \label{eqn:intervention}
\end{equation}
where $\boldsymbol{\hat{c}}$ and $\boldsymbol{c}$ refer to the predicted and ground truth concept vectors, respectively. Note the notation abuse in the order of the arguments in $f_{\boldsymbol{\hat{\theta}}}(\cdot)$.

\paragraph{Semi-supervised multiview concept bottleneck models}

As mentioned before, the set of $K$ concepts given at the training time might be incomplete, for instance, due to the high cost of annotation, the lack of knowledge or the unethicality of measuring certain variables. More formally, concept bottlenecks implicitly assume that concepts are a sufficient statistic for the target variable \cite{Yeh2020}; in other words, $\boldsymbol{x}\perp \!\!\! \perp y\,\vert\,\boldsymbol{c}$. When this is not the case, the predictive performance of the CBM is limited since the model solely relies on the predefined set of concepts which is insufficient. To address this limitation, we propose a semi-supervised variant of the MVCBM (Figure~\ref{fig:workflow}) that additionally learns representations complementary to the concepts and relevant to the downstream prediction task.

\newpage

Next to the feature extraction and concept prediction, SSMVCBM includes an unsupervised module mapping views $\left\{\boldsymbol{x}_i^v\right\}^{V_i}_{v=1}$ to the representation $\boldsymbol{\hat{z}}_i\in\mathbb{R}^J$. To predict the label, $\boldsymbol{\hat{c}}_i$ and $\boldsymbol{\hat{z}}_i$ are concatenated and fed into the target model. Thus, a forward pass of the SSMVCBM is given by 
\begin{subequations}
    \begin{align}
        \textrm{(i) Feature}&\textrm{{ }extraction: }\notag \\
        \boldsymbol{h}^{\boldsymbol{c},v}_{i}&=\boldsymbol{h}_{\boldsymbol{\psi}^{\boldsymbol{c}}}\left(\boldsymbol{x}_i^{v}\right),\,1\leq v\leq V_i,\label{eqn:ssmvcbm_interim}\\
        \boldsymbol{h}^{\boldsymbol{z},v}_{i}&=\boldsymbol{h}_{\boldsymbol{\psi}^{\boldsymbol{z}}}\left(\boldsymbol{x}_i^{v}\right),\, 1\leq v\leq V_i,\notag \\
        \textrm{(ii) Feature}&\textrm{{ }fusion: } \notag \\ 
        \boldsymbol{\bar{h}}^{\boldsymbol{c}}_i&=\boldsymbol{r}_{\boldsymbol{\xi}^{\boldsymbol{c}}}\left(\left\{\boldsymbol{h}^{\boldsymbol{c},v}_{i}\right\}_{v=1}^{V_i}\right),\label{eqn:ssmvcbm_fusion}\\
        \boldsymbol{\bar{h}}^{\boldsymbol{z}}_i&=\boldsymbol{r}_{\boldsymbol{\xi}^{\boldsymbol{z}}}\left(\left\{\boldsymbol{h}^{\boldsymbol{z},v}_{i}\right\}_{v=1}^{V_i}\right),\notag \\
        \textrm{(iii) Concep}&\textrm{t{ }and representation prediction: } \notag \\ 
        \boldsymbol{\hat{c}}_i&=\boldsymbol{s}_{\boldsymbol{\zeta}^{\boldsymbol{c}}}\left(\boldsymbol{\bar{h}}^{\boldsymbol{c}}_i\right),\label{eqn:ssmvcbm_concepts}\\
        \boldsymbol{\hat{z}}_i&=\boldsymbol{s}_{\boldsymbol{\zeta}^{\boldsymbol{z}}}\left(\boldsymbol{\bar{h}}^{\boldsymbol{z}}_i\right),\notag \\
        \textrm{(iv) Label p}&\textrm{rediction: } \notag \\ 
        \hat{y}_i&=f_{\boldsymbol{\theta}}\left(\left[\boldsymbol{\hat{c}}_i, \boldsymbol{\hat{z}}_i\right]\right),\label{eqn:ssmvcbm_target}
    \end{align}
    \label{eqn:ssmvcbm}
\end{subequations}
where variables and parameters superscripted by $\boldsymbol{c}$ and $\boldsymbol{z}$ correspond to the concept and representation learning modules, respectively.

To avoid learning a representation redundant to the concepts, it is desirable that $\boldsymbol{\hat{c}}\perp \!\!\! \perp\boldsymbol{\hat{z}}\,\rvert\, y$, i.e. the predicted concepts and unsupervised representations should be statistically independent conditional on the label. Concretely, we use another neural network $\boldsymbol{a}_{\boldsymbol{\tau}}:\:\mathbb{R}^J\rightarrow \mathbb{R}^K$, parameterized by weights $\boldsymbol{\tau}$, to quantify the degree of statistical dependence as $\max_{\boldsymbol{\tau}}\mathrm{corr}\left(\boldsymbol{a}_{\boldsymbol{\tau}}\left(\boldsymbol{\hat{z}}\right),\,\boldsymbol{\hat{c}}\right)$ \citep{Adeli2021}. Thus, network $\boldsymbol{a}_{\tau}$ is used to adversarially regularize representation $\boldsymbol{\hat{z}}$. Empirically, we observed that this regularization scheme helps de-correlate $\boldsymbol{\hat{z}}$ from concept predictions and improves the model's intervenability (see Appendix~\ref{app:ssmvcbm_abl} in the Supplementary for an ablation study).

The procedure to train SSMVCBMs is outlined in Supplementary Algorithm~\ref{alg:opt_ssmvcbm} (Appendix~\ref{app:ssmvcbm}). Similar to the sequential optimization for (MV)CBMs (Equations~\ref{eqn:opt_seq_1}--\ref{eqn:opt_seq_2}), it consists of multiple steps. First, parameters $\boldsymbol{\phi}^{\boldsymbol{c}}=\left\{\boldsymbol{\psi}^{\boldsymbol{c}},\,\boldsymbol{\xi}^{\boldsymbol{c}},\,\boldsymbol{\zeta}^{\boldsymbol{c}}\right\}$ involved in concept prediction are optimized using the loss function analogous to Equation~\ref{eqn:opt_seq_1}. Then, we fix $\boldsymbol{\hat{\phi}}^{\boldsymbol{c}}$ and optimize parameters $\boldsymbol{\phi}^{\boldsymbol{z}}=\left\{\boldsymbol{\psi}^{\boldsymbol{z}},\,\boldsymbol{\xi}^{\boldsymbol{z}},\,\boldsymbol{\zeta}^{\boldsymbol{z}}\right\}$ by solving the following problem:
\begin{equation}
    \boldsymbol{\hat{\phi}}^{\boldsymbol{z}},\boldsymbol{\tilde{\theta}}=\arg\min_{\boldsymbol{\phi}^{\boldsymbol{z}},\boldsymbol{\theta}}\max_{\boldsymbol{\tau}}\sum_{i=1}^Nw_i^t\mathcal{L}^t\left(\hat{y}_i, y_i\right)-\lambda\sum_{i=1}^N\sum_{k=1}^Kw_i^{c_k}\mathcal{L}^{c_k}\left(\left[a_{\boldsymbol{\tau}}\left(\boldsymbol{\hat{z}}_i\right)\right]_k,\hat{c}_{i,k}\right),
    \label{eqn:opt_ssmvcbm_minimax}
\end{equation}
where $\lambda>0$ is a tuning parameter corresponding to the weight of the adversarial regularizer. The loss function above can be extended with further regularization terms, e.g. to de-correlate individual dimensions of $\boldsymbol{\hat{z}}$ \citep{Cogswell2016}, facilitating a more straightforward interpretation. In practice, the minimax objective is optimized using adversarial training similarly to the generative adversarial networks \citep{Goodfellow2020} (Supplementary Algorithm~\ref{alg:opt_ssmvcbm}). Last but not least, parameters of the target model are re-optimized (cf. Equation~\ref{eqn:opt_seq_2}), treating $\boldsymbol{\hat{\phi}}^{\boldsymbol{c}}$ and $\boldsymbol{\hat{\phi}}^{\boldsymbol{z}}$ as fixed: $\boldsymbol{\hat{\theta}}=\arg\min_{\boldsymbol{\theta}}\sum_{i=1}^Nw_i^t\mathcal{L}^t\left(f_{\boldsymbol{\theta}}\left(\left[\boldsymbol{\hat{c}_i},\boldsymbol{\hat{z}_i}\right]\right),y_i\right)$.

\subsection*{Experimental setup}

The purpose of our experiments was twofold: \mbox{(i) to} provide a proof of concept for the introduced extensions of the CBMs on simple benchmarks and \mbox{(ii) to} apply our techniques to a real-world medical imaging dataset. In the following, we provide a more detailed overview of the experimental setup.

\paragraph{Datasets and validation scheme}

As the simplest benchmark, we set up a synthetic nonlinear classification problem (Appendix~\ref{app:synthetic}). The generative process of this dataset was defined directly based on the classical concept bottleneck model: \mbox{(i) a} design matrix is sampled, \mbox{(ii) features} are mapped to concepts, and \mbox{(iii) concepts} are used to construct the labels. In addition, multiple ``views'' are constructed, each comprising a subset of the original feature set. Naturally, this dataset favors multiview approaches. Its essential advantage over the conventional benchmarks from the literature, such as the UCSD Birds, is the presence of reliable per-data-point concept labels.

As mentioned, we also adapted a popular natural image attribute-based  \emph{Animals with Attributes 2} dataset \citep{Lampert2009} (Appendix~\ref{app:mvawa}) to the multiview classification. Namely, we randomly cropped four patches from each original image, thus, constructing multiview data points. Like in the UCSD birds experiment by Koh et al. \citep{Koh2020}, herein, the attributes are given per class rather than per data point, e.g. all polar bears are assumed to have white fur. The main challenge of this dataset is that only some concepts may be identifiable from every view because cropping may remove an image region with the input relevant to a specific concept. During model comparison, for both the synthetic data described above and multiview animals, we trained and evaluated classifiers by performing a train-test split on several independent simulations, i.e. replicates. 

Finally, to demonstrate the utility of the proposed methods on real-world data, we employed ultrasound imaging and tabular clinical, lab and score data from pediatric patients with suspected appendicitis. We explored three different target variables encompassing the diagnosis, treatment assignment and complications. For a detailed description of this dataset, we refer the reader to the previous sections and Appendix~\ref{app:appendicitis_data}. For model validation and comparison, we divided the data according to the \mbox{90\%-10\%} train-test split. Hyperparameter tuning was performed only on the training set using five-fold cross-validation (the final hyperparameter values are reported in Supplementary Tables~\ref{tab:parameters_synthetic}--\ref{tab:parameters_app_severity}, Appendix~\ref{app:imp}).

\paragraph{Ablations, baselines and evaluation}

In the experiments, we compared several variations of the proposed multiview concept bottlenecks to better understand the role of design choices made. Namely, we trained models using the sequential \mbox{(MVCBM-seq)} and joint \mbox{(MVCBM-joint)} optimization procedures (Equations~\ref{eqn:opt_seq_1}--\ref{eqn:opt_joint}). We also compared the semi-supervised extension \mbox{(SSMVCBM)} to the basic MVCBM (Equation~\ref{eqn:ssmvcbm}). To facilitate meaningful comparison, we purposefully trained models under insufficient concept sets to observe if the SSMVCBM could achieve any performance improvement over the MVCBM. Furthermore, two fusion functions were investigated: the arithmetic mean \mbox{((SS)MVCBM-avg)} and LSTM \mbox{((SS)MVCBM-LSTM)}. Last but not least, like Koh et al. \citep{Koh2020}, we experimented with intervening on the concept bottlenecks by replacing the predicted concept values with the ground truth at test time. The goal was to investigate whether a medical practitioner utilizing a concept-based model could improve its predictions interactively.

In addition to the ablations above, we benchmarked the performance of the (SS)MVCBMs against several baselines. Across all datasets, we applied single-view neural-network-based classifiers. Namely, we trained MLPs for tabular data and fine-tuned ResNet-18 on images. Vanilla CBMs served as an interpretable single-view baseline. Notably, for CBMs and (SS)MVCBMs, the same architectures of individual modules were used to facilitate fair comparison. As a black-box multiview baseline, we employed a neural network with the same architecture as for the MVCBM but trained without concept supervision in the bottleneck layer. We refer to this model as a multiview bottleneck (MVBM). Similarly, as for its interpretable counterpart, we compared two ways of aggregating per-view representations: averaging and LSTM. Lastly, specific to the pediatric appendicitis dataset, we also investigated a random forest classifier \cite{Breiman2001} trained on the radiomics features extracted from a single US view.

Since the intended use case of our models in healthcare applications is decision support rather than decision-making, in our experiments, we focused on evaluating concept and label predictions as scores using AUROCs and AUPRs. Notably, for pediatric appendicitis, differing performance metrics may be relevant depending on the target variable, e.g. for the diagnosis and severity, a low false negative rate is crucial, whereas, for the management, a low false positive rate might be desirable to avoid negative appendectomies \citep{Kryzauskas2016}.

\subsection*{Implementation details}

We implemented MVCBM and SSMVCBM in PyTorch (v 1.11.0) \citep{Paszke2019}. Across all experiments and models, when applicable, we fine-tuned pretrained \mbox{ResNet-18 \citep{He2016}} as the shared view encoder. For the concept encoder and target model, we utilized MLPs with ReLU hidden activations. Detailed architecture specifications are provided in Appendix~\ref{app:imp}. We used the \emph{PyRadiomics} \mbox{package \citep{vanGriethuysen2017}} for radiomic feature extraction. Features were extracted from the whole images without prior segmentation of the region of interest since image segmentation is beyond the scope of the current work. We computed first-order statistics, gray level size zone and gray level run length matrix features from the original and square-filtered images. Random forests were trained with a cost-sensitive loss function to account for class imbalance. ANOVA $F$-value-based feature selection was performed using nested cross-validation to improve the performance of this baseline further. The remainder of the implementation details can be found in Appendix~\ref{app:imp} and within the published code and documentation.
\section*{Results}\label{sec:results}

\subsection*{Cohort and dataset description}

In our retrospective study, we analyzed the data from a cohort of 579 patients, aged 0--18 years (Figures~\ref{fig:data_overview}b and \ref{fig:data_overview}d) and admitted with abdominal pain to Children's Hospital St. Hedwig in Regensburg, Germany. The current study builds on and extends the initial analysis of the tabular data from a smaller cohort of patients published earlier \citep{Marcinkevics2021}. Figure~\ref{fig:data_overview} provides an overview of the dataset used in the final analysis. 

We acquired (potentially multiple) abdominal B-mode ultrasound images for each patient, in total 1709 images were included. The number of views per subject ranges from 1 to 15 (Figure~\ref{fig:data_overview}c); the images depict various regions of interest, such as the abdomen's right lower quadrant (RLQ), appendix, intestines, lymph nodes and reproductive organs (Extended Data Figure~\ref{fig:us_example}). Alongside multiple US images, for each subject, we retrieved information encompassing laboratory tests, physical examination results, clinical scores, such as Alvarado (AS) and pediatric appendicitis (PAS) scores, and expert-produced ultrasonographic findings represented by categorically-valued features (Figure~\ref{fig:data_overview}a). A subset of the latter was identified as high-level concepts relevant to decision support (Extended Data Table~\ref{tab:concept_distribution}). 

\begin{figure}[ht!]%
\centering

\subfigure[]{
    \includegraphics[width=1.0\linewidth]{src/figures/data_summary.pdf}
}

\subfigure[]{
    \input{src/tables/contingency.tex}
}

\subfigure[]{
    \includegraphics[width=0.46\linewidth]{src/figures/num_views_dist.pdf}
}
\subfigure[]{
    \includegraphics[width=0.45\linewidth]{src/figures/age_dist.pdf}
}

\caption{Overview of the pediatric appendicitis dataset. \mbox{(a) The} dataset structure: for each pediatric patient from the cohort, we acquired multiple ultrasound images, aka views, tabular data comprising laboratory, physical examination, scoring results and ultrasonographic findings extracted manually by the experts, and three target variables, namely, diagnosis, management and severity. \mbox{(b) The contingency} table of the \mbox{management (M)} by \mbox{severity (S)} stratified by the \mbox{diagnosis (D)}. (c--d) Distributions of the number of US images acquired per patient and subjects' age.}
\label{fig:data_overview}
\end{figure}

Lastly, the subjects were labeled w.r.t. three target variables: \mbox{(i) diagnosis} (\emph{appendicitis} vs. \emph{no appendicitis}), \mbox{(ii) management} (\emph{surgical} vs. \emph{conservative}) and \mbox{(iii) severity} (\emph{complicated} vs. \emph{uncomplicated or no appendicitis}). The diagnosis was confirmed histologically in the patients who underwent appendectomy. Subjects treated conservatively were labeled as having appendicitis if their appendix diameter was at least \mbox{6 mm} and either AS or PAS were at least 4. The management label reflects the decision made by a senior pediatric surgeon based on clinical, laboratory and US data. For the severity, complicated appendicitis includes cases with abscess formation, gangrene or perforation. In Appendix~\ref{app:appendicitis_data}, we provide a more comprehensive description of the dataset and its acquisition. Note that the analysis below utilizes only ultrasound images and findings extracted from them. Our goal was to explore US image analysis and its benefits for predictive models for pediatric appendicitis. Nevertheless, we publicize the entire dataset, including modalities other than imaging.

\newpage

\subsection*{Proof of concept on (semi-)synthetic data}

To test the feasibility of the proposed concept-based multiview classification approaches, we first applied them to toy (semi-)synthetic data, comprising multiview nonlinear synthetic (Appendix~\ref{app:synthetic}) and multiview animals with attributes (MVAwA; Appendix~\ref{app:mvawa}) datasets. Both problems feature binary concepts that are identifiable from the given multiview observations. Although, herein, concept and target prediction are classification problems, all methods present are easily extendable to regression. In these experiments, we assessed the models' performance at \mbox{(i) target}, \mbox{(ii) concept} prediction and \mbox{(iii) the} effectiveness of interventions on the predicted concepts. Furthermore, to explore the scenario where the set of concepts is incomplete, we purposefully trained the models on concept subsets of varying sizes. We compared the performance against single- and multiview black-box classifiers and the vanilla concept bottleneck \citep{Koh2020}. Figure~\ref{fig:synthetic_res} contains the summary of the results for both datasets.

\paragraph{Synthetic nonlinear data}

The first benchmark we considered was tabular synthetic nonlinear data simulated from a model very similar to that of MVCBM (Appendix~\ref{app:synthetic}). As expected, black-box and concept-based multiview approaches are consistently more accurate than their single-view counterparts at target (Figure~\ref{fig:synthetic_res}a) and concept prediction (Figure~\ref{fig:synthetic_res}b). Namely, a multiview bottleneck model without concept supervision (MVBM) performs considerably better than a multilayer perceptron trained on a single view (MLP); similarly, a multiview concept bottleneck (MVCBM) outperforms a simple CBM. Notably, the target prediction accuracy for CBM and MVCBM increases with the number of concepts given (Figure~\ref{fig:synthetic_res}a). When almost a complete concept set is provided, the performance of the  multiview CBM is comparable to that of the multiview black-box classifier. The semi-supervised MVCBM (SSMVCBM) performs well even when very few concepts are known and is close to the black-box baseline in all settings. 

For the concept prediction (Figure~\ref{fig:synthetic_res}b), MVCBM and SSMVCBM attain comparable performance with higher areas under receiver operating characteristic curves (AUROC) than the single-view model. Lastly, we observe that similarly to the classical CBM, both multiview variants are intervenable (Figure~\ref{fig:synthetic_res}c), i.e. their predictive performance improves when replacing predicted concepts with the ground truth at test time. In addition to the results above, Appendix~\ref{app:ssmvcbm_abl} contains an ablation study exploring the SSMVCBM in more detail.

\paragraph{Multiview animals}

Another benchmark dataset we have used for model comparison is multiview animals with attributes comprising cropped animal images (Supplementary Figure~\ref{fig:mvawa_ex}) accompanied by 85 high-level binary attributes and a 50-class label. For this dataset,  the experiment results are very similar to the ones above: \mbox{(i) multiview} techniques perform superior to single-view techniques (Figures~\ref{fig:synthetic_res}d--e); \mbox{(ii) when} given the complete concept set, MVCBM is comparable to an end-to-end black-box (Figure~\ref{fig:synthetic_res}d); and \mbox{(iii) the} proposed multiview and semi-supervised extensions of the CBM are intervenable (Figure~\ref{fig:synthetic_res}f). Herein and above, for MVCBMs, we focused on a simple approach to aggregating multiple views and a single optimization procedure; however, other design choices are plausible. Extended Data Table~\ref{tab:mvawa_full_res} reports additional results with alternative fusion functions and optimization schemes for the MVAwA experiment under the complete concept set.

\subsection*{Application to pediatric appendicitis}

As mentioned before, our multiview concept bottleneck models are readily applicable to medical imaging datasets, which, in practice, often include multiple views and heterogeneous data types. In the following, we explore the application of the multiview CBMs to the pediatric appendicitis dataset. In addition to deep-learning- and concept-based approaches, we considered an alternative baseline predictive model: a random forest (RF) \cite{Breiman2001} fitted on radiomic features \cite{vanGriethuysen2017} extracted from the sonograms. 

\paragraph{Predicting high-level ultrasound features}

We first evaluated the ability of all concept-based models to predict high-level appendix ultrasound features (Extended Data Table~\ref{tab:concept_distribution}) from (multiple) abdominal US images. Table~\ref{tab:appendicitis_concept_res} contains test-set AUROCs and areas under precision-recall curves (AUPR) achieved by the different variants of the concept bottleneck. In addition to comparing vanilla CBMs to their multiview and semi-supervised extensions, we investigated the effect of the optimization procedure, sequential vs. joint, and view-specific feature fusion, averaging vs. long short-term memory (LSTM). The models included in Table~\ref{tab:appendicitis_concept_res} were trained to predict the diagnosis (\emph{appendicitis} vs. \emph{no appendicitis}); however, we 
\begin{landscape}
\begin{figure}[h!]
\centering

\subfigure[]{
    \includegraphics[width=0.3\linewidth]{src/figures/synthetic_target_prediction.pdf}
}
\subfigure[]{
    \includegraphics[width=0.3\linewidth]{src/figures/synthetic_concept_prediction.pdf}
}
\subfigure[]{
    \includegraphics[width=0.3\linewidth]{src/figures/synthetic_interventions.pdf}
}

\subfigure[]{
    \includegraphics[width=0.31\linewidth]{src/figures/mvawa_target_prediction.pdf}
}
\subfigure[]{
    \includegraphics[width=0.3\linewidth]{src/figures/mvawa_concept_prediction.pdf}
}
\subfigure[]{
    \includegraphics[width=0.3\linewidth]{src/figures/mvawa_interventions.pdf}
}

\caption{Target and concept prediction results on (semi-)synthetic data for the proposed multiview concept bottleneck (MVCBM) and semi-supervised multiview concept bottleneck (SSMVCBM) models alongside several baselines. \mbox{(a--c) The} top row of panels shows the results for the tabular synthetic dataset. \mbox{(d--f) The} bottom row displays the results for the semi-synthetic multiview animals with attributes (MVAwA). \mbox{(a,d) One-}vs-all AUROCs for predicting the target on the test data under the varying number of observed concepts. Box plots were produced across ten independent simulations. \mbox{(b,e) AUROCs} for predicting concepts on the test data under the varying number of observed concepts. AUROCs were averaged across the observed concepts; box plots were produced across ten independent simulations. \mbox{(c,f) AUROCs} for predicting the target on the test data after intervening on the varying number of concepts. Confidence bands correspond to interquartile ranges across ten independent simulations and several randomly sampled concept subsets. The intervention experiment was performed for 5/30 and  10/85 observed concepts for synthetic and MVAwA datasets, respectively.}
\label{fig:synthetic_res}
\end{figure}
\end{landscape}
\input{src/tables/concepts_diagnosis.tex}
\noindent observed similar results for the management and severity (Extended Data Tables~\ref{tab:appendicitis_concept_res_management} and \ref{tab:appendicitis_concept_res_severity}). Minor discrepancies across the targets are attributable to the differences in the weights assigned to data points in the cost-sensitive loss function (Equations~\ref{eqn:opt_seq_1}--\ref{eqn:opt_joint} and \ref{eqn:opt_ssmvcbm_minimax}) and the choice of hyperparameter values (Supplementary Tables~\ref{tab:parameters_synthetic}--\ref{tab:parameters_app_severity}, Appendix~\ref{app:imp}).

Across all target variables, most concepts could be predicted by at least one of the models significantly better than by a fair coin flip (one-sample two-sided $t$-test $p$-value $<0.05$, adjusted using the Benjamini--Yekutieli procedure with the FDR of $q=0.05$). Surprisingly, some of the variables with relatively few cases present in the dataset could be captured by some models, e.g. \emph{coprostasis} ($c_8$) and \emph{meteorism} ($c_9$) by the LSTM-based variants of MVCBM and SSMVCBM. On the other hand, the \emph{thickening of the bowel wall} ($c_7$) was particularly challenging to model, likely due to its low prevalence and the lack of predictive power in the downstream classification task: some models trained with the severity as the target variable were able to perform significantly better than random (Extended Data Table~\ref{tab:appendicitis_concept_res_severity}). 

Predictably, sequentially optimized models (seq) were more performant at the concept prediction than the ones optimized jointly (joint), in agreement with the findings reported in the literature \citep{Koh2020}. Similar to the experiments on the \mbox{(semi-)synthetic} data (Figures~\ref{fig:synthetic_res}b and \ref{fig:synthetic_res}e), the models aggregating multiple views tended to have higher AUROCs and AUPRs. However, by contrast, LSTM-based aggregation consistently and noticeably outperformed simple averaging (avg), especially for predicting the visibility of the appendix, one of the most important diagnostic concepts \cite{Marcinkevics2021}. This could be associated with the loose spatiotemporal ordering among the US images acquired for each subject. Last but not least, semi-supervised bottlenecks were comparable to the sequentially optimized MVCBMs. Thus, learning complementary representations disentangled from the concepts did not hurt the model's performance at concept prediction.

\paragraph{Predicting diagnosis, management and severity}

As mentioned, the end goal of the developed models was the prediction of the \mbox{(i) diagnosis}, (ii) management and \mbox{(iii) severity} among suspected appendicitis patients based on the multiview US images. Test-set performance for these three target variables is reported in Table~\ref{tab:appendicitis_target_res}. Similar to the results discussed above, different concept bottleneck variants were trained with various view aggregation and optimization approaches. In addition to the concept-based methods, we considered several black-box baselines, namely, random forests fitted on the radiomics features, ResNet-18 \cite{He2016} fine-tuned on a single view (chronologically, the first captured image for each subject) and the multiview bottleneck architecture (MVBM), the same as for the MVCBM but trained without concept supervision. 

\input{src/tables/predictive_perf_app.tex}

All models were able to predict all target variables better than the naive baseline. Among the concept-based approaches, multiview models offered a consistent improvement over the vanilla CBM for diagnosis and severity. Moreover, the best-performing concept-based classifiers often achieved AUROCs and AUPRs comparable to those of the black-box MVBM. For the diagnosis, on average, multiview concept bottlenecks with the LSTM-based fusion outperformed averaging-based approaches. However, for management, we saw the opposite. For the diagnosis and management prediction, we also observed that neural-network-based methods, overall, outperformed RFs fitted on radiomics features. The latter result is not surprising, given that we did not utilize manually segmented regions of interest for radiomics feature extraction. Lastly, across all targets, the semi-supervised extension of the MVCBM achieved higher AUROCs and AUPRs or comparable to the approaches that purely relied on the concepts. 

In summary, concept-based classification on multiview US data is encouragingly effective at predicting the diagnosis. For management, aggregating multiple US images offers no improvement over simple single-view classification. We attribute this to the \emph{diagnostic} nature of the chosen concepts and their limited predictive power for the treatment assignment. Likewise, accurately predicting appendicitis severity is challenging, likely, due to the low prevalence of complicated appendicitis cases in the current dataset. Last but not least, in all tasks, the proposed SSMVCBM mitigated the poorer performance of concept-based approaches by learning representations complementary to the probably incomplete concept set.

\paragraph{Interacting with the model}

The practical utility of CBMs lies in the ability of the human user, in the current use case, the physician, to intervene on the concepts predicted by the model, thus affecting the model's behavior at test time. Similarly to the proof-of-concept experiments, we intervened on the bottleneck layers of the  CBM, MVCBM and SSMVCBM trained on the pediatric appendicitis data. Figure~\ref{fig:intervention_res} summarizes these results. Since LSTM-based and sequentially trained classifiers generally captured the concepts better (Table~\ref{tab:appendicitis_concept_res}), we only considered this specific configuration.

Figure~\ref{fig:intervention_res} shows the effect of interventions on the three models for the diagnosis (Figures~\ref{fig:intervention_res}a and \ref{fig:intervention_res}d), management (Figures~\ref{fig:intervention_res}b and \ref{fig:intervention_res}e) and severity (Figures~\ref{fig:intervention_res}c and \ref{fig:intervention_res}f). The lines show changes in AUROCs and AUPRs when intervening on randomly chosen concept subsets of varying sizes. For the diagnosis, the intervention effect is similar to the behavior of the models on the \mbox{(semi-)synthetic} data (Figures~\ref{fig:synthetic_res}c and \ref{fig:synthetic_res}f). Namely, AUROC and AUPR increase steadily with the number of concepts intervened on: for all models, the maximum median AUROC and AUPR achieved are approx. 0.85 and 0.94, respectively. Being the best-performing model (Table~\ref{tab:appendicitis_target_res}), SSMVCBM demonstrates only a slight or nearly no increase in predictive performance after intervening on the full concept set. Similarly, for management, we observed an increase in AUROC and AUPR. However, for predicting this target, a single-view CBM performed surprisingly well and overtook multiview models after interventions. Last but not least, interventions yielded no visible performance improvement for severity, possibly, due to considerable variance across initializations and randomly sampled concept subsets. 

\subsection*{Online prediction tool}

As a first step towards enabling clinicians and other interested parties to benefit from ML-based decision support, we developed and published an online decision support tool based on the abovementioned methods\footnote{Available at \url{https://papt.inf.ethz.ch/mvcbm}. The current implementation uses limited computational resources. We plan to improve the inference speed in the future.} (Extended Data Figure~\ref{fig:tool_gui}). The tool utilizes the multiview CBM model for predicting the diagnosis in suspected appendicitis patients (Figure~\ref{fig:workflow}). The user may upload several ultrasonography images, each representing a different view of the same patient. Image preprocessing, described in the \emph{Methods} section (Extended Data Figure~\ref{fig:us_example}), may be optionally executed. In addition to predicting the diagnosis, the tool allows the user to intervene on the concept predictions (Extended Data Table~\ref{tab:concept_distribution}). Uploaded images are protected using server-side sessions, which are only temporarily stored on the server and are purged after 30 minutes (Supplementary Figure~\ref{fig:webtool}). See Appendix~\ref{app:tool} for more information.

\begin{landscape}
\begin{figure}[h]%
\centering

\subfigure[]{
    \includegraphics[width=0.3\linewidth]{src/figures/app_interventions_diag.pdf}
}
\subfigure[]{
    \includegraphics[width=0.3\linewidth]{src/figures/app_interventions_trt.pdf}
}
\subfigure[]{
    \includegraphics[width=0.3\linewidth]{src/figures/app_interventions_comp.pdf}
}

\subfigure[]{
    \includegraphics[width=0.31\linewidth]{src/figures/app_interventions_diag_aupr.pdf}
}
\subfigure[]{
    \includegraphics[width=0.3\linewidth]{src/figures/app_interventions_trt_aupr.pdf}
}
\subfigure[]{
    \includegraphics[width=0.3\linewidth]{src/figures/app_interventions_comp_aupr.pdf}
}

\caption{Intervention experiment results for the pediatric appendicitis dataset. Interventions were performed by replacing the concept values predicted by the concept-based models with the ground truth for the \mbox{(a,d) diagnosis}, \mbox{(b,e) management} and \mbox{(c,f) severity} as target variables. Lines correspond to median \mbox{(a--c) test-set} AUROCs and \mbox{(d--f) AUPRs} attained by intervened models across ten initialization and three randomly sampled concept subsets.}
\label{fig:intervention_res}
\end{figure}
\end{landscape}
\section{Pediatric appendicitis dataset}\label{app:appendicitis_data}

In addition to the data acquisition and preprocessing discussed in the \emph{Methods} section, below we describe further details related to the pediatric appendicitis dataset. As mentioned, this study presents a retrospective analysis. The patients included in the cohort were managed according to the procedure 
summarized in Supplementary Figure~\ref{fig:app_management_algo}. For the concept variables in Extended Data Table~\ref{tab:concept_distribution}, missing entries were imputed with the negative findings. A comprehensive dataset summary with detailed variable explanations is available online.\footnote{\url{https://docs.google.com/spreadsheets/d/1L-5dkMyd_tSV2zKCZqacqcfVgAtAm3X2hiMgIzkEvYM/edit?usp=sharing}}

\begin{figure}[H]
    \centering
    \begin{center}
        \includegraphics[width=0.9\textwidth]{src/figures/appendicitis_management_algorithm.png}
    \end{center}
    \caption{A schematic of the routine management procedure for patients with suspected appendicitis. The decision on conservative management and operation was made by a senior pediatric surgeon based on clinical presentation combined with laboratory and ultrasound findings. This procedure is exercised at the Department of Pediatric Surgery and Pediatric Orthopedics, Hospital St. Hedwig of the Order of St. John of God, University Children's Hospital Regensburg (KUNO), Regensburg, Germany, where the dataset was acquired. The figure was adapted from the Supplementary Material of the article by Marcinkevics et al. \cite{Marcinkevics2021}. \label{fig:app_management_algo}}
\end{figure}

\section{Synthetic dataset\label{app:synthetic}}

To compare different variants of the proposed model and baselines in a controlled manner, we designed a simple synthetic multiview dataset with nonlinear relationships between the covariates, concepts and labels. Let $N$, $p$, $V$ and $K$ denote the number of data points, covariates per view, views and concepts respectively. Below we outline the generative process behind the data:
\begin{enumerate}
    \item Let $\boldsymbol{\mu}\in\mathbb{R}^{pV}$ be a randomly drawn vector where each component $\mu_j\sim\mathrm{Uniform}\left(-5,\,5\right)$ for $1\leq j\leq pV$.
    \item Let $\boldsymbol{\Sigma}\in\mathbb{R}^{pV\times pV}$ be a randomly generated symmetric, positive-definite matrix.
    \item Let $\boldsymbol{X}\in\mathbb{R}^{N\times pV}$ be a randomly generated feature matrix where $\boldsymbol{X}_{i,:}\sim\mathcal{N}_{pV}\left(\boldsymbol{\mu},\,\boldsymbol{\Sigma}\right)$.
    \item Let $\boldsymbol{x}_i^v=\boldsymbol{X}_{i,\left(1+p(v-1)\right):pv}$ for $1\leq i\leq N$ and $1\leq v\leq V$.
    \item Let $\boldsymbol{g}:\:\mathbb{R}^{pV}\rightarrow\mathbb{R}^K$ and $f:\:\mathbb{R}^K\rightarrow\mathbb{R}$ be randomly initialized MLPs with ReLU nonlinearities.
    \item Let $c_{i,k}=\boldsymbol{1}_{\left\{\left[\boldsymbol{g}\left(\boldsymbol{X}_{i,:}\right)\right]_k\geq m_k\right\}}$, where $m_k=\mathrm{median}\left(\left\{\left[\boldsymbol{g}\left(\boldsymbol{X}_{l,:}\right)\right]_k\right\}_{l=1}^N\right)$, for $1\leq i\leq N$ and $1\leq k\leq K$.
    \item Let $y_i=\boldsymbol{1}_{\left\{f\left(\boldsymbol{c}_i\right)\geq m_y\right\}}$, where $m_y=\textrm{median}\left(\left\{f\left(\boldsymbol{c}_i\right\}_{l=1}^N\right)\right)$, for $1\leq i\leq N$.
\end{enumerate}
Observe that the procedure above results in $N$ triples $\left(\left\{\boldsymbol{x}_i^{v}\right\}_{v=1}^{V},\, \boldsymbol{c}_i,\,y_i\right)$, for $1\leq i\leq N$. By contrast with the appendicitis ultrasonography dataset (Appendix~\ref{app:appendicitis_data}), herein, all data points have the same number of views. In our experiments, we set $N=\mbox{8000}$, $p=500$, $V=3$ and $K=30$. \mbox{2000} data points were held out as a test set. The simulation was repeated across ten independent replications.

\section{Multiview animals with attributes\label{app:mvawa}}

In addition to the purely synthetic classification task described above, we adapted a popular attribute-based classification dataset \emph{Animals with Attributes 2}\footnote{Available at \url{https://cvml.ista.ac.at/AwA2/}.} (AwA) \citep{Lampert2009} to the multiview scenario. The original AwA consists of 37322 images of 50 animal classes with $K=85$ binary-valued concepts, i.e. attributes. Similar to the UCSD Birds experiment for vanilla CBMs \citep{Koh2020}, the concepts are labeled per class and \emph{not} per data point. We extended AwA by randomly cropping $V=4$ patches, 60$\times$60 px\textsuperscript{2} big, from each image $i$ to produce multiple ``views'' (Supplementary Figure~\ref{fig:mvawa_ex}). Note that, while the concepts are only partially observable from individual images, there is no ordering among the patches, and, for simplicity, we generate the same number of views for each data point. Nevertheless, compared with the original AwA, classification based on a single view becomes markedly more challenging (Figure~\ref{fig:synthetic_res}). During the experiments, we divided the dataset according to the 60\%-20\%-20\% train-validation-test split. Simulations were repeated ten times independently.

\begin{figure}[H]
    \centering
    \begin{center}
        \includegraphics[width=0.5\textwidth]{src/figures/MVAwA_ex_1.png}
        
        \vspace{0.5cm}
        
        \includegraphics[width=0.5\textwidth]{src/figures/MVAwA_ex_2.png}
        
        \vspace{0.5cm}
        
        \includegraphics[width=0.5\textwidth]{src/figures/MVAwA_ex_3.png}
    \end{center}
    \caption{Three examples of the four-view data points from the multiview AwA dataset. Each row corresponds to a single data point. Every view (column) constitutes a random patch of the original AwA image. Therefore, in this dataset, the views are exchangeable. Moreover, some concepts can be identified only from certain views, e.g., in the bottom row, attributes referring to the background cannot be detected from the second (counting from the left) view.\label{fig:mvawa_ex}}
\end{figure}

\newpage

\section{Optimization procedure for the SSMVCBM\label{app:ssmvcbm}}

\input{src/supplementary/ssmvcbm_algo.tex}

\newpage

\section{Further implementation details\label{app:imp}}

\subsection{Architectures}

Supplementary Table~\ref{tab:architectures} provides a detailed description of the MVCBM architectures implemented in our experiments. Herein, $B$ denotes the batch size, $V$ is the maximum number of views, $K$ is the number of concepts, $H$ is the number of units in the hidden layer of $f_{\boldsymbol{\theta}}(\cdot)$, and $N_o$ is the number of output units. Supplementary Table~\ref{tab:architectures}a shows the architectures for the tabular synthetic data, and Supplementary Table~\ref{tab:architectures}b shows the architectures utilized for the image datasets. As can be seen, the encoder network $\boldsymbol{h}_{\boldsymbol{\psi}}(\cdot)$ is different: in  the first case, it is fully connected, whereas, in the latter, it is comprised of the ResNet-18 without the penultimate fully connected layer. Notably, the number of output units $N_o$ and the activation function depend on the number of classes of the target variable.

Note that, in the appendicitis dataset, all US image sequences were padded to the length of $V=20$. However, as intended, fusion layers discard the padding and can be applied to variable-length sequences. As mentioned in Appendices~\ref{app:synthetic} and \ref{app:mvawa}, we considered $V=3$ and $4$ views for the synthetic and MVAwA datasets, respectively. The number of concepts was $K=30$, $85$ and $9$ for the synthetic, MVAwA and appendicitis datasets, respectively. Notably, in MVAwA, the input images were 224$\times$224 px\textsuperscript{2}, while US images were 400$\times$400 px\textsuperscript{2} big. For the synthetic dataset and MVAwA, we fixed $H=100$, and for the appendicitis data, it was set to $5$. For MVAwA, the output layer was $N_o=50$ units wide and had \texttt{Softmax} activation. Since all labels in the pediatric appendicitis dataset were binary, we set $N_o=1$ and used \texttt{Sigmoid} activation.

For the SSMVCBM, we had to choose architectures for the concept prediction and representation learning ``branches'' of the model (Equation~\ref{eqn:ssmvcbm} in the main text). For both, we utilized architectures similar to those from Supplementary Table~\ref{tab:architectures}. For representation learning, instead of the sigmoid, we applied the hyperbolic tangent activation function at the output of $\boldsymbol{s}_{\boldsymbol{\zeta}}^{\boldsymbol{z}}(\cdot)$. Another architectural hyperparameter of the SSMVCBM is the number of dimensions of the vector $\boldsymbol{\hat{z}}_i$, denoted by $J$. In the experiments reported in Figure~\ref{fig:synthetic_res}, $J$ was set to the difference between the number of the ground truth concepts and the number of the concepts given to the model during training. For the experiments from Extended Data Table~\ref{tab:mvawa_full_res}, we set $J=24$. Finally, for pediatric appendicitis, we fixed $J=5$ across all target variables. Lastly, another architectural difference from the MVCBM was that the number of inputs in the target model $f_{\boldsymbol{\theta}}(\cdot)$ had to be $K+J$. Exact architecture specifications for the SSMVCBM experiments not covered above can be found in our code.\footnote{\url{https://github.com/i6092467/semi-supervised-multiview-cbm}}

\input{src/tables/architectures.tex}

\newpage

\subsection{Hyperparameters}

In all experiments, deep learning models were trained using the Adam optimizer. To avoid potential overfitting on the moderately-sized appendicitis dataset, throughout training, we applied on-the-fly data augmentation with Gaussian noise addition, random black rectangle insertion and one additional randomly chosen transformation: brightness adjustment, rotation, shearing, resizing, change of image sharpness or gamma correction. 

Applicable model hyperparameter values used for the synthetic, MVAwA and appendicitis datasets are provided in Supplementary Tables~\ref{tab:parameters_synthetic}--\ref{tab:parameters_app_severity}. The numbers of training epochs and learning rates were selectively tuned on the training set using five-fold cross-validation. In the tables below, by $E_y$ and $\eta_y$, we denote the number of epochs used to train a model and the initial learning rate, respectively. Note that sequentially trained MVCBMs allow for a separate hyperparameter configuration for the concept model $\boldsymbol{g}_{\boldsymbol{\phi}}(\cdot)$. We exploit this possibility for the number of epochs ($E_{\boldsymbol{c}}$) and the initial learning rate ($\eta_{\boldsymbol{c}}$). Due to the lack of this freedom, we have found that jointly trained MVCBMs sometimes require tuning $E_y$ and $\eta_y$ for the model weights to converge. Recall that parameter $\alpha$ controls the trade-off between the target and concept loss terms in the jointly trained concept bottleneck models. We did not explore the influence of this hyperparameter, fixing it to $\alpha=1.0$.  The remaining parameters belong to the semi-supervised variant of the MVCBM (Algorithm~\ref{alg:opt_ssmvcbm}, Appendix~\ref{app:ssmvcbm}): $C$ denotes the number of iterations for the adversarial training; $E_{\boldsymbol{z}}$ and $\eta_{\boldsymbol{z}}$ are the number of training epochs and learning rate, respectively, for the representation learning module; $E_a$ and $\eta_a$ are the number of epochs and learning rate for training the adversary network; and, finally, $\lambda$ is the parameter controlling the weight of the adversarial penalty in the loss function for optimizing the representation learning module and target model parameters.

\input{src/tables/hyperparameters_1.tex}

\input{src/tables/hyperparameters_2.tex}

\newpage

\section{Further results\label{app:res}}

\subsection{SSMVCBM ablation\label{app:ssmvcbm_abl}}

As mentioned before, the semi-supervised variant of the proposed multiview concept bottleneck model includes an adversarial regularizer (Equation~\ref{eqn:opt_ssmvcbm_minimax}) to de-correlate learned representations $\boldsymbol{\hat{z}}\in\mathbb{R}^J$ and concept predictions $\boldsymbol{\hat{c}}\in\mathbb{R}^K$. To better understand the impact of this regularization on the model's predictive performance and intervenability, we performed an ablation study on the \mbox{(semi-)synthetic} datasets by training semi-supervised concept bottlenecks under varying values of the regularization parameter $\lambda\in\{0,\, 0.01,\, 0.1\}$. 

In this experiment, we assessed the predictive performance (Supplementary Figure~\ref{fig:ssmvcbm_ablation}a) and intervenability (Supplementary Figure~\ref{fig:ssmvcbm_ablation}c) of the resulting models and the correlation among the individual dimensions of the concept predictions and representations (Supplementary Figures~\ref{fig:ssmvcbm_ablation}a and \ref{fig:ssmvcbm_ablation}c). For the latter, we have utilized Pearson's correlation coefficient conditional on the target variable; in particular, we have looked at the median absolute value of the pairwise correlation coefficient given by $\mathrm{median}_{i,j,k}\: \left\vert\widehat{\mathrm{corr}}\left(\hat{c}_i,\hat{z}_j\,\vert\,y=k\right)\right\vert$, where $\hat{c}_i$ and $\hat{z}_j$ denote the $i$-th and $j$-th components of the concept and representation vectors, respectively. For both datasets, the experiment was run under the incomplete concept set: $K=5$ (out of 30) observed concepts and $J=25$ for synthetic data and $K=10$ (out of 85) and $J=75$ for MVAwA. All results reported below correspond to the multiview CBMs with the averaging-based fusion.

Supplementary Figure~\ref{fig:ssmvcbm_ablation} summarizes the results of the ablation study. It appears that stronger regularization expectedly hurts the performance at predicting the target variable but allows learning representations de-correlated from the given concepts (Supplementary Figures~\ref{fig:ssmvcbm_ablation}a and \ref{fig:ssmvcbm_ablation}b). However, even in the absence of the adversarial regularization ($\lambda=0$), $\boldsymbol{\hat{c}}$ and $\boldsymbol{\hat{z}}$ are already relatively weakly correlated. Importantly, regularized models demonstrate a steeper increase in predictive performance during interventions on predicted concepts (Supplementary Figure~\ref{fig:ssmvcbm_ablation}c). Moreover, when most of the concepts have been intervened on, the unregularized model predicts the target variable more poorly than the regularized ones.

In summary, we observed that the adversarial regularizer in the SSMVCBM's loss function helps de-correlated representation learning and improves the model's intervenability, albeit it may reduce the non-intervened model's predictive performance. In future work, it would be interesting to seek alternative regularization techniques for disentangling concepts and representations, possibly focusing explicitly on intervenability.

\begin{figure}[H]
    \centering

    \subfigure[]{
        \input{src/tables/ablation_results.tex}
    }
    
    \subfigure[]{
        \includegraphics[width=1.0\textwidth]{src/figures/ssmvcbm_ablation_mawa.pdf}
    }
    
    \subfigure[]{
        \includegraphics[width=0.5\textwidth]{src/figures/ssmvcbm_ablation_mawa_interventions.pdf}
    }
    
    \caption{Results of the ablation study on the effect of the adversarial regularization in the semi-supervised multiview concept bottleneck models. \mbox{(a) Target} and concept prediction results alongside correlation between the predicted concepts and representations conditioned on the target variable. SSMVCBM models were trained on the synthetic and MVAwA datasets under varying  regularization parameter $\lambda=0,\,0.01,\,0.1$. AUROCs are reported as averages and standard deviations; for the correlation, we report median values and interquartile ranges. Statistics were computed across ten independent simulations. \mbox{(b) Conditional} correlation among the predicted concepts and representations for class $y=50$ in the MVAwA dataset. \mbox{(c) Intervention} experiment results on the MVAwA dataset for varying regularization strength. Predicted concepts were iteratively replaced with ground-truth values while measuring changes in AUROC for predicting the target variable. \label{fig:ssmvcbm_ablation}}
\end{figure}

\newpage

\section{Online prediction tool\label{app:tool}}

Below, we provide details on the implementation of our online pediatric appendicitis prediction tool. It is important to note that the current version is a research prototype which must only be used for non-commercial, educational purposes and not for clinical decisions. The web tool deploys a multiview concept bottleneck model trained to predict the diagnosis using the sequential optimization procedure and LSTM to fuse the views (MVCBM-seq-LSTM from Table~\ref{tab:appendicitis_target_res} in the main text). We use a single set of parameters obtained after training from one of the initializations included in the experiments. Note that the model \emph{was not} re-trained on the complete dataset.

\paragraph{Workflow}

Figure~\ref{fig:webtool} contains a workflow diagram for the website. Herein, the \emph{worker thread} handles incoming requests and creates a new server-side session if it does not exist for the current user. The images uploaded by the user are saved in the session. If requested, UI element regions are masked and filled, and CLAHE is applied to the input images. Note that due to the characteristics of the images, the effectiveness of preprocessing may be limited. In particular, UI artifacts, such as text, logos and diagrams, which differ considerably from those in our collected dataset, may not be completely masked and filled. The images are then passed into the trained MVCBM network, and the concept values and the diagnosis label are predicted and displayed. The user may intervene if they choose and re-calculate the final prediction using adjusted concept values. In the background, the \emph{session cleanup thread} is started along with the web application. It iterates every 60 seconds over all stored session objects. If a session object is older than 30 minutes, it is deleted along with all associated data. After this, no data provided by the user or data resulting from processing the user's uploaded data are retained.

\begin{figure}[h!]%
\centering
\includegraphics[width=0.9\textwidth]{src/figures/online_tool_workflow.pdf}
\caption{Workflow diagram of the online appendicitis prediction tool.}\label{fig:webtool}
\end{figure}











