\section{Introduction}
Artificial intelligence~(AI) systems that are built on machine learning~(ML) and deep neural networks~(DNNs) are increasingly deployed in numerous application domains such as military, cybersecurity, healthcare, etc. Further, ML and DNN models are applied to solving complex and emerging biomedical research problems, covering text mining, drug discovery, single-cell RNA sequencing, early disease diagnosis and prognosis. Disciplines such as systems medicines apply holistic approaches to understand complex physiological and pathological processes, thereby forming a basis for the development of innovative methods for the diagnosis, therapy, and prevention strategies mitigating disease conditions. Clinical data can be combined and fully analyzed by employing ML methods for new biomarker discovery. Further, the paradigm of evidence-based precision medicine has evolved toward a more comprehensive analysis of disease phenotype and their induction through underlying molecular mechanisms and pathway regulation. Another common application of AI in precision medicine is predicting what treatment protocols are likely to succeed on a patient based on patient phenotype, demographic, and treatment contexts~\cite{davenport2019potential}. 

Biomedical data science includes various types of data such as sequencing data, multi-omics, imaging, EMRs/clinical outcomes, and structured and unstructured biomedical texts data~\cite{han2022challenges}, where ML methods are typically used for the analysis and interpretation of multimodal data~(e.g., multi-omics, imaging, clinical outcomes, medication, disease progression, etc.), in a multimodal data setting. Further, many datasets, including bioimaging and omics are of increasing dimensionality. The surge of the massive amounts of biomedical data not only brings unprecedented progress in bioinformatics and opportunities to perform predictive modelling at scale~\cite{han2022challenges}, it also introduces challenges to existing AI methods and tools such as data heterogeneity, high-dimensionality, and volume~\cite{karim2020deep}.
Principal component analysis~(PCA) and isometric feature mapping~(Isomap) are extensively used as dimensionality reduction techniques~\cite{fournier2019empirical}. However, the representations learned by these methods often lose essential properties~\cite{clusteringAlgo}, making them less effective against a known phenomenon called \emph{curse of dimensionality}, particularly in high-dimensional datasets~\cite{fournier2019empirical}. 

A complex DNN model may tackle this issue very well and benefit from higher pattern recognition capabilities when it comes to feature extraction and learning useful representation from high dimensional data. A concrete example is using autoencoders~(AEs) for unsupervised learning tasks. AEs architectures with multiple hidden layers and non-linear activation functions learn to encode complex and higher-order feature interactions. Learning non-linear mappings allows transforming input feature space into a lower-dimensional latent space. The resultant embeddings that can capture important contextual information of underlying data can be used for different downstream tasks~\cite{karim2020deep}. However, latent factors learned by AEs are not easily interpretable, whilst disentangling them can provide insight into what features are captured by the representations and what attributes of the samples the task is based on~\cite{karim2020deep}. 

AI systems that are already deployed in selected application areas in healthcare and are already outperforming medical experts at selected tasks such as spotting malignant tumours, modelling disease progression, survival analysis, and treating cancerous conditions. However, the adoption of AI-driven approaches in many clinical settings has been hampered by the lack of efficient AI solutions that are able to deal with large-scale high dimensional biomedical data, and have high levels of uncertainty. Although DNN models can solve very complex problems, with increasing complexity owing to non-linearity and higher-order interactions among a large number of features, they tend to be less and less interpretable and may end up as \emph{black-box} methods. Their internal working principles are unavoidably opaque and algorithmic complexity is simply beyond humans to comprehend. Mathematical certainty means it should be possible to transparently show that no hidden behaviour or logic influencing the behaviour of a model~\cite{molnar2020interpretable}. 

Since the way how the predictions made by such a \emph{black-box} can neither be traced back to the input, nor it is clear why outputs are transformed in a certain way. If an AI system is built on such complex models and employed in biomedical data science, it will not only lack explainability but also have trustworthiness and transparency issues. Further, these limitations will evolve into another serious drawback: the inability to answer follow-up questions by human users and get contrastive or causal explanations. The emerging explainable artificial intelligence~(XAI) field, aims to mitigate the opaqueness of \emph{black-box} models by interpreting how AI systems should make decisions. The primary objectives of XAI are improving human understandability, decision reasoning capability, transparency, and accountability of AI systems~\cite{han2022challenges}. An interpretable ML model can identify factors~(e.g., statistically significant features) that can be used to explain the way they affect the model's outcome and the way they interact. Although linear models, decision trees~(DTs), and rule-based systems are less complex and intrinsically interpretable, they are less accurate compared to the tree-based ensemble and DNNs models that have been shown to outperform simple models. This phenomenon called the \emph{accuracy vs.\ interpretability trade-off} is shown in \cref{fig:acc_vs_xai}. 

\begin{figure*}[h]
	\centering
 \captionsetup{justification=centering}
		\includegraphics[width=0.6\textwidth]{images/accuracy_vs_interpretability_v2.pdf}
		\caption{Accuracy vs. interpretability trade-off - complex models tend to be less and less interpretable~\cite{karim_phd_thesis_2022}}
        \label{fig:acc_vs_xai}
\end{figure*}

Recently, XAI has gained a lot of attention from both academia and industry~\cite{tjoa2020survey}. Numerous model-specific and model-agnostic interpretable ML methods have been proposed, with view to improve local and global interpretability of ML models~\cite{wachter2017counterfactual}. However, majority of these methods have not been developed in a domain-agnostic way~\cite{han2022challenges}. Therefore, may not be inherently suitable for all biomedical and bioinformatics problems and diverse data types. Therefore, these methods need to be customized or extended in order to fit individual needs in bioinformatics and a variety of data types~\cite{han2022challenges}. 

In this paper, we discuss importance and advantages of explainability with a focus on bioinformatics. First, we provide a comprehensive overview of different interpretable ML methods. Then, via several case studies covering bioimaging, cancer genomics, and biomedical text mining, we show how bioinformatics research could benefit from XAI methods and how they could help improve decision fairness. Finally, we provide a brief overview of interpretable ML tools and libraries. 

\section{Importance of XAI in Bioinformatics}\label{sec:needs}
Large-scale biomedical data is not only heterogeneous, highly dimensional, and unstructured, but also has noise and high levels of uncertainty. However, not only the data-driven nature and technical complexity, the adoption of data-driven approaches in many bioinformatics application scenarios could be hampered due to the lack of efficient ML models that are able to tackle these challenges. Yet, an AI system is expected to perform in a reliable and safe manner to provide trustworthy decisions. 

Although not all the predictions need to be explained, an ML model itself needs to be interpretable since higher interpretability of the model means easier comprehension for the targeted users~\cite{stiglic2020interpretability}.  
Therefore, the need for efficient ML models with explainability are getting stronger, so that transparency, fairness, and accountability of AI can be enforced for mission-critical situations~\cite{das2020opportunities}. As outlined in \cref{fig:xai_importance}, interpretable ML methods provide several advantages over their \emph{black-box} counterparts, irrespective of the example introduced in the next section. 

\subsection{Helps avoid practical consequences}
A critical application of AI is aiding the treatment of cancerous conditions, having different types and subtypes. An early diagnosis of cancer, by classifying the patients into high or low-risk groups is a key to facilitating subsequent clinical management of patients~\cite{kourou2015machine}. Let's introduce a more concrete example of a clinical diagnosis of cancer for a breast cancer patient: a doctor suspects a patient could have breast cancer. Since breast cancer is one of the leading causes of death in women, the patient case requires careful investigation. By acquiring deep insights into omics data~(e.g., somatic mutations, gene expression~(GE), DNA methylation, copy number variations~(CNVs), and miRNA expressions), accurate diagnosis, as well as treatment, can be focused on preventive measures. Let's assume a multimodal DNN model trained on multi-omics data can distinguish~(classify) cancerous samples from healthy samples with an accuracy of 95\%. Assume a patient is diagnosed with the trained model with breast cancer with a probability of 70\%, and asks several follow-up questions: i) \emph{``why do I have breast cancer?''}, ii) \emph{``how did the model reach this decision?''}, iii) \emph{``which biomarkers or factors are responsible for it?''}. However, since the representations learned by the multimodal DNN architectures may not be easy to interpret, it may not be possible to transparently show what features the representations captured and what attributes of the samples the classification would be based on. 

\begin{figure*}
	\centering
    \captionsetup{justification=centering}
	\includegraphics[width=0.8\textwidth]{images/doctor_patient_v2.pdf}
	\caption{Example of practical consequence: a black-box model cannot explain diagnosis decisions~\cite{karim_phd_thesis_2022}}
    \label{fig:doctor_patient}
\end{figure*}

The diagnosis may further depend on several distinct molecular subtypes and factors like estrogen-, progesterone-, and human epidermal growth factor receptors. Therefore, the AI system may require using multimodal data~(e.g., multi-omics, bioimaging, and clinical data) and creating a multimodal DNN architecture. Further, image-guided pathology may need to employ in order to analyze imaging data~(e.g., breast histopathological image analysis), as shown in \cref{fig:real_Life_example}. Inability to: i) explain the decision by considering multimodal data settings, and ii) to answer such follow-up questions means the model is technically a \emph{black-box model}, as shown in \cref{fig:doctor_patient}. Any AI systems built on \emph{black-box} models will be \emph{black-box} too, where internal logic is either intentionally or strategically hidden from users~\cite{das2020opportunities}. 
A \emph{black-box} model would not allow tracing how and why an input instance is mapped to a decision and vice versa. Opacity is one of the major issues in \emph{black-box} ML -- a problem involving theoretical, practical, and legal consequences~\cite{zednik2019solving}. Therefore, such a \emph{black-box} model cannot be used in a clinical setting. 

\begin{figure*}[h]
	\centering
    \captionsetup{justification=centering}
	\includegraphics[width=0.6\textwidth]{images/real_example.pdf}
	\caption{AI for cancer diagnosis in a clinical setting~\cite{karim_phd_thesis_2022}}
    \label{fig:real_Life_example}
\end{figure*}

An interpretable ML model that emphasizes transparency and traceability for its underlying logic can reason why and how it reached certain decisions, thereby avoiding practical consequences. For our cancer diagnosis example, the local interpretability explains why a specific decision was made for an instance or by providing similar examples of instances to the target instance~\cite{stiglic2020interpretability}.  Thus, we can emphasize specific characteristics of a patient representing a smaller group, yet different from other patients. In contrast, global interpretability outlines the overall behaviour of the model, at an abstract level. Another concrete example in the same context could be as follows: suppose an ML model is trained to predict if a gene is up-regulated~(i.e., label) after some treatment based on the presence or absence of a set of regulatory sequences~(i.e., features). Global interpretability will tell us how important regulatory sequence is for predicting up-regulation across all genes in the dataset, while local interpretation strategy will tell us how important the sequence is for predicting a gene as up-regulated. 

\begin{figure*}
	\centering
    \captionsetup{justification=centering}
	\includegraphics[width=0.7\textwidth]{images/local_vs_global_exp.pdf}	
    \caption{An interpretable model can explain decisions locally, outlining global behaviour~(conceptually recreated based on~\cite{stiglic2020interpretability}}	
	\label{fig:local_vs_global_ex_2}
\end{figure*}

\subsection{Reduces complexity and improves accuracy}\label{subsec:biomarker_discovery}
The goal of genomics data analysis is to disseminate biologically meaningful information about genes and proteins since it will provide a clearer understanding of the roles a certain set of genes play in cancer development. However, biological processes in humans are not single-gene-based mechanisms, but complex systems controlled by regulatory interactions between thousands of genes~\cite{han2022challenges}. Omics data are very high-dimensional, where a large number of genes may be irrelevant for a learning task, e.g., cancer type prediction. For example, with more than 30,000 genes in GE data, the size of the feature space is so huge that the training data will be sparse. Besides, sample sizes are barely composed of a few tens to hundreds, e.g., in the case of clinical trials. The inclusion of all features not only introduces unwanted noise impeding predicting power of a model but also increases the computational complexity. 

Therefore, selecting biologically significant features with high individual correlation w.r.t classes of interest is important such that no correlation exists among the genes. Accurate identification of such cancer-specific biomarkers: i) helps improve classification accuracy, ii) allows biologists to study the interaction of relevant genes, and iii) helps study their functional behaviour to facilitate further exploration and discovery of other genes. Once such biomarkers are identified based on some individual feature attributions, they can be ranked according to their relative or absolute importance. Further, the identified genes can be treated as cancer-specific marker genes that characterize specific or multiple tumour classes.

\begin{figure*}
		\centering
        \captionsetup{justification=centering}
		\includegraphics[width=0.7\textwidth]{images/biases.pdf}
        \label{fig:bias_in_stages}
	\caption{Different types of biases involved in a typical ML pipeline~(recreated based on~\cite{GoogleBiasList})} 
\end{figure*}

\subsection{Improves decision fairness}
With the wide adoption of AI, it is also important to take fairness issues into consideration since AI systems can be used in many sensitive environments to make important and life-changing decisions~\cite{stiglic2020interpretability}. Biases are one of the main obstacles to fair decision-making. Therefore, fighting against bias and discrimination has a long history in philosophy and psychology~\cite{fairness_survey}. Statistically, bias means an inaccurate representation of what is true w.r.t the underlying population. Biases can manifest in any part of an ML pipeline~\cite{GoogleBiasinMLPipeline,fairness_survey}: from data collections, feature selection, model training, setting hyperparameters, and the way results interpreted to affected users, e.g., patients. For example, in healthcare and biomedical domains, manifesting representative-, selection-, and discriminatory biases can easily be ingested in into biophysical data, which will raise fairness issues and can lead to unfairness in different learning tasks~\cite{fairness_survey}. 

\begin{figure*}
	\centering
    \captionsetup{justification=centering}
	\includegraphics[width=0.8\textwidth]{images/xai_importance.pdf}	
    \caption{Advantages of XAI in improving interpretability, trust, transparency, and fairness in algorithmic decision-making}	
	\label{fig:xai_importance}
\end{figure*}

ML algorithms are always a form of statistical discrimination that could become objectionable when it places certain privileged groups at a systematic advantage and certain unprivileged groups at a systematic disadvantage. A model trained on biased data will be biased. If an AI system is developed based on ML models, it will produce biased decisions too. A recent study~\cite{obermeyer2019dissecting} has shown that a widely used algorithm can affect millions of patients, and exhibits significant racial bias. At a given risk score, they showed that black patients are considerably sicker than white patients, pointing out that remedying this disparity would increase the percentage of black patients receiving additional help from 17.7 to 46.5\%. Similar scenarios like these can further undermine the trust of healthcare experts and other affected bodies. As for our cancer example, predictions based on a biased model can disproportionately impact the diagnosis, which can tend a doctor to make the wrong treatment too.

Since humans are naturally biased, data collected or prepared by humans will always have minimal biases. Therefore, being aware of common human biases that could manifest in the data is important, so that proactive measures can be taken to mitigate their effects before training a model. Fair decision-making requires extensive contextual understanding~\cite{fairness_survey}, where interpretability and explainability can help identify sensible attributes and variables that could drive unfair outcomes. Once identified, proactive measures can be taken to ensure that the decisions would not reflect discriminatory behaviour toward certain groups or populations due to those variables~\cite{mehrabi2019survey}. 

\subsection{Internal governance and legal compliance}
With increasing adaptions of AI across domains, the need to explain the decisions made is important for ethical, judicial, and safety reasons~\cite{das2020opportunities}. For example, in a sensitive area like healthcare, explainability and accountability are not only some desirable properties but also legal requirements -- especially where AI may have an impact on human lives. The \emph{European Union General Data Protection Regulation}~(EU GDPR)~\cite{kaminski2019right} advocates important realization of AI such as ethics, accountability, and robustness~(e.g., adversarial attacks could fool an ML model during decision-making). Further, it enforces that processing based on automated decision-making tools should be subject to suitable safeguards, including \textit{``right to obtain an explanation of the decision reached after such assessment and to challenge the decision''} and individuals \emph{``have the right not to be subject to a decision based solely on automated processing and whenever human subjects have their lives significantly impacted by an automatic decision-making machine''}. 

Since AI plays a central role in the interaction between organizations and individuals~\cite{meske2022explainable}, explaining AI-assisted decisions to affected individuals require those within an organization to understand the overall decision-making processes~\cite{kazim2020explaining}. GDPR prohibits the use of AI for automated decision-making unless clear explanations of the logic involved are provided in a human-interpretable way. To avoid any legal consequences, how the ML model makes predictions should be as transparent as possible in an interpretable manner. As an iterative model design can generate better results, interpretability can facilitate better learning, in such scenarios~\cite{das2020opportunities}. 
By enforcing \emph{explainability} a key requirement: i) an organization will have better oversight of what an AI system does and why, ii) ensures the AI system continues to meet objectives and support in refining them to increase precision, iii) helps an organization to comply with parts of GDPR, iv) and to adhere to external policies, practical consequence, and processes that regulate business practices~\cite {kazim2020explaining}. 

\section{Techniques and Methods for Interpretable ML} \label{sec:xai_methods}
A wide range of model-specific and model-agnostic interpretable ML methods have been proposed and developed~\cite{azodi2020opening}. These methods largely fall into 3 main categories: probing, perturbing, and model surrogation. Further, depending on the level of abstractions, they can be categorized as local interpretability and global interpretability methods\footnote{\scriptsize{GitHub repo:~\url{https://github.com/rezacsedu/XAI-for-bioinformatics}}}. We categorize these interpretable ML methods based on papers, books, and tools. 

\subsection{Terminologies and notations}
Interpretability and explainability are interchangeably used in literature. However, the former signifies cause and effect of a certain outcome can be determined in a system, the latter is the extent to which the internal working mechanism of an AI system can be explained. According to Cambridge Dictionary\footnote{\scriptsize{\url{https://dictionary.cambridge.org/dictionary/english/interpretable}}}, \emph{``if something is interpretable, it is possible to find its meaning or possible to find particular meaning in it''}. Miller \textit{et al.}~\cite{miller2018explanation} define \emph{explanation} as the answer to `why' questions. Das \textit{et al.}~\cite{das2020opportunities} define interpretation as a \emph{``simplified representation of a complex domain, such as outputs generated by an ML model, to meaningful concepts which are human-understandable and reasonable''}. Interpretability, on the other hand, \emph{``is the degree to which a human can understand the cause of a decision''}~\cite{miller2018explanation}. Interpretability gives an ML system the ability to explain or present a model's predictions to humans in an understandable way~\cite{doshi2017towards}. Less technically, the interpretability of the ML model is the extent to which the cause and effect can be observed. 

\begin{table*}[h]
    \centering
    \caption{Interpretable ML techniques, categorized based on agnosticism, scopes, underlying log, and data types}
    \scriptsize{
    \begin{tabular}{l|l|l|l|l}
    \hline
        \textbf{Approach} & \textbf{Scope} & \textbf{Agnosticism} & \textbf{Methodology} & \textbf{Datatype} \\ \hline
        CAM~\cite{he2016computer, izadyyazdanabadi2018weakly} & Local & Model agnostic & Back-propagation & Image \\ \hline
        Grad-CAM~\cite{selvaraju2017grad} & Local &  Model agnostic & Back-propagation & Image \\ \hline
        Grad-CAM++~\cite{chattopadhay2018grad} & Local & Model agnostic & Back-propagation & Image \\ \hline
        Guided Grad-CAM~\cite{tang2019interpretable} & Local & Model agnostic & Back-propagation & Image \\ \hline
        Respond CAM~\cite{zhao2018respond} & Local &  Model agnostic & Back-propagation & Image \\ \hline
        LRP~\cite{LRP1,bach2015pixel} & Local &  Model agnostic & Back-propagation & Image \\ \hline
        DeepLIFT~\cite{shrikumar2017learning} & Global/local &  Model agnostic & Back-propagation & Image \\ \hline
        Prediction difference analysis~(PDA)~\cite{zintgraf2017visualizing} & Local & Model agnostic & Perturbation & Image \\ \hline
        Slot activation vectors~\cite{jacovi2018understanding} & Global & Model agnostic & Back-propagation & Text \\ \hline
        Peak response mapping~(PRM)~\cite{zhou2018weakly} & Local & Model agnostic & Back-propagation & Image \\ \hline
        LIME~\cite{LIME,ribeiro2016should} & Global/local & Model agnostic & Perturbation & Image, text, tabular \\ \hline
        MUSE~\cite{lakkaraju2019faithful} & Global & Model agnostic & Perturbation & Text \\ \hline
        DeConvolutional nets~\cite{zeiler2014visualizing} & Local & Model agnostic & Back-propagation & Image \\ \hline
        Guided back-propagation~\cite{springenberg2014striving, izadyyazdanabadi2018weakly} & Local & Model agnostic & Back-propagation & Image \\ \hline
        Activation maximization~\cite{erhan2010understanding} & Local & Model agnostic & Back-propagation & Image \\ \hline
        Bayesian averaging over decision trees~\cite{schetinin2007confident} & Global & Model specific & Bayesian & Tabular \\ \hline
        Generative discriminative models~(GDM)~\cite{caruana2015intelligible} & Global & Model specific & Discriminative & Tabular \\ \hline
        Bayes rule lists~\cite{letham2015interpretable} & Global & Model specific & Rule-based & Tabular \\ \hline
        Shapley sampling~\cite{SHAP} & Global/local & Model agnostic & Perturbation & Image, text, tabular \\ \hline
        Gradient-based saliency maps~\cite{simonyan2013deep} & Local & Model agnostic & Back-propagation & Image \\ \hline
        Bayesian case model~(BCM)~\cite{kim2014bayesian} & Global & Model specific & Bayesian & Image, text, tabular \\ \hline
        Deep Taylor decomposition~\cite{montavon2017explaining} & Local & Model agnostic & Decomposition & Image \\ \hline
        Deep attribution maps~\cite{ancona2017towards} & Local & Model agnostic & Back-propagation & Image, text \\ \hline
        Axiomatic attributions~\cite{sundararajan2017axiomatic} & Local & Model agnostic & Back-propagation & Image, text \\ \hline
        PatternNet and pattern attribution~\cite{kindermans2017learning} & Local & Model agnostic & Back-propagation & Image \\ \hline 
        Neural additive models~(NAMs)~\cite{agarwal2021neural} & Global & Model specific & Others & Image \\ \hline
        ProtoAttend~\cite{arik2019protoattend} & Global & Model agnostic & Others & Image \\ \hline
        Concept activation vectors~(CAVs)~\cite{kim2018interpretability} & Global & Model agnostic & Others & Image \\ \hline
        Global attribution mapping~(GAM)~\cite{ibrahim2019global} & Global & Model agnostic & Perturbation & Image \\ \hline
        Randomized sampling for explanation~(RISE)~\cite{petsiuk2018rise} & Local & Model agnostic & Perturbation & Image \\ \hline
        Spectral relevance analysis~(SpRAy)~\cite{lapuschkin2019unmasking} & Global & Model agnostic & Back-propagation & Image \\ \hline
       
        Salient relevance~(SR) map~\cite{li2019beyond, wang2021saled} & Global/local & Model agnostic & Back-propagation & Image \\ \hline
        Automatic concept-based explanations~(ACE)~\cite{ghorbani2019towards} & Global & Model agnostic & Others & Image \\ \hline
        Causal concept effect(CaCE)~\cite{goyal2019explaining} & Global & Model agnostic & Others & Image \\ \hline
    \end{tabular}}
    \label{tab:methods_scope_agnosticism_methodology_hi_dt_xai}
\end{table*}

If the parameters of an ML model and its architecture are known, the model is considered a \emph{white-box}. An ML model is a \emph{black-box} if its parameters and network architectures are hidden from the end user. In contrast, an interpretable ML refers to methods that make the behaviour and predictions of a system understandable to humans~\cite{molnar2020interpretable}. Algorithmic transparency suggests \emph{factors that influence the decisions made by algorithms should be visible, or transparent, to the people who use, regulate, and are affected by systems that employ those algorithms}~\cite{diakopoulos2017algorithmic}. An ML model is transparent if it is expressive enough to be human-understandable, where transparency can be a part of the algorithm itself or using external means~(e.g., model decomposition or simulations)~\cite{das2020opportunities}. 

Although a \emph{white-box model} improves model debugging and promotes trust, knowing its architecture and parameters alone won't make the model explainable~\cite{das2020opportunities}. Thereby, an interpretable model can outline how input instances are mapped to certain outputs by identifying statistically significant features. While explainability is using the knowledge of what those features represented and their relative importance in explaining the predictions to humans in an understandable way or terms. We refer to some terminologies and notations from our early work~\cite{karim2022interpreting} that are used in the remainder of the paper.

\begin{definition}[\textbf{Dataset}] \label{def:dataset}
    $D=(\tilde{X}, \tilde {Y})$ is a dataset, where $\tilde{X}$ be an $N$-tuple of $M$-instances, ${X}$ be the set of all instances in $\tilde{X}$, and $\tilde{Y}$ be $N$-tuple of labels $l \in L$. 
\end{definition}

\begin{definition}[\textbf{Model and prediction}] \label{def:classifier_and_prediction}
    Let the pair $(name, value)$ be a parameter and $\Theta$ be a set of parameters. A \emph{model} $f$ is a parameterized function $f:X \times \Theta \rightarrow {\mathbb{R}}$ that maps an input instance $x$ from a feature space $X$ to a decision $y \in L$ and returns an output called \emph{prediction} $\hat y$. A prediction $\hat y=f(x_i, \theta)$ is accurate for model $f$ and parameter $\theta \in \Theta$ if and only if $\hat y \approx \tilde{Y}\left[i\right]$ for $x=\tilde{X}\left[i\right]$, where $1 \leq i \leq M$. 
\end{definition}

\begin{definition}[\textbf{Black-box and interpretable models}] \label{def:black_box_and_white_box_models}
    Let $f$ be a model and $\Theta$ be a set of parameters. Model, $f$ is a {black-box} if its internal working principle and ${\theta}$ are hidden or uninterpretable by humans owing to lack of traceability of how $f$ makes predictions. Model $f$ is interpretable if the parameters ${\theta}$ are known and there exists a mathematical interpretation $\lambda_i$ showing how and why a certain prediction $\hat y$ is generated by $f$.  
\end{definition}

\begin{definition}[\textbf{Algorithmic transparency}] \label{def:algo_trans_1} 
    A model $f$ is transparent if there exists a mathematical interpretation $\lambda_t$ by which the learning algorithm learns model $f$ from $D$ by mapping relations between $X$ and $Y$ to make predictions.
\end{definition}

\begin{definition}[\textbf{Feature importance}] \label{def:feature_importance}
    Let $a_i$ be a feature in instance $x$ and $A$ be the set of all features. Importance function $h: A \rightarrow [0,1]$ assigns each element of $A$ a non-negative number between 0 and 1: the larger the number, the higher the importance of $a_i$. Local feature importance for $x$ is a set of feature and importance pairs $I_x=\left \{(a_1, h(a_1)), (a_2, h(a_2)), \dots, (a_M, h(a_M))\right\}$ for all $a_i \in x$. {Global feature importance} for $X$ is a set of feature and importance pairs $\bar I_X=\left \{(a_1, \bar p_1), (a_2, \bar p_2), \dots, (a_k, \bar p_M)\right\}$, where $\bar p_i$ is the mean local feature importance of $a_i$. 
\end{definition}

\begin{definition}[\textbf{Feature impacts}] \label{def:feature_impacts}
    Let $a_i$ be a feature of instance $x$, and $A$ is the set of all feature names. Impact function $g: A \rightarrow [-1, 1]$ takes an element of $A$ as input and results in a real number between -1 and 1. Local feature impact for $x$ is a set of feature and impact pairs $T_x=\left \{(a_1, g(a_1)), (a_2, g(a_2)), \dots, (a_M, g(a_M))\right\}$ for all $a_i \in x$. Global feature impact for $X$ is a set of feature and impact pairs $\bar T_X=\left \{(a_1, \bar q_1), (a_2, \bar q_2), \dots, (a_k, \bar q_M)\right\}$, where $\bar q_i$ is the mean of all local feature impacts of feature for $a_i$. 
\end{definition}

\begin{definition}[\textbf{Top-k and bottom-k features}] \label{def:top_k_important_features}
    Let $f$ be a model, $I$ be {global feature importance} for $D$, and $k$ be an integer. The {top-$k$ features} is a $k$-tuple such that for all $i \leq k \leq m$, $I[k[i]] \geq I[k[m+1]]$, and $I[k[i]] \geq I[k[i+1]]$, where $k$ is the number of top features used to explain the decision. A {bottom-k features} is a $k$-tuple such that for all $i \geq k \geq m$, $I[k[i]] \leq I[k[m+1]]$, and $I[k[i]] \leq I[k[i+1]]$\footnote{\scriptsize{Since the learning algorithm for a model involves stochasticity and the way they are computed could be different, feature importance scores~($p$) could be different if $I$ is sorted in ascending or descending orders of $p$.}}. 
\end{definition}

\begin{definition}[\textbf{Top-k and bottom-k impactful features}] \label{def:top_k_impactful_features}
    Let $f$ be a model, $T$ be {global feature impacts} for $D$, and $k$ be an integer. The {top-$k$ features} is a $k$-tuple such that for all $i \leq k \leq m$, $T[k[i]] \geq T[k[m+1]]$, and $T[k[i]] \geq T[k[i+1]]$, where $k$ is the number of top features used to explain a decision. A {bottom-k impactful features} is a $k$-tuple such that for all $i \geq k \geq m$, $T[k[i]] \leq T[k[m+1]]$, and $T[k[i]] \leq T[k[i+1]]$\footnote{\scriptsize{Since the learning algorithm for a model involves stochasticity and the way they are computed could be different, feature impact scores~($q$) could be different if $T$ is sorted in ascending or descending orders of $q$.}}.  
\end{definition}

\begin{definition}[\textbf{Interpretability}] \label{def:interpretability}
    \emph{Interpretability} is the degree to which humans can understand the cause of a decision~\cite{miller2018explanation}.  
    Global interpretability refers to an explanation $E$ explaining why model $f$ has predicted $\hat Y$ for all instances in $X$, outlining overall conditional interactions between dependent- and independent variables using some functions $\sigma(\cdot,\cdot)$. Local interpretability $e$ reasons why $\hat y$ has been predicted for an instance $x$, showing conditional interactions between dependent variables and $\hat y$, focusing on individual predictions. 
\end{definition}

\begin{definition}[\textbf{Algorithmic fairness}] \label{def:algo_fairness} 
    An algorithm is fair if the predictions do not favour or discriminate against certain individuals or groups based on sensitive attributes. 
\end{definition}

\begin{definition}[\textbf{Explanations}] \label{def:explanation}
    An explanation $e$ for a prediction $\hat y = f(x)$ for an instance $x \in X$ is an object derived from model $f$ using some function $\sigma(\cdot,\cdot)$ that reasons over $f$ and $x$ for $y$ such that $e = \sigma(f,x)$ and $e \in E$, where $E$ is the human-interpretable domain and $\sigma$ is an explanation function. 
\end{definition}

\begin{definition}[\textbf{Decision rules}] \label{def:decision_rules}
    A {decision rule} $r$ is a formula $p_1 \wedge p_2 \wedge \cdots \wedge p_k \rightarrow y$, where $p_i$ are boolean conditions on feature values in an instance $x$ and $\hat y$ is the \emph{}{decision}. Each decision rule $r$ is evaluated w.r.t $x$ by replacing the feature with a feature value from $x$, evaluating the boolean condition and reaching a conclusion $y$ if the condition is evaluated to be true.  
\end{definition}

\begin{definition}[\textbf{W-perturbations}] \label{def:w_perturbation}
    Let $D=(\tilde{X}, \tilde {Y})$ be a dataset, let $X$ be the set of all instances in $\tilde{X}$, and let $x$ be an instance of an $m$-tuple in $X$. Let $x^\prime$ for $x$ be a resulting vector by applying minimum change $\Delta x$ to some feature values $v_i$ using an optimization method~(e.g., ADAM~\cite{wachter2017counterfactual}) such that $y^{\prime}=f(x^\prime)$ against original prediction $y=f({x})$, where $x^\prime = x + \Delta x$. Then, $x^\prime$ is called a ${W}$-perturbation of $x$ and $y^\prime$.  
\end{definition}

\begin{definition}[\textbf{Counterfactual rules}] \label{def:Counterfactual_rules}
    Let $r: p \rightarrow y$ be a decision rule for an instance $x$ and $x^{\prime}$ is the perturbed vector of $x$ by applying W-perturbation\footnote{\scriptsize{Making changes to certain features may lead to a different outcome.}}. A \textit{counterfactual rule} $r^\updownarrow$ for the boolean conditions $p$ is a rule of the form $r^\updownarrow: p[\delta] \rightarrow y^{\prime}$ for $y^{\prime}=f(x^{\prime})$ s.t. $y^{\prime} \neq y$, where $\delta$ is a \textit{counterfactual} for original decision $y=f(x)$ for model $f$. 
\end{definition}

\begin{definition}[\textbf{Surrogate model}] \label{def:surrogation}
    Let $x \in X$ be an instance of an $m$-tuple, $\tilde{Y}$ be $N$-tuple of labels $l \in L$, and $f_b$ be a black-box model. Model $f$ is a \emph{surrogate model} of $f_b$ if the variance $R^2$ of $f_b$ captured by $f$ is $\approx 1$. 
\end{definition}

\begin{definition}[\textbf{Causally interpretable model}] \label{def:casually_interpretable_model}
    Let $f$ be an interpretable model and $q$ be a causal or counterfactual question. Then, the model $f$ is casually interpretable if there exists at least one answer $\hat y_a$ to question $q$. 
\end{definition}

\begin{definition}[\textbf{Local explanations}] \label{def:local_explanation_and_domain}
     Let $f$ be an interpretable model, $\hat y=f(x)$ be the prediction for instance $x$, $r$ be a \emph{decision rule} for $\hat y$, $\Phi$ be the set of \emph{counterfactual rules} for $r$, and $I$ be the set of \emph{local feature importance} for $\hat y$. A local explanation $e$ explaining decision $\hat y$ for $x$ is a triple $(I, r,\Phi)$. The domain $E_\ell$ of $e$ is a $N$-tuple of triples $(e_1, e_2,\cdots, e_N)$, where $N$ is the number of instances in $X$. 
\end{definition}

\begin{definition}[\textbf{Global explanations}] \label{def:global_explanation_and_domain}
    Let $f$ be a model, $\hat y=f(x)$ be the prediction for instance $x$, $\bar I$ and $\bar T$ be the sets of global feature \emph{importances} and \emph{impacts} for the set of predictions $\hat Y$ for instances $X$, $\bar I_{k_{t}} = \{(a_1, \bar q_{1_{t}}), (a_2, \bar q_{2_{t}}), \dots, (a_k, \bar q_{k_{t}})\}$ be the set of top-$k$ features, and $\bar I_{k_{b}} = \{(a_1, \bar q_{1_{b}}), (a_2, \bar q_{2_{b}}), \dots, (a_k, \bar q_{k_{b}})\}$ be the set of bottom-$k$ features. A global explanation $e_g$ is a pair $\langle \bar I_{k_{t}}, \bar I_{k_{b}} \rangle$. The domain $E_g$ of $e_g$ is a pair $(\bar I, \bar T)$. 
\end{definition}

The methods we discuss in the following sub-sections largely operate by approximating the outputs of a \emph{black-box} model via tractable logic or local approximation via a linear model, thereby putting emphasis on transparency and traceability of the \emph{black-box}~\cite{miller2018explanation}. Besides, tree-, rules-, and knowledge-based interpretable methods have been proposed. 


\begin{figure*}
	\centering
    \captionsetup{justification=centering}
	\begin{subfigure}{.48\linewidth}
		\centering
    \captionsetup{justification=centering}
		\includegraphics[width=0.9\textwidth]{images/probing.pdf}
		\caption{Probing a black-box model}
        \label{fig:probing}
	\end{subfigure}
	\hspace{2mm}
	\begin{subfigure}{0.48\linewidth}
		\centering
    \captionsetup{justification=centering}
		\includegraphics[width=0.9\textwidth]{images/purturbing.pdf}
		\caption{Interpreting black-box model with perturbing}
        \label{fig:purturbing}
	\end{subfigure}
		\begin{subfigure}{0.48\linewidth}
		\centering
    \captionsetup{justification=centering}
		\includegraphics[width=0.9\textwidth]{images/surrogation.pdf}
		\caption{Interpreting black-box model with model surrogation}
        \label{fig:surrogation}
	\end{subfigure}
	\caption{Overview of probing, perturbing, and model surrogation interpretability methods~(based on Azodi \textit{et al.}~\cite{azodi2020opening})}
	\label{fig:pro_per_surroga}
\end{figure*}

\subsection{Probing black-box models} 
Some interpretable methods do not take into account the inner working principles of \emph{black-box} models. Therefore, probing techniques have been proposed to probe and understand the underlying logic of opaque models. Saliency and gradient-based methods such as gradient-weighted class activation mapping~(Grad-CAM++)~\cite{chattopadhay2018grad} and layer-wise relevance propagation~(LRP)~\cite{LRP1} are examples of of this category, where first-order gradient information of a \emph{black-box} model is used to produce heatmaps~(HMs) indicating the relative importance of input features, e.g., relevant regions can be localised with HMs w.r.t importance to each feature(e.g., pixels in a CT/MRI/X-Ray image). 

These approaches are particularly suitable for bioimaging by which a convolutional neural network~(CNN) can learn features like class-discriminating pixels using edge detectors and filters across convolutional layers. Thereby, it can learn more abstract features in the deepest layer to provide attention to some areas using attention maps, explaining where the model put more attention to where the most important pixels are located at. 

\subsubsection{Saliency map and gradient-based methods} \hspace{0.1mm}
saliency maps~(SMs) and gradient-based methods are used to identify relevant regions, followed by assigning importance to each feature, e.g., a pixel in an image. Guided backpropagation~\cite{springenberg2014striving}, class activation maps~(CAM)~\cite{zhou2016learning}, Grad-CAM/++~\cite{selvaraju2017grad,chattopadhay2018grad}, and LRP are examples of this category. In GB, absolute output gradients are shown as heat maps~(HMs) and input nodes in which negative gradients are set to zero at the rectified linear layers of the network, thereby ``rectifying” the gradient,s in the backward pass leads to more focused HMs~\cite{bohle2019layer}. 

\begin{figure*}[h]
	\centering
    \captionsetup{justification=centering}
	\includegraphics[width=\textwidth]{images/xai_timeline.pdf}	
    \caption[Timeline of XAI methods and algorithms]{Timeline and evolution of interpretable ML methods, covering scopes, methodology, and usage level~\cite{karim_phd_thesis_2022}}	
	\label{fig:xai_timeline}
\end{figure*}

Class-discriminating attention map is applied to visualize the weighted combination of the feature maps~(FMs). To visualize where a CNN architecture provides more attention, CAM computes weights w.r.t each FM from the final convolutional layer. The contribution to prediction $y_c$ is calculated at location $(i,j)$. To obtain $L_{ij}^{c}$ that satisfies $y^{c}=\sum_{i, j} L_{ij}^{c}$, the last FM $A_{ijk}$ and prediction $y_c$ are represented in a linear relationship. A linear layers consist of global average pooling~(GAP) and fully connected layer~(FCL): GAP outputs $F_{k}=\sum_{i,j} A_{ijk}$ and FCL holds weight $w_{k}^{c}$ to generate the final prediction as~\cite{kim2020extending}: 
 
 \begin{align}
     y^{c}=\sum_{k} w_{k}^{c} F_{k}=\sum_{k} w_{k}^{c} \sum_{i, j} A_{i j k}=\sum_{i, j} \sum_{k} w_{k}^{c} A_{i j k}.
 \end{align}

where $L_{i j}^{c}=\sum_{k} w_{k}^{c} A_{i j k}$~\cite{kim2020extending}. However, if linear layers replace the classifier architecture, re-training of the network is required and the non-linearity of the classifier vanishes. An improved generalization of CAM called Grad-CAM is proposed~\cite{114}, where instead of pooling, globally average gradients of FMs are used as weights for the target class $c$. 

Guided backpropagation in Grad-CAM helps generate more human-interpretable but fewer class-sensitive visualizations than SM. Since SM use true gradients, network weights are likely to impose a stronger bias toward specific subsets of input pixels. Accordingly, class-relevant pixels are highlighted instead of producing random noise~\cite{nie2018theoretical}. Therefore, Grad-CAM uses HMs to provide attention on and to localize discriminating image regions, where class-specific weights of each FM are collected from the final convolutional layer through globally averaged gradients~(GAG) instead of pooling~\cite{chattopadhay2018grad}: 

\begin{equation}
    \alpha_k^c=\frac{1}{Z}\sum_{i}\sum_{j}\frac{\partial y^c}{\partial A_{ij}^k}.
    \label{eq:alpha}
\end{equation}

where $Z$ is the number of pixels in a FM, $c$ is the gradient of the class and $A_{ij}^k$ is the value of $k^{th}$ FM~\cite{chattopadhay2018grad}. Having gathered the relative weights, the coarse SM $L^c$ is computed as the weighted sum of $\alpha_k^c*A_{ij}^k$ of rectified linear unit~(ReLU) activation and applied to a linear combination of FM~\cite{chattopadhay2018grad}. Since features with positive influence on the class are of interest, negative pixels that belong to other categories are discarded~\cite{114}: 

\begin{equation}
    L^c=\operatorname{ReLU}(\sum_{i}\alpha_k^cA^k).
    \label{3.11}
\end{equation}

If an image contains multiple occurrences with slightly different orientations of the same class, several objects fade away from the SMs using Grad-CAM. Due to its overlooking of significance disparity among pixels, parts of the objects are rarely localised by Grad-CAM. To overcome these, Grad-CAM++ is proposed~\cite{chattopadhay2018grad}, where GAG is replaced with a weighted average of the pixel-wise gradients since the weights of pixels contribute to the final prediction by applying the following iterators over the same activation map $A^k$, $(i,j)$ and $(a,b)$~\cite{114}: 

\begin{align}
    \begin{aligned}
        w_{k}^{c}=\sum_{i} \sum_{j} \alpha_{i j}^{k c} \cdot \operatorname{ReLU}\left(\frac{\partial y^{c}}{\partial A_{i j}^{k}}\right) \\
        y^{c}=\sum_{k} w_{k}^{c} \cdot \sum_{i} \sum_{j} A_{i j}^{k} \\
        \alpha_{i j}^{k c}=\frac{\frac{\partial^{2} y^{\ell}}{\left(\partial A_{i j}^{k}\right)^{2}}}{2 \frac{\partial^{2} y^{c}}{\left(\partial A_{i j}^{k}\right)^{2}}+\sum_{a} \sum_{b} A_{a b}^{k} \frac{\partial^{3} y^{c}}{\left\{\left(\partial A_{i j}^{k}\right)^{3}\right\}}}.
    \end{aligned}
\end{align}

\Cref{fig:viz}, shows an example of using Grad-CAM++, which starts from taking a radiograph as input and passing to convolutional layers before getting rectified convolutional feature maps with guided-backdrop and Grad-CAM++ to be fed into a fully-connected softmax layer for the classification. Finally, critical image areas are localized with heatmaps~(HMs). Although CAM variants back-propagate gradients all the way up to inputs, gradients are essentially propagated until the final convolutional layer. Besides, they are  architecture-specific.

\begin{figure*}
    \centering
    \captionsetup{justification=centering}
    \includegraphics[width=0.7\textwidth]{images/network.pdf}    
    \caption{Example of Grad-CAM++ showing pixel relevances using heatmaps to localize critical regions in knee radiography~\cite{karim2021deepkneeexplainer}}    
    \label{fig:viz}
\end{figure*}

In contrast, LRP is based on the assumption that the likelihood of a class can be traced backward through a network to the individual layer-wise nodes of the input~\cite{LRP2}, making it suitable for imaging. For an image recognition task, LRP results in an HM that indicates what pixels of the image are important for a model's prediction. It identifies important pixels by running a backward pass in a CNN. The backward pass is a conservative relevance redistribution procedure, where nodes contribute the most to the higher layer receive the most relevance: an image $x$ is classified in a forward pass and relevance $R_{t}^{(L)}$ is then back-propagated to generate a relevance map $R_{LRP}$. The relevance $R_{n}^{(l)}$ at node $n$ in layer $l$ is recursively calculated as~\cite{LRP2}: 

\begin{align}
    R_{n}^{(l)}=\sum_{m} \frac{a_{n}^{(l)} w_{n, m}^{+(l)}}{\sum_{n^{\prime}} a_{n^{\prime}}^{(l)} w_{n^{\prime}, m}^{+(l)}} R_{m}^{(l+1)},
   
\end{align}

where it is assumed for a $L+1$ layered-network that each of first $L$ layers consists of $l$, $1,2,...,N$ nodes and $1,2,..,M$ are the nodes in layer $l+ 1$. The node-level relevance in the case of negative values is calculated using ReLU as~\cite{LRP2}:

\begin{align}
    R_{n}^{(l)}=\sum_{m} \frac{x_{n}^{(l)} w_{n, m}^{(l)}-b_{n}^{(l)} w_{n, m}^{+(l)}-h_{n}^{(l)} w_{n, m}^{-(l)}}{\sum_{n^{\prime}} x_{n^{\prime}}^{(l)} w_{n^{\prime}, m}^{(l)}-b_{n^{\prime}}^{(l)} w_{n^{\prime}, m}^{+(l)}-h_{n^{\prime}}^{(l)} w_{n^{\prime}, m}^{-(l+1)}}.
    \label{eq:rn_neg}
\end{align}

Output layer's relevance is calculated as~\cite{LRP2}:

\begin{align}
    R_{n}^{(L)}=\left\{\begin{array}{ll}
    {z_{t}^{(L)}} & {n=t}, \\
    {0} & {\text { otherwise.}}
    \end{array}\right.
    \label{eq:rn}
\end{align}

Class-discrimination regions highlighted by Grad-CAM++~(e.g., on CT/MRI/X-ray images) are found to be less precisely localized and scattered, while LRP highlights them more accurately~\cite{karim2021deepkneeexplainer}. The reason is Grad-CAM++ replaces the GAG with the weighted mean instead of individual parts, hence can highlight conjoined features more precisely. Therefore, if the diagnosis has to provide based on microscopic histopathology images, an AI-guided image analysis tool could confirm the presence of breast cancer. Since class discriminative areas can be localized w.r.t pixel relevance, Grad-CAM++ and LRP can be applied to ensure diagnosis reliability. \Cref{fig:explain_with_lrp} shows an example of an explainable diagnosis of osteoarthritis from MRT and X-ray images. As seen, Grad-CAM++ and LRP generate reliable HMs to show critical image regions and precise localization while depicting similar regions as important.

\begin{figure*}
	\centering
    \captionsetup{justification=centering}
	\includegraphics[width=0.8\textwidth]{images/lrp_gcam.pdf}	
    \caption{Example of using Grad-CAM++ and LRP for osteoarthritis diagnosis from knee MRI and X-ray images, highlighting critical knee regions to emphasise plus textual explanations~(based on~\cite{karim2021deepkneeexplainer})}	
	\label{fig:explain_with_lrp}
\end{figure*}

\subsubsection{Attention-based probing techniques} \hspace{0.1mm}
Attention mechanisms are proposed to detect larger subsets of features. Therefore, they improve accuracy in a variety of language modelling tasks. The concept of propositional self-attention networks~(PSAN) is an earlier method based on attention~\cite{vaswani2017attention}. In PSAN, attention heads are used, where the number of heads represents distinct relations between input features. Besides, the attention mechanism is widely used in NLP, computer vision, speech recognition, and CV tasks. In particular, transformer language models~(TLMs) such as bidirectional encoder representations from transformers~(BERT)~\cite{devlin2018bert} apply it to represent distinct relations between input features and identify important tokens for the next word prediction\footnote{\scriptsize{BERT utilizes bidirectional attention mechanism and large unsupervised corpora to obtain effective context-sensitive representations~\cite{xue2019fine}.}}. Xu \textit{et al.}~\cite{xu2020building} have shown that TLMs have great potential for biomedical text mining tasks such as named entity recognition~(NER) from unstructured texts with high accuracy~\cite{anantharangachar2013ontology}.

\begin{figure*}
	\centering
    \captionsetup{justification=centering}
	\includegraphics[width=0.8\textwidth]{images/ner_ex1.pdf}
	\caption{Example of explainable NER, where the model classifies and highlights relevant biomedical entities}
    \label{fig:bio_ner}
\end{figure*}

\begin{figure*}
	\centering
    \captionsetup{justification=centering}
	\includegraphics[width=0.65\textwidth]{images/san.pdf}	
	\caption{Example of using SAN for cancer type prediction from gene expression data~(based on~\cite{karim_phd_thesis_2022})}
	\label{fig:k_attention}
\end{figure*}

TMLs are effective for domain-specific fine-tuning in a transfer learning setting. Subsequently, they become the de-facto standard for representation learning for text mining and information extraction in NLP. Besides, the attention mechanism is also increasingly applied. Self-attention network~(SAN)~\cite{vskrlj2020feature} is proposed to identify important features from datasets having a large number of features, indicating having not enough data can distil the relevant parts of the feature space~\cite{vskrlj2020feature}. TabNet~\cite{arik2021tabnet} is another approach, which uses a sequential attention mechanism to choose a subset of semantically meaningful features to process at each decision step. It visualizes feature importance and how they are combined to quantify individual feature contributions leveraging local and global interpretability. In these approaches, attention layers~(ALs) are placed as hidden layers that map real values to parts of the human-understandable input space~\cite{vskrlj2020feature}. An AL can be represented as~\cite{vskrlj2020feature}: 

\begin{equation}
    l_{2}=\sigma\left(W_{2} \cdot\left(\alpha \left(W_{|F|} \cdot \Omega(X)+{b}_{l_{1}}\right)\right)+{b}_{l_{2}}\right),
\end{equation}

where $\alpha$ is an activation function, $b_{li}$ represents layerwise bias, and $\Omega$ represents the  first network layer that maintains the connection with the input features $X$~\cite{vskrlj2020feature}: 

\begin{equation}
   \Omega(X)=\frac{1}{k} \bigoplus_{k}\left[X \otimes \operatorname{softmax}\left(W_{l_{\mathrm{att}}}^{k} X+{b}_{l_{\mathrm{att}}}^{k}\right)\right],
\end{equation}

where $X$ are first used as input to a softmax-activated layer by setting: i) its number of neurons equal to $|F|$, and ii) a number of attention heads representing relations between input features to $k$, and $W_{l_{\mathrm{att}}}^{k}$ represents a set of weights in individual attention heads. The softmax function applied to ${i}$-th element of a weight vector $v$ is defined as~\cite{vskrlj2020feature}:

\begin{equation}
    \operatorname{softmax}\left({v}_{i}\right)=\frac{\exp \left({v}_{i}\right)}{\sum_{i=1}^{|F|} \exp \left({v}_{i}\right),}
\end{equation}

where $W_{l_{\mathrm{att}}}^{k} \in \mathbb{R}^{|F| \times|F|}$, $v \in \mathbb{R}^{|F|}$, and $l_{\text {att}}$ represents a single AL. In an AL, an element-wise product with $X$ is computed in the forward pass to predict labels $\hat{y}$ in which two consecutive dense layers $l_{1}$ and $l_{2}$ contribute to predictions, $\otimes$ and $\oplus$ are Hadamard product- and summation across $k$ heads. As $\Omega$ maintains a bijection between the set of features and weights of attention heads, weights for the $|F| \times|F|$ matrix represent the relations between features. Once the training is finished, weights of the AL are activated using softmax as~\cite{vaswani2017attention}:

\begin{equation}
    R_{l}=\frac{1}{k} \bigoplus_{k}\left[\operatorname{softmax}\left(\operatorname{diag}\left(W_{l_{\mathrm{att}}}^{k}\right)\right)\right], 
   
\end{equation}

where $W_{l_{\mathrm{att}}}^{k} \in \mathbb{R}^{|Z| \times|Z|}$. Then, the top-k features can be extracted as diagonal of $W_{l_{\mathrm{att}}}^{k}$ and ranked them w.r.t. their respective weights and conceptualised as \emph{feature importance}.

\subsection{Perturbing black-box model}
Sensitivity analysis, game theoretic approach, and feature-based attribution methods fall in this category.

\subsubsection{Feature-based attribution methods} \hspace{0.1mm}
Knowing what features are statistically most important to a model help achieve and generate human-level interpretability. Some features have a higher impact than others. This notion of feature importance~(ref.~\cref{def:feature_importance}) can be computed as permutation feature importance~(PFI). PFI works by randomly permuting or shuffling a single column in the validation dataset leaving all the other columns intact, where a feature is considered \emph{important} if and only if the model's accuracy drops significantly, thereby increasing the prediction error. A feature is considered \emph{unimportant} if switching its position does not significantly affect the accuracy or performance of the model. 

FI can be conceptualized both locally and globally: effects of a feature for a single prediction, over a large number of samples, or for the overall predictions. Let $x_i$ be a feature in instance $x$. FI methods define an explanation function $g: f \times \mathbb{R}^{d} \mapsto \mathbb{R}^{d}$, returns importance scores $g(f,x) \in \mathbb{R}^{d}$ for all features for $f$ and a point of interest in $x$. Besides, FI can be extracted from tree-based models, e.g., DTs or tree ensembles.

\begin{figure}
	\centering
    \captionsetup{justification=centering}
	\includegraphics[width=0.45\textwidth]{images/fi.pdf}
	\caption{Example of using SHAP for ranking globally important features w.r.t feature importance~\cite{karim2019onconetexplainer}}
    \label{fig:shap_gene_ranking}
\end{figure}

\begin{figure*}
	\centering
    \captionsetup{justification=centering}
	\includegraphics[width=\textwidth]{images/global_fi_1.pdf}
	\caption{Example of using SAN for ranking globally important features w.r.t feature impacts}
    \label{fig:san_gene_ranking}
\end{figure*}

\subsubsection{Game theoretic approach}\hspace{0.1mm}
For a non-linear function~(e.g., a linear model), the order in which features are observed by the model matters. Scott M. \textit{et al.}~\cite{SHAP} showed that tree-based methods like DTs or tree ensembles are similar to using a single ordering defined by a tree's decision path, yielding inconsistency in PFI~\cite{SHAP}. A widely used perturbation-based method is SHapley Additive exPlanations~(SHAP). SHAP is a game theory-based approach and is based on Shapley values~(SVs). 
An SV is the average marginal contribution of a feature and a way to distribute the gains to its players. Therefore, SVs give an idea of how to fairly distribute the payout~\cite{molnar2020interpretable}. 
SHAP connects game theory with local explanations by uniting several previous methods and representing the only possible consistent and locally accurate additive feature attribution method based on expectations~\cite{SHAP}. SHAP is the only possible consistent and locally accurate approach~\cite{SHAP}. 

Let $f$ be an interpretable model, $x_i$ be a feature in features in instance $x$ in a dataset $D$, and let $\hat y$ be the prediction, for the instance $x$. Then, SHAP explains the prediction $\hat y$ by computing the contribution of each feature $x_i$ in $x$ w.r.t SVs. SVs are based on coalitional game theory~\cite{branzei2008models}\footnote{\scriptsize{In game theory, a cooperative game played with competition between groups of players known as ``coalitions'' as there is a possibility of external enforcement of cooperative behaviour.}} and used as the measure of feature attributions, where each feature $x_i$ acts as a player in a coalition. SHAP models an explanation as~\cite{molnar2020interpretable}:

\begin{equation}
    f\left(z^{\prime}\right)=\phi_{0}+\sum_{i=1}^{C} \phi_{i} z_{i}^{\prime},
\end{equation}

where $z^{\prime} \in\{0,1\}^{C}$ is the coalition vector\footnote{\scriptsize{Coalition vector is a simplified feature representation for tabular data.}} such that the effect of observing or not observing $x_i$ is calculated by setting $z_{i}^{\prime}=1$ or $\left.z_{i}^{\prime}=0\right)$, $\mathrm{C}$ is the maximum coalition size equal to the number of input features $M$ and $\phi_{i} \in \mathbb{R}$ is the feature attribution for $x_i$ or SVs. Taking into the fairness aspect, the computation is repeated for all possible orders. The importance $\phi$ of a feature $x_i$ is computed by comparing what model $f$ predicts with and without $x_i$ for all possible combinations of $M-1$ features~(i.e., except for feature $x_i$) in $x$ as~\cite{lundberg2017consistent}. 

\begin{align}
    \phi_{x_i}=\sum_{S \subseteq N \backslash\{i\}} \frac{|S| !(M-|S| !-1)}{M !}\left[f_{x}(S \cup\{i\})-f_{x}(S)\right]. 
    \label{eq:shap_local}
\end{align}

SHAP values explain the output of a function as a sum of the effects $\phi_i$ of each feature being introduced into a conditional expectation. SHAP averages over all possible orderings for computing the mean SVs. If $x_i$ has no or almost zero effect on the predicted value, it is expected to produce an SV of 0. If two features $x_i$ and $x_{i+1}$ contribute equally to the prediction, SVs should be the same~\cite{SHAP}. To compute global importance, absolute SVs per feature across instances are averaged as~\cite{lundberg2017consistent}. 


\subsubsection{Sensitivity analysis} \hspace{0.1mm}
For a \emph{black-box} or interpretable model $f$, sensitivity analysis~(SA) is used to explain a prediction $\hat y$ based on the model's locally evaluated gradient or partial derivatives. Sensitivity $R_{x_i}=\left\|\frac{\partial}{\partial x_{i}} f({x})\right\|$ quantifies the importance of an input feature $x_i$ at a low level~(e.g., image pixel). This measure assumes that the most relevant features are those for which the output is sensitive. Often HMs are plotted to visualize which pixels need to be changed to make the image look similar to the predicted class. However, such an HM does not indicate which pixels are pivotal for a specific prediction, making them not suitable for quantitative evaluation and to validate globally important features. Therefore, SA is more suitable for tabular data to inspect which features a model is more sensitive to. 

To perform SA for a tabular dataset, the value for a feature $x_i$ is changed by keeping other features in the dataset unchanged. If any change in its value significantly impacts the prediction, the feature is considered to have a high impact and is statistically significant. For this, a new test set $\hat X^{*}$ is often created by applying $w$-perturbation over feature $x_i$, followed by measuring its sensitivity at the global level. To measure the change in the predictions, the mean square error~(MSE) between actual and predicted labels are compared, i.e., sensitivity $R_{x_i}$ for feature $x_i$ is the difference between the MSE at original feature space $X$ and sampled $\hat X^{*}$. However, SA requires a large number of calculations~(i.e., $N \times M$; $N$ and $M$ are the number of instances and features) across predictions\footnote{\scriptsize{Therefore, some approaches~\cite{karim2022interpreting} recommends making minimal changes to top-k features only, thereby reducing the computational complexity.}}. 

\begin{figure*}
	\centering
     \captionsetup{justification=centering}
		\includegraphics[width=0.7\textwidth]{images/intercept.pdf}
	    \caption{Explaining a diagnosis decision to a patient is difficult unless it is interpreted simpler way: a) explaining\\ with the plot, b) explaining with a rule, c) explaining in natural human language~\cite{karim_phd_thesis_2022}}
	\label{problem_of_xai:1}
\end{figure*}

\subsection{Tree, textual, and rules-based approaches} \label{subsec:tree_rules}
For the cancer diagnosis example introduced earlier, let a DT classifier is used for cancer-type prediction. As shown in \cref{problem_of_xai:1}, suppose for a test instance, the model predicts from the patient's GE profile that the selected patient has breast cancer with a probability of 75.3\%, showing an average response~(i.e., intercept) of +0.452. Let the patient's gender, age, and some marker genes be the most contributing feature w.r.t feature impact score. However, explaining this decision by combining all these concepts may not be well-understood by users from everyone, e.g., patients. The reason is that explaining a decision with plots and charts are good for exploration and discovery, but interpreting them for patients may be difficult. 

Using a \emph{decision rule}~(DR), it would be easier to explain this decision intuitively to humans with the ability to look up the reasons. A DR that maps an observation to a prediction, can have several characteristics: i) follow a general \emph{structure} - IF conditions are met THEN make a certain prediction, ii) \emph{number of condition} - at least one feature=value statement in the condition, with no upper limit and more statements can be added with an `AND' operator, iii) \emph{single or multiple DRs} - although a combination of several rules can be used to make predictions, sometimes a single DR is enough to explain the outcome. Since an explanation relates the feature values of a sample to its prediction, rule-based explanations are easier to understand for humans~\cite{ming2018rulematrix}. Further, the IF-THEN structures semantically resemble natural language and the way how humans think~\cite{molnar2020interpretable}. The same decision now can be translated into rules: \emph{``IF gender = female AND age $>$ 50 AND top gene=~(EPHA6, NACA2) AND gType = Oncogene, THEN type = BRCA''}.

However, explaining this to a patient may still be difficult unless they are explained in a human-interpretable way. A simple explanation for this, in a natural language, could be \emph{``Increased breast cancer is associated with risk for developing lymphedema, musculoskeletal symptoms, and osteoporosis. Being a 50 years old woman and having two highly expressed EPHA6 and NACA2 oncogenes highly mutated in your tumour cells, you are diagnosed with breast cancer positive, with a probability of 75.3\%.''}. The doctor could further explain why it is the case saying \emph{``the model learns attention on the patches from the image and localizes the malignant and normal regions, based on which the diagnosis decision has been made''}.  

Local interpretable model-agnostic explanations~(LIME) approximates a \emph{black-box} model via an interpretable model to explain the decisions locally. Anchor~\cite{ribeiro2018anchors} is another rule-based method that extends LIME. It computes DRs by incrementally adding equality conditions on antecedents w.r.t precision threshold~\cite{guidotti2018local}. DRs are arguably the most interpretable predictive models as long as they are derived from intelligible features and the length of the condition is short. However, a critical drawback of rule-based explanations could arise due to overlapping and contradictory rules. Sequential covering and Bayesian rule lists are proposed to deal with these issues. Sequential covering iteratively learns a single rule covering the entire training data rule-by-rule, by removing data points already covered by new rules~\cite{molnar2020interpretable}, while Bayesian rule lists combine pre-mined frequent patterns into a decision list using Bayesian statistics~\cite{molnar2020interpretable}. 

\begin{figure*}[h]
	\centering
    \captionsetup{justification=centering}
		\includegraphics[width=0.7\textwidth]{images/tree_example_path_v2.pdf}
	    \caption[Decision rules extracted from a root-leaf path]{An example showing how a tree-based model has arrived at decision~(bladder urothelial carcinoma~(BLCA))} 
	    \label{fig:example_DT_path}
\end{figure*}

Tree-based approaches exploit tree structures, where internal nodes represent feature values w.r.t boolean conditions and leaf nodes represent predicted class labels. Each branch in a DT represents a possible outcome, while paths from the root to leaf nodes represent classification rules. The outcome of a DR signifies the predicted label of a leaf node, while the conjunctions of conditions in \emph{IF} clause correspond to different conditions in the path~\cite{guidotti2018survey}. DT iteratively splits $X^{*}$ into multiple subsets w.r.t to threshold values of features at each node until each subset contains instances from one class only. The relation between prediction $\hat{y}^{*}_{i}$ and feature $x^{*}_i$ is defined as~\cite{di2019surrogate}: 

\begin{align}
    \hat{y}^{*}_{i}=f\left(x^*_{i}\right)=\sum_{j=1}^{N} c_{j} I\left\{x^*_{i} \in R_{j}\right\},
\end{align}

where each sample $x^{*}$ reaches exactly one leaf node, $R_{j}$ is the subset of the data representing the combination of rules at each internal node, and $I\{.\}$ is an identity function~\cite{di2019surrogate}. The mean importance for $x_i$ is computed by going through all the splits for which $x_i$ was used and adding up how much it has improved the prediction in a child node~(than its parent node) w.r.t decreased measure of Gini impurity $I_{G}$~\cite{hatwell2020chirps}: 

\begin{align}
    I_{GQ}=\sum_{k=1}^{N} p_{k} \cdot\left(1-p_{k}\right), 
\end{align}

where $p_{k}$ is the proportion of instances having label $y_{k}$ in a node $Q$ in DT. The sum of all values of importance is scaled to 1: a mean importance of 0 means $x_i$ is not important, while an importance value of 1 means $x_i$ is highly important. As for tree ensembles~(i.e., boosted trees~(GBT) or random forest~(RF)), the prediction function $f(\mathrm{x})$ is defined as the sum of individual feature contributions plus the average contribution for the initial node in a DT and $K$ possible class labels that change along the prediction paths after every split w.r.t $I_{G}$~\cite{al2021cdrgi}:

\begin{align}
    f(x)=c_{full}+\sum_{k=1}^{M} \sigma(x, k),
\end{align}

where $c_{full}$ is the average of training set $X$ and $\mathrm{M}$ is the number of features. Thus, DRs can naturally be derived from a root-leaf path in a DT: starting at the root and satisfying the split condition of each decision node, it requires to traverse until a leaf node is reached, as shown in \cref{fig:example_DT_path}. 

\begin{figure*}
	\centering
    \captionsetup{justification=centering}
		\includegraphics[width=0.9\textwidth]{images/wf_v3.pdf}
	    \caption{Creation of interpretable surrogate model based on the multimodal black-box model and using the surrogate model to explain the decision~(grey circles with red numbers represent different steps of the process)~\cite{karim_phd_thesis_2022}}
	    \label{fig:workflow}
\end{figure*}

\subsection{Model surrogation strategies} 
Since interpretability comes at the cost of accuracy vs. complexity trade-off, research has suggested learning a simple interpretable model to imitate a complex model~\cite{molnar2020interpretable}. A \emph{surrogate} or a simple proxy model is often developed to learn a locally faithful approximation of the \emph{black-box}~\cite{stiglic2020interpretability}. Model surrogation is a model interpretation strategy, which involves training an inherently interpretable model for making  predictions\footnote{\scriptsize{Approximating the predictions of a \emph{black-box}.}} of the \emph{black-box} model~\cite{molnar2020interpretable}. Depending on the complexity of the problem, the surrogate model is trained on the same data the \emph{black-box} was trained on or on sampled data, as shown in \cref{fig:workflow}. 

Since most important features can be identified by the \emph{black-box}, training an interpretable model on top-k feature space~(ref.~\cref{def:top_k_important_features}) is reasonable. Let $X^{*}$ be a sampled~(e.g., a simplified version of the data containing top-k features only) data for the original dataset $X$ and ${Y}$ be the ground truths. A surrogate model $f$ can be trained on $X^{*}$ and for $Y$. The advantage of model surrogation is that any interpretable model can be used~\cite{molnar2020interpretable}, e.g., LR, DTs or GBT or RF  classifiers\footnote{\scriptsize{Although tree ensembles are complex and known to be \emph{black-boxes}, decision rules can be extracted and FI can be computed from them.}}. 

\subsection{Casual inference and contrastive  explanations}
ML or DNN models usually generate statistical outputs that are based on correlational rather than causal inference~(i.e., focus on the association instead of the causality). Let a set of relevant input features $x$ be mapped to a target variable $y$ based on established association or correlation between them. Humans tend to think in a counterfactual way by asking a question such as \emph{``How would the prediction have been if input $x$ had been different?''}. While we would conventionally say that features in $x$ are correlated to $y$, it is not justified to say that features in $x$ cause $y$, i.e., \emph{``correlation does not imply causation''}. However, recent interpretable ML methods try to answer questions related to causality such as \emph{``Was it a specific feature that caused the diagnosis decision made by the model?''}. Statistical interpretability aims to uncover statistical associations, while causal interpretability is designed to answer \emph{what-if} and \emph{why} questions, yielding the highest level of interpretability~\cite{moraffah2020causal}. 

Kim \textit{et al.}~\cite{kim2019learning} proposes to learn an oracle model $f_c$ to estimate the causal effects covering all observed instances and learning an interpretable model $f$ to approximate the oracle model $f_c$. We would argue that a \emph{black-box} can be trained to learn casual effects, followed by generating an interpretable model by employing model surrogation strategies. Lipton \textit{et al.}\cite{lipton2018mythos} argues that a causally interpretable model~(ref. \cref{def:casually_interpretable_model}) is often indispensable to guarantee fairness. Therefore, by using a set of DRs and counterfactuals, it is possible to explain the decisions directly to humans with the ability to comprehend the reason so that users can focus on understanding the learned knowledge instead of underlying data representations~\cite{ming2018rulematrix}. \emph{Local rule-based explanations}~(LORE)~\cite{guidotti2018local} is an approach that learns an interpretable model by computing a neighbourhood using a genetic algorithm. It derives explanations from the interpretable model in order to explain the decisions in the form of DRs and \emph{counterfactuals}~(ref.~\cref{def:Counterfactual_rules}). Partial dependence plot~(PDP) is another way that depicts the marginal effect of features on the predicted outcomes. PDP allows measuring the change in predictions after making an intervention~(w-perturbations), which can help to discover the features’ causal relationship. 

\subsection{Human-in-the-loop and knowledge-based methods}
Since an explanation is an ``interface'' between humans and a user~\cite{guidotti2018survey}, interactive ML systems can add the component of human expertise to AI processes by enabling them to re-enact and retrace AI/ML results~\cite{holzinger2020measuring}. Therefore, the human-computer interaction~(HCI) community have a long-standing interest in algorithmic transparency towards making AI systems more human-friendly~\cite{liao2021question}. As shown in \cref{fig:decision_reasoning_with_rules}, an interactive human–AI interface can be developed to evaluate the quality of explanations for an AI system~\cite{holzinger2020measuring}. A more concrete example could be developing an explainable chatbot to enable interaction between humans and an AI system. Via a chatbot users can ask questions and get answers from the AI system interactively, e.g., explanations about cancer decisions made by ML models. 

In statistical learning theory, ML models are usually trained on data and iteratively optimized. However, except for the data, no domain knowledge is incorporated. In many scientific discoveries, where the goal is rather better understanding, employing sophisticated ML methods and getting high accuracy is not enough. Even though an ML model may be better than humans at making more accurate predictions and processing patterns, it may not be able to reason or explain underlying decisions. Therefore, the model would not be able to act as humans do, e.g., infer symbols, abstractions, and functional connections~\cite{kapanipathi2020question}. If trained on data accompanying expert knowledge or metadata, a well-fitted \emph{black-box} model would have sufficient know-how of what statistically significant features it has learned from a domain point of view. 

By interpreting a complex \emph{black-box} model, we would know what are the most important features in a dataset. The model could tell us, for our cancer diagnosis example, which oncogenes or protein-coding genes are most important. However, the model is not capable of validating if they are really of biological significance~(e.g., are all features in the antecedents biologically significant?). Since the generated explanations are based on statistical learning theory, we cannot confidently say that the decisions are reliable. Unless the recommendation made by an AI system is validated from the domain knowledge, such unreliable decisions can disproportionately impact the diagnosis and could lead a medical doctor to make the wrong treatment. Further, in cancerogenesis, it is also important to understand the biological mechanism. An interpretable model alone cannot build the foundation of a reliable and trustworthy AI system. Besides, rigorous clinical validation w.r.t domain expertise is required before deploying an AI system in a clinical setting.

\begin{figure*}
	\centering
    \captionsetup{justification=centering}
	\includegraphics[width=0.8\textwidth]{images/explain_in_text.pdf}
	\caption{Decision reasoning with rules in textual and natural human language based facts from a domain-knowledge graph~\cite{karim_phd_thesis_2022}}
     \label{fig:decision_reasoning_with_rules}

\end{figure*}

The integration of an ML model with a knowledge-based system would allow human operators the reasoning and question-answering over\footnote{\scriptsize{This paradigm, which is being emerged as \emph{neuro-symbolic AI} combines both connectionist AI and symbolic AI paradigms together.}}. Further, domain experts require to rely on up-to-date knowledge and findings from scientific literature. Knowledge and facts about drugs, genes, protein, and their mechanism are spread across a huge number of structured~(knowledge bases) and unstructured~(e.g., scientific articles)~\cite{karim_phd_thesis_2022,karim2022question}. These articles are a large-scale knowledge source and can have a huge impact on disseminating knowledge about the mechanisms of certain biological processes, e.g., diseases to design strategies in order to develop prevention and therapeutics decision-making process~\cite{karim2022question}. From such structured and unstructured sources, facts containing simple statements like \textit{``TP53 is an oncogene"} or quantified statements:\textit{``Oncogenes are responsible for cancer"} can be extracted and integrated into a knowledge graph~(KG) in three steps: i) \emph{NER} that recognizes the named entities scientific articles, ii) \emph{entity linking} to link extracted named entities, and iii) \emph{relation extraction}. 

NER is recognizing domain-specific proper nouns in a biomedical corpus. It can be performed by fine-tuning a domain-specific BERT variant such as BioBERT~\cite{BioBERT} and SciBERT~\cite{SciBERT} on relevant articles. For example, for the abstract: \emph{``Cyclooxygenase~(COX)-2 mRNA and protein expression were found to be frequently elevated in human pancreatic adenocarcinomas and cell lines derived from such tumours. Immunohistochemistry demonstrated cytoplasmic COX-2 expression in 14 of 21~(67\%) pancreatic carcinomas. The level of COX-2 mRNA was found to be elevated in carcinomas, relative to the histologically normal pancreas from a healthy individual, as assessed by reverse transcription-PCR.''}, an interpretable BioBERT~\cite{lee2020biobert}-based NER model~\cite{alonso2021named} would be able to recognize named entities classified as diseases, chemical or genetic, as shown in, where named entities are highlighted with different HMs.  

These facts can be extracted as a triplet fact in the form of $(u,e,v)=(\mathit{subject},\mathit{predicate},\mathit{object})$, where each triple forms a connected component of a sentence for the KG, e.g., for the sample text: \emph{``TP53 is responsible for a disease called BRCA\footnote{\scriptsize{Breast cancer}}. TP53 has POTSF\footnote{\scriptsize{Proto-oncogenes with tumour-suppressor function.}} functionality, which is mentioned in numerous PubMed articles''}, \emph{TP53}, \emph{disease}, \emph{BRCA}, \emph{POTSF}, and \emph{PubMed} are the extracted named entities. Then, by linking entities with the concept in a domain-specific ontology, the following relation triples can be generated and integrated into a KG: \texttt{(TP53, causes, BRCA), (TP53, hasType, POTSF), (BRCA, a, Disease), (POTSF, hasEvidence, PubMed)}. 

The full potential of an AI system can only be exploited by integrating both domains- and human knowledge~\cite{tocchetti2022role,zhu2022reasonchainqa,rajabi2022knowledge}. Inference rules~(IRs) are a straightforward way to provide automated access to deductive knowledge~\cite{hogan2020knowledge}. An IR encodes IF-THEN-style consequences: $p \rightarrow y$, where both body and head follow graph patterns in KG. Reasoning over KGs, using a reasoner help deduce implicit knowledge from existing facts~\cite{futia2020integration}. For example, assuming two facts \emph{``TP53 is an oncogene; oncogenes are responsible for cancer''} are already present, reasoning over the KG would entail an extended knowledge \emph{``TP53 is responsible for cancer''}. Then, as shown in \cref{fig:decision_reasoning_with_rules}, a doctor with their own expertise and by combining the facts from the KG, can explain the decision and provide further interpretation~(e.g., biomarkers and their relevance w.r.t specific cancer types)~\cite{karim_phd_thesis_2022}. Further, Kim \textit{et al.}~\cite{xian2019reinforcement} has shown that a neuro-symbolic system could be an effective means to generate casual reasoning if the KG contains sufficient knowledge. 

\subsection{Measure of interpretability and explainability} \label{sec:xai_measure}
AI explainability should have rigorous metrics to satisfy different needs in biomedical applications~\cite{han2022challenges}. However, when it comes to providing qualitative or quantitative measures of explainability, there is little consensus on what interpretability in ML is and how to evaluate it for benchmarking. Current interpretability evaluation typically falls into two categories. The first category evaluates explainability in a quantifiable way: a domain user/expert first claims that some model family, e.g., linear models, rule lists, RF, GBT are interpretable and then presents algorithms to optimize within that family~\cite{miller2018explanation}. The second category evaluates the explainability of an ML model in the context of its applications from a qualitative point of view.  

These approaches rely on the notion of \emph{``you will know it when you see it''}, where interpretability meet the first test of having face validity on a correct test set of subjects: human operators. However, this is too naïve, leaving many questions unanswerable, e.g., \emph{``are all models in all defined-to-be-interpretable''?} or \emph{``are all model classes equally interpretable?''}. However, answers to these questions can only be realized in terms of some metrics that can be used to evaluate the quality of explanations by an AI system. Kusner \textit{et al.}~\cite{kusner2017counterfactual} proposed a metric for measuring how fair decisions w.r.t counterfactuals. A decision $\hat y$ is fair for instance $x$ if the prediction is the same for both the actual world and the counterfactual world in which the instance belonged to a different demographic group. 

Since surrogate models are often used to explain the decisions, the quality of the explanations depends on the surrogate's predictive power~\cite{karim2022interpreting}. To measure how well a surrogate $f$ model has replicated a \emph{black-box} $f_b$, R-squared measure~($R^2$) is calculated as an indicator of goodness-of-fit. It is conceptualized as the percentage of variance of $f_b$ captured by the surrogate~\cite{molnar2020interpretable}: 

\begin{equation}
    R^{2}=1-\frac{SSE^{*}}{SSE} = 1 - \frac{{\sum_{i=1}^{N}}\left(\hat{y}^{*}_{i}-\hat{y}_{i}\right)^{2}}{\sum_{i=1}^{N}\left(\hat{y}_{i}-{\hat{y}}\right)^{2}},
    \label{eq:r_squared_1}
\end{equation}

where $\hat{y}^{*}_{i}$ is the prediction for $f$, $\hat{y}_{i}$ is the prediction for $f_b$ for $X^{*}$, and $SSE$ and $SSE^{*}$ are the sum of squares errors for $f$ and $f_b$, respectively~\cite{molnar2020interpretable}. The surrogate model $f$ can be used instead of the \emph{black-box} $f_b$, based on $R^2$ value:

\begin{itemize}
   
    \item If $R^2$ is close to $1$~(low $\mathrm{SSE}$), the surrogate model approximates the behavior of the \emph{black-box} model very well. Hence, $f_b$ can be used instead of $f$. 
    \item If $R^2$ is close to 0~(high $\mathrm{SSE}$), the surrogate fails to approximate the \emph{black-box}, hence cannot replace $f_b$. 
\end{itemize}

Comprehensiveness and sufficiency that is proposed to measure the quality of explanations in NLP. These metrics are based on the concept of rationales introduced by Zaidan \textit{et al.}~\cite{zaidan2007using} in NLP: where human annotators would highlight a part of the texts that could support their labelling decision\footnote{\scriptsize{For example, to justify why a review about a movie is positive, an annotator can highlight the most important words and phrases that would tell someone why to see the movie. While to justify why a review is negative, the annotator can highlight words and phrases that would recommend to someone why not to see the movie.}}. This concept is found useful across NLP tasks like sentiment analysis~\cite{herrewijnenmachine}. 
Rational explanation is key to understanding an AI system since translating the rationales results into usable and easily understandable formats. Let $f\left(x_{i}\right)_{c}$ be the original prediction probability for model $f$ for class $c$. To measure the sufficiency, they proposed to create contrasting example $\tilde{x}_{i}$ for each sample $x_{i}$, by removing predicted rationales $r_{i}$ as~\cite{deyoung2019eraser}: 

\begin{align}
    s  =  f\left(x_{i}\right)_{c} f\left(r_{i}\right)_{c},
    \label{eq:sufficiency}
\end{align}

where $s$ measures the degree to which extracted rationales are adequate for the model $f$ for making the prediction~\cite{deyoung2019eraser}. Let $f\left(x_{i} \backslash r_{i}\right)_{c}$ be the predicted probability of $\tilde{x}_{i}\left(=x_{i} \backslash r_{i}\right)$. Thus, it is expected the prediction to be lower on removing the rationales, where comprehensiveness $e$ is calculated as~\cite{deyoung2019eraser}: 

\begin{align}
    e  = f\left(x_{i}\right)_{c}-f\left(x_{i} \backslash r_{i}\right)_{c}.
\end{align}

\begin{table*}
    \centering
    \caption{SCT and it's usability questionnaires with interpretation~(adopted from~\cite{holzinger2020measuring})}
    \label{table:usability_questionnaires}
    \scriptsize{
    \begin{tabular}{l|l}
        \hline
         \textbf{Usability question}  & \textbf{Interpretation}  \\
        \hline
        \textbf{Factors in data}  & Data included all relevant known causal factors with sufficient precision and granularity \\ \hline
        \textbf{Understood} & Explanations are within the context of my work \\  \hline
        \textbf{Change detail level} & Level of detail can be changed on demand \\  \hline
        \textbf{Understanding causality} & No support was necessary to understand the explanations \\  \hline
        \textbf{Use with knowledge} & Explanations helped me understand the causality \\  \hline
        \textbf{No inconsistencies} & Explanations could be used with my knowledge base \\  \hline
        \textbf{Learn to understand} & No inconsistencies between explanations \\  \hline
        \textbf{Needs references} & Additional references were not necessary in the explanations \\  \hline
        \textbf{Efficient} & Explanations were generated in a timely and efficient manner. \\
        \hline
    \end{tabular}}
\end{table*}

The same can be conceptualized for our cancer example w.r.t leave-one-feature-out: the rationale can be computed based on the number of extracted features~(e.g., top-k genes) divided by the number of features in $x$. A prediction $\hat y$ is considered a match if the overlap with any of the ground truth rationales $r_i \geq 0.25$. A high value of comprehensiveness implies that the rationales were influential w.r.t prediction $\hat y$. If an AI system is useful, then it must somehow be interpretable and able to provide human-understandable explanations~\cite{bhatt2020explainable}. Hoffman \textit{et al.}~\cite{hoffman2018metrics} proposed following metrics to evaluate an AI system: 

\begin{itemize}
    \item The goodness of the explanations
    \item Whether users are satisfied by explanations
    \item How well users understand the AI systems
    \item How curiosity motivates the search for explanations
    \item Whether the user's trust on the AI is appropriate
    \item How the human-XAI work system performs.
\end{itemize}

System casuability scale~(SCS)~\cite{holzinger2020measuring} is another measure based on the notion of causability~\cite{holzinger2019causability}. It is proposed to quickly determine whether and to what extent an explainable user interface, an explanation, or an explanation process itself is suitable for the intended purpose~\cite{holzinger2020measuring}. 
SCS combines with concepts adapted from a widely accepted usability scale and SCS is computed based on responses from ten usability questionnaires listed in \cref{table:usability_questionnaires}. SCS system uses a 5 point scale: rating 1 = strongly disagree; 2 = disagree; 3 = neutral; 4 = agree; 5 = strongly agree. SCS is measured by dividing the acquired ratings by the total ratings, i.e., $SCS  = \sum_{i=1}^{10} \text {Rating}_{i} / 50$. 

\section{Interpretable ML frameworks and libraries} \label{sec:xai_library}
Following the timeline of XAI methods~(ref. \cref{fig:xai_timeline}), numerous frameworks and libraries are developed that are based on interpretable ML methods such as LIME~\cite{LIME}, model understanding through subspace explanations~(MUSE)~\cite{lakkaraju2019faithful}, SHAP~\cite{SHAP}~(its variants kernel SHAP and tree SHAP), PDP, individual conditional expectation~(ICE), permutation feature importance~(PFI), and counterfactual explanations~(CE)~\cite{wachter2017counterfactual}. We provide a list of some widely used interpretable tools, libraries, and links to their GitHub repositories: 

\begin{itemize}
    \scriptsize{
        \item \textbf{DeepLIFT}: \url{https://github.com/kundajelab/deeplift}
        \item \textbf{Xplique}: \url{https://github.com/deel-ai/xplique/}
        \item \textbf{DALEX}: \url{https://dalex.drwhy.ai/python/}
        \item \textbf{Alibi}: \url{https://github.com/SeldonIO/alibi}
        \item \textbf{SHAP}: \url{https://github.com/slundberg/shap}
        \item \textbf{LIME}: \url{https://github.com/marcotcr/lime}
        \item \textbf{ELI5}: \url{https://github.com/TeamHG-Memex/eli5}
        \item \textbf{InterpretML}: \url{https://github.com/interpretml/interpret}
        \item \textbf{Interpret-Text}: \url{https://github.com/interpretml/interpret-text}
        \item \textbf{ExplainerDash}: \url{https://github.com/oegedijk/explainerdashboard}
        \item \textbf{CNN viz}: \url{https://github.com/utkuozbulak/pytorch-cnn-visualizations}
        \item \textbf{iNNvestigate}: \url{https://github.com/albermax/innvestigate}
        \item \textbf{DeepExplain}: \url{https://github.com/marcoancona/DeepExplain}
        \item \textbf{Lucid}: \url{https://github.com/tensorflow/lucid}
        \item \textbf{TorchRay}: \url{https://facebookresearch.github.io/TorchRay/}
        \item \textbf{Captum}: \url{https://captum.ai/}
        \item \textbf{AIX360}: \url{https://github.com/Trusted-AI/AIX360}
        \item \textbf{BERTViz}: \url{https://github.com/jessevig/bertviz}
        \item \textbf{Bio-NER}: \url{https://librairy.github.io/bio-ner/}.
    }
\end{itemize}

These tools and libraries are developed with a view to improving the interpretability and explainability of \emph{black-box} ML models, covering general-purpose problems. In particular, the majority of these tools are developed for computer vision, text mining, or structured data, hence are not specialized to tackle bioinformatics/biomedical problems by default. Further, not each form~(e.g., image-, tree-, rule-, evidence-, or text-based) of explanations can be generated using a single tool. Therefore, these tools may need to be customized or even multiple tools may need to be combined, in order to develop XAI applications for bioinformatics by handling diverse data types.  


\section{Conclusions and outlook}\label{sec:con}
In this paper, we discussed the importance of interpretability in bioinformatics and provided a comprehensive overview of interpretable ML methods. Interpretability is a key to generating insights on why and how a certain prediction has been made by the model~(e.g., most important biomarkers exhibit shared characteristics while clustering patients having a certain cancer type, which can help in recommending more accurate treatments and drug repositioning). Through several examples covering bioimaging, genomics, and biomedical text mining, we showed how bioinformatics research could be benefited from interpretability and different means of explanations~(e.g., rules, plots, heatmaps, textual, and knowledge-based). Besides, we analyzed existing interpretable ML tools and libraries that can be employed to improve interpretability while solving complex bioinformatics research problems and use cases. 

Although interpretability could contribute to trustworthy AI~\cite{kim2019learning}, the benefits of explainability still need to be proven in practical settings~\cite{kazim2020explaining}. We argue that only interpretability alone cannot make an AI system fully reliable and trustworthy. There are other important considerations and potential challenges in the development and deployment phases -- especially for mission-critical application scenarios~(e.g., clinical). 
First, before building the interpretable model, decision-makers and domain experts should discuss: i) what kind of data to be used - imaging, texts, tabular, graphs?, ii) what types of explanations need to provide, e.g., visual-, tree-, rule-based, or textual?, iii) how could the full potentials of global interpretability be achieved in order to tailor and generalize the model better for unseen data?, and iv) how could the local interpretability be used if the model fails~(e.g., wrong diagnosis) so the model can be diagnosed before re-training? Besides, it is important to outline the decision-making process and potential impacts the model could have on the targeted users~\cite{kazim2020explaining}. 

Second, one of the first steps to improving an AI system is to understand its weaknesses. The better we understand which factors cause a model to make what prediction or which factors caused it to make wrong predictions, the easier it becomes to improve it~\cite{miller2018explanation}. However, such weakness analysis on \emph{black-box} or even interpretable models is not straightforward~\cite{bhatt2020explainable}, but requires close monitoring and debugging by zooming individual data points. 
Third, as outlined in co-badged guidance of XAI~\cite{kazim2020explaining}, explanations by AI-assisted decisions could potentially disclose commercially sensitive information as well as how an ML model works. Besides, there are potential risks with the rationale, fairness and data explanation types – information on how other similar individuals were treated and detail on input data for a particular decision. Therefore, it is important to limit detail such as feature weightings or importance, as well as carefully assess this risk as part of data protection. 

Fourth, AI systems may be vulnerable to imperceptible attacks~(e.g., adversarial attacks), biased against underrepresented groups~(e.g., lack of fairness in decision-making), and lack individual privacy protection that not only degrades user experience but also erodes society's trust. Therefore, an AI system should be robust to adversaries, which can only be achieved by taking reactive and proactive measures against adversarial attacks. Since an adversarially robust model can generate consistent and reliable predictions, the robustness property of an AI system may need to be formulated such that the predictions remain stable to small variations in inputs~(e.g., adding minor noise to inputs should not flip the prediction to completely different predictions). 
Fifth, an AI system should not only be able to provide \emph{meaningful information}~(e.g., clinical outcomes) and \emph{supporting explanations} in a human-interpretable form outlining why and how ML models reach a decision, but \emph{insights}~(e.g., treatment recommendations). 
Sixth, when using interpretable models, the biological relevance of top-ranked features~(e.g., cancer-specific biomarkers) needs to be validated both clinically and based on domain knowledge~(e.g., oncologists can combine their expertise with evidence from a domain knowledge graph). 

In this regard, EU guideline on robustness and explainability of AI~\cite{hamon2020robustness} emphasizes three aspects for the right deployment of AI: transparency, reliability, and individual data protection in AI models. We believe the techniques and methods discussed in this paper provide valuable insights into interpretable ML and XAI in general. We hope domain experts~(e.g., doctors, data scientists), lay persons~(e.g., patients), and stakeholders will be benefited too in order to accelerate bioinformatics research.  

\section*{{\textbf{Key points}:}}
\begin{itemize}
    \item \textbf{Post-hoc or ante-hoc interpretable methods?} Since linear models are easy to interpret and comprehend, the first choice should be building and explaining using inherently interpretable models, in an ante-hoc manner. However, depending on modelling complexity, a simple or surrogate model may not be able to fully capture the behaviour of a \emph{black-box} model. Consequently, it could lead to wrong decisions -- especially if the model surrogation process is not thoroughly evaluated~\cite{karim2022interpreting}. Therefore, it is recommended to build a \emph{black-box} model, followed by embedding interpretable ML logic into it and using the interpretable model to explain the decisions. 
   
    The latter approach can combine both ante-hoc and post-hoc approaches in a single pipeline. 
    
    \item \textbf{Local or global explainability?} 
    Global explainability is important for monitoring and having a holistic view of a model's interpretability~(e.g., what features are more important to the model?). Local explainability is important for individual instances without providing a general understanding of the model, e.g., what factors contributed most for a certain prediction?. Further, in case of wrong predictions, local explainability helps in diagnosing why the model has made wrong predictions for certain instances. 

    \item \textbf{What ML models/methods and explanations types?} The choice depends on problems, requirements, data types, as well as target users ~(decision recipients). For example, decision rules and counterfactuals are more effective at providing intuitive explanations compared to visualization-based explanations. The former is suitable for tabular data, while the latter is for imaging~\cite{molnar2020interpretable}. Further, since one-size-may-not-fit-all, several explanation types out of visual-, tree-, rule-, or text-based may need to be used. An AI system needs to build such that relevant information can be extracted for a range of explanation types~\cite{karim2022interpreting}. 
    
    \item \textbf{Accuracy or explainability?} Since achieving mediocre performance is not acceptable in many critical situations, accuracy and robustness are the two most important factors. However, depending on the complexity level of the problem at hand, data types and volume, balancing the performance vs. explainability trade-off could be difficult. As long as explainability is not critical, solving a complex problem and building an efficient model is more important.  

    \item \textbf{Can explainability help mitigate unfairness in the decision-making process?} Interpretability are two important properties that can help identify sensitive features that could drive a model to generate unfair decisions. This help takes proactive measures to ensure a decision would not reflect discriminatory behaviour toward certain groups or populations because of such attributes. 
    
    \item \textbf{Is explainability always necessary?} Higher interpretability means easier comprehension and easier explaining the predictions to target users~\cite {stiglic2020interpretability}. However,  all predictions may not require to explain. Solving AI learning security or fixing learning flaws could be more important than building an interpretable model, in application domains like disease diagnosis in translational bioinformatics~\cite{han2022challenges}.
    
    \item \textbf{Human-in-the-loop and domain knowledge?} HCI researchers should closely collaborate with domain experts~(e.g., data scientists, and doctors) -- especially during the development phase and utilize only state-of-the-art algorithms and interpretable ML methods. Such collaborations would help the AI developers design question-driven user interfaces capable of asking ``why'', ``how'', and ``what-if'' types of questions by the human operators from diverse domains and generate casual and contrastive  explanations. 

\end{itemize}

\section*{{\textbf{Conflict of Interest}}}
No conflict of interest to declare for this article.

\section*{{\textbf{Acknowledgment}}}
This article is loosely based on Ph.D. dissertation titled \emph{``Interpreting black-box machine learning models with decision rules and knowledge graph reasoning''}~\cite{karim_phd_thesis_2022} by the first author. 

\section*{\textbf{Abbreviations and Acronyms}}

{\small{
\begin{itemize}
 
    \item Attention layers~(ALs) 
    \item Artificial intelligence~(AI) 
    \item Autoencoders~(AEs)
    \item Bayesian rule lists~(BRL)
    \item Bayesian case Model~(BCM)
    \item Convolutional neural network~(CNN)
    \item Concept activation vectors~(CAVs)
    \item Counterfactual explanations~(CE)
    \item Contrastive layer-wise relevance propagation~(CLRP)
    \item Convolutional autoencoder~(CAE)
    \item Class activation mapping~(CAM)
    \item Causal concept effect(CaCE)
    \item Casual inferencing~(CI)
    \item Deep learning~(DL)
    \item Deep neural networks~(DNNs)
    \item Decision rules~(DRs)
    \item Decision tree~(DT)
    \item Explainable artificial intelligence~(XAI)
    \item Feature maps~(FMs)
    \item General data protection regulation~(GDPR)
    \item Generative discriminative models~(GDM)
    \item Global attribution mapping~(GAM)
    \item Gene expression~(GE)
    \item Guided back-propagation~(GB)
    \item Gradient-weighted class activation mapping~(Grad-CAM++)
    \item Global average pooling~(GAP)
    \item Globally averaged gradients~(GAG)
    \item Gradient boosted trees~(GBT)
    \item Graph neural networks~(GNNs)
    \item Heat maps~(HMs)
    \item Isometric mapping~(Isomap)
    \item Individual conditional expectation~(ICE)
    \item K-nearest neighbour~(KNN)
    \item Layer-wise relevance propagation~(LRP)
    \item Local interpretable model-agnostic explanations~(LIME)
    \item Local rule-based explanations~(LORE) 
    \item Long short-term memory~(LSTM)
    \item Machine learning~(ML)
    \item Model-agnostic counterfactual explanation~(MACE)
    \item Model understanding through subspace explanations~(MUSE)
    \item Natural language processing~(NLP)
    \item Neural additive models~(NAMs)
    \item Principal component analysis~(PCA)
    \item Partial dependence plots~(PDP)
    \item Peak response mapping~(PRM)
    \item Propositional self-attention networks~(PSAN)
    \item Prediction difference analysis~(PDA)
    \item Permutation feature importance~(PFI)
    \item Random forest~(RF)
    \item Rectified linear unit~(ReLU)
    \item Saliency maps~(SM)
    \item Self-attention network~(SAN)
    \item System causability scale~(SCS)
    \item Sequential covering~(SC)
    \item SHapley Additive exPlanations~(SHAP)
    \item Sensitivity analysis~(SA)
    \item Salient relevance~(SR)
    \item Spectral relevance analysis~(SpRAy)
    \item Shapely values~(SVs)
    \item Singular value decomposition~(SVD).
    \end{itemize}
    }
}

\section*{Author details}
\noindent \textbf{Md. Rezaul Karim} is a Senior Data Scientist at Fraunhofer FIT and a Postdoctoral Researcher at RWTH Aachen University, Germany. Previously, he worked as a Machine Learning Researcher at Insight Centre for Data Analytics, University of Galway, Ireland. Before that, he worked as a Lead Software Engineer at Samsung Electronics, South Korea. He received his PhD from RWTH Aachen University, Germany; M.Eng. degree from Kyung Hee University, South Korea; and B.Sc. degree from the University of Dhaka, Bangladesh. His research interests include machine learning, knowledge graphs, and explainable AI~(XAI) with a focus on bioinformatics. \\

\noindent \textbf{Tanhim Islam} is a Data Scientist at CONET Solutions GmbH, Germany. He received his BSc. degree in Computer Science from the BRAC University, Bangladesh and MSc. from RWTH Aachen University, Germany. His research interests include applied machine learning, NLP, and explainable AI. \\

\noindent \textbf{Oya Beyan} is a Professor of medical informatics at University of Cologne, Faculty of Medicine, and University Hospital Cologne. Prof. Beyan is the director of Institute for Biomedical Informatics which aims to support data-driven medicine and digital transformation in healthcare. Prof. Beyan's research team focuses on health data reusability, semantic interoperability, and data science towards continuous improvement of healthcare through innovation and creating new knowledge. Prof. Beyan co-leads the MEDIC data integration center at Medical Faculty of Cologne, where multi-modal health data is integrated and reused for research. She is also affiliated with Fraunhofer FIT, where she leads a research and innovation group for FAIR data and distributed analytics. \\

\noindent \textbf{Christoph Lange} is the Head of Data Science and Artificial Intelligence department at Fraunhofer FIT, Germany, and a Senior Researcher at RWTH Aachen University. He received his Ph.D. in computer science from Jacobs University Bremen, Germany. His research is broadly concerned with knowledge engineering and data infrastructures: choosing the right formalism domain to represent domain knowledge in an expressive and scalable way following Linked Data and FAIR principles, thus enabling and facilitating machine support with sharing, publishing, and collaborative authoring. \\

\noindent \textbf{Michael Cochez} is an Assistant Professor at Vrije Universiteit Amsterdam, the Netherlands. He was formerly a Senior Researcher at Fraunhofer FIT, Germany. He received his Ph.D. and M.Sc. degrees in Mathematical Information Technology from the University of Jyv\"askyl\"a, Finland, and B.Sc. degree in Information Technology from the University of Antwerp, Belgium. His research interests include knowledge representation, deep learning, machine learning, and question answering over knowledge graphs. \\

\noindent \textbf{Dietrich Rebholz-Schuhmann} is a Professor of Medicine at the University of Cologne and the Scientific Director of the Information Center for Life Sciences, German National Library of Medicine~(ZB MED), Germany. He was formerly the Director of the Insight Centre for Data Analytics, and Professor of Informatics at the University of Galway, Ireland. Before that, he was the Director of Healthcare IT at a Heidelberg, Germany-based bioscience company and Group Leader of Semantic Data Analytics at European Bioinformatics Institute~(EMBL-EBI). His research interests include data science, semantics-driven data analytics, biomedical text mining, and bioinformatics. \\

\noindent \textbf{Stefan Decker} is the Chair and Professor for Information Systems and Databases at RWTH Aachen University and Managing Director of Fraunhofer FIT, Germany. He was formerly the Director of the Insight Centre for Data Analytics, and Professor of Informatics at University of Galway, Ireland. Even before that, he worked as Assistant Professor at the University of Southern California and as a Postdoctoral Researcher at Stanford University, USA. His research interests include Semantic Web and linked data, and knowledge representation. 

\bibliographystyle{natbib}
