\section{Introduction}
Science requires reproducibility, but many sub-fields of science have recently experienced a reproducibility crisis, eroding trust in processes and results and potentially influencing the rising rates of scientific retractions \cite{baker_1500_2016,cokol_retraction_2008,vasilevsky_reproducibility_2013}.
Reproducibility is also critical for machine learning research, whose goal is to develop algorithms to reliably solve complex tasks at scale, with limited or no human supervision. Failure of a machine learning system to consistently replicate an intended behavior in a context different from which that behavior was defined may result in dramatic, even fatal, consequences~\cite{levin_tesla_2018}.
Ranking prominently among machine learning applications that may put human lives at stake are those related to Machine Learning for Health (\textsc{ML4H}\xspace). In a field where applications are meant to directly affect human health, findings should undergo heavy scrutiny along the validation pipeline from research findings to applications deployed in the wild. For example, in 2018, 12 AI tools using \textsc{ML4H}\xspace algorithms to inform medical diagnosis and treatment were cleared by Food and Drug Administration (FDA) and will be marketed to and potentially used by millions of Americans~\cite{muoio_roundup:_2018}. Verifying the reproducibility of the claims put forward by the device manufacturer should thus be a main priority of regulatory bodies~\cite{parikh2019regulation}, extending the need for reproducible \textsc{ML4H}\xspace results beyond the machine learning research community.




Unfortunately, several  factors relating to the availability, quality, and consistency of clinical or biomedical data make reproducibility especially challenging in \textsc{ML4H}\xspace applications.
In this work, we make several contributions. First, we present a taxonomy of reproducibility tailored to \textsc{ML4H}\xspace applications, and designed to capture reproducibility goals more broadly. Second, we use this taxonomy to define several metrics geared towards quantifying the particular challenges in reproducibility faced within ML4H, and conduct a comprehensive review of the published literature to support our claims and compare \textsc{ML4H}\xspace to machine learning more generally. Finally, we build on this analysis by exploring promising areas of further research for reproducibility in \textsc{ML4H}\xspace.


\section{A Reproducibility Taxonomy}
The common understanding of reproducibility in machine learning can be summed up as follows:
A machine learning study is \textit{reproducible} if readers can fully replicate the exact results reported in the paper.
We will call this concept \textit{technical replicability}, as it is centrally concerned with whether or not one can replicate the precise, technical results of a paper under identical conditions. Though intuitive, we argue technical replicability is actually only a small part of the goal of ``reproducibility'' more generally. This discrepancy has been noted historically in various ways \cite{drummond_replicability_2009,gundersen_2018,joelle_pineau_reproducible_2018,plesser_reproducibility_2018} and is made apparent by the use of the term in the natural and social sciences, where attempted reproductions will often occur at different labs, using different equipment/staff, etc.
We argue that in order for a study to be fully reproducible, it must meet three tiers of replicability:

\SetLabelAlign{parright}{\parbox[t]{\labelwidth}{\raggedleft#1}}
\begin{description}[
leftmargin=!,labelwidth=\widthof{\bfseries Conceptual Replicability},align=parright
]
    \item[Technical Replicability] Can results be replicated under \emph{technically} identical conditions?
    \item[Statistical Replicability] Can results be replicated under \emph{statistically} identical conditions?
    \item[Conceptual Replicability] Can results be replicated under \emph{conceptually} identical conditions
\end{description}

\emph{Technical replicability} refers to the ability of a result to be fully technically replicated, yielding the precise results reported in the paper. For example, this entails aspects of reproducibility related to code and dataset release. \emph{Statistical replicability} refers to the ability of a result to hold under re-sampled conditions that yield different technical configurations, but should not statistically affect the claimed result (e.g., a different set of random seeds, or train/test splits). Note this is related, but not identical, to the notion of \emph{internal validity}~\cite{campbell1986relabeling} commonly used in social science research. Similarly, \emph{conceptual replicability} is heavily related to \emph{external validity}~\cite{pearl2014external}, as it describes the notion of how well the desired results reproduces under conditions that mach the conceptual description of the purported effect. Note that this is task-definition dependent; claiming a task has a greater conceptual horizon of generalizability makes it harder to satisfy this reproducibility requirement.

All three of these replicability criteria are central for full reproducibility: without technical replicability, one's result cannot be demonstrated. Without statistical replicability, one's result will not reproduce under increased sampling and the presence of real-world variance. And lastly, without conceptual replicability, one's result does not depend on the desired properties of the data, but instead depends on potentially unobserved aspects of the data generation mechanism that, critically, \emph{will not reproduce when deployed in practice}.

Under each of these lenses, ML4H differs from general machine learning domains in critical ways and presents unique challenges.

\section{Core Reproducibility Challenges in ML4H}
In this section, we illustrate both through qualitative arguments and a quantitative literature review that ML4H lags behind other subfields of machine learning on various reproducibility metrics. Our literature review procedure entailed manually extracting and annotating over 300 papers from various venues, spanning ML4H, Natural Language Processing (NLP), Computer Vision (CV), and general machine learning (general ML).\footnote{NLP and CV were chosen to represent broad fields with a significant applied focus, much like ML4H. General ML was chosen to have an unbiased comparison to the field more broadly.} The full procedure used for this statistical review is detailed in the Appendix (Section~\ref{sec:review_procedure}), and final quantitative results on several key metrics can be seen in Figure~\ref{fig:paper_stats}, though they are also detailed in the text where appropriate. The full set of results is available at https://github.com/evidation-health/ICLR2019.

\begin{figure}[t]
    \centering
    \includegraphics[width=1\textwidth]{paper_stats.pdf}
    \caption{Fraction of papers satisfying certain conditions by ML field. See the Appendix (Section~\ref{sec:review_procedure}) for detailed descriptions of the underlying data collection procedure. Note that ML4H consistently lags other subfields of machine learning on all measures of reproducibility save inclusion of proper statistical variance.
    }
    \label{fig:paper_stats}
\end{figure}

\subsection{Technical Replicability}
ML4H faces several key challenges with regards to technical replicability. Firstly, health data is privacy sensitive, making it difficult to release openly without either incurring risks of re-identification, or severely diminishing utility by applying aggressive de-identification techniques~\cite{dwork_ullman_2018}. As a result, few public datasets are available, and those that are available are used extremely frequently, leading to a risk of dataset-specific over-fitting. To this point, approximately only 51\% of the ML4H papers we examined used public datasets, as compared to over 90\% of both CV and NLP papers, and approximately 77\% of general ML papers.

ML4H scores even more poorly when it comes to code release, preprocessing specification, and cohort description; only approximately 13\% of the papers we analyzed released their code publicly, compared to approximately 37\% in CV and 50\% in NLP. A previous study that reviewed the prevalence of code release in AI reported that only 6\% of papers released code \cite{gundersen_2018}, which is lower than all of our estimates. This discrepancy may be due to sampling papers from different conferences and different time periods (2013-2016 vs. 2018 for the current study). Specific prior works have also already examined the prevalence of code release and dataset cohort sub-selection within ML4H, finding that even when restricting focus to public datasets, studies often do not release code or include sufficiently informative text to enable a full technical reproduction \cite{hegselmann_reproducible_2018,johnson2017reproducibility}.  Note that code release is itself not necessarily sufficient for full technical replicability; even when code is released, it may not run correctly, it may exclude critical details, or it may fail to generate the results reported in the paper.


\subsection{Statistical Replicability}
To assess the state of statistical replicability in ML4H, we quantified how often papers describe the variance around their results (e.g., by listing both the mean and the standard deviation of a performance metric over several random splits). Interestingly, while ML4H's rate of this is still relatively low (approximately 38\%), it is higher than that of CV, NLP, or the general domain (17\%, 33\%, and 32\%, respectively).

Though this is an encouraging sign, we feel there is still significant room for improvement.
Even in other fields of machine learning, with arguably less complex data types, repeated studies have shown that published models fail to statistically reproduce when appropriate statistical procedures are implemented and fair hyperparameter search/preprocessing methods are used: in \cite{recht_imagenet_2018}, researchers find that published models evaluated using the public ImageNet test set show consistent performance drops when trained/tested on other random splits within ImageNet, and in \cite{melis_state_2017,lucic_are_2017}, researchers find that published performance deltas between various model architectures for language models and generative adversarial networks, respectively, fail to persist under more robust hyperparameter search and statistical comparison techniques. In \cite{henderson_deep_2017}, researchers examine practices in deep reinforcement learning which can limit or enhance studies' reproducibility. Due to our inability to technically replicate many ML4H works given dataset and code release issues, we cannot assess the extent of these issues to the same degree in ML4H. However, it seems doubtless that similar problems affect our field, perhaps to an even greater degree given that our datasets are frequently relatively small, high dimensional, sparse/irregularly sampled, and suffer from high rates of noise.

\subsection{Conceptual Replicability}
The critical issue in ML4H preventing conceptual replicability is the lack of multi-institution datasets in healthcare and the lack of usage of those that do exist. Whereas approximately 83\% of CV studies and approximately 66\% of NLP studies used multiple datasets to establish their results, only approximately 19\% of ML4H studies did. Using only a single dataset is potentially dangerous, as it is known that developing ML4H models that generalize over changing care practices or data formats is challenging. In \cite{gong_predicting_2017}, researchers demonstrate the dangers of training a model on raw data from one institution and transferring to another (using a simulated institutional divide within the MIMIC dataset), and in \cite{nestor_rethinking_2018}, researchers establish that without using manually engineered representations, ML4H models exhibit significant degradation in performance over time as care patterns evolve. These results are not surprising; health data is rife with hidden confounders, differs significantly between data collection and deployment environments, drifts over time, and further differs in structure and concept between different healthcare institutions~\cite{caruana2015intelligible}.

\section{Opportunities for Improvement}
In this section, we present several practical suggestions for enhancing reproducibiltiy for ML4H from the perspective of three main \textsc{ML4H}\xspace stakeholders: the \textsc{ML4H}\xspace research community, data providers, and related journals and conferences.  Figure~\ref{fig:recommend} summarizes these suggestions.

\begin{figure}
    \centering
    \includegraphics[width=0.6\columnwidth]{recommendations.pdf}
    \caption{Summary of recommendations for different stakeholders.
    }
    \label{fig:recommend}
\end{figure}

\subsection{Create Shared Research Resources}
Data providers such as hospitals, clinical research centers, and government agencies produce vast amounts of valuable data. Unfortunately, as suggested by our literature review, few datasets are available for a wide range of researchers to explore. We call for more instances of large data trusts where medical institutions can anonymously pool data for researchers to use and create algorithms. Several prominent examples already exist for the field to draw on in creating such repositories, including MIMIC~\cite{johnson2016mimic}, the U.K and Japan Biobanks~\cite{nagai2017overview,sudlow2015uk}, eICU~\cite{pollard_eicu_2018}, the Temple University Hospital EEG Corpus~\cite{harati2014tuh}, and Physionet~\cite{goldberger2000physiobank}. This recommendation is especially compelling as corporate entities increasingly invest in the ML4H space, particularly with regards to data~\cite{rajkomar_scalable_2018,wood_improving_2019}. Anchoring our research progress to large scale datasets in non-public or non-academic hands poses a dangerous precedent, especially in light of recent patent filings~\cite{bresnick_google_2019,google_inc._system_nodate}.

\subsection{Integrate Multi-Institute Datasets}
Multi-institute datasets enable studies to assess their ability to translate to new contexts, a critically understudied facet of ML4H research. Recent strides have been taken in this domain with the release of the eICU dataset~\cite{pollard_eicu_2018}, one of ML4H's first large scale, multi-institution EHR datasets, and researchers are already analyzing how to form generalizable models using this resource \cite{johnson_generalizability_2018}. Lastly, Observational Health Data Sciences and Informatics (OHDSI)~\cite{hripcsak2015observational} provides a mechanism to run observational health studies across multiple institutions and countries.
We encourage more collaborative efforts in this area from data providers to improve conceptual replicability.

\subsection{Prospectively Collect Data}
Data collected as a by-product of care, then later released for research purposes (e.g., MIMIC \cite{johnson_mimic-iii_2016}), presents serious privacy risks and contains many confounding variables. The landscape of these privacy risks and the nature of the confounding variables change (though they do not necessarily lessen) if data is instead prospectively collected directly from new, consented participants. These types of data collection regimes are logistically challenging, but are possible.
The NIH's All of Us Research Program \cite{national_institutes_of_health_all_nodate}, Evidation's DiSCover Project \cite{evidation_health_digital_2018}, and Google's Project Baseline \cite{verily_why_2017}
are examples of these approaches.


\subsection{Adopt Rigorous Statistical Methods}
ML4H researchers should be more rigorous in the development, refinement, and dissemination of statistical best practices (e.g., the proper procedures for model comparisons, etc.). Holding ourselves to high standards of statistical rigor, potentially including periodic statistical audits of our own statistical replicability, will help ensure we mitigate the problems other fields have found with ``over-fitting via publication,'' e.g. \cite{recht_imagenet_2018}.  Several commonly-used challenges already exist in ML4H with fixed train/test sets (e.g., \cite{ghassemi_you_2018,stubbs_annotating_2015}) which could be used for these kind of post-hoc replication studies.

\subsection{Develop New Privacy-Preserving Analysis Techniques}
Technological solutions can also be employed to help mitigate privacy concerns. ML4H researchers can explore noised, fully- or partially- simulated, or encrypted datasets. In cases where data cannot be released, techniques to train distributed models without sharing data have been proposed~\cite{vepakomma2018split}. Synthetic data can be an excellent tool to help enable researchers to still meaningfully release their code with full end-to-end realizations of their pipeline, as is done in, e.g., \cite{zink_fair_2019}. Technology for producing synthetic patient-level data also already exists \cite{walonoski_synthea:_2018}.

\subsection{Propose Pre-Registration Alternatives}
In the biomedical sciences, ``observational studies,'' under which definition nearly all ML4H research falls, undergo intense scrutiny to ensure they are not susceptible to statistical artifacts---in particular, increasingly these studies are required to be \emph{pre-registered},\footnote{
Meaning they are required to report their goal and planned preprocessing/modeling scheme before they run any experiments in order to avoid intentional or unintentional statistical fraud. This idea is not without its detractors \cite{lash_commentary:_2012,editors_registration_2010}.
} a move promoted by scientists and major journals alike \cite{lancet_should_2010,loder_registration_2010,williams_registration_2010}. Such prospective checks make unintentional statistical fraud more difficult, but are utterly absent in ML4H. A verbatim application of the pre-registration practice in use in the epidemiology community is likely an impractical solution due to the intrinsic exploratory nature of model development, but other techniques, such as systematic re-release of new data and/or rotation of official train/test splits have been found to help reduce the presence of statistical errors in other fields. ML4H researchers and academic publishers should engage in serious conversation on what the best vehicle for these checks and balances are.



\subsection{Establish Reporting Requirements and Require Code Release}
If conferences and journals required greater rates of data/code release, or the use of additional data/code availability statements at publication, this would instill a significant pressure on the field to address some of the foundational barriers to reproducibility. Conferences and journals have the additional ability to insist on high standards for the reporting of statistical variance around results, hyperparameter search procedures, and evaluation mechanisms, each of which would help ensure that we maintain high standards of technical and statistical replicability.

\subsection{Develop Data Standards and Reporting Standards}
Collaborative efforts in developing data standards and reporting standards is another avenue for improving reproducibility. Healthcare analytics organizations have developed data standards like the OMOP standard \cite{overhage_validation_2012} or the FHIR standard \cite{hl7_introducing_2018}, but they are not commonly adopted in \textsc{ML4H}\xspace research. Increased use of standards would make it easier to technically and conceptually replicate \textsc{ML4H}\xspace studies. Similarly, when \textsc{ML4H}\xspace datasets are created, we need greater descriptions of their contents, potential confounders and biases, missing data prevalence and distribution, and how they were created. Increasing the use of ``specs'' or ``datasheets'' describing datasets would help allay these concerns---efforts in the broader machine learning community are also approaching this goal \cite{gebru_datasheets_2018} and we should vigorously adopt these practices in \textsc{ML4H}\xspace.



\section{Conclusion}
In this work, we have framed the question of reproducibility in \textsc{ML4H}\xspace around three foundational lenses: technical, statistical, and conceptual replicability. In each of these areas, we argue both qualitatively and quantitatively, through a manual, extensive review of the literature, that \textsc{ML4H}\xspace performs worse than other machine learning fields in several reproducibility metrics we have identified. While keeping in mind the intrinsic challenges of data acquisition and use that plague the field, we highlight several areas of opportunities for the future, focused around improving access to data, expanding our trajectory of statistical rigor, and increasing the use of multi-source data to better enable conceptual reproducibility.

\section*{Acknowledgments}
This paper benefited substantially from the help of many people. Most notably, Bret Nestor, Amy Lu, Denny Wu, Elena Sergevea, and Di Jin all helped annotate papers for our analysis.

Additionally, this work was funded in part by National Institutes of Health: National Institutes of Mental Health grant P50-MH106933, a Mitacs Globalink Research Award, as well as a CIFAR AI Chair at the Vector Institute, and an NSERC Discovery Grant.

