Clinical AI Research Is More Than Building a Model: Why We Built ClinAI
Updated: Oct 8
Artificial intelligence in healthcare is advancing rapidly. New models are being developed for diagnosis, prediction, triage, clinical decision support and many other applications.
But developing the model is often only one part of the challenge.
Behind almost every clinical AI project is a much larger research journey: finding and understanding the data, defining the cohort, preparing the analysis environment, reproducing methods, tracking model versions, interpreting results clinically, documenting limitations and eventually determining whether the model performs beyond the population in which it was developed.
It was this broader challenge that led us to develop ClinAI.
The clinical AI research journey is fragmented
Clinical AI projects rarely happen within a single system.
The dataset may be stored in one environment. Analysis code may sit in a repository. Computing may occur locally or in the cloud. Results appear in spreadsheets, notebooks and presentations. Research documentation sits somewhere else again.
As a project evolves, it can become increasingly difficult to answer seemingly simple questions:
Which dataset was used?
How was the final cohort created?
Which version of the analysis generated these results?
What model was evaluated?
What were its limitations?
Could another researcher reproduce the work?
These questions become even more important when an AI model moves beyond development towards external validation or potential clinical use.
We encountered this ourselves
ClinAI did not begin as an abstract software concept. It evolved from challenges we encountered while conducting our own clinical AI research.
One example was our work reproducing a published sepsis study using the MIMIC-IV critical care database.
At first glance, reproducing a published study sounds straightforward: obtain the dataset, run the published code and compare the results.
The reality was considerably more complicated.
The project required reconstructing the research environment, preparing the database, managing very large clinical tables, troubleshooting published code, and carefully reviewing derived clinical variables and time relationships.
Some processing exceeded the available computing resources and needed to be redesigned or rerun. More importantly, apparently technical decisions often required clinical interpretation.
By the end of the process, we had much more than a model or a set of results. We had a chain of research decisions connecting the original dataset to the cohort, analysis and final research outputs.
That chain needed to be visible and reproducible.
From that experience came ClinAI
We began developing ClinAI as a structured environment around the clinical AI research lifecycle.
The idea is not to replace the tools researchers already use for statistics, coding or machine learning.
Instead, ClinAI is intended to connect the work around those tools.
A project can maintain a structured record of its research question, datasets, computing environment, analyses, models and outputs—providing a clearer view of how the research has progressed.
At its simplest, we think of the journey as:
Dataset → Cohort → Analysis → Model → Evaluation → Research Outputs
As projects mature, that pathway becomes more extensive:
Intended use → Validation population → Performance → Subgroup performance → Limitations → Reproducibility → External validation → Clinical evaluation → Deployment monitoring
Why reproducibility matters
A model reporting good performance is not enough.
To interpret that performance, researchers need to understand the data's source population, inclusion and exclusion criteria, missing data, preprocessing decisions, model configuration, and the context in which the outcome was defined.
Without this information, even impressive performance metrics can be difficult to interpret and even harder to reproduce.
This becomes particularly important when a model is evaluated in another hospital or patient population.
External validation is not simply running the same algorithm on another dataset. Differences in patient populations, clinical workflows, documentation, measurement practices and healthcare systems can all affect performance.
The evidence surrounding the model therefore matters as much as the model itself.
Building around real research
We are deliberately developing ClinAI alongside real clinical AI projects.
Our MIMIC-Sepsis work provided one early use case, allowing us to test how a published research project could be reconstructed and represented within a structured workspace.
Our other clinical AI research is helping us identify different requirements and refine the platform further.
This approach has been important to us.
Rather than deciding what researchers should need and building around assumptions, we want ClinAI to evolve from the practical problems encountered while conducting clinical research. There is still much to build and learn.
Where we hope ClinAI can contribute
Healthcare does not necessarily need more isolated AI models.
It needs better ways of determining which models are credible, reproducible, appropriately validated and suitable for the clinical environments in which they may eventually be used.
That requires clinicians, researchers, data scientists and health services to understand not only the final algorithm, but the evidence and decisions that surround it.
Our aim with ClinAI is to help make that journey more visible.
From a clinical question, through data and analysis, towards reproducible evidence, validation and ultimately clinical translation.
ClinAI is still evolving, and our own research projects continue to shape its development.
If you are working with a healthcare dataset, reproducing published clinical AI research, developing a model or considering external validation, we would be interested in hearing about your experience.
Comments