mimic iv

[netanelcyber]

Hi MIT-LCP team,

I’m reaching out specifically in the context of MIMIC’s role as a foundational infrastructure for reproducible clinical research.

While developing a reproducible ICU pathogen prediction pipeline using MIMIC-IV (https://github.com/netanelcyber/PenuX), we encountered what appears to be a systematic evaluation gap that may be relevant to the broader MIMIC ecosystem.

Core observation

Across multiple standard modeling approaches built on MIMIC-IV-derived cohorts, we consistently observed that:

• AUROC and similar discrimination metrics remain relatively stable across pipelines
• however, model reliability significantly varies under:
  • temporal validation (vs random splits)
  • subgroup stratification (ICU type / severity / missingness regimes)
  • minor but realistic preprocessing changes (windowing, aggregation, imputation strategy)

Most importantly, these effects appear not model-specific, but instead evaluation-design dependent.

Why this is relevant to MIMIC’s role

Given that MIMIC is widely used as a benchmarking substrate for clinical ML, this suggests a potential gap between:

standardized cohort/feature construction (MIMIC Code)
and
standardized evaluation under temporal and operational shift

In practice, this means that two studies using identical MIMIC Code pipelines may still report substantially different “robustness” depending on evaluation protocol choices that are currently not standardized.

Proposal (infrastructure-level, not project-level)

Rather than addressing this at the level of individual models, it may be useful to consider whether the MIMIC ecosystem could benefit from an optional evaluation extension layer, for example:

• standardized temporal split utilities for ICU tasks
• calibration reporting templates (beyond AUROC/PR-AUC)
• subgroup robustness