Challenge
The task was to identify customers at risk of a late refill from historical transactions. The rare positive class, time-dependent features and changing customer behaviour make a random train/test split misleading.
Selected work
Applied ML case study · 2026
Late refill risk modelling
A leakage-aware temporal modelling pipeline for prescription refill risk, with calibration and explicit analysis of dataset shift.
My role
I built temporal features, designed a chronological validation strategy and compared logistic regression, random forest and XGBoost models with probability calibration.
Outcome
The later test period exposed a substantial performance drop. Instead of presenting an optimistic validation score, the analysis concludes that the current model should not be deployed.
At a glance
0.0866test PR-AUC
0.0517test prevalence
0test F1 at chosen threshold
System view
The workflow.
A simplified view of the stages and boundaries that shape the project.
- 01Transactions
- 02Temporal features
- 03Chronological split
- 04Model training
- 05Calibration
- 06Shift analysis
Approach
Decisions that shaped the work.
Make time part of the evaluation
Features only use information available at prediction time, and validation/test periods follow training chronologically.
Optimise for the real class balance
PR-AUC, calibration and threshold behaviour are more informative than accuracy on a heavily imbalanced target.
Treat failure as evidence
A threshold selected on validation produced no useful test predictions. That is a deployment blocker, not a metric to hide.
Findings
What the evidence says.
- The model retained some ranking signal above test prevalence but not enough for the selected operating point.
- The validation-to-test gap is consistent with temporal dataset shift.
- A production iteration would require monitoring, re-training policy and a cost-sensitive threshold agreed with stakeholders.
Engineering reflection
This project is valuable because it demonstrates restraint: the right outcome of an ML experiment can be a well-supported decision not to deploy.