Data Science and ML Feasibility: The Model We Recommended They Not Build

The engagement deliberately did not start with modelling. It started with whether the data could support a model at all.

Label quality and outcome definition: the client had 41,000 historical loans — a respectable volume — but default was recorded inconsistently across three system migrations, and loans restructured rather than defaulted were coded differently depending on which relationship manager handled them. After normalization, the number of cleanly labelled defaults was 780, concentrated in two sectors, in a single macroeconomic period.

Feature availability at decision time: this was the finding that changed the engagement. Many of the strongest historical predictors — updated management accounts, covenant compliance data, banking transaction feeds — were present in the archive only because they had been collected during monitoring, after the loan was issued. They would not exist at the moment of the underwriting decision. Envion demonstrated this by building the leaky model and the honest one side by side: 0.88 AUC versus 0.61.

Regulatory constraint: adverse credit decisions require explanation under the client's national regulator — not fatal, but it eliminated several architectures the vendor's proposal had assumed. Baseline comparison: Envion also measured the existing manual process, which the client had never done — underwriter decisions were more consistent than assumed, and a simple scorecard reproduced 84% of them.

The assessment ran five weeks and covered the data readiness audit, the ML feasibility analysis and the technology strategy. The two data scientists already hired were retained and redeployed onto the data foundation work — which is what they spent their first year doing anyway, just with a clear mandate instead of a stalled model project.

Practical rules from the project: check whether features will be available at decision time — target leakage is the single most common cause of a model that dazzles in backtest and disappoints in production, trivially detectable in advance and almost never checked; count labelled events, not rows — a large dataset with a rare, inconsistently recorded outcome is a small dataset wearing a costume; and measure the existing manual process before replacing it — teams routinely discover their humans are more consistent than assumed, which reframes the problem from accuracy to throughput.