Auto finance · delinquency and charge-off · a label that changed meaning mid-year

Credit Risk Early Warning & Collections Prioritisation

Collections is a prioritisation problem disguised as a prediction problem. Knowing an account will charge off is worth nothing on its own. The value is entirely in knowing early enough that an intervention still changes the outcome, and in ranking accounts by how much the intervention is worth rather than by how likely the loss is. Then, halfway through the year, a payment-holiday programme detached the delinquency flag from the behaviour it had always proxied for, and the models were fluent in a vocabulary that no longer described the portfolio. The rebuild is the interesting part of this project.

Reconstructed

I do not hold the source for this system. The architecture and the reasoning are my own account; the diagram is a reconstruction rather than a rendering of a repository.

My résumé is the source of truth for outcomes. Outcome figures are labelled attested; scale figures are labelled illustrative.

2020mldataevaluation

What I owned

Primary architect and modeller. I owned the end-to-end design: the feature layer over servicing and payment history, the model topology, the imbalance and threshold strategy, the ranking-to-treatment handoff, the drift monitoring, and the executive reporting, and rebuilt the segmentation when forbearance invalidated the original target.

open the diagram →

Model accuracy: 87%attestedModel accuracy
Recall: 81%attestedRecall
Portfolio loss: −20%attestedPortfolio loss

10components
2request paths
3load stages
3decisions, argued
1failures, owned

Built onPythonscikit-learn / XGBoostSQLTableau

01Problem 02My role 03Architecture 04Decisions 05What broke 06Impact

The problem

What was actually wrong

An auto-finance portfolio loses money in a predictable sequence: an account misses a payment, then misses more, then reaches a point where the loss is already determined and collections activity only affects recovery rather than prevention. The window in which an intervention changes the outcome opens and closes early.

The existing process worked the portfolio by days past due, which is a lagging indicator of exactly the thing you want to get ahead of. By the time an account is prominent in a days-past-due queue, the useful window has often shut.

Then the constraint changed. A pandemic payment-holiday programme let customers defer payments without penalty, which meant the delinquency flag (the target variable, the primary feature, and the operational trigger, all at once) stopped meaning what it had meant when the training data was collected.

Before this existed

Collections queues ordered by days past due and exposure, with outreach effort spread evenly across the delinquent population.

Constraints

The conditions the design had to hold under

Constraints are the interesting part of an architecture. Without them any diagram looks reasonable.

The useful prediction window is early and narrow: A model that is accurate at 60 days past due is describing a loss that has already happened. Accuracy at the point of intervention is the only accuracy that pays.
The portfolio is severely imbalanced: Charge-offs are a small fraction of accounts. Any model optimised for accuracy will predict 'no charge-off' for everything and score extremely well.
Outreach capacity is fixed and small: The collections team can contact a bounded number of customers per cycle. The model's output is a ranking against that capacity, not a classification of the portfolio.
The target variable's meaning was not stable: Mid-year, forbearance decoupled the delinquency flag from payment distress in both directions: some deferred customers were fine, and some paying customers were not.

My role

Primary architect and modeller. I owned the end-to-end design: the feature layer over servicing and payment history, the model topology, the imbalance and threshold strategy, the ranking-to-treatment handoff, the drift monitoring, and the executive reporting, and rebuilt the segmentation when forbearance invalidated the original target.

Designed

The split between a delinquency-roll model and a charge-off model, because they answer different operational questions at different points in the account's life
The feature layer over payment history, exposure, contract terms and vehicle-value position
The ranking-to-treatment handoff: the model orders accounts against a fixed outreach capacity rather than classifying the portfolio
The threshold strategy for a severely imbalanced target, chosen on the cost asymmetry rather than on a symmetric metric
The forbearance-aware segmentation that replaced the original target definition when the programme launched
The drift monitoring that made the target-definition break visible as a data problem rather than as a modelling mystery

Built

The delinquency and charge-off models, at 87% accuracy and 81% recall
Three production models above 85% accuracy for 30- and 45-day delinquency segmentation under the forbearance regime
The scoring pipeline and the ranked collections queue it feeds
The executive dashboards, including the panels that state what the numbers do not support

Led

Presented the portfolio risk position and the model's limits to the Executive Committee, including the period where the original models could not be trusted
Made the case for recall over precision by pricing the two error types rather than arguing about metrics
Established that a model whose target definition has changed is retired, not recalibrated

Architecture

How it is put together

A scoring pipeline over servicing data that produces a ranked collections queue against fixed outreach capacity. The design decision that matters is that the model ranks and the collections team decides; the model never triggers an action on a customer's account by itself.

The same architecture, in prose

Loan and lease servicing systems provide account, contract and payment history on a scheduled cycle.

The feature layer builds payment-behaviour features, exposure and contract position, and the vehicle-value position that determines loss severity if the account does charge off.

A forbearance and programme-status layer marks accounts under payment holiday, because their payment history means something different from an identical history without the flag.

The delinquency-roll model predicts whether a currently-performing or early-stage account will roll further past due within the horizon.

The charge-off model predicts terminal loss for accounts already in distress, and pairs the probability with the expected severity from the vehicle-value position.

The ranker combines probability and severity into expected loss, then orders accounts against the collections team's fixed capacity for the cycle.

The collections platform receives the ranked queue; treatment assignment and the customer contact itself remain human decisions.

Outcomes flow back as labels, and drift monitoring compares feature and target distributions against the training period.

Executive dashboards report the portfolio position, the model's performance, and explicitly what the current numbers do not support.

Every path, written out (2)· the walkthroughs above, as text

A collections cycle

The portfolio is scored, ranked against the team's capacity, and worked, with the model ordering attention and a person making every decision that touches a customer.

01
Servicing data lands
Account, contract and payment history for the cycle arrive from the servicing systems.
Payment behaviour is the signal; everything else modifies how to read it.
Fails by: A servicing change alters how a payment event is recorded and the feature layer reads it as behaviour.
02
Mark programme status
Accounts under payment holiday or relief are flagged before any feature is computed.
A missed payment under an approved deferral is a different event. This has to be known before the features, not after the scores.
Fails by: Enrolment data lags the payment data it explains, so an account looks distressed for the duration of the lag.
03
Build the account view
Payment behaviour, exposure, contract position and vehicle-value position are computed per account.
Severity matters as much as probability, and severity comes from the vehicle position rather than from the payment history.
04
Predict the roll
Early-stage accounts are scored for whether they will roll further past due within the horizon.
This is the model that operates inside the window where an intervention still changes the outcome.
Fails by: Tuned on a symmetric metric, this model predicts 'no roll' for everything and reports excellent accuracy.
05
Price the distressed tail
Accounts already in distress get a terminal-loss probability paired with expected severity.
A different question at a different point in the account's life, deliberately kept as a separate model.
06
Rank by expected loss
Probability and severity combine into expected loss, and accounts are ordered against the cycle's outreach capacity.
Ranking by probability alone spends the team on many small likely losses rather than on fewer large ones.
Fails by: If the queue is only ever worked to a fixed depth, rank silently becomes exclusion.
07
A person decides
The collections team works the queue and chooses the treatment and the contact.
The model orders attention. It has no authority to act on a customer's account, and that boundary is in the topology rather than in a policy.
08
Wait for the horizon, then label
Realised outcomes become training labels once the prediction horizon closes.
The label cannot exist before the horizon does, which sets a hard floor on the learning rate of the whole system.
Fails by: Contacted accounts have different outcomes than uncontacted ones: the intervention working and a selection bias, simultaneously.

Failure: the label changes meaning

A payment-holiday programme launches. The models keep scoring, keep reporting good validation metrics, and are describing a portfolio that no longer exists.

01
Payments stop arriving, legitimately
Large numbers of accounts stop making payments under an approved deferral.
Nothing is wrong. The customers are complying with a programme the business designed.
02
The features read distress
Payment-behaviour features move sharply toward the patterns that historically preceded charge-off.
The features are correct about what happened and wrong about what it means. There is no signal in the data itself that distinguishes the two.
Fails by: Validation metrics on held-out historical data remain excellent, because the historical data is from the old regime.
03
The model is confidently wrong
Roll probabilities rise across the portfolio and the ranked queue fills with customers who are fine.
The model is faithfully applying a relationship that was true when it was learned and is not true now.
04
Drift monitoring names it
Feature drift is obvious, but the decisive observation is that the target rate itself has moved: the event being predicted is not the event being observed.
Monitoring features alone would have shown a problem without identifying it. Monitoring the target definition is what made it diagnosable.
05
Make the programme a first-class input
A programme-status layer is introduced ahead of the feature layer, so relief enrolment is known before behaviour is interpreted.
The distinction cannot be recovered downstream. It has to enter the system before the features are built.
06
Retire and re-segment
The original models are retired rather than recalibrated, and replaced by three models segmented by days past due under the new regime.
Recalibrating a model whose target has changed meaning produces a well-calibrated answer to the wrong question.

What changes as load grows· the scale stages, as text

Steady state

illustrative

Scheduled portfolio scoring per collections cycle

Batch scoring comfortably fits the cycle. The binding constraint is not compute; it is the fixed number of accounts the collections team can contact.

Gives first: Outreach capacity, which is why the output is a ranking rather than a classification.

Regime shift

illustrative

Forbearance enrolment across a large share of the portfolio

The target variable's meaning changes. Feature distributions move, but more importantly the label itself now describes a different event.

Gives first: Not throughput, but validity. Drift monitoring becomes the load-bearing component, and the correct action is to retire the models rather than to rescore faster.

Rebuild under the new regime

attested

Three replacement models on a compressed timeline

Segmentation by days past due replaces a single target, so the models can describe distinct populations under the new programme rather than one population that no longer exists.

Gives first: Label availability. The new regime has only months of history, which caps how much any model can be expected to know.

Decisions

The calls I would defend

Each one with the alternatives I rejected, what the choice cost, and how it turned out.

Decision

The operating threshold was set for recall at a substantial precision cost, chosen by pricing the two error types rather than by optimising a symmetric metric.

Context

On a severely imbalanced target, every convenient metric rewards predicting the majority class. Accuracy in particular is actively misleading and was the number stakeholders arrived already anchored to.

Alternatives, and why not

Optimise F1 or AUC and take the threshold from the curveBoth treat the two error types as comparable. Here they differ by orders of magnitude in cost, and the metric should say so.
Match the historical alert volumeAnchors the new system to the capacity of the old process rather than to where the value is.

Rationale

A false positive is a supportive phone call to a customer who did not need one. A false negative is a charge-off that was preventable in a window that has now closed. Writing both costs down ended the metric argument in about ten minutes.

What it cost

The collections team contacts more customers who would have been fine. That cost is real, bounded, and small relative to the losses avoided.

Outcome

87% accuracy at 81% recall, and a 20% reduction in portfolio loss through earlier targeting.

What I would do today

The same, and I would express the threshold directly as an expected-cost optimisation in code rather than as a chosen operating point, so the cost assumption is visible and challengeable.

What it moved in the diagram

Delinquency-roll model
Expected-loss ranker

What broke

Failures, and what they changed

Every one of these is a thing that went wrong in a system I own. They are here because the architecture is largely a record of them.

A payment-holiday programme made the target variable mean something else

What happened: Customers stopped paying under an approved deferral. Payment-behaviour features moved into the ranges that had historically preceded charge-off, roll probabilities rose across the portfolio, and the collections queue filled with people who were complying with a programme the business had designed.
Root cause: The delinquency flag had always been a proxy for payment distress, and the models treated the proxy as the thing. When policy separated them, nothing in the system could tell the difference, because the distinction had never existed in the data.
What I did: Feature drift was visible immediately and was not diagnostic on its own. What identified the problem was monitoring the target rate: the event being predicted had moved independently of the population, which is a definitional break rather than distribution shift.
What changed in the architecture: A programme-status layer was added ahead of the feature layer, so relief enrolment is known before behaviour is interpreted rather than joined on afterwards. The original models were retired and replaced with three segmented by days past due.
What it taught me: Monitor the target definition, not just the inputs. Feature drift tells you the world moved; a moving target rate tells you your question did. And when the question changes, recalibration gives you a well-calibrated answer to something nobody asked.

Evaluation

How I knew whether it worked

Evaluated on the decision the model informs rather than on the prediction it makes, because the prediction is an intermediate quantity and the decision is the product.

Recall at the operating threshold, per segment

A portfolio-level recall number hides that the model may be strong on the dense middle and weak exactly where the exposure is concentrated. Reported per segment, including per programme-status group after forbearance.

Loss avoided per contacted account, by rank decile

The ranking is the product. If the top decile does not avoid materially more loss than the fifth, the ordering is decorative.

The rank at which work actually stopped

Ranking is only legitimate if the tail is eventually worked. Tracking where the team stopped is what distinguishes prioritisation from quiet exclusion.

Target-rate stability as a release gate

Added after the forbearance break. A model does not ship if the target rate in the recent period is inconsistent with the training period, regardless of how good the validation metrics look.

Impact

What changed, and how it is known

Every figure carries its basis. Nothing here is rounded up, and nothing modelled is presented as a result.

87%attested

Model accuracyDelinquency and charge-off models

81%attested

RecallChosen over precision on cost asymmetry

−20%attested

Portfolio lossThrough earlier collections targeting

3 at >85%attested

Replacement models30/45-day segmentation under the forbearance regime

Business: Losses fell 20% by moving collections effort earlier in the account's life, into the window where an intervention still changes the outcome.
Engineering: A scoring pipeline that survived a mid-year change in what its target variable meant, because the break was detected as a definitional problem rather than debugged as a performance problem.
People using it: Collections worked a queue ordered by where their fixed capacity was worth the most, rather than by how overdue an account already was.

Leadership and hindsight

What I influenced, and what I would change

Technical leadership

Told the Executive Committee that the existing models could not be trusted during the transition, and what was being done about it, rather than continuing to render numbers that still looked plausible.
Ended a metric argument by pricing the two error types instead of debating F1 against AUC.
Established that a model whose target definition has changed is retired rather than recalibrated, a rule that costs weeks in the moment and saves a false sense of coverage.

What I would do differently

I monitored feature distributions and not the target definition. The features told me something was wrong; only the target rate told me what, and I added that monitor after I needed it rather than before.
The programme-status layer should have existed from the start in some form. Any portfolio subject to policy interventions will eventually have a policy intervention that changes what its data means.
The 'rank at which work stopped' metric came late. Without it I could not have distinguished prioritisation from exclusion, and I was reporting on a ranking whose tail behaviour I could not see.

Go deeper

The detail, for people who want it

Collapsed by default. The case study stands without any of it.

Why expected loss and not probability of loss

A ranked queue by probability puts the most likely losses first. With fixed capacity, that spends the whole team on the accounts easiest to predict, which are usually small, late-stage and already determined.

Expected loss (probability times severity) reorders that completely. A moderately likely loss on a large exposure with a weak vehicle-value position outranks a near-certain small one.

The severity half comes from the vehicle position rather than the payment history, which means the two most important inputs to the ranking come from different systems with different refresh rates. Getting the staleness of the severity input under control mattered more to the ranking than any modelling improvement.

Stack

What it is built on, and why that

A technology list without reasons is a list of things I have heard of.

Python: Feature engineering, model training and the scoring pipelineThe modelling ecosystem, and the standard for analytics work in the team.
scikit-learn / XGBoost: The delinquency-roll and charge-off modelsGradient-boosted trees on tabular credit data, with the explainability a risk function needs.
SQL: The feature layer over servicing and payment historyThe volume made computing features where the data lived the only sensible option.
Tableau: Executive dashboards, including the confidence and limits panelsIt was the reporting layer the Executive Committee already read.

5 more sections are written and hidden: constraints, evaluation, leadership, deeper, stack.