Annuity Lapse Prediction on Databricks

The actuarial gap that targeted retention can close

A variable annuity carrier’s exposure to lapse and surrender behavior runs through nearly every planning document the business produces. Lapse rates feed into reserving assumptions, hedging programs, and profitability models. When experience diverges from assumption — when the book lapses faster or slower than modeled — the consequences hit capital ratios, hedge costs, and reported margins at the same time.

The traditional instrument for managing this is the aggregate lapse assumption: a set of rates segmented by policy year, surrender charge period, or interest-rate environment, estimated from historical experience and updated periodically. The aggregate assumption tells an organization how much lapse to expect across its book. What it doesn’t tell anyone is which specific in-force contracts are most likely to surrender in the next quarter — or the quarter after that.

That gap is where targeted retention becomes possible, and where it typically breaks down. Retention outreach costs money. Applied broadly, it’s expensive and imprecise. Applied to the wrong contracts — policies that were going to stay regardless, or policyholders who have already decided to leave — it’s wasted. What effective retention needs is a score per contract: a continuously updated estimate of lapse propensity for each policy in the in-force book, surfaced early enough that an outreach can still change the outcome.

That score — and the governed MLOps pipeline to produce, monitor, and maintain it — is what this engagement was built to deliver.

Why governance is harder than the model

The client — a large US life insurance and annuity carrier — was not starting from zero. Their Databricks Lakehouse was already in place. Source data was flowing. ML tooling existed. What had been missing was a production-grade, governed scoring capability: one that could run on a schedule, be retrained without disrupting production, and be promoted or rolled back through a traceable process the business and its risk function could stand behind.

The framing that shaped the engagement from the first week: the model is almost the easy part. The discipline around it — how you define model readiness, how you promote a new version, how you ensure a deployed score is reproducible — is where most production ML systems eventually fail. We’ve seen the organizational version of this failure documented in Why Your AI Pilot Stalls at 80%; the version that hits after a pilot succeeds is subtler and more expensive.

The governance design was committed before a single experiment was run:

• One agreed primary metric, decided in advance, that governs whether a model version can be promoted to production
• A frozen evaluation set, version-controlled in the Lakehouse, that no one can edit without a documented change process
• A model registry as the single authority on what version is in production at any point in time
• A model bundle that pins model version, feature code version, and evaluation configuration together — so any deployed score traces back to a known, reproducible state

These are not model decisions. They are governance decisions. Our data science consulting practice treats them as prerequisites to any production ML engagement in a regulated industry, not features to add when the model is “done.”

The Databricks architecture: end-to-end MLOps on the Lakehouse

The entire pipeline — data engineering, feature engineering, training, evaluation, scoring, monitoring, and actuarial output — runs natively on the client’s Databricks Lakehouse. No external serving infrastructure. No data copies to a separate ML platform. Everything lives in the same governed environment the carrier already operated.

Data layer (Delta tables, Unity Catalog)

Source policy records and transaction data from the policy administration system land in the raw layer as partitioned Delta tables, ingested by Databricks Workflows on a scheduled cadence. A data engineering pipeline then transforms the raw layer into the prepared layer — normalized, schema-stable, model-agnostic Delta tables that represent what the data says about each in-force contract, not what any specific model needs.

The prepared layer is the handoff contract between data engineering and the ML track. Data engineering owns everything upstream; the ML track owns everything downstream. Feature engineering — transforming prepared-layer records into model inputs across the contract, policyholder, and behavioral attribute families — lives in versioned Python code under the ML team’s ownership, not embedded in the data pipeline. The failure modes that occur when this boundary dissolves are documented in Medallion Architecture: 5 Failure Modes.

Unity Catalog governs all three layers and every artifact above them. Access policies sit at the catalog level: the retention team reads the scored output layer; the actuarial team reads the actuarial handoff layer; neither has write access to the prepared or raw layers. Table-level lineage is automatic — a scored output traces back through Unity Catalog to the model version that produced it, the feature code version that shaped the inputs, and the prepared-layer snapshot it ran against.

MLflow: experiment tracking and production registry

MLflow is the backbone of the model lifecycle. Every training run logs to MLflow automatically: hyperparameters, the prepared-layer date slice used for training, evaluation metrics against the frozen eval set, and the model artifact including the versioned feature code and evaluation configuration.

The Model Registry holds the source of truth on what is in production. Named model stages — Staging, Production, Archived — are not administrative categories; they are load-bearing. The scoring job loads the model currently in the Production stage at runtime. There is no “copy the artifact to the scoring server” step, no informal promotion via Slack, no notebook output emailed to an engineer. The registry is the authority.

Promotion from Staging to Production follows a champion/challenger pattern: the new candidate model runs a full evaluation pass against the frozen eval set before promotion is considered. If it clears the primary metric threshold and beats or matches the current Production model on that metric, it is eligible for promotion. The evaluation record — metric value, eval set version, comparison against the champion — is logged to MLflow and linked to the model version before promotion executes. When an auditor asks “why was this model promoted?” the answer is a query, not a memory.

Databricks Workflows: orchestration and scheduling

Databricks Workflows ties the end-to-end run into a single observable DAG:

1. Prepared-layer data quality gate (null rates, schema validation, out-of-range checks)
2. Feature engineering job (versioned feature code applied to the prepared layer)
3. Batch scoring job (loads the Production model version from MLflow, runs inference, writes propensity scores to the output Delta table)
4. Actuarial handoff job (extracts model coefficients and feature contribution summaries; writes to the actuarial output Delta table in Unity Catalog)
5. Monitoring refresh (updates data drift and model output drift dashboards)

Every step runs in isolation and has its own retry and alerting configuration. A failure in any step alerts the owning team and halts downstream steps. There is no “it probably ran, let me check the notebook.”

MLOps observability: staying honest about model health in production

A model that ships without observability is a model that degrades silently. For a lapse prediction system that feeds into retention prioritization and sits adjacent to actuarial assumptions, silent degradation is a specific and serious risk.

Data drift monitoring tracks the distribution of model inputs between scoring runs, compared to a baseline snapshot from training time. For a variable annuity book, input distributions shift for real and predictable reasons: contracts age through surrender charge periods, interest-rate environments change, product cohorts mature at different rates. Databricks Lakehouse Monitoring computes distributional distance metrics per feature on every scoring run. When a feature’s distribution moves past a configured threshold, the system flags it before it becomes a model accuracy problem.

Model output drift monitoring tracks the distribution of predicted propensity scores across the book over time. A sudden shift in the score distribution — the entire book moving toward high lapse probability, or a persistent narrowing of the score range — is a model signal worth investigating before labeled experience arrives to confirm it. Output drift can surface a real change in policyholder behavior, a data pipeline issue, or a model assumption that no longer holds. The monitoring layer makes the distinction visible.

Prediction logging writes every batch scoring run’s output to a versioned partition in the output Delta table, with the model version, feature code version, and run timestamp attached to each row. This means the scoring history is queryable: you can pull up what score any contract received on any past run, under which model version, and with what feature inputs. In a regulated context, that property is what makes the audit trail meaningful rather than theatrical. Our experience with a similar observability setup — where every prediction carries enough metadata to reconstruct the decision — is described in insurance claims classification with LLMs.

Retraining triggers are monitored conditions, not calendar events. Retraining is queued when: (a) data drift exceeds threshold on features with high model importance, (b) a pre-defined A/E check period has elapsed and experience materially diverges from the model’s predicted rates, or (c) the production model’s champion evaluation metric has degraded below the promotion threshold when re-evaluated against the current prepared layer. Each of these is a measured condition rather than a scheduled assumption that the model is stale.

From lapse score to actuarial input: the A/E feedback loop

The scoring pipeline doesn’t stop at a propensity score. The engagement specifically designed the output to feed into the actuarial workflow — closing the loop between predictive model and actuarial assumption.

The actuarial handoff table in Unity Catalog receives, on each scoring run, a structured output that includes the contract-level propensity score alongside a feature contribution breakdown from the production model. This gives the actuarial team visibility not just into what the model predicts but which contract attributes are driving the prediction for any cohort of interest — cohorts defined by product type, policy year, or surrender charge status.

A/E integration is where predictive ML and traditional actuarial practice connect. The model’s predicted lapse rate for a defined cohort becomes the “expected” side of the actual-to-expected ratio for that cohort. Actual surrender counts over the same period become the “actual” side. An A/E ratio persistently above 1.0 for a cohort means the model is underpredicting lapse in that segment; below 1.0 means it is overpredicting. This per-cohort A/E analysis does two things:

1. It validates the model’s assumption quality at a granularity the aggregate lapse table never could — surfacing systematic misfit by product, policy year, or behavioral segment before it accumulates in reserves.
2. It feeds the retraining signal — a persistent A/E divergence in a cohort is a model-improvement opportunity, with the A/E diagnostic pointing directly at which segment to oversample or re-weight in the next training cycle.

Downstream pricing input is the longer-cycle use. The model’s segment-level predictions, validated through A/E experience over successive measurement periods, produce a more granular lapse assumption than the traditional rate table. The actuarial team can incorporate these predictions — as an experience study input rather than a black-box override — into the next pricing or reserve assumption update. The model’s coefficient structure and feature importances, accessible through the actuarial handoff table, make that incorporation transparent and auditable: actuaries can see which contract attributes drove the assumption shift, not just that the model said so.

The model bundle: what makes a deployed score trustworthy

The design decision that the risk and audit function cared most about was the model bundle — the deliberate practice of pinning together the model version, the feature code version, and the evaluation configuration so that any deployed score can be reproduced exactly.

The alternative is the situation most production ML systems actually live in: a model in the serving layer, feature code that has been edited since training, and an evaluation set that exists somewhere in a notebook. When an auditor asks “which model produced these lapse scores from three months ago?” the honest answer involves git blame and memory.

For a scoring system that informs retention prioritization and feeds into actuarial assumptions at a regulated carrier, that answer isn’t acceptable. The bundle approach means:

• The production model version corresponds to a specific, recorded training run
• That training run executed against a specific feature code version, logged as an artifact
• Its evaluation was measured against a specific frozen evaluation set, version-controlled in the Lakehouse
• The primary metric threshold that governed promotion is documented in the registry alongside the version

Rolling back means pointing the scoring job at the previous bundle. Because the previous bundle’s feature code version is pinned, the rollback reproduces scores that match the previous deployment — not an approximation based on “probably the same code.”

What the engagement delivered

The carrier has a production-grade, repeatable lapse prediction pipeline on their existing Databricks Lakehouse — not a trained model handed off as a notebook, but the full operational lifecycle: governed training, evaluation, promotion, scheduled scoring, drift monitoring, and actuarial handoff. Every component of that lifecycle runs on a single governed platform, observable end-to-end, with no external infrastructure to maintain.

MLOps governance replaced informal model management. Promotion is gated by a frozen benchmark. Every version is logged. Drift conditions trigger retraining rather than accumulating silently. A/E experience feeds back into the model’s retraining signal rather than sitting in a separate actuarial spreadsheet.

The actuarial team gained a new input layer for experience analysis and pricing decisions — contract-level propensity scores validated through A/E ratios, with model coefficient transparency that makes the assumption change defensible to product and pricing committees.

Contract-level retention targeting turned the aggregate lapse assumption into an actionable, ranked signal. The retention team receives a scored list prioritized by lapse propensity — concentrating outreach where an intervention can still change the outcome, not where the book happens to be large.

Related work

The same MLOps discipline — frozen evaluation sets, versioned promotion, audit-ready lineage, prediction logging — governed our engagement on insurance claims classification with LLMs, where a multi-label document triage system had to be deployable under regulatory scrutiny. The argument is identical across use cases: in a regulated industry, the governance around the model is the system.

The data architecture follows the medallion discipline documented in Medallion Architecture: 5 Failure Modes — particularly around protecting the data engineering boundary from feature engineering drift.

For the broader question of why ML systems fail to reach production even after a successful pilot, see Why Your AI Pilot Stalls at 80%.

Our data science consulting practice runs this playbook — governance-first ML lifecycle on Databricks, observability and drift monitoring, actuarial handoff design — for carriers and other regulated-industry clients where model trustworthiness is not optional.