A stolen card number is only valuable to a fraudster for a few seconds. By the time a batch job flags a suspicious transaction the next morning, the money is already gone. That gap between “when the fraud happens” and “when the system notices” is the single biggest lever banks, card networks, and payment providers have for cutting fraud losses and it’s why so many financial services teams are rebuilding their fraud stack around real-time data, not overnight reports.
Databricks has become a common landing point for this shift. Not because it’s a fraud-detection product out of the box, but because it lets teams run streaming ingestion, feature enrichment, ML scoring, and governance on one platform instead of stitching together a patchwork of specialized systems. This post looks at how that architecture actually works, what it delivers in practice, and what it takes to get there.
1. Why Batch Scoring Isn’t Good Enough Anymore
Traditional fraud pipelines score transactions in overnight or hourly batches. That model made sense when fraud review was largely manual, but it breaks down against modern payment fraud for a simple reason: authorization decisions happen in milliseconds, and a batch job that runs hours later can only tell you fraud happened, not stop it.
● Card-present and card-not-present fraud both rely on exploiting the window before a transaction is reviewed.
● Real-time payment rails (RTP, FedNow, instant ACH) settle in seconds, leaving no time for after-the-fact review.
● Analysts working from stale, batch-scored alerts spend more time chasing false positives than catching real fraud.
The fix isn’t a better batch model — it’s moving the scoring decision to sit inside the transaction path itself.
2. The Reference Architecture: Streaming, Feature Enrichment, and Scoring in One Platform
The pattern that’s emerged across banks, card networks, and crypto exchanges building on Databricks follows a consistent shape: ingest events continuously, enrich them with fresh context, score them with a model, and route the decision all within a single governed platform.
Reference architecture for real-time fraud detection on Databricks
| Layer | What It Does | Databricks Component |
| Ingestion | Streams card swipes, ACH entries, and wire requests as they happen | Kafka → Spark Structured Streaming |
| Real-time processing | Processes events continuously instead of in micro-batches, with latency in the tens of milliseconds | Spark Real-Time Mode (RTM) |
| Stateful tracking | Tracks per-card and per-account velocity (e.g. transactions per minute) with automatic state expiration | transformWithState API |
| Feature enrichment | Looks up merchant risk profiles and cardholder history without the overhead of broadcast joins | Lakebase (managed Postgres, online feature store) |
| Scoring | Applies a trained fraud model to each transaction as it flows through the pipeline | MLflow model served as a Spark UDF |
| Decision routing | Approves, flags, or blocks the transaction based on the score | Structured Streaming sink / downstream service |
| Analyst monitoring | Gives fraud analysts a live view of flagged transactions and case context | Databricks Apps (e.g. Streamlit dashboard) |
| Governance | Applies access control and keeps an auditable trail across data, features, and models | Unity Catalog |
3. What Changed: Spark Real-Time Mode and Lakebase
Two relatively recent additions to the Databricks platform are what make this architecture practical without bolting on a separate streaming engine:
● Spark Real-Time Mode (RTM) extends Structured Streaming with a trigger that processes events continuously as they arrive, instead of on a fixed micro-batch schedule. Databricks reports end-to-end latency as low as five milliseconds for supported workloads, with fraud detection called out as a primary use case.
● Lakebase is a fully managed, serverless Postgres database built into Databricks, used here as a low-latency online feature store. It serves merchant and cardholder risk context to the streaming pipeline in place of slower broadcast joins, and can also store the resulting fraud scores for sub-millisecond reads by downstream applications.
Because both run on the same platform as the rest of the lakehouse, teams get streaming ingestion, feature serving, model scoring, governance, and analyst-facing apps without maintaining a separate real-time stack alongside their existing Databricks environment.
4. Batch vs. Real-Time: The Practical Difference
| Dimension | Batch Scoring (Legacy) | Real-Time Scoring (RTM + Lakebase) |
| Typical latency | Hours (overnight or hourly jobs) | Tens of milliseconds to sub-second |
| Decision point | After the transaction has already settled | Before the transaction is authorized |
| Feature freshness | Stale — reflects yesterday’s activity | Live — reflects the current session and velocity |
| Infrastructure | Separate batch and streaming stacks, duplicated governance | Single governed platform for streaming, features, and ML |
| Analyst experience | Alerts arrive after the fact, high false-positive review load | Live dashboard with enriched, contextual alerts |
| Best fit | Historical trend analysis, model retraining | Card authorization, instant payments, account takeover checks |
5. Real-World Results
Financial institutions building this pattern on Databricks have reported meaningful latency gains over their prior architectures:
● A global bank processes credit card transactions from Kafka and flags suspicious activity within 200 milliseconds, without having to replatform its existing pipelines.
● An open-source fraud detection accelerator built on RTM and Lakebase demonstrates sub-300ms detection using stateful velocity tracking and dictionary-based enrichment in place of costly broadcast joins.
● A crypto exchange’s fraud and anti-money-laundering platform moved from an optimized micro-batch setup (roughly 800-900ms freshness) to RTM, reaching 150ms for stateless and 250ms for stateful streaming feature aggregations while reducing the operational burden of maintaining that performance.
The consistent theme: teams aren’t just getting faster scoring, they’re getting it without running a second specialized streaming system alongside their lakehouse.
6. Governance and Compliance Don’t Get Left Behind
Speed matters, but fraud systems in financial services live under strict regulatory and audit requirements. Any real-time architecture needs to answer for itself in a compliance review, not just a latency benchmark.
● Unity Catalog provides a single governance layer across the raw transaction data, the engineered features in Lakebase, and the ML models used for scoring so access control and lineage don’t fragment across separate systems.
● Reg E and dispute-handling processes require logs detailed enough to support timely investigation of unauthorized transactions; keeping the pipeline on one governed platform makes that evidence easier to produce.
● AML and consortium data-sharing use cases require clear controls over permissible use and retention, which are easier to enforce when there’s one governance boundary instead of several.
This is also where human-in-the-loop design still matters: real-time scoring should feed analysts better, faster context not fully remove them from high-stakes decisions.
7. Getting Started: A Practical Rollout Path
● Start narrow. Pick one transaction type (e.g. card-not-present) and one fraud pattern before expanding to ACH, wires, or account takeover.
● Prove the latency budget. Benchmark Spark RTM against your current pipeline on real transaction volumes before committing to a full rebuild.
● Stand up Lakebase as your online feature store early. Feature freshness usually matters more to model accuracy than model complexity.
● Wire in Unity Catalog governance from day one. Retrofitting access control and audit logging after launch is far more expensive than designing for it upfront.
● Give analysts a live view before you go live. A Databricks Apps dashboard for case review closes the loop between automated scoring and human judgment.
● Plan for quarterly model refresh and drift monitoring. A fraud model’s accuracy decays as fraud patterns evolve; real-time infrastructure doesn’t fix a stale model.
8. The Algoscale Perspective
Real-time fraud detection isn’t really a machine learning problem, it’s a data engineering and governance problem with a model attached at the end. The teams getting the biggest wins from Databricks aren’t the ones with the most sophisticated model; they’re the ones who got feature freshness, latency, and governance right first.
At Algoscale, we help financial services teams design and implement streaming architectures like this one from Kafka ingestion and Spark Real-Time Mode pipelines to Lakebase feature stores and Unity Catalog governance sized to your actual transaction volumes and compliance requirements.
Looking to cut fraud detection latency on your own data stack? Talk to Algoscale’s data engineering team about a real-time architecture assessment.
