Most lakehouse performance problems don’t announce themselves. There’s rarely a single dramatic failure – instead, queries get a little slower each month, costs creep up gradually, and by the time someone notices, the cause is buried under months of accumulated table maintenance nobody was doing. These mistakes are quiet specifically because each individual instance seems minor; it’s the compounding that eventually becomes a real problem.
This post catalogs the lakehouse-specific mistakes that cause this kind of gradual degradation – not the broad “what changes at volume” issues any data lake faces, but the ones specific to how open table formats like Delta Lake, Iceberg, and Hudi actually need to be maintained. For a deeper technical grounding in these formats, our companion post on Delta Lake, Iceberg, and Hudi: comparing open lakehouse formats is a useful reference.
If you’re seeing unexplained performance degradation on your own lakehouse, our Data Lake Services team at Algoscale diagnoses exactly this kind of quiet accumulation regularly.
The Quiet Performance Mistakes
Skipping Regular Compaction
Open table formats accumulate many small files over time, especially with frequent streaming or micro-batch writes. Without regular compaction – merging small files into appropriately sized ones – query engines spend increasing time on file-open overhead rather than actual data scanning. This is arguably the single most common cause of gradual lakehouse performance decline, precisely because a table works fine for months before the small-file count crosses a threshold where it starts noticeably slowing every query against it.
Ignoring Metadata and Snapshot Bloat
Every write to a Delta or Iceberg table creates new metadata and, depending on configuration, retains historical snapshots for time travel. Without periodic cleanup – vacuuming old snapshots, expiring unneeded metadata – this overhead grows continuously, and query planning itself (figuring out what to read, before reading anything) starts taking longer. This is a particularly quiet problem because it affects every query uniformly rather than causing an obvious specific failure.
Poor Partitioning Choices
Both over-partitioning (too many small partitions, creating excessive metadata and file overhead) and under-partitioning (partitions too large for queries to prune effectively) degrade performance, just through different mechanisms. Partitioning decisions made early, before real query patterns were well understood, often turn out to be a poor fit once actual usage develops – and revisiting this decision periodically, rather than treating an initial choice as permanent, is worth building into your maintenance routine.
Not Using Clustering for Common Query Patterns
Modern table formats support data clustering or similar techniques (sometimes called Z-ordering) that physically organize data within files to make common filter patterns dramatically more efficient. Tables that see consistent, predictable query patterns but were never clustered around those patterns leave meaningful performance on the table – this is a lower-effort optimization that’s easy to skip simply because it’s not required for a table to function, only to function well.
Orphaned Files and Uncleaned Old Versions
Failed or interrupted write operations, along with normal table format versioning, can leave orphaned files that are no longer referenced by any active table version but still consume storage and, in some cases, slow down maintenance operations that need to scan the underlying storage location. Periodic cleanup of these orphaned files is easy to overlook precisely because they don’t affect query correctness – only cost and, indirectly, the performance of maintenance operations themselves.
Wrong File Format or Size for the Query Engine
Even within the “use Parquet” baseline that most lakehouse architectures follow correctly, file size still matters – files that are too small create the overhead already discussed, while files that are excessively large can hurt parallelism for certain query engines. The “right” target file size varies somewhat by engine and workload, which is exactly why this often goes unaddressed: there’s no single universal number to apply, so it quietly gets skipped rather than tuned specifically for the actual engine in use.
Catalog Under-Provisioning or Latency at Scale
As the number of tables and the frequency of metadata operations grow, catalog performance itself can become a bottleneck – slow catalog responses add latency to every single query, regardless of how well-optimized the underlying table data is. This is easy to miss because it doesn’t show up as a table-specific problem; it shows up as generally, uniformly slower performance across the entire lakehouse, which teams often initially misattribute to something else entirely.
How to Detect These Issues Before They Become Painful
Watch for a few specific signals rather than waiting for users to complain. Query times trending upward over months, even for queries whose logic hasn’t changed, is a strong indicator of accumulating small files or metadata bloat. Growing storage costs disproportionate to actual data volume growth often points to orphaned files or unmanaged historical versions. Query engine logs or diagnostics that explicitly flag high file counts per query are worth monitoring directly rather than waiting for a general performance complaint. Building simple, regular checks for these signals into your operational monitoring catches this kind of quiet degradation well before it becomes a significant, user-visible problem.
A Simple Maintenance Cadence
Most of these issues are prevented by establishing a routine maintenance schedule rather than addressing each one reactively. Weekly or daily compaction (depending on write frequency) for high-traffic tables keeps small-file accumulation under control. Periodic snapshot expiration and orphaned file cleanup – typically weekly or monthly, depending on your time-travel retention needs – prevents metadata and storage bloat from accumulating silently. A quarterly review of partitioning and clustering choices against actual query patterns catches decisions that no longer fit as usage evolves. None of this needs to be elaborate, but it does need to actually happen on a schedule, rather than being addressed only when someone notices a problem.
For a broader look at how these issues intensify specifically as data volume grows, our post on big data lakes vs standard data lakes: what changes at volume covers the underlying dynamics in more depth.
How This Compounds in Practice
Picture a table receiving hourly streaming writes, each producing a handful of small files. In isolation, one hour’s worth of small files is trivial. After six months without compaction, that same table might have accumulated hundreds of thousands of small files, and every query against it now spends more time on file-open overhead than on actually scanning data. Layer on six months of uncleaned old snapshots from time-travel retention, and query planning itself slows down before a single row has been read.
None of this required a mistake in the traditional sense – the pipeline was working correctly the entire time, writing valid data on schedule. The problem is purely the absence of routine maintenance, which is exactly why it’s easy to overlook: nothing in the pipeline logs or dashboards flags “table needs compaction” the way an actual pipeline failure would trigger an alert. It just gets slower, gradually, until someone finally investigates why a report that used to take ten seconds now takes two minutes.
Getting Ahead of Quiet Performance Decline
None of these mistakes are dramatic on their own, which is exactly why they’re worth taking seriously – a lakehouse that quietly gets 20% slower every few months for a year has a genuinely different performance profile than the one it started with, even though no single event caused it. At Algoscale, our Data Lake Services team builds table maintenance discipline into every lakehouse implementation from the start, and diagnoses exactly this kind of accumulated drift for existing architectures that have quietly degraded over time.
To see the broader range of data engineering and analytics work we do beyond performance tuning specifically, take a look at what Algoscale builds across the data stack.
Why Algoscale
A few things shape how we actually deliver on data lake and data engineering work, beyond the architecture and practices covered above:
● Pre-built accelerators. We don’t start every engagement from a blank slate – proprietary accelerators built from prior implementations speed up common data source integration and analytics patterns.
● Faster time to value. For a focused initial scope covering core data sources and first analytics use cases, our accelerators typically compress development timelines to around four weeks, rather than the several months a from-scratch build often takes.
● Built on a scalable framework. Our implementation approach follows a repeatable, scale-ready framework, so the architecture built for your first use case extends cleanly as data volume and teams grow, rather than requiring a redo.
● Microsoft Solution Partner for Data & AI. Algoscale holds Microsoft Solution Partner status for Data & AI, including specific expertise implementing Microsoft Fabric as a modern data warehouse.
Frequently Asked Questions
1. How do we know if compaction is actually needed on a specific table?
Check the average file size and total file count for the table against your query engine’s recommended ranges – most engines and platforms provide table statistics or diagnostic commands that surface this directly, making it straightforward to identify tables that have drifted into small-file territory.
2. How often should compaction actually run?
It depends on write frequency – tables receiving frequent streaming or micro-batch writes benefit from daily or even more frequent compaction, while tables updated less often may only need weekly or monthly maintenance. The right cadence is proportional to how quickly small files accumulate for that specific table.
3. Is clustering or Z-ordering worth the effort for every table?
Not necessarily – it’s most valuable for tables with high query volume and consistent, predictable filter patterns. Tables queried infrequently or with highly variable access patterns may not justify the maintenance overhead of clustering.
4. Can these performance issues be fixed without downtime?
Generally yes. Compaction, snapshot expiration, and orphaned file cleanup are typically designed to run without disrupting concurrent reads, though it’s worth confirming this for your specific table format and platform before running maintenance operations against critical production tables.
5. How do we know if catalog latency specifically is the bottleneck, rather than the underlying table data?
If query slowness is broad and uniform across many tables rather than concentrated on specific large or poorly-maintained ones, catalog performance is worth investigating directly, since a table-specific optimization won’t address a catalog-level bottleneck.
6. Should table maintenance be automated, or handled manually?
Automated wherever possible. Manual maintenance depends on someone remembering to run it consistently, which is exactly the kind of dependency that leads to the gradual degradation this post describes in the first place.
Noticing your lakehouse getting slower over time without an obvious cause? Talk to Algoscale’s Data Lake Services team about diagnosing what’s actually accumulating.
