We’ve covered the business-level comparison between a data lake and a lakehouse elsewhere – what changes, when each fits, how to decide. This post is different: it’s about what’s actually happening mechanically when you write to and query each one. If you’ve read that a lakehouse “adds ACID transactions” and wanted to know what that actually means at the level of files sitting in a storage bucket, this is for you.
For the business-level version of this comparison – differences that matter for decision-making rather than implementation – see our earlier post on data lake vs. data lakehouse: what’s actually different. This post assumes that context and goes underneath it.
If you’re implementing this kind of architecture and want help getting the mechanics right, our Data Lake Services team at Algoscale works with these internals directly.
What Actually Happens on Write
Writing to a Plain Data Lake
On a plain data lake, a write is just a file write. A Spark job or ETL process writes one or more Parquet files to a path in cloud storage – that’s it. There’s no coordination layer tracking that this write is “one logical unit.” If the job fails halfway through writing ten files, five files exist and five don’t, and nothing in the storage layer knows or cares that this is an incomplete state. A reader querying that path mid-write, or after a partial failure, sees whatever files happen to physically exist at that moment – potentially a mix of old and new data, or missing data entirely, with no signal that anything is wrong.
Writing to a Lakehouse Table
A write to a Delta Lake or Iceberg table happens in two distinct phases. First, the actual data files (still Parquet, under the hood) get written to storage – mechanically similar to the plain data lake case. Second, and critically, a new metadata entry gets committed that atomically points to exactly which files constitute the table’s new valid state. Until that second step completes, readers continue seeing the previous valid version of the table – the new data files exist in storage but aren’t yet “part of the table” from any reader’s perspective. If the job fails before that metadata commit, the orphaned data files simply sit unreferenced; the table itself remains in its last known-good state, untouched.
Under the Hood: The Metadata Layer
Delta Lake’s Transaction Log
Delta Lake tracks every change to a table in a directory of JSON files (commonly _delta_log), where each file represents one atomic transaction – a record of which data files were added and which were removed. To know the current state of a table, a reader (or the underlying engine) replays this log, applying each transaction in order, to arrive at the current set of valid files. Periodically, Delta checkpoints this log into a more efficient format so readers don’t need to replay the entire history from the beginning every time. New writes are committed by atomically creating the next numbered JSON file in this sequence – the atomicity of that single file creation operation is what gives the whole system its transactional guarantee.
Iceberg’s Metadata Tree
Iceberg takes a related but structurally different approach: a hierarchy of metadata files. A metadata.json file points to the current snapshot; each snapshot points to one or more manifest list files; each manifest list points to individual manifest files, which in turn list the actual data files and their statistics. A new write creates new manifests and a new snapshot, then atomically updates a pointer so the metadata.json file (or the catalog’s reference to it) now points to this new snapshot. Old snapshots and their associated manifests remain in storage, still valid and readable, until explicitly expired – which is the underlying mechanism that makes time travel possible.
How This Enables Atomicity
Both approaches rely on the same underlying trick: making the “did this write succeed” question answerable by a single atomic operation, rather than requiring many files to all succeed together. Cloud object stores generally support atomic operations at the level of a single file – either a file write fully succeeds and becomes visible, or it doesn’t. By reducing “did my multi-file write succeed” down to “did this one pointer-update file get written successfully,” both formats get genuine atomicity out of storage systems that were never designed to support multi-file transactions natively.
How Time Travel Actually Works
Because old snapshots (Iceberg) or historical log entries (Delta) aren’t deleted immediately – they’re retained until an explicit cleanup operation runs – querying a table “as of” a previous point in time is mechanically just a matter of resolving which snapshot or log position was current at that timestamp, and reading the files that snapshot pointed to. Nothing needs to be reconstructed or restored; the old version’s file references simply still exist and are still valid, sitting alongside the newer ones, until a VACUUM (Delta) or snapshot expiration (Iceberg) operation actually removes the now-unreferenced old data files and cleans up the historical metadata pointing to them.
How Schema Enforcement Actually Works
Schema information is stored as part of the table’s metadata – in Delta’s log entries and in Iceberg’s metadata.json – rather than being inferred fresh from the data files on every read. When a new write occurs, the writer checks the incoming data’s schema against this stored schema before committing the write’s metadata entry. If it doesn’t match (and schema evolution rules don’t explicitly allow the difference), the write is rejected before the commit happens – meaning the table’s valid state never includes the mismatched data in the first place, rather than accepting it and causing problems for readers downstream.
Query Planning Differences
Querying a plain data lake typically requires listing files in a directory (or scanning across a broader storage location) to figure out what exists to read, then reading each file’s own header to understand its schema and statistics – real work that has to happen before a query can even begin executing. Querying a lakehouse table means reading the relevant manifest or log entries first, which already contain file paths and, in Iceberg’s case, column-level statistics (min/max values, null counts) that let the query engine skip entire files without opening them at all. This is the mechanical basis for the performance difference discussed at a higher level elsewhere – it’s not that lakehouse storage is inherently faster, it’s that query planning has far more information available up front, without needing to touch the actual data files to get it.
Concurrency Control Mechanics
Both Delta and Iceberg use optimistic concurrency control: a writer reads the current table version, prepares its changes, and attempts to commit assuming no conflict – then checks, at commit time, whether the table version has changed since it started. If another writer commits in the meantime, the commit is retried against the new current state (sometimes automatically, sometimes requiring the writer to reconcile the conflict, depending on whether the changes actually overlap). This is different from traditional database locking, where a writer would block other writers for the duration of a transaction – lakehouse formats generally allow concurrent write attempts to proceed and only resolve conflicts at the final commit step, which tends to work well for the batch and micro-batch write patterns common in analytical workloads.
Why This Matters in Practice
Understanding these mechanics isn’t just academic – it explains specific operational behaviors that are otherwise confusing. Why does a failed pipeline run sometimes leave orphaned files taking up storage space without corrupting the table? Because the metadata commit never happened, so those files were never part of the table’s valid state. Why does VACUUM or snapshot expiration matter for cost, not just cleanliness? Because old file versions genuinely remain in storage, fully intact, until explicitly removed. Why can two engines write to the same table concurrently without corrupting it, when they couldn’t safely do that against plain files? Because the atomic metadata commit – not application-level coordination – is what actually enforces consistency.
Getting the Implementation Right, Not Just the Architecture Diagram
Understanding these mechanics matters when things go wrong – diagnosing why a table is accumulating storage cost, why a concurrent write pattern is producing conflicts, or why query performance doesn’t match expectations all require reasoning about what’s actually happening at this level, not just the architecture diagram. At Algoscale, our Data Lake Services team works with these internals directly, both when building new lakehouse architectures and when diagnosing issues in existing ones.
To see the broader range of data engineering and analytics work we do beyond table format internals specifically, take a look at what Algoscale builds across the data stack.
Why Algoscale
A few things shape how we actually deliver on data lake and data engineering work, beyond the architecture and practices covered above:
● Pre-built accelerators. We don’t start every engagement from a blank slate – proprietary accelerators built from prior implementations speed up common data source integration and analytics patterns.
● Faster time to value. For a focused initial scope covering core data sources and first analytics use cases, our accelerators typically compress development timelines to around four weeks, rather than the several months a from-scratch build often takes.
● Built on a scalable framework. Our implementation approach follows a repeatable, scale-ready framework, so the architecture built for your first use case extends cleanly as data volume and teams grow, rather than requiring a redo.
● Microsoft Solution Partner for Data & AI. Algoscale holds Microsoft Solution Partner status for Data & AI, including specific expertise implementing Microsoft Fabric as a modern data warehouse.
Frequently Asked Questions
1. Does understanding these internals matter for someone just using a lakehouse day to day?
Not for routine use, but it becomes valuable quickly once something unexpected happens – unexplained storage growth, write conflicts, or performance that doesn’t match expectations are all much easier to diagnose with a mental model of what’s actually happening at the metadata level.
2. Are Delta Lake’s and Iceberg’s approaches to metadata interchangeable?
Conceptually similar – both use a layer of metadata that atomically tracks valid table state – but structurally different enough that tooling built for one doesn’t directly read the other’s metadata format, even though both achieve the same fundamental guarantees.
3. What actually happens to “deleted” data in a lakehouse table?
It isn’t immediately removed from storage. The metadata is updated so the file is no longer part of the table’s current valid state, but the underlying file remains in storage – still fully intact – until a cleanup operation like VACUUM or snapshot expiration actually deletes it.
4. Why does optimistic concurrency control work well for lakehouse formats specifically?
Because analytical workloads are typically dominated by append-heavy, batch, or micro-batch writes rather than many small, frequently-conflicting transactions – conditions where optimistic concurrency (assume no conflict, check and retry if wrong) performs well, compared to workloads with constant write contention where pessimistic locking might be preferred.
5. Does query planning against manifest or log metadata always beat directly scanning files?
In almost all realistic cases, yes, since it avoids the overhead of listing and opening files just to determine what exists – the exception would be a trivially small table where the overhead difference is negligible either way.
6. Can I read a lakehouse table’s raw Parquet files directly, bypassing the metadata layer?
Technically yes, since the underlying files are standard Parquet, but doing so means bypassing the transactional guarantees, schema enforcement, and consistent versioning the metadata layer provides – you’d be reading whatever files happen to physically exist, with none of the reliability benefits the format is designed to provide.
