Data Lake Migration: A Practical Framework for Moving to the Cloud

Moving a data lake to the cloud sounds like a storage problem: copy the files from one place to another. In practice, it rarely is. We hear some version of “it’s just a copy job” at the start of almost every Hadoop conversation. Legacy on-premises systems carry years of undocumented dependencies, inconsistent data quality, and pipelines nobody has touched since the person who built them left.

Think of it like moving house. Loading the truck takes a day; reconnecting the utilities, forwarding the mail and finding what broke takes weeks. Copying HDFS files is the truck. The work is in everything around the files: the Hive metastore, Spark and Oozie jobs, Kerberos and Ranger permissions, and the reports that quietly depend on all of it. This framework covers that work phase by phase, so the migration you planned for six months does not turn into twelve.

Why Companies Migrate Their Data Lakes to the Cloud

Most data lake migrations start with one of four pressures:

• Aging on-premises hardware. Clusters near end-of-life force a choice between a like-for-like refresh and the cloud.
• The cost of running Hadoop. Legacy big data clusters are expensive to operate, and engineers who want to run them are getting harder to hire.
• Fixed capacity. On-prem clusters are sized for peak load, so you pay for idle capacity most of the year and still hit limits during spikes.
• AI and modern analytics. Managed Spark, serverless SQL and lakehouse table formats are far easier to adopt on cloud platforms than on a legacy cluster.

If these sound familiar, the same warning signs often apply to your warehouse too. See our signs your enterprise needs data warehouse modernization.

Choose Your Migration Strategy Before Anything Moves

The first real decision is not which tool copies the files. It is how much you change on the way. Three strategies cover almost every data lake migration:

Strategy	What changes	Best when	Main risk
Rehost (lift and shift)	Hadoop workloads move to a managed cluster service such as Amazon EMR, Azure HDInsight or Google Cloud Dataproc, with minimal code change	Deadlines are tight and you have a lot of Hadoop-specific code	You carry the old design, and its costs, into the cloud
Replatform to a cloud-native lake	Data moves to object storage; jobs run on managed Spark and serverless SQL	You want elastic compute and less cluster administration	Job rewrites take longer than expected
Modernize to a lakehouse	Data lands in open table formats (Apache Iceberg, Delta Lake, Apache Hudi) with central governance	You need reliable updates, schema evolution, BI and AI on one copy of the data	Largest design effort up front

These are not mutually exclusive. A common pattern is to rehost the most fragile workloads first, then replatform or modernize them wave by wave once they are running in the cloud.

A hybrid end state is also legitimate. Some datasets stay on-premises for regulatory or latency reasons, while new workloads run in the cloud. If that is the plan, design the catalog and access model to span both from the start.

You will also see these choices described as the “7 Rs” of cloud migration (AWS migration strategy guidance): rehost, relocate, replatform, refactor, repurchase, retain and retire. For a data lake, “retire” matters more than most teams expect. Phase 1 usually finds datasets that nobody uses and that never need to move.

The 7-Phase Data Lake Migration Framework

The phases run in order, but planning for the later ones starts on day one. Each phase ends with a gate: a short list of things that must be true before the next phase starts.

Phase 1: Assess and Inventory

Before any data moves, you need an honest picture of what actually exists. This means cataloging data sources, volumes, dependencies, and, critically, who actually uses each dataset and how. Skipping this step is the single most common reason migrations run over budget: teams discover mid-migration that a “dead” table is actually feeding a report an executive checks weekly.

In our migration projects, this is where most of the surprises hide. For a Hadoop-based lake, the inventory needs to go well beyond file counts:

Area	What to capture
HDFS storage	Used capacity, file counts, growth rate, replication factor, small-file hotspots
Hive metastore	Databases, managed vs. external tables, partitions, schemas, table locations
Processing jobs	Spark, MapReduce, Hive SQL and custom Java or Python jobs, with runtimes and SLAs
Orchestration	Oozie or Airflow workflows, schedules, retry logic, job dependencies
Ingestion	Kafka, NiFi, Sqoop, Flume, CDC feeds and batch file drops
Security	Kerberos principals, Ranger or Sentry policies, HDFS ACLs, encryption zones, PII locations
Consumers	BI dashboards, notebooks, APIs, ML models and downstream systems, each with an owner

This phase should also assess data quality. Migrating messy, ungoverned data to the cloud without addressing quality issues just moves the same problems to a more expensive environment.

Gate: every dataset and job has an owner, a business criticality rating, and a decision: migrate, rebuild, archive or retire.

Phase 2: Define the Target Architecture

Decide what you are migrating to, not just where. That means picking a cloud provider, a storage layout, a table format and a catalog. If you are adopting lakehouse capabilities as part of the move, make that decision now rather than migrating into another plain data lake.

The core building blocks map across the three major clouds:

Capability	AWS	Microsoft Azure	Google Cloud
Object storage	Amazon S3	Azure Data Lake Storage Gen2	Cloud Storage
Managed Spark / Hadoop	Amazon EMR, Databricks	Azure Databricks, Microsoft Fabric, HDInsight	Dataproc, Databricks
SQL on the lake	Amazon Athena, Amazon Redshift	Microsoft Fabric, Azure Synapse	BigQuery
Catalog and governance	AWS Glue Data Catalog, AWS Lake Formation	Microsoft Purview, Unity Catalog	Dataplex, Unity Catalog
Identity	AWS IAM	Microsoft Entra ID	Google Cloud IAM

Table format is the other decision that is expensive to reverse, so we push teams to settle it before the first pipeline is written. Delta Lake is the default on Databricks and Microsoft Fabric. Apache Iceberg is the most engine-neutral choice and works well with Snowflake, AWS and multi-engine setups. Apache Hudi fits heavy streaming and upsert workloads. Our Iceberg vs Delta vs Hudi in 2026 breakdown goes deeper.

For a deeper look at what the target should provide, see what is a data lake architecture. If the migration also replaces a data warehouse, compare Snowflake, BigQuery, Azure Synapse and Redshift before you commit.

Gate: a written target architecture, a table format choice, and a migration pattern (rehost, replatform or modernize) for every workload group.

Phase 3: Build the Landing Zone

The landing zone is the secure, governed environment that receives the data. Build it before any production data moves, not after.

• Networking: private connectivity (AWS Direct Connect, Azure ExpressRoute or Google Cloud Interconnect), DNS, firewall rules and bandwidth monitoring.
• Identity and access: Kerberos principals and Ranger policies do not carry over to cloud IAM automatically. Map every role and policy to the target access model and test it explicitly.
• Storage zones: raw, cleaned and curated layers (often called bronze, silver and gold), with encryption, lifecycle and retention rules.
• Compliance: if the lake holds regulated data, compliance and security teams sign off on the controls for HIPAA, SOX or PCI-DSS before that data lands.
• Cost controls: budgets, resource tags and alerts for storage, compute and data transfer from the first day.

Our post on data lakehouse security best practices covers what to get right from day one.

Gate: the environment can receive data securely, run a test job, log access for audit, and recover from a defined failure.

Phase 4: Prioritize and Sequence the Waves

Not everything needs to move at once, and not everything should. Group datasets and jobs into migration waves based on business criticality, technical complexity and dependency chains:

Wave	What moves	Why in this order
0 (pilot)	A representative slice: one partitioned Hive table, one complex Spark job, one ingestion feed, one report	Proves the process end to end
1	Low-risk batch datasets and jobs with few dependencies	Builds speed and surfaces tooling issues
2	Shared datasets, core Hive tables and the pipelines that depend on them	Moves the core data products
3	Business-critical daily pipelines, streaming and BI consumers	Needs the most careful cutover
4	Archive data and retirement candidates	Lowest urgency, often the biggest volume

Gate: a wave plan with owners, dates and a rollback plan for each wave.

Phase 5: Run a Pilot Migration

The pilot is wave 0 run as a full rehearsal. Google Cloud’s Hadoop-to-lakehouse migration demo is a useful reference for what one covers. Pick a slice that is small enough to finish in a few weeks but representative enough to break things: at least one partitioned Hive table, one complex Spark job, one ingestion feed and one downstream report.

1. Record the source baseline: row counts, file counts, schemas, partition layouts, job runtimes and the output of the report.
2. Transfer the data and recreate the table definitions in the target catalog.
3. Port the job: update storage paths, credentials, libraries and scheduler references.
4. Run the job on both platforms and compare the outputs.
5. Write down every surprise, then fix the runbook and the estimates before wave 1.

In our experience, the pilot is where your estimates become real. Teams that skip it usually learn the same lessons in wave 2, with production data and less time.

Gate: the pilot’s outputs match the source, and the runbook is updated.

Phase 6: Migrate Data, Metadata and Pipelines

This is the phase most people picture when they think “migration,” but if the earlier phases were done properly it should be the most mechanical part of the project. It covers three separate moves.

Data. The transfer method depends on volume, available bandwidth and how long the source keeps changing:

• Online copy with Hadoop DistCp, AWS DataSync, Azure Data Factory or Google Cloud’s Storage Transfer Service, for most volumes when the network allows it.
• Offline transfer appliances (AWS Snowball, Azure Data Box, Google Transfer Appliance) when the network would take too long.
• Incremental sync or CDC after the bulk copy, so the source can stay live until cutover.

To pick between online and offline, estimate the transfer time first:

\text{Transfer time} = \frac{\text{Data volume}}{\text{Bandwidth} \times \text{Effective utilization}}

At 80% effective utilization, the numbers look like this:

Data volume	At 1 Gbps	At 10 Gbps
10 TB	about 1.2 days	about 3 hours
100 TB	about 11.6 days	about 1.2 days
500 TB	about 58 days	about 5.8 days
1 PB	about 116 days	about 11.6 days

These are best-case figures for one continuous copy. Millions of small files, shared bandwidth and retries all push them up. If the online estimate runs past your cutover window, price out an appliance.

Metadata. Copying files does not move your tables. Hive table definitions, partitions and locations must be recreated in the target catalog: AWS Glue Data Catalog, Unity Catalog, Dataplex or Microsoft Purview. On Databricks, Parquet tables can be converted to Delta Lake in place with CONVERT TO DELTA, without rewriting the data.

Pipelines. Jobs need new storage paths, new credentials and often new engines. Where a job is being rewritten anyway, rebuild it for the target platform rather than porting the old logic line by line. Run new pipelines alongside the legacy ones during the transition window.

Each Hadoop component has a usual landing spot in the cloud:

Hadoop component	Typical cloud target	What the work looks like
HDFS	Amazon S3, Azure Data Lake Storage Gen2, Cloud Storage	Copy data; change hdfs:// paths to s3://, abfss:// or gs://
Hive metastore	AWS Glue Data Catalog, Unity Catalog, Dataproc Metastore	Recreate tables, partitions and locations
HiveQL	Spark SQL, Amazon Athena, Databricks SQL, BigQuery	Most queries port with small changes; review custom UDFs by hand
Pig scripts and MapReduce jobs	Spark (PySpark or Spark SQL)	Usually rewritten, and often simpler afterwards
Oozie workflows	Apache Airflow (Amazon MWAA, Cloud Composer), Databricks Workflows, Azure Data Factory	Rebuild the schedules and dependencies as new workflows
Sqoop imports	AWS DMS, Azure Data Factory, Google Datastream or a CDC tool	Replace; Apache Sqoop itself has been retired
Kafka	Amazon MSK, Azure Event Hubs, Confluent Cloud	Re-point producers and consumers; plan offset handling at cutover
Kerberos, Ranger, Sentry	AWS IAM + Lake Formation, Entra ID + Unity Catalog or Purview, Cloud IAM + Dataplex	Redesign roles and policies; test them explicitly

Open-source SQL translators such as SQLGlot can convert much of the HiveQL syntax automatically. Edge cases, custom functions and performance tuning still need an engineer.

Throughout this phase, security is part of the migration itself, not something applied once data has landed.

Gate: each wave’s data, tables and jobs are running in the cloud, with the legacy versions still available.

Phase 7: Validate, Cut Over and Decommission

Before cutting over, prove the migrated data matches the source. Row counts and checksums alone are not enough: matching counts do not prove the data is identical, and matching checksums do not prove a report still gives the same answer.

Validation area	What to check
Completeness	File and table inventories, row counts, partition counts
Integrity	Checksums where supported, schema and data type matches
Data quality	Null rates, duplicates, referential rules, key business totals
Job output	The same job on both platforms produces the same result
Reports	Priority dashboards show the same numbers on both sides
Performance	Runtimes and freshness meet the agreed SLAs
Security	Access tests pass, including denied-access tests, and audit logs are written

Run both platforms in parallel for at least one full business cycle. For finance data, that should include a month-end close.

Agree on sign-off criteria before the migration starts, so cutover is not a judgment call on the day. For example: every in-scope table reconciles on row counts, financial totals match exactly, the top 20 reports show the same numbers, and no access test fails.

Once validation passes and data owners sign off, redirect production pipelines and reporting to the new environment. Keep the legacy system available in a read-only state for a defined grace period, often 30 to 90 days, before decommissioning it. That gives you a safety net if something unexpected surfaces after cutover.

A cutover runbook keeps the day itself predictable (Microsoft’s Cloud Adoption Framework describes the same sequence for Azure):

When	What happens
T-2 weeks	Freeze schema changes on the source; confirm owners and the rollback plan
T-3 days	Bulk copy complete; incremental sync or CDC running
T-1 day	Reconciliation passes the agreed criteria; go/no-go meeting
T-0	Pause source writes, run the final sync, re-point pipelines, BI connections and applications
T+1 to T+14 days	Hypercare: daily checks on freshness, failed jobs, report numbers and cloud spend
T+30 to T+90 days	Legacy stays read-only, then is decommissioned after sign-off

Decide the rollback triggers in advance. Examples: a priority report stops reconciling, a critical job misses its SLA two days running, or an access test fails.

After cutover, optimize. Migration does not end at decommission. The first few months in the cloud are when the savings are made or lost:

• Right-size compute and turn on autoscaling
• Use spot or preemptible capacity for non-critical batch jobs
• Compact small files and run regular table maintenance
• Set storage lifecycle rules so cold data moves to cheaper tiers
• Tag resources and review spend against budget every month
• Retire the unused datasets that Phase 1 flagged

Gate: owners have signed off, production runs on the new platform, and the decommission date is set.

Big Bang vs Phased Migration: Which Is Safer?

Two broad patterns dominate real-world migrations:

	Big bang migration	Phased migration
How it works	Everything moves in one tightly scheduled cutover window	Data moves in waves over weeks or months, with old and new systems running in parallel
Speed	Faster to finish overall	Takes longer on the calendar
Risk	Problems surface all at once, under time pressure	A problem in one wave does not threaten the rest
Cost profile	Short dual-running period	You pay for both environments during the transition
Fits	Small lakes with few consumers and a hard deadline	Most organizations beyond a fairly small scale

Our recommendation for most organizations is phased. The extra calendar time is usually a worthwhile trade for the lower risk of a large cutover failure.

Data Lake Migration Cost and Timeline: What Actually Drives the Bill

Migration cost usually gets estimated from data transfer and target-environment provisioning, but that is only part of the bill:

• Pipeline rebuild. Engineering time to rebuild pipelines, rather than just relocate data, is often the largest single cost in the project.
• Dual-running. A phased migration means paying for the legacy and cloud environments at the same time, sometimes for months.
• Egress fees. Moving large volumes out of an existing cloud provider (cloud-to-cloud, not on-prem to cloud) can be substantial and is easy to underestimate.
• Transfer setup. Network upgrades or appliance fees for very large volumes.
• Governance and security. Rebuilding access controls, lineage and audit logging in the new platform.
• Contingency. Budget for the undocumented dependencies Phase 1 is designed to find. Migrations that only budget for the happy path are the ones that blow through timeline and budget together.

For the business case, split the costs into three buckets:

Cost type	What it includes	When you pay it
One-time migration	Assessment, pipeline rebuild and code conversion, transfer setup or appliances, testing	During the project
Transition	Dual-running both environments, egress fees, partner time, training	Until decommission
Ongoing cloud operations	Storage, compute, data transfer, catalog and governance tools, monitoring	Every month after cutover

Compare the ongoing line against what the cluster costs today, including hardware, licenses, data center space and admin time. That comparison, not the migration quote alone, is the business case.

On timing, a focused migration for a single business unit might take a couple of months. A full enterprise migration across dozens of systems often runs six months to a year with a phased approach. Volume matters less than the number of pipelines, consumers and undocumented dependencies.

At Algoscale, data lake engagements usually start around $30,000, and most begin with a 2-week audit that ends in a fixed-scope estimate. Cloud storage and compute are billed separately by AWS, Azure or Google Cloud. For a quick range for your own project, try our engagement estimator.

Who Needs to Be in the Room

A data lake migration touches more people than the engineering team expects:

• Report consumers need to know when their sources change, so a broken dashboard the morning after cutover is not a surprise.
• Data owners sign off on each wave’s validation results.
• Security and compliance approve access controls before sensitive data moves, and confirm HIPAA, SOX or PCI-DSS obligations still hold in the new environment.
• The budget owner needs regular progress against the wave plan. Migrations that go quiet for months tend to lose executive support right when they need it most.

Treat stakeholder communication as a defined workstream with an owner and a cadence, not something that happens in hallway conversations.

Plan for skills and adoption, not just cutover. Engineers who ran Hadoop need time with object storage, cloud IAM and open table formats before they can run the new platform alone. Analysts need to know where their data now lives and which connections changed. A technically clean migration that nobody uses delivers no return.

Common Data Lake Migration Pitfalls (and How to Avoid Them)

• Hidden dependencies. Legacy systems collect undocumented downstream consumers over years, and we see this trip teams up more often than any technical problem. A thorough Phase 1 is the only real defense.
• Assuming permissions carry over. Kerberos identities, Ranger policies and HDFS ACLs do not translate to cloud IAM on their own. Untested access models are a common reason cutovers get rolled back.
• Forgetting the metastore. Files land in object storage, but nobody can query them until table definitions and partitions are recreated in the new catalog.
• Losing the old data checks. Row-count thresholds, null checks and reconciliation logic often live inside old ETL scripts. Rebuild them in the new pipelines before the legacy jobs are switched off. One missing row-count check can quietly duplicate months of data, as this Fabric incremental-load bug shows.
• The small-file problem. HDFS estates often hold millions of tiny files, which slow both the transfer and queries in object storage. Compact them before or during the move.
• Migrating technical debt. A migration is a natural moment to fix long-standing quality and governance issues. Moving them unchanged just relocates the problem. See common data warehouse migration challenges, many of which apply to data lakes too.
• No rollback plan. Decide how a wave rolls back before you need to.

Choosing Your Cloud Platform: AWS, Azure, Google Cloud or Databricks

The right platform depends on your existing tooling, team skills and workloads more than on any universal “best” answer:

• AWS suits teams that want the broadest data lake tooling, with Amazon S3, AWS Glue, AWS Lake Formation and Amazon EMR (see our AWS data lake architecture guide).
• Microsoft Azure suits organizations already invested in Microsoft, with Azure Data Lake Storage Gen2, Microsoft Fabric OneLake and Power BI.
• Google Cloud suits teams with heavy BigQuery usage, with Cloud Storage and Dataproc.
• Databricks runs on all three and suits teams that want one lakehouse for data engineering, SQL and machine learning. Our Microsoft Fabric vs Databricks comparison covers that choice.

The framework above stays the same whichever you choose. What changes is the tooling in Phases 3 and 6.

Data Lake Migration Tools by Step

Step	Tools
Discovery and inventory	AWS Glue crawlers, Microsoft Purview, Dataplex, Apache Atlas
Bulk transfer	Hadoop DistCp, AWS DataSync, AzCopy, Storage Transfer Service; appliances: AWS Snowball, Azure Data Box, Google Transfer Appliance
Incremental sync and CDC	AWS DMS, Azure Data Factory, Google Datastream, Debezium, Fivetran
Code conversion	SQLGlot, plus manual review of custom functions
Orchestration	Apache Airflow (Amazon MWAA, Cloud Composer), Azure Data Factory, Databricks Workflows
Validation	Great Expectations, dbt tests, custom reconciliation queries
Governance and catalog	AWS Lake Formation, Microsoft Purview, Dataplex, Unity Catalog

Real-World Case Studies: An Azure Data Lake and a Databricks Lakehouse

ECM Energy Services, a field services and fleet operator, went from kickoff to production in 14 weeks. Algoscale built a medallion data lake on Azure Data Lake Storage Gen2 with Azure Data Factory. It unified Oracle NetSuite, Fleetio, Samsara, KPA Flex and ADP, was governed with Microsoft Purview, and serves eight Power BI modules. Read the case study.

For Juice Ventures, Algoscale brought HubSpot, Azure SQL and Amazon S3 data into one Databricks lakehouse governed with Unity Catalog. Weekly manual report pulls were replaced with real-time dashboards, cutting manual effort by 80%. Read the case study.

Why Algoscale for Data Lake Migration

Algoscale has guided organizations off legacy Hadoop clusters, aging on-premises warehouses and fragmented multi-system data estates, using the phased approach above.

• Certified on the platforms you are moving to. AWS Advanced Tier Services Partner, Microsoft Solutions Partner for Data & AI (Azure), Snowflake Select Partner and Databricks Bronze Partner.
• A fixed scope before you commit. Most engagements start with a 2-week audit that ends in a written assessment and a fixed-scope estimate.
• A faster start. S.C.A.L.E.™, our Terraform-driven enterprise data platform, gets the target environment ready in weeks rather than months.
• 12+ years and 150+ projects across data engineering, warehousing, lakehouses and analytics.

If you are planning a migration now, our data lake migration services team can start with the audit.

Conclusion

A data lake migration is won or lost before the first byte moves. Teams that inventory honestly, pick their target architecture early, run a real pilot and validate against the source tend to finish close to plan. Teams that treat it as a copy job usually find the hidden dependencies in production.

If you are weighing a move off Hadoop or an aging on-premises lake, start small: do the transfer math for your volume, and scope a pilot wave. That is the smartest way to find out what your migration really involves.

Have a data problem worth solving? Our team can start with a 2-week audit and a fixed-scope estimate. Schedule a consultation.

Frequently Asked Questions