Multi-Terabyte Data Warehouse and ETL Migration to AWS, Wave by Wave
A multi-terabyte on-premise data warehouse — the system internal analysts and downstream consumers pulled reports from during business hours — had to move to AWS while month-end close made a multi-day freeze impossible in any month of the year. Envion ran the migration in controlled waves against the five stages of the engagement model: assess, sequence, migrate, validate, operate. The ETL dependency graph was reconstructed from scheduler logs rather than documentation, cutover was ordered by consumer blast radius rather than table size, source and target ran in parallel with the legacy warehouse authoritative until each wave passed reconciliation, ETL was rewritten rather than lifted, and a full cost model — compute, storage tiering, transfer, steady-state operations — was built before a single workload moved.
The challenge
Reporting was the product. Internal analysts and downstream consumers pulled from the warehouse during business hours, and month-end close made a multi-day freeze impossible in any month of the year. A hardware refresh quote forced the decision, but the migration could not buy new infrastructure by breaking the old one.
Three risks were named at kickoff. Silent data drift — a migrated warehouse that returns slightly different numbers is more damaging than one that is down, because nobody catches it for weeks. ETL job interdependency — the nightly jobs had accumulated undocumented ordering dependencies over years. And egress and storage cost surprise — multi-terabyte workloads punish teams who model compute and forget transfer, storage class, and cross-AZ traffic.
Decision path
Assessment inventoried every table, job, and consumer before any architecture was proposed, and produced three artifacts. A dependency graph of the ETL estate, reconstructed from scheduler logs rather than documentation — which surfaced jobs nobody owned and jobs that could be retired outright. A consumer map — which report, dashboard, and downstream system reads which table — because cutover order is set by consumer blast radius, not by table size. And a cost model covering compute, storage tiering, data transfer, and steady-state operations, projected over three years against the on-premise refresh baseline — built before a single workload moved, so the business decision was made on numbers rather than on a cloud-by-default assumption.
Sequencing was set by risk, lowest first: retired jobs, then internal-only reporting tables, then shared dimensions, then customer-facing marts last. Each wave was small enough to roll back inside a single maintenance window. The target landed raw history in an S3 landing zone with Redshift as the query layer — the lake as landing zone means raw history survives independently of whatever query engine sits on top. That decoupling is what keeps the second migration cheap.
Envion contribution
Envion owned the migration design and execution end to end: the assessment artifacts, the wave plan, the transfer tooling, the ETL rewrite, the reconciliation harness, and the operational handover.
Migration ran dual-run, not big-bang. Source and target ran in parallel, with the on-premise warehouse remaining authoritative until each wave passed validation. The historical load moved by bulk transfer into the S3 landing zone; ongoing change data capture kept Redshift current through the parallel period. ETL was rewritten into cloud pipelines rather than lifted, because lifting scheduler-era job logic into the cloud reproduces the original coupling and forfeits most of the cost benefit.
The validation rule: no wave cut over on the strength of "the job ran successfully." Every migrated table passed row-count and checksum parity per load; production reports were executed against both systems and compared field by field, with a variance threshold of zero for financial measures; and performance baselines captured pre-migration made "slower than before" a measurable claim rather than a perception.
Delivery
Handover included infrastructure-as-code for the full environment, runbooks for the recurring failure modes, and cost alerting tuned to the workload. Storage lifecycle policies were set at migration time rather than retrofitted — which is where most multi-terabyte cloud bills quietly go wrong.
Outcome and evidence
The warehouse and its pipelines moved to AWS wave by wave, each wave reversible inside its own maintenance window, with rollback available per wave until the final cutover. Outcome instrumentation — reporting downtime during migration, the nightly ETL window before and after, three-year TCO against the on-premise refresh quote, jobs retired, and query performance on the top reports — is measured by the client against pre-migration baselines rather than asserted here.
The migration architecture
Bulk load, CDC sync, parallel run — the design in one diagram
Evidence gate. This page publishes only what Envion's project records and client disclosure permissions support. Outcomes are added once verified against a baseline, a measurement period, and an approved source.
FAQ
Questions about this case
Facing a similar challenge?
A warehouse too critical to freeze and too expensive to keep? Envion migrates it in waves — costs modeled first, parity proven per wave, rollback never more than a window away.
Discuss a Similar ChallengeKeep exploring
Similar case studies
Executive Technology Leadership
Support for high-stakes product and AI decisions
Bring senior technology leadership into the business when the roadmap is unclear, delivery is at risk, an AI initiative needs stronger ownership, or the company needs an experienced technical voice before hiring a permanent CTO.
Discuss Interim CTO SupportCore responsibilities
- Align product and technology priorities with business goals and measurable outcomes.
- Review architecture, delivery risks, data foundations, security needs, and AI readiness.
- Lead internal teams and external partners through a practical execution plan.
- Clarify team structure, ownership, decision rights, and delivery cadence.
- Support investor, board, partner, and due-diligence conversations with credible technical judgment.
New experience
Prompt-to-Page — try it right here
Describe the landing page you want, in your own words. We turn it into a finished page and email you a private link in 5–10 minutes — no briefs, no calls, $0 to see the result.
- Describe what you want to create.
- We structure, write, and compose the page.
- You receive a private link when it is ready.
Start with a sentence — the interactive builder takes it from there.
Generate My PageSafe, respectful content only. No obligation.