# Migrating 50 Production Databases to Aurora with Under 3 Minutes of Downtime

> The methodology behind 50+ production MySQL-to-Aurora migrations — because the downtime budget is a design input, not something you measure afterward.

---

LLMS index: [llms.txt](/llms.txt)

---

Over about two years, we migrated around 50 production databases from on-premises MySQL to AWS Aurora. The largest were multi-terabyte, tier-0 systems with 24/7 write traffic. Most cutovers took under 3 minutes of downtime.

That number is not the result of careful clicking during a maintenance window. It's the result of treating the downtime budget as a **design input**: once "3 minutes" is a requirement rather than a hope, every downstream decision — replication topology, rehearsal gates, connection draining, DNS weights — is forced into a specific shape. This post is the methodology that fell out of that constraint, plus what actually bit us along the way.

One framing note, because it shapes everything below: migrating one database is a project. Migrating fifty is a platform. One-off migration plans don't scale to fifty databases owned by fifty different teams, so the unit of work was never "a migration" — it was a parameterized pipeline that each workload got tuned into.

---

## The Core Move: Don't Migrate Data, Replicate It

The key insight: the cutover should have almost no work left to do.

By establishing replication from on-premises to Aurora and letting it catch up long before cutover day, the cutover itself shrinks to a brief, mechanical operation:

1. Stop writes to source
2. Verify replication complete (lag = 0)
3. Promote Aurora
4. Redirect traffic
5. Resume operations

Total downtime: the time to verify, promote, and redirect — typically under 3 minutes, because every slow step happened days or weeks earlier.

---

## The Pipeline

### Phase 1: Assessment (Weeks Before per Database, Months for the Fleet)

**Capacity planning**
- Analyze query patterns and peak loads
- Size Aurora instances for the write workload first — the writer is yours to size, and no amount of reader auto-scaling fixes an undersized writer
- Plan *read* burst capacity with Aurora Auto Scaling, which adds or removes reader instances — it does nothing for the writer

**Compatibility testing**
- Run application test suites against Aurora
- Identify SQL syntax differences (there are fewer than you'd think)
- Test character sets and collations with production *data*, not just production schemas (this bit us once — more below)

**Infrastructure preparation**
- Provision the Aurora cluster via Terraform
- Configure security groups and networking
- Set up monitoring dashboards before you need them

### Phase 2: Data Migration

**Initial sync.** We used AWS DMS for the initial bulk load and native MySQL binlog replication for ongoing sync.

A word of honesty about DMS, because migration guides tend to sanitize it: DMS does the job, but it is not a set-and-forget tool. Full-load tasks on large, write-heavy sources need watching; LOB handling and task settings matter more than the console wizard suggests; and CDC depends on the source's binlog retention being long enough to survive your slowest day. We treated DMS as a bootstrap mechanism for the bulk load and leaned on native replication — which we could inspect, measure, and reason about directly — for the part that actually had to be trustworthy at cutover.

**Replication lag management.** Multi-terabyte replication takes time. We monitored replication lag (seconds behind primary), binary log position, and disk usage on the Aurora cluster.

> [!TIP] **Pro tip**
> InnoDB compression on the source can slow replication. We addressed this by temporarily increasing the Aurora instance size during initial sync.

### Phase 3: Validation

Never trust, always verify:
- Row counts match between source and target
- Checksum comparisons on critical tables
- Read traffic testing against the Aurora replica
- Synthetic transactions validating end-to-end flow

We ran read traffic against Aurora (as a replica) for weeks before cutover, gradually increasing the share.

### Phase 4: Cutover — Rehearsed, Not Heroic

Every cutover was executed against a restored copy first, with the same scripts, the same checks, and the same rollback points as production. Rehearsal was a gate, not a suggestion: a cutover that hadn't been rehearsed to boring didn't get a production date.

The production runbook:

**Pre-cutover (T-60 min)**
- Verify replication lag < 1 second
- Confirm rollback path tested
- Alert stakeholders
- Quiet period — no deployments

**Cutover (T-0)**
- Stop application writes (graceful)
- Verify replication complete (lag = 0)
- Promote Aurora to primary
- Drain connections and shift DNS weights — automated and timed, because "redirect traffic" done by hand is where 3-minute budgets go to die
- Resume application traffic

**Post-cutover (T+30 min)**
- Monitor error rates
- Verify transaction success rates
- Confirm no replication to old primary
- Update documentation

### Phase 5: Cleanup

After running on Aurora for a week with the old primary as fallback: decommission the old primary, update disaster recovery documentation, conduct a retrospective. The retro fed the next migration — the runbook was a living artifact, not a one-off.

---

## The Connection Stampede Problem

This one deserves its own section, because it's the failure mode that only exists at platform scale.

When you redirect traffic to a new endpoint, every application reconnects simultaneously. Fifty application pools all stampeding a freshly promoted writer at once can produce a self-inflicted outage in the middle of a textbook cutover — the database is fine, the migration is "done," and the site is down anyway.

We absorbed this at the routing layer: applications held long-lived connections to Vitess VTGate rather than to the database endpoint directly, so the cutover redirect happened at the proxy, and the applications never had to mass-reconnect at all. If you don't have a proxy layer, the poor man's version is staggered restarts and jittered reconnects — but the real lesson is that the cutover plan isn't complete until it includes the *reconnection* plan, and that decision belongs in the platform, not in each team's migration checklist.

---

## What We Learned

### Things That Worked

**1. Obsessive documentation.** We maintained a 21-page migration guide updated after every migration. When something unexpected happened, we documented it. By migration #50, surprises were rare.

**2. Practice runs.** Every migration step was tested in QA first. Not just "we ran it once" — we ran the full cutover procedure multiple times until it was boring.

**3. Rollback testing.** We tested rollback scenarios as thoroughly as the forward path. In one case we actually rolled back a production cutover due to an application issue discovered post-cutover — and it was smooth precisely because we'd practiced. The downtime budget math has to include the reverse path, or you don't have a budget, you have a hope.

**4. Over-communication.** Stakeholders knew exactly when the migration would happen, what to expect, and who to contact. Updates every 15 minutes during cutover.

### Things That Bit Us

**1. Character set edge cases.** Aurora was stricter about character sets than our old MySQL. Some tables with legacy data needed careful handling. Lesson: test with production data, not just production schemas.

**2. Timezone tables.** Our Aurora build didn't come with MySQL timezone data loaded. Applications relying on `CONVERT_TZ()` failed until we loaded it. Now it's in the pre-migration checklist.

**3. Connection pool stampedes.** Covered above — the most platform-shaped problem of the three.

---

## The Infrastructure

Everything was provisioned through Terraform:

```hcl
module "aurora_mysql" {
  source = "path/to/aurora-module"

  cluster_name   = "critical-database"
  engine_version = "8.0.mysql_aurora.3.04.0"

  # Production-ready defaults
  instance_class = "db.r6g.2xlarge"
  instances      = 3  # Writer + 2 readers

  # Built-in monitoring (Database Insights)
  database_insights_enabled = true
  cloudwatch_alarms_enabled = true
}
```

The module encodes years of lessons learned into defaults — timezone data loaded at provision time, monitoring on by default, reader sizing guidance baked in — so the fifty-first team to migrate doesn't rediscover what bit the first fifty. That, more than any individual runbook, is what made this a platform: the accumulated scar tissue lives in the module.

One honest note on versions: the engine version above is pinned because version skew between on-prem MySQL and Aurora was one of our real compatibility variables. Pin deliberately, upgrade deliberately.

---

## The Trade-Off Nobody Puts in the Migration Guide

Aurora has been rock-solid for us, but it is not a free upgrade, and two costs deserve to be named.

**Aurora bills for I/O.** On high-write workloads, the per-million-request I/O line item can be a genuine surprise coming from on-prem, where IOPS are a sunk hardware cost. For most of our fleet it was unremarkable; for the chattiest write-heavy databases it materially changed the cost model. Model it before you migrate, not after the first bill.

**You are buying into Aurora's operational model.** Managed failover, storage autoscaling, and point-in-time recovery are excellent — and they are also a boundary. Deep engine-level tuning options shrink; your levers become instance class, parameter groups, and architecture. If your workload depends on exotic MySQL configuration, verify it survives the managed boundary before the migration, not during.

Neither of these would have changed our decision. Both would have changed somebody's, and a migration guide that doesn't say so is marketing.

---

## By The Numbers

Around 50 databases migrated over two years. Average cutover downtime was under 3 minutes for tier-0 systems. One rollback due to an application issue discovered post-cutover — the rollback itself went smoothly because we'd tested that path. Zero data-loss incidents.

---

## Conclusion

The interesting question was never "how do you migrate a database to Aurora" — AWS will happily sell you that answer. It's "how do you make the fiftieth migration as safe as the first one was supposed to be." Our answer: the downtime budget is a design input, the cutover is rehearsed until it's boring, the reconnection plan lives in the platform layer, and every lesson gets compiled back into the module so the next team starts where we ended.

The most boring migration is the best migration. The platform's job is to make boring the default.

---

*The platform context behind this post: [Aurora Migration Toolkit](/projects/aurora-migration-toolkit/).*