The cloud is a hostile environment: machines vanish mid-request, networks blip, services throttle you, and instances are killed and replaced constantly. Part 2 of Design for the Cloud covers the habits that let an app survive that — statelessness, transient-fault handling, retries done right, liveness vs readiness health checks, graceful shutdown, autoscaling, partitioning, idempotency, monitoring, and the twelve-factor mindset. These are the practices that separate "works on my laptop" from "survives Black Friday."
In Part 1 we picked a shape for our system. Now we make it survive. Here's the mental shift that the cloud forces and that most outages trace back to: in the cloud, failure is not an exception — it's the steady state. An instance you're using right now might be killed in thirty seconds for a routine deploy, a node drain, or a spot-instance reclaim. A network call might time out not because anything is broken but because a switch was busy for 200ms. A downstream service might return 429 "slow down." None of this is a bug. It's Tuesday.
An app designed for one stable server treats all of this as catastrophe. An app designed for the cloud treats it as normal and keeps serving. The difference is a set of habits — not clever code, just discipline applied consistently. This article is that checklist, with the why behind each item, grounded in the failure modes teams actually hit when systems that were fine at 1,000 users start falling over at 100,000.
1. Be stateless — externalize everything
The habit. Your application instances should hold no important state in their own memory or local disk. No in-process session store, no "uploads land on this server's disk," no counters that live only in one process. Anything that must survive should live in an external store: session and cache in something like Redis, files in object storage (S3/Blob), data in a database.
Why it's non-negotiable. The cloud's whole scaling model assumes instances are interchangeable and disposable. If any instance can handle any request, a load balancer can spread traffic freely, autoscaling can add or remove instances at will, and a crashed instance can be replaced with zero user impact. The moment a user's session lives only on "server 3," you've broken all of that — now traffic must be pinned to server 3 (sticky sessions), scaling is lumpy, and if server 3 dies the user is logged out. Statelessness is the foundation every other practice here stands on.
2. Expect transient faults — and handle them
The habit. A transient fault is a brief, self-correcting failure: a momentary network blip, a service that's busy for a second, a timeout under load. They resolve themselves in milliseconds. Your code should anticipate them and retry, rather than surfacing every blip to the user as an error.
Why. At scale, transient faults are constant — with enough requests, the rare becomes routine. An app that throws a 500 every time a call blips will look unreliable even when nothing is actually broken. But — and this is the trap — handling them wrong causes the Retry Storm from Part 3, where your retries become the outage. So transient-fault handling and retry discipline are the same topic.
3. Retry correctly (this is where teams go wrong)
Retrying sounds trivial and is full of footguns. The rules, distilled from how cloud providers tell you to do it:
| Rule | Why |
|---|---|
| Only retry retryable errors | Retry 429 (too many requests) and 5xx (server errors / timeouts). Never retry 4xx client errors like 400, 401, 403, 404 — they'll fail identically every time. Retrying them is pure waste. |
| Exponential backoff | Wait longer each attempt (1s, 2s, 4s…) instead of hammering. Gives the struggling service room to recover. |
| Add jitter | Randomize the waits so thousands of clients don't retry in perfect lockstep and re-spike the service at the same instant. |
| Cap the attempts | Use a finite retry count. Infinite retries turn a brief outage into a permanent self-inflicted one. |
| Don't nest retry layers | The subtle killer. If your HTTP client retries 3×, and your service layer retries 3×, and the caller retries 3×, one logical request becomes 27 attempts. Retry at one level, not every level. |
4. Make operations idempotent
The habit. An idempotent operation can be performed many times with the same result as performing it once. "Set status to shipped" is idempotent. "Charge the card $50" is not — do it twice and you've double-charged.
Why it's the hidden prerequisite for retries. The moment you retry, you risk doing the same thing twice — because the first attempt might have succeeded and you just didn't get the response (the timeout happened on the way back). If the operation isn't idempotent, your well-intentioned retry double-charges the customer. Every retryable operation must be safe to repeat.
5. Health checks: liveness vs readiness (not the same thing)
The habit. Give your orchestrator (Kubernetes, ECS, App Service) two separate health signals, because it's asking two different questions:
- Liveness: "Is this process alive, or is it wedged and needs restarting?" A failed liveness check → kill and restart the instance.
- Readiness: "Is this instance ready to receive traffic right now?" A failed readiness check → stop sending it requests, but don't restart it.
Why conflating them breaks production. An instance can be perfectly alive but not ready — during startup while it warms caches, or when a dependency is briefly down. If you only have a liveness check and it fails during a slow startup, the orchestrator keeps killing and restarting a process that just needed a moment — a crash loop. If you route traffic based on liveness, you send requests to instances that aren't ready and users get errors. Two questions, two endpoints.
Fig 1 — Liveness asks "should I restart you?"; readiness asks "should I send you traffic?" Conflating them causes crash loops or errors-to-users.
6. Shut down gracefully
The habit. When an instance is told to stop (every deploy, every scale-down, every node drain), it should: stop accepting new requests, finish the ones in flight, flush anything buffered, and then exit. This is graceful shutdown, and paired with readiness it gives you zero-downtime deploys.
Why. Instances are killed constantly in the cloud — a deploy alone might replace every instance you have. If an instance just drops dead mid-request when it gets the stop signal, every user it was serving gets an error, on every deploy. The fix: catch the termination signal (SIGTERM), immediately fail your readiness check so the load balancer drains you (stops sending new traffic — "connection draining"), finish in-flight work, then exit. Done right, users never notice a deploy happened.
7. Autoscale to demand — don't size for peak
The habit. Let the platform add instances when load rises and remove them when it falls, driven by real metrics (CPU, queue depth, request rate, custom signals). Don't provision for peak 24/7, and don't scale by hand.
Why. The cloud's signature advantage is elasticity — you pay for what you use. Sizing for peak means paying peak prices at 3am when traffic is a trickle. Scaling by hand means you're always too late: you react after the spike has already caused errors. Autoscaling on the right metric is what makes the cloud economical and responsive.
The catch. Autoscaling only works if you're stateless (practice #1) — new instances must be able to take traffic immediately — and it has limits. You can scale stateless app servers easily; you usually can't scale your database the same way. Which is why the database is the choke point at scale, and why the next practice matters.
8. Partition data before the database becomes the bottleneck
The habit. Split data and load across multiple stores or partitions (shards) so no single database is asked to do everything. Partition by a key (customer ID, region, tenant), or split by usage pattern (the cure for Part 3's Monolithic Persistence).
Why. Nearly every "we worked fine at 1,000 users and fell over at 100,000" story ends at the database. App servers scale out cheaply; a single relational database scales up (a bigger box) and eventually hits a ceiling. When reads dominate, add read replicas and a cache; when writes dominate or data is huge, partition it. The point is to plan the seams before you hit the wall, because re-sharding a live database under load is one of the most painful operations in this field.
9. Make it observable — you can't fix what you can't see
The habit. Build in the three pillars of observability from day one: metrics (numbers over time — latency, error rate, throughput, saturation), logs (structured records of what happened), and traces (the path of a single request across services). Watch percentiles (p95, p99), not just averages.
Why. Every antipattern in Part 3 is invisible until load reveals it, and you only see it if you're measuring. Averages hide the cliff — a p50 of 50ms can sit next to a p99 of 8 seconds, and it's the p99 users are screaming about. Distributed tracing is what turns "the app is slow" into "this one downstream call is the problem." Without observability you debug by guessing, and in a distributed system guessing is hopeless.
10. Adopt the twelve-factor mindset
Most of the above isn't new — it was codified years ago as the Twelve-Factor App, a set of principles for building software that runs well on cloud platforms. You don't need to memorize all twelve, but a few are load-bearing:
| Factor | In plain words |
|---|---|
| Config in the environment | Never hardcode secrets, URLs, or connection strings. Read them from environment variables, so the same build runs in dev, staging, and prod with different config. |
| Stateless processes | Practice #1. Processes share nothing; state lives in backing services. |
| Backing services as attached resources | Treat your database, cache, and queue as swappable resources reached by URL — not things baked into the app. |
| Disposability | Start fast, shut down gracefully (practices #5–6). Instances are cattle, not pets. |
| Logs as event streams | Write logs to stdout and let the platform collect them. Don't manage log files inside the app. |
The pre-launch checklist
Run this before anything serious goes live. If you can't tick a box, you have a known failure mode waiting for traffic.
| ✓ | Check |
|---|---|
| ☐ | Can I kill any instance right now and lose no data? (stateless) |
| ☐ | Do I retry only 429/5xx, with backoff + jitter, capped, at one layer only? |
| ☐ | Are my mutating operations idempotent (safe to retry)? |
| ☐ | Do I have separate liveness and readiness checks? |
| ☐ | Does the app drain and shut down gracefully on SIGTERM? |
| ☐ | Does autoscaling trigger on a metric that reflects real load? |
| ☐ | Do I know where the database becomes the bottleneck, and have a partition plan? |
| ☐ | Can I trace one request end to end and see p99 latency? |
| ☐ | Is all config in the environment, with no secrets in code? |
FAQ
If I autoscale, do I still need to worry about performance?
Yes — autoscaling hides inefficiency, it doesn't fix it. A chatty, cacheless app just autoscales to a bigger bill (and the database, which usually can't autoscale, still falls over). Autoscaling handles legitimate load growth; it can't save you from the antipatterns in Part 3.
Why can't I just retry everything to be safe?
Two reasons. Retrying non-retryable errors (4xx) is wasted effort that'll fail anyway. And retrying non-idempotent operations can double-charge, double-ship, or double-send. Retries are only safe when the error is transient and the operation is idempotent.
Liveness and readiness seem like the same check — why two?
They answer different questions. Liveness: "should you restart me?" Readiness: "should you send me traffic?" An instance can be alive but not ready (warming up, or a dependency is down) — you want traffic to stop without a restart. Using one check for both causes crash loops or errors to users.
Is twelve-factor still relevant in 2026?
The principles are, even if the article is old. Config in the environment, stateless processes, disposability, logs as streams — these are exactly what containers, Kubernetes, and serverless assume. Modern platforms basically enforce twelve-factor; following it is how you stop fighting them.
Takeaways
- Failure is the steady state. Design assuming instances die, networks blip, and services throttle — because they do, constantly.
- Statelessness is the foundation. Externalize state and every other practice (scaling, replacement, zero-downtime deploys) becomes possible.
- Retries are a loaded gun. Backoff, jitter, cap, retry only retryable codes, at one layer — and make operations idempotent first.
- Liveness ≠ readiness, and graceful shutdown is mandatory. Together they're what make deploys invisible to users.
- Plan for the database wall. App tiers scale out cheaply; databases don't. Know where the choke point is before you hit it.
- Observe everything, at p99. You can't fix what you can't see, and averages hide the pain.
You've now got the shapes (Part 1) and the habits (Part 2). Next we look at what happens when these habits are missing — Part 3: Performance antipatterns, the specific, named traps that turn a fine app into a 3am page, and exactly how to fix each one.
References
- Transient fault handling · Azure Architecture Center
- Recommendations for handling transient faults · Well-Architected Framework
- Retry pattern · the canonical retry guidance
- The Twelve-Factor App · the cloud-app principles, in full
Extra reads
- Timeouts, retries, and backoff with jitter · Amazon Builders' Library
- Liveness, Readiness & Startup Probes · Kubernetes docs
- Idempotent requests · Stripe — idempotency keys in practice
- Part 3 — Cloud performance antipatterns · what happens when these habits are missing