The whole design, in writing
Learn system design by building a text-paste service like Pastebin step by step. An interactive guide covering unique key generation, separating metadata from blob storage, caching and CDN for read-heavy traffic, TTL expiration, privacy, and async analytics.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is Pastebin?
Paste some text, get a short link, share it; anyone who opens the link sees the text. Dead simple — but at scale it’s a sharp lesson in read-heavy design: pastes are written once and read thousands of times, sometimes from all over the world.
Mint a unique short key, store the (potentially large) text in a blob store with only a pointer in the database, and make the read path fly with caching and a CDN. It’s the URL shortener’s cousin — but with real content to store and expire.
What the new pieces do
- Creatorclient
- Pastes some text — a log, a snippet, an error — and gets back a short URL to share. Writes are rare compared to reads.
- Readerclient
- Opens a shared paste link. Expects the text to load instantly, possibly thousands of times for one popular paste.
Step 1 · The skeleton
Store text, return a key
Two operations: create (take text, give back a key) and read (take a key, give back text). A paste can be a few bytes or a few megabytes, so where the text lives matters from the start.
A paste can be a few bytes or a few megabytes. Where do you put the text?
Multi-megabyte bodies bloat the DB, slow every query, and waste its indexed storage on data you only fetch by key. Keep big bodies out of the database.
The DB holds tiny queryable rows (key → location, TTL, flags) while the object store holds the bulk. Same split as photos/videos — small fast database, bulk where bulk belongs.
In-memory pastes vanish on restart and can’t scale past one machine’s RAM. Pastes must be durably stored, with memory used only as a cache later.
The API Server writes the text to an Object Store and a small row to the Metadata DB mapping the key to that text’s location. Reads resolve the key in the DB, then fetch the body from the blob store.
Why this piece earns its place
Two stores means two writes and no transaction spanning them, so the order is part of the design. Write the body to the object store first, then the metadata row. If the process dies in between, you are left with an orphan body nobody can reach — invisible to readers, costing only storage, and reclaimable later by the same background sweep that handles expiry. Do it the other way and a crash leaves a row whose pointer resolves to nothing: a live key that hands readers a 500 instead of a paste. Deletion runs in the mirror image — drop the row first so the key stops resolving, then remove the body — because the window you can tolerate is “the bytes exist but nothing points at them”, never “the key still works but the bytes are gone”. The same asymmetry decides what a read does when a pointer misses: treat it as gone, alarm on it, and never retry into the store hoping the body turns up.
What the new pieces do
- API Serverbackend
- Handles both operations: store a new paste and return its key, or look up a key and return the text. The single front door.
- Metadata DBstore
- Small rows: the key, where the text lives, expiry time, and privacy flag. Queried on every read to resolve a key to its blob.
- Object Storestore
- The actual paste bodies. Kept out of the database so large pastes don’t bloat it; the DB holds only a pointer here.
Step 2 · The short link
Mint a unique key
Every paste needs a unique, short, URL-safe code like /aZ9k2. Hashing the content collides on identical pastes and yields long codes; pure random forces a “taken?” check on every write.
Every paste needs a unique short URL-safe code like /aZ9k2. How do you mint it without collisions?
Hashing collides on identical pastes (two people paste the same snippet → same key, overwriting), and full hashes are long. You need short codes unique regardless of content.
Random codes force a "is it taken?" lookup on every create, and collisions rise as you fill the space. Uniqueness by construction avoids the check entirely.
Counting up then base62-encoding guarantees no collisions without any lookup; ~7 chars covers trillions of pastes. (Or hand out pre-generated unused keys.) Unique by construction.
Use a Key Gen Service: take a counter and encode it in base62, or hand out pre-generated unused keys. Codes are unique by construction, so creating a paste never needs a collision check.
Why this piece earns its place
A counter is one number, and every create in the system queues behind it — you have traded a collision check for a single point of contention, and if that counter is one row somewhere, a single point of failure too. The fix is to stop dealing keys one at a time: each Key Gen instance claims a block of the range in one atomic increment and then hands ids out of memory, so the shared counter is touched once per block instead of once per paste. Block size is the knob. Big blocks mean almost no coordination, at the cost of a bigger hole in the keyspace every time an instance dies with a block half spent — and holes are exactly what a space of 3.5 trillion keys can afford to lose. Small blocks waste fewer codes and put you back on the shared row under load. I’d rather burn keys than serialize creates. The one property a block allocator can never lose is that no block is ever handed out twice, so that increment has to be genuinely atomic, not read-then-write.
- base62encoding
- ~7chars
- trillionsof keys
What the new pieces do
- Key Genservice
- Mints a unique short code (base62) for each paste so two pastes never collide — uniqueness by construction, no “is it taken?” check.
Back of the envelope
- base62, ~7 chars
- 62^7 ≈ 3.5 trillion keys
- counter ⇒ unique by construction
- no collision check on create
- short + URL-safe
- a–z A–Z 0–9, no escaping
Step 3 · Reads dwarf writes
Cache hot pastes
A paste shared in a popular thread gets read thousands of times. Hitting the metadata DB and blob store for every one of those identical reads is wasteful and slow.
A paste in a popular thread is read thousands of times. How do you avoid hitting the DB + blob store every time?
Replicas add throughput but every hot read still does a full store fetch for identical, unchanging content. When the same paste repeats, serve it from memory instead of re-fetching.
A hit returns the text in <1ms; a miss falls through to the blob store and warms the cache. Pastes are write-once, so there’s no invalidation — caching is pure upside from the second read on.
You can’t control reader behavior, and the goal is to serve many readers fast. The system must make repeat reads cheap server-side via caching, not push work onto users.
Put a Cache (Redis) on the read path keyed by paste id. A hit returns the text in under a millisecond; a miss falls through to the blob store and warms the cache. Because a paste’s text never changes, it’s perfectly cacheable.
Why this piece earns its place
The cache is a fixed memory budget and paste bodies do not share a size, which is the trap here. A handful of multi-megabyte pastes admitted by value will evict thousands of small hot ones, and that eviction is silent: hit-rate sags, latency climbs, nothing errors. So cap what is cacheable as content — small bodies go in whole, and anything above the cap caches only its metadata row, so the read still skips the DB and streams the body from the object store. That keeps most of the win, because the expensive part of a large paste is moving the bytes, not finding them. The other thing worth a cache entry is the negative answer. Keys are never reused, so a key that resolves to nothing today resolves to nothing tomorrow, and a link shared widely keeps getting clicked long after the paste is gone. Hold those misses briefly too, or every dead link in an old thread is a database lookup you pay for forever.
- <1mscache hit
- 100:1reads : writes
- noinvalidation
What the new pieces do
- Cachecache
- Redis holding the most-read pastes. A popular link is served from memory in under a millisecond instead of fetching the blob every time.
Back of the envelope
- ~100:1 reads : writes
- write once, read thousands of times
- hit ≈ <1ms RAM
- vs a DB lookup + blob fetch
- immutable ⇒ no invalidation
- caching is pure upside
Step 4 · Readers everywhere
Go global with a CDN
Your cache and store sit in one region, but readers are worldwide. A paste opened from another continent pays a long round-trip on every fetch.
Your cache and store sit in one region but readers are worldwide. How do you cut the cross-continent round-trip?
More nodes in one region don’t help a reader on another continent — they still pay the long round-trip on every fetch. You need content closer to the reader, geographically.
Hand-rolling global replication of all content is expensive and heavy when most pastes are never read in most regions. A CDN already does edge caching on demand.
The first reader in a region pulls through the edge; everyone after is served locally. Immutable pastes cache at the edge with long TTLs — the cheapest scaling for read-heavy payloads.
Front the read path with a CDN. The first reader in a region pulls the paste through the edge; everyone after is served locally. Immutable content plus a long TTL means the edge does the heavy lifting.
Why this piece earns its place
The long TTL rests on an assumption worth naming: a paste is immutable, so a copy of it can never go wrong. That holds until a paste has to stop existing — an expiry passes, an author changes its visibility, someone reports content that has to come down. From this step on, the origin is no longer the only place the bytes live, and an edge copy has its own lifetime that nothing in the metadata DB governs. Two defenses, and you want both. Bound the edge TTL by the paste’s remaining life so a cached copy can never outlive the paste it came from — free for pastes with no expiry, and it makes short-lived ones cheap to reason about. And purge explicitly on delete instead of waiting a TTL out. Then be honest about what a purge is: a request that propagates across an edge fleet, not a transaction. The real guarantee is “gone at the origin now, gone at the edge shortly”, so anything that must be unreachable the instant you say so should never have been edge-cacheable at all.
What the new pieces do
- CDNcache
- Caches paste content near readers worldwide. Since a paste’s text is immutable, it can sit at the edge with a long TTL.
Back of the envelope
- 1st regional read pulls through edge
- every reader after is served locally
- immutable + long TTL
- the edge does the heavy lifting
- offloads latency + bandwidth
- origin barely touched
Step 5 · Don’t keep it forever
Expiration & TTL
Many pastes are meant to be temporary (“expire in 1 hour / 1 day”), and keeping every paste forever steadily fills storage with dead content nobody will ever read again.
Many pastes are meant to be temporary ("expire in 1 hour"). How do you stop storage filling with dead content?
An expiry on the record makes cleanup routine: expired keys return 404/410, lazy deletion drops them on access, and a background worker (or native store TTL) sweeps the rest. Lifetime becomes data.
Manual cleanup doesn’t scale and risks deleting the wrong thing under pressure. Expiry should be declared per paste and enforced automatically, not firefought.
Keeping every temporary paste forever fills storage with content nobody will read again, and ignores users who explicitly wanted expiry. TTL is both a feature and a cost control.
Store an optional TTL with each paste. An Expiry Worker (or the store’s native TTL) removes expired pastes from the blob store and metadata, after which the key returns a clean 404/410.
Why this piece earns its place
Be clear about which half of this carries the guarantee, because it changes how hard the other half has to work. The check on read is what makes an expired paste unservable: it sits in the request path, it reads the expiry off the row it already fetched, and it cannot lag. The worker is only reclaiming space, which means it is allowed to be late — and treating it as a batch job rather than a deadline is what keeps it affordable. That matters because the naive version scans every paste hunting for dead ones, turning a chore into a full pass over a table that only grows. Index by expiry time, or bucket keys into the hour they die in, so the worker reads only what is actually due. Then rate-limit it, because expiry is bursty by nature: a spike of one-hour pastes created together comes due together, and a worker deleting as fast as it can will aim that burst at the same store serving reads, converting a storage chore into a latency incident.
What the new pieces do
- Expiry Workerworker
- Removes pastes whose TTL has passed from the store and metadata, reclaiming space so expired links return 404.
Step 6 · Who can see it?
Privacy & limits
Not every paste should be public. Some are unlisted (only those with the link) or private (only the author). And without size limits, one user can paste a gigabyte and wreck your storage and bandwidth.
Some pastes should be unlisted (link-only) or private (author-only). How do you enforce that?
An obscure URL is "security by obscurity" — unlisted is not private. Links leak, get shared and indexed, so true privacy needs a real access check, not just a random key.
Hiding from a listing does nothing if anyone with the key can still GET the content. The read path itself must verify permission, not just the discovery surface.
Store a per-paste flag; public serves freely, unlisted needs the key, private requires the author’s auth — checked on every read. Pair with size caps and rate limits so abuse can’t balloon costs.
Add a privacy flag per paste, enforced at the API on every read, and require the right key/auth for private ones. Cap paste size and rate-limit creation so abuse can’t balloon costs or hammer the service.
Why this piece earns its place
Privacy changes something upstream of the check itself: after the last two steps the read path has several entrances, and only one of them can enforce anything. A request answered at the edge never reaches the API server, so a private paste that is edge-cacheable is a paste whose permission check has been designed out — and a shared cache can hand a body fetched for one authorized reader to the next person who asks. So the flag has to decide cacheability before it decides access. Public and unlisted bodies can be cached anywhere, since their only protection was ever the key. Private bodies are marked uncacheable end to end, skip the shared tiers, and are read from the store after the author is authenticated. You are knowingly giving up the sub-millisecond path for that class of paste, which is the right trade: private pastes are rare and read by one person. The size cap belongs at the same boundary — enforce it while the upload streams, not after you have already accepted and stored the bytes.
Step 7 · Counts & scale
Analytics, async
Owners want view counts, but writing an analytics row on every read would slow the one thing that must stay fast — and billions of small rows pile up in one database over time.
Record views asynchronously (fire an event, aggregate later) so reads never block on counting. Shard the Metadata DB by key and lean on the cache/CDN so the read path stays flat as paste count grows into the billions.
Why this piece earns its place
The count has a measurement problem created by this design’s own success. The CDN exists precisely to stop reads reaching the origin at all, and it works — most views of a popular paste are answered at the edge, and the API server never sees them. A Redis hit is not the same thing: it is served by the API server, so it is precisely the read the origin does see and can count. But a counter incremented only where the request happens to land undercounts exactly the pastes that are read most, and it gets worse every time the edge does its job better. The owner ends up watching a number that tracks edge caching rather than readership. Count where the reads actually are: treat the CDN’s access logs as the primary source, aggregate them on a schedule, and use the origin’s own events only for the traffic the edge never sees. That also explains why this number is best-effort by construction — logs arrive in batches, late, and sometimes not at all — so present it as a view count and never let anything that has to be exact, like billing or a rate limit, depend on it.
What the new pieces do
- Analyticsstore
- Tracks how often each paste is viewed, updated asynchronously so counting a view never slows the read itself.
You did it
You just designed Pastebin.
Everything you assembled, in order
- Create/read over an API, with text in a blob store and a pointer in the DB.
- Base62 keys from a Key Gen Service — unique by construction.
- A Redis cache makes the read-heavy, immutable path sub-millisecond.
- A CDN serves pastes from the edge for global readers.
- TTL-based expiry (lazy delete + sweep) keeps storage from filling with dead pastes.
- Per-paste privacy flags, size caps and rate limits for the real world.
- Async analytics and a sharded metadata DB keep the read path flat at scale.