Designing Twitter's Home Timeline

1.1What it has to do

We'll call a short post a tweet, the name Twitter used until 2023 (X now calls them posts). A user's home timeline is the list of recent tweets from the accounts that user follows, newest first. It's what Kabir sees when he opens the app. Twitter also had a second kind of list, the user timeline: every tweet one account has sent, as shown on Ananya's profile page. Twitter's own engineers described the home timeline as a merge of the user timelines of everyone you follow, and that definition is a good one to hold on to, because every design in this chapter is a different way of doing that merge.

What we're designing comes down to a short list:

1. Post a tweet, and store it so it's never lost.
2. Deliver it to the home timeline of every follower, within a few seconds.
3. Serve the home timeline fast whenever someone opens the app or scrolls.
4. Search every tweet by its words, seconds after it's posted.
5. Rank the timeline, so the posts a user most likely wants come first (Twitter added this in 2016; before then the timeline was strictly newest first).

And the qualities it needs:

• Fast reads: opening the app should show a timeline in a fraction of a second.
• Fresh: a tweet should reach followers' timelines within about five seconds, which was Twitter's own target in 2012.
• Durable where it matters: a tweet, once posted, must never disappear. A timeline is a different matter, as we'll see: it can be rebuilt.
• Available under spikes: the moments when everyone posts at once, a goal in a football final or a famous scene in a film, are exactly when people most want to read.

1.2How big is it?

The most detailed public numbers come from a talk called Timelines at Scale, given at QCon San Francisco in November 2012 by Raffi Krikorian, who then led Twitter's platform services group. At the time Twitter had about 150 million active users, who sent about 400 million tweets a day. That averages about 5,000 tweets a second, with a daily peak around 7,000 and more than 12,000 during big events.

What shapes everything else is on the other side. Krikorian put timeline reads at about 300,000 queries a second, against about 6,000 requests a second of writes. People read Twitter far more than they write to it. He described Twitter as "primarily a consumption mechanism, not a production mechanism".

In August 2013 Twitter's engineering blog gave newer figures: more than 500 million tweets on a typical day, about 5,700 a second on average, and a record one-second peak of 143,199 tweets a second, set on 3 August 2013 when viewers in Japan watching a TV broadcast of the film Castle in the Sky all posted at the same moment. That spike was about 25 times the normal rate. In 2023, the last time Twitter published a comparable figure, it was still about 500 million tweets a day.

2.1The obvious design

Start with two tables in a relational database. One holds tweets: an ID, the author, the text and the time it was posted. A second holds follows: one row for each pair "Kabir follows Ananya". When Kabir opens the app, the server runs one query:

SELECT * FROM tweets
WHERE author_id IN (SELECT followee_id FROM follows WHERE follower_id = :kabir)
ORDER BY created_at DESC
LIMIT 20;

Read in plain words, it says: find everyone Kabir follows, find all their tweets, sort them by time and keep the newest twenty. This is the merge from section 1.1, done fresh every time someone looks. Doing the work when someone reads is called fan-out on read: the reader's request fans out to every account they follow. Twitter in 2006 and 2007 was a Ruby on Rails web application in front of a MySQL database, and it built timelines roughly this way.

Writing is as cheap as it can be here: Ananya's tweet is one row, whether she has three followers or three million. All the cost lands on reads. If Kabir follows 300 accounts, his query has to look up the recent tweets of 300 authors and sort them together, and the database has to do it again for his next refresh, and for the next person's, 300,000 times a second. Krikorian's talk summed up this approach in one line: the naive version is "a massive select statement over all of Twitter", and "it was tried and died".

2.2The Fail Whale years

Slow reads weren't the only problem, and Twitter's 2013 account of these years lists the others. Tweets lived in a single MySQL database with one primary server, and when it filled up, Twitter started a new database for the newest tweets. That's called temporal sharding: splitting data by time, so each database holds one period. It seems tidy, but think about where the traffic goes. Every new tweet is written to the newest database, and most reads ask for recent tweets, which also live in the newest database. So the newest database is always the hottest, and adding more databases for older periods doesn't help at all.

Even the web servers had a ceiling. Each Rails server ran as a set of separate processes, each handling one request at a time, and Twitter measured them at 200 to 300 requests a second per machine. At that rate, keeping up with growth meant buying machines faster than anyone could install them.

When it all fell over, users saw an illustration by the designer Yiying Lu: eight small orange birds lifting a whale out of the sea in a net, captioned "Too many tweets! Please wait a moment and try again." Users called it the Fail Whale, and it became famous enough that the New York Times Magazine wrote about it in February 2009. In November 2013 Twitter's head of engineering told Wired that the company had taken it out of production that summer. It came to a head at the 2010 football World Cup. Twitter's engineering blog described how every shot on goal, penalty and card sent a flood of tweets that "repeatedly took its toll and made Twitter unavailable for short periods of time", while engineers worked through the nights for efficiency gains that the growth swallowed straight away.

2.3Off the monolith

After the World Cup, Twitter decided to rebuild instead of patching. Its 2013 write-up lists three goals: reduce the number of machines needed by ten times, isolate failures so one broken part couldn't take down the whole site, and let small teams ship features independently.

First came the servers themselves. Twitter already ran large services on the JVM, the Java virtual machine that runs Java and Scala programs: its search engine was written in Java, and its social graph store in Scala. A JVM server can handle many requests at once with threads, where each Rails process handled one. Twitter estimated a rewrite would give more than ten times the throughput on the same hardware, and by 2013 its JVM services were serving 10,000 to 20,000 requests a second per machine, against 200 to 300 for Rails.

Second, Twitter split the single Rails application, which Twitter called the monolith, into separate services, starting with what it called the "core nouns": a tweet service, a timeline service and a user service. Each service has one owner and one network interface, and teams agree on the interfaces, then build independently. For the calls between services, Twitter wrote Finagle, a library that gives every service the same connection pooling, retries, timeouts and load balancing, after early services each handled failures differently and set off storms of retries against each other.

From here on we follow the timeline service and the services around it. Its first problem was the one from 2.1: building every timeline from scratch on every read is far too slow. If reads outnumber writes fifty to one, we should do the work once, when the tweet is written, and make each read cheap.

3.1A mailbox per user

Here's the idea. Give every user a precomputed home timeline, like a mailbox, holding the IDs of the recent tweets they should see. When Ananya posts, look up her followers and drop her tweet's ID into each of their mailboxes. When Kabir opens the app, there's nothing to compute: read his mailbox, which already holds his timeline in order.

Doing the work when a tweet is written is called fan-out on write, because one write fans out into one insert per follower. It's the opposite of section 2's fan-out on read, and it turns the cost around: a read becomes one lookup, and a write becomes as many inserts as the author has followers. Since Twitter's reads outnumbered writes fifty to one, that trade was clearly worth making. By 2012 this was how Twitter built every home timeline, with the mailboxes kept in Redis, an in-memory key-value store whose values can be lists (chapter 22).

Notice the queue between the write API and the fan-out. Ananya's tap is answered as soon as the tweet is stored; delivery happens afterwards, in the background. In 2012 that was a necessity as much as a choice: the tweet-writing path was still Ruby, with only about 45 to 48 processes per machine, so each process had to hand the work off and hang up as fast as it could. It's also why Twitter's goal was phrased as delivery "in under 5 seconds" and not instantly. Fan-out takes time, and the design accepts a few seconds' delay in exchange for never making a reader wait.

3.2What's in the mailbox, and how much memory it takes

Each entry in a home timeline is tiny. Krikorian's talk listed what it held: the tweet's ID, the ID of the user who wrote it, and 4 bytes of flags marking whether it's a retweet, a reply or something else. An ID is a 64-bit number, so an entry is 8 + 8 + 4 = 20 bytes. Ananya's words aren't there at all.

Each list was capped at 800 entries. Scroll back far enough and you hit the end. That cap is a design choice, and it's what makes the whole cache affordable:

Only active users got a timeline in memory. Twitter defined active as having logged in within the last 30 days, and the talk noted that the window could change depending on how much cache was available. If you hadn't logged in for a month, fan-out skipped you, and when you came back, Twitter rebuilt your timeline with a process it called reconstruction: ask the social graph who you follow, fetch each of their recent tweets from disk, merge them and load the result into Redis. That's version 1's fan-out on read, kept as the slow path for the rare user who needs it.

3.3Why a timeline can live in memory

Keeping something important only in memory sounds reckless, so it's worth being clear about why it's fine here. A home timeline doesn't record anything that isn't stored somewhere else: tweets are kept durably in their own store (section 7), and so is the follow graph. A home timeline is derived data: given who Kabir follows and what they've posted, it can always be computed again, as reconstruction does. Losing it costs time, not data.

Twitter still protected it. Each timeline was stored on three different machines in each data centre, each found through its own hash ring (chapter 29 explains how a hash ring maps keys to machines). A read asks whichever of the three answers fastest. If one machine dies, the other two still answer, so Twitter doesn't have to reconstruct every timeline that lived on it. Krikorian reported the timeline service's response time as about 5 milliseconds at the median and 100 at the 99th percentile (the time that 99 requests in 100 beat); only at the 99.9th percentile, when a timeline had to be rebuilt from disk, did it take a couple of hundred milliseconds.

4.1The operations a timeline needs

Look at what fan-out and reads do to Kabir's list. Fan-out pushes a new ID onto the front. Reads take the first 20 or so from the front, then the next 20 as Kabir scrolls. And when the list grows past 800, the oldest entries fall off the back. In Redis terms that's LPUSH to add at the head, LTRIM to cut it back to 800, and LRANGE to read a slice. Chapter 22 covers Redis itself; here the question is what structure sits behind those three commands, because at 20 bytes an entry, the overhead of the structure can be bigger than the data.

Your first instinct is probably a linked list: each entry is a separate small block of memory holding the value plus a pointer to the next block, and usually one to the previous block as well. Pushing at the head and dropping from the tail are both cheap, because only a couple of pointers change.

Its weakness is memory. Each node needs its two pointers, 16 bytes on a 64-bit machine, plus whatever the memory allocator adds to track each separate block. For values this small, the bookkeeping is about as large as the data, which would roughly double the terabytes from 3.2. Yao Yue, who worked on Twitter's cache team, made exactly this point in her 2014 talk on scaling Redis at Twitter: pointer overhead is high when each item is a small ID.

4.2One block, then many small blocks

At the other extreme, a structure can pack all the entries into one contiguous block of memory, one after another, with no pointers at all. Redis had such a structure, called a ziplist, and Twitter's original timeline design used ziplists exclusively. Memory-wise it's ideal: 800 entries cost about 800 × 20 bytes and almost nothing else.

So one structure wastes memory and the other wastes time. Twitter's fix was to combine them. A hybrid list is a linked list whose nodes are small ziplists. Each node holds a run of entries packed together, up to a fixed size in bytes, and the nodes are linked to each other with pointers. An insert at the head touches only the first small block, and if that block is full, a new block is linked in front of it. Trimming the oldest entries drops whole blocks from the tail. Now pointer overhead is paid once per block instead of once per entry.

Yue's 2014 talk gave the scale of this structure in one data centre: the timeline service's hybrid lists took about 40 TB of allocated memory, served about 30 million queries a second, and ran on more than 6,000 Redis instances. By 2017 Twitter's infrastructure blog named the cluster Haplo, "the primary cache for Tweet timelines", backed by a customised Redis implementing the hybrid list, written by the fan-out and timeline services and read by the timeline service. It handled 40 to 100 million Redis commands a second across the cluster, from about 800,000 requests a second to the service.

5.1When one tweet is 100 million inserts

Fan-out on write handles Ananya beautifully: 300 followers, 300 inserts, done in well under a second. Now Nova posts at the same moment.

This wasn't hypothetical. In 2012 Lady Gaga had 31 million followers, and Katy Perry and Justin Bieber 28 million each. Krikorian's talk reported that fan-out to a million followers took about 3.5 seconds at the median, but the slowest 1% of large fan-outs took up to five minutes, and that it could take that long for a tweet from Lady Gaga to reach all her followers.

Delay also caused a stranger problem. A follower near the front of Lady Gaga's fan-out sees her tweet early and replies. The reply is an ordinary tweet with an ordinary fan-out, so it can reach some of her other followers before her original tweet does. Krikorian described exactly this: replies being seen before the tweets they reply to, a race condition that confused users. Sorting timelines by tweet ID couldn't fix it, because the original wasn't in the list yet.

5.2Measuring the trade

To see the trade in numbers, this program builds a small Twitter: 50,000 accounts, each following 50 others, chosen so that a few accounts are followed by almost everyone and most by almost nobody, the shape of real follower counts. Then it compares four policies. "Pure push" fans out every tweet. "Pure pull" fans out nothing and merges at read time, like version 1. In between, we fan out on write for everyone except accounts above a follower limit, and merge those accounts' tweets in at read time.

For each policy it reports the average number of timeline inserts per tweet, the worst single tweet (the most inserts any one tweet causes), the number of lists a read has to fetch, and the total work per tweet, counting each insert or list fetch as one unit and assuming 50 reads for every tweet, the 2012 ratio. random.choices with cum_weights picks followed accounts with probability proportional to 1/rank, so account 0 is the most popular. f & big is the set of big accounts one reader follows.

import random
random.seed(65)
 
# A small Twitter: 50,000 accounts, each following 50 others.
# Popular accounts get picked far more often (account 0 most of all).
USERS, FOLLOWS = 50_000, 50
weights = [1 / (rank + 1) for rank in range(USERS)]
cum, total = [], 0.0
for w in weights:
    total += w
    cum.append(total)
 
followers = [0] * USERS
following = []
for user in range(USERS):
    picks = set(random.choices(range(USERS), cum_weights=cum, k=FOLLOWS))
    picks.discard(user)
    following.append(picks)
    for account in picks:
        followers[account] += 1
 
ordered = sorted(followers)
print(f"followers: median {ordered[USERS // 2]}, top account {ordered[-1]:,}")
 
# Every account tweets equally often; reads outnumber tweets 50 to 1
READS_PER_TWEET = 50
print(f"{'skip fan-out above':>18} | {'inserts/tweet':>13} | {'worst tweet':>11} | {'lists/read':>10} | {'work/tweet':>10}")
for limit in [None, 40_000, 5_000, 0]:
    if limit is None:
        big = set()
    else:
        big = {a for a in range(USERS) if followers[a] > limit}
    inserts = sum(followers[a] for a in range(USERS) if a not in big) / USERS
    lists = sum(1 + len(f & big) for f in following) / USERS
    worst = max((followers[a] for a in range(USERS) if a not in big), default=0)
    work = inserts + READS_PER_TWEET * lists
    label = "never (pure push)" if limit is None else f"{limit:,}"
    if limit == 0:
        label = "always (pure pull)"
    print(f"{label:>18} | {inserts:13.1f} | {worst:11,} | {lists:10.1f} | {work:10.0f}")

followers: median 9, top account 49,492
skip fan-out above | inserts/tweet | worst tweet | lists/read | work/tweet
 never (pure push) |          42.4 |      49,492 |        1.0 |         92
            40,000 |          40.5 |      38,626 |        2.9 |        185
             5,000 |          30.9 |       4,896 |       12.5 |        655
always (pure pull) |           0.0 |           0 |       43.4 |       2169

Start with the first line: the median account has 9 followers and the top one has 49,492, nearly everyone. Now read the table top to bottom. Pure push does the least total work by far, 92 units per tweet against 2,169 for pure pull, because reads are fifty times more common and pure push makes every read a single list. That's the 2012 argument for fan-out on write, reproduced in a few lines. Pure pull makes each read fetch about 43 lists (the 50 follows, minus a few duplicates), and with fifty reads per tweet that dominates everything.

But look at the "worst tweet" column. Under pure push, one tweet from the top account costs 49,492 inserts, a thousand times the average of 42. That's Nova's tweet, and it's the column that hurts. Skipping only the two accounts above 40,000 followers costs a couple of extra list fetches per read and doubles the total work, but nobody's tweet is ever bigger than 38,626 inserts. Push the limit down to 5,000 and the worst tweet is under 5,000 inserts, at seven times pure push's total work. So the hybrid doesn't save work on average. What it buys is a bound on the worst single tweet. That bound decides how late Nova's post arrives and how long Ananya's waits behind it.

5.3The hybrid: merge celebrities in at read time

Twitter's answer, described in the 2012 talk as work in progress, was exactly this hybrid. For "high value users" with huge followings, stop fanning out. Keep their tweets only in their own user timelines, and when a follower reads, merge those few lists into the follower's precomputed timeline. Krikorian gave Taylor Swift as the example of an account whose tweets would be merged in at read time, and said that balancing the read and write paths this way saved "10s of percents of computational resources". Twitter never published the follower limit it used, or how many accounts were above it.

That last step is a classic algorithm, the k-way merge. Each list is already sorted, newest first. Look at the head of each list, take whichever is newest, and advance that list. With a small structure called a heap, which always keeps the largest of its items on top, picking the newest of k heads takes about log₂ k steps, so merging three lists costs barely more than reading one.

This program does Kabir's merge. It needs tweet IDs that sort by time, so it makes them the way Twitter does, with the Snowflake layout that section 6 explains: the time in milliseconds goes in the top bits, so a bigger ID means a newer tweet. Python's heapq.merge is a k-way merge over a heap, and reverse=True tells it the inputs are sorted largest first. islice takes the first five results.

import heapq
from itertools import islice
 
EPOCH = 1288834974657                  # Snowflake's custom epoch, in ms
 
def snowflake(ms, machine, seq=0):     # 41 bits time | 10 bits machine | 12 bits sequence
    return ((ms - EPOCH) << 22) | (machine << 12) | seq
 
def seconds(tweet_id):                 # read the time back out of an ID
    return ((tweet_id >> 22) + EPOCH) / 1000
 
NOW = 1_791_658_800_000                # 10 Oct 2026, 19:00:00 UTC, in ms
author = {}
 
def tweet(who, seconds_ago, machine):
    tid = snowflake(NOW - seconds_ago * 1000, machine)
    author[tid] = who
    return tid
 
# Kabir's cached home timeline: IDs fanned out at write time, newest first
home = [tweet("Ananya", 2, 17), tweet("Dev", 95, 4), tweet("Meera", 300, 9),
        tweet("Ananya", 1800, 17)]
# Nova has 100M followers, so her tweets were not fanned out
nova = [tweet("Nova", 1, 31), tweet("Nova", 600, 22)]
# A second large account Kabir follows, also merged at read time
league = [tweet("CricketLeague", 45, 8)]
 
# Each list is already sorted newest first, so a k-way merge yields a sorted timeline
merged = heapq.merge(home, nova, league, reverse=True)
for tid in islice(merged, 5):
    print(f"{tid}  {author[tid]:<14} {NOW / 1000 - seconds(tid):6.0f} s ago")

2108995977737269248  Nova                1 s ago
2108995973542907904  Ananya              2 s ago
2108995793187799040  CricketLeague      45 s ago
2108995583472582656  Dev                95 s ago
2108994723640283136  Meera             300 s ago

Kabir's merged timeline interleaves the three lists correctly: Nova's post from one second ago, then Ananya's from two seconds ago, then the league's, then the older entries from Kabir's cached list. Nothing in the program compares timestamps. It compares IDs, and the IDs are in time order because of how they're built. You can see it in the left column: each ID is smaller than the one above it. Notice also that heapq.merge stopped after five results without reading the rest of Kabir's list. A merge only reads as far as the page it's building, so merging a few lists at read time is cheap.

6.1Why the IDs matter

The merge in 5.3 only worked because tweet IDs sort by time. Fan-out sorts by them too, and so does every client that shows a timeline. In version 1 that came for free: MySQL handed out IDs from a counter that went up by one with each new row, so a newer tweet always had a bigger ID.

That stopped working once tweets were spread over many databases. Twitter's 2013 account explains the chain. To get away from the hot newest database of 2.2, Twitter used Gizzard, its own framework for sharding data across many MySQL servers, and built a tweet store called T-Bird on it: each new tweet's ID was hashed to pick a database, so writes and reads spread evenly over all of them. But a counter inside one database can't hand out IDs for the others, and two databases counting independently would hand out the same numbers. Twitter needed a new source of IDs, and in June 2010 it announced one called Snowflake.

6.2What Snowflake had to do

Twitter's 2010 announcement set out the requirements. Snowflake had to generate tens of thousands of IDs a second, with high availability, which ruled out a single central counter that every tweet had to wait for. The IDs had to be "roughly sortable": tweets posted around the same time should have IDs close together, because "this is how we and most Twitter clients sort tweets". Twitter aimed to keep IDs ordered to within a second. And the IDs had to fit in 64 bits, because changing the size of tweet IDs had already been painful once, with over 100,000 different programs built on Twitter's API.

Twitter rejected three alternatives. Ticket servers, a MySQL table that does nothing but hand out numbers, as Flickr used, didn't give the ordering without extra machinery. UUIDs, random identifiers that need no coordination, take 128 bits. ZooKeeper's sequential nodes, a counter kept by a coordination service, were too slow, and making every tweet wait on a coordinated service would lower availability.

Twitter's answer was to build each ID out of three parts: a timestamp, a worker number and a sequence number. Chapter 43 walks through the layout and what happens when a machine's clock jumps backwards (section 6, on deep dives), so here's the short version. The top 41 bits hold milliseconds since a custom starting moment (an epoch), 4 November 2010; the next 10 bits identify the machine (5 bits for the data centre and 5 for the worker, in the original code); and the last 12 bits count IDs within one millisecond on that machine.

Because the time sits in the most significant bits, comparing two IDs as numbers compares their times first. The merge in 5.3 relied on exactly this. Two machines can't produce the same ID because their machine fields differ, and one machine can't repeat one because the time or the counter has moved on. Nobody has to talk to anybody to get an ID.

?Why only "roughly" sorted?

Because the machines' clocks don't agree exactly, and because the time is taken when the ID is created, not when the tweet becomes visible. A tweet created on a machine whose clock runs a few milliseconds fast can get a bigger ID than one created a moment later elsewhere. Twitter's announcement put it formally: the IDs are k-sorted, meaning every ID is at most a bounded distance from its sorted position, and Twitter aimed to keep that bound under a second. For a timeline, a few milliseconds of disorder between unrelated tweets is invisible.

6.3Reading a real ID

Since the time is inside the ID, you can read it back out of any tweet's ID with a shift and an add. This program decodes the ID of the selfie Ellen DeGeneres posted during the 2014 Academy Awards: shift right by 22 bits to drop the machine and sequence fields, add the epoch, and convert milliseconds to a date. & 0x1F keeps the low 5 bits after a shift, and & 0xFFF the low 12.

from datetime import datetime, timezone
 
EPOCH = 1288834974657   # Snowflake's epoch: 4 Nov 2010, 01:42:54.657 UTC
 
def decode(tweet_id):
    ms = (tweet_id >> 22) + EPOCH          # top 41 bits: milliseconds since the epoch
    datacenter = (tweet_id >> 17) & 0x1F   # 5 bits
    worker = (tweet_id >> 12) & 0x1F       # 5 bits
    sequence = tweet_id & 0xFFF            # low 12 bits
    when = datetime.fromtimestamp(ms / 1000, timezone.utc)
    return when, datacenter, worker, sequence
 
# The Oscars selfie, posted during the 2014 Academy Awards
when, dc, worker, seq = decode(440322224407314432)
print(f"created   {when:%Y-%m-%d %H:%M:%S.%f}"[:-3] + " UTC")
print(f"machine   datacenter {dc}, worker {worker}")
print(f"sequence  {seq}")
print(f"bits used {(440322224407314432).bit_length()} of 63")

created   2014-03-03 03:06:13.747 UTC
machine   datacenter 1, worker 0
sequence  0
bits used 59 of 63

Line one gives 03:06 UTC on 3 March 2014, which is just after 7 pm on 2 March in Los Angeles, during the ceremony. Line two says which generator made the ID, and the sequence of 0 means it was the first ID that worker issued in that millisecond. And the last line shows there's room left: 41 bits of milliseconds last about 69 years from the 2010 epoch, so the top bit stays clear (signed 64-bit integers stay positive) until around 2080.

Those 64 bits caused one well-known problem. JavaScript stores numbers as 64-bit floating point, which holds integers exactly only up to 2⁵³, and Snowflake IDs are bigger. A web page that parsed a tweet ID as a number would silently round it to a different tweet. Twitter's API answered by sending every ID twice, as a number in id and as a string in id_str, and telling developers to use the string.

7.1Hydration

A timeline holds IDs, so before Kabir sees anything, each ID has to be turned back into a tweet: the text, the author's name and picture, the counts of likes and retweets. That step is called hydration. The timeline service takes the 20 IDs it's about to return and asks the tweet service for all of them at once, in parallel, and the user service for the authors. In 2012 the tweet service, Tweetypie, kept about the last six weeks of tweets in a memcached cluster (a plain in-memory key-value cache), and the user service, Gizmoduck, kept every user in cache. Krikorian's talk pointed out that the text of a tweet is "almost irrelevant" to most of the infrastructure: fan-out, timelines and ranking all work on IDs, and the words are only fetched at the end.

Behind the caches sits the store that holds every tweet ever posted, and that has to be durable. In 2012 it was T-Bird, the Gizzard-sharded MySQL from 6.1. From 2014 it was Manhattan.

7.2Manhattan

Twitter announced Manhattan in April 2014 as "our real-time, multi-tenant distributed database". Its motivation was operational: Twitter had many separate storage clusters, each built and run for one feature, and engineers spent too much time firefighting them and waiting for new capacity. Manhattan was built as one storage service for many teams, where an engineer could ask for space and throughput and start writing in seconds.

Its design choices follow Twitter's needs. It's a key-value store: you write a value under a key and read it back. It's eventually consistent by default, meaning a write is accepted by some replicas and reaches the others shortly afterwards, so a read can briefly return an older value. The announcement says most of Twitter's use cases "strongly favor availability over consistency", and that many of them couldn't accept even a few seconds of unavailability while a new primary server was elected. To keep replicas converging, Manhattan runs three repair mechanisms: a background process that continuously compares and reconciles replicas, read repair (fixing a stale replica when a read notices the difference), and hinted handoff (holding writes for a replica that's down and delivering them when it returns). For the cases that need it, strong consistency is available as an opt-in service built on a consensus algorithm (chapter 27), with compare-and-set operations within one data centre or across several.

Underneath, Manhattan offered three storage engines for different access patterns: seadb, a read-only format for data computed in batch jobs on Hadoop; a B-tree engine for read-heavy data; and an engine based on a log-structured merge tree (LSM tree) for write-heavy data. An LSM tree never updates data in place. It collects writes in memory, writes them out as sorted files, and merges files in the background, which turns many small random writes into a few large sequential ones. Chapter 18 covers how it works and what it costs.

By January 2017 Twitter's infrastructure blog described Manhattan as "the backend for Tweets, Direct Messages, Twitter accounts, and more", with read-only clusters handling tens of millions of queries a second and read-write clusters millions. Cassandra, which Twitter had adopted around 2010, had been deprecated in Manhattan's favour.

7.3How one tweet is laid out

Twitter's 2023 open-source release of Tweetypie includes the code that writes tweets to Manhattan, and its comments document the layout. Each tweet is stored as many small key-value pairs under one primary key, the tweet ID, with a second-level key naming each piece:

/[TweetId]/fields/internal/1          core fields, packed together
/[TweetId]/fields/internal/9 … /99    other built-in fields, one key each
/[TweetId]/fields/external/100 …      fields added later
/[TweetId]/metadata/delete_state      hard delete
/[TweetId]/metadata/soft_delete_state
/[TweetId]/metadata/scrubbed_fields/[FieldId]

A few details show the reasoning. The core fields that almost every read needs (the code says field IDs 2 to 8 of the tweet structure) are packed into one value under fields/internal/1, so a typical read is one lookup. Rarer fields get their own keys, so adding a feature means adding a key, with no change to existing tweets. Deletion is recorded as a state under metadata, separate from the content, so a delete can be undone or audited. And the tweet ID itself isn't stored inside the value, because it's already the key.

One more line in that file connects back to section 6. Manhattan orders keys as strings, and the string "10" sorts before "9". So the code pads both the tweet ID and the field ID with leading zeros to a fixed width, which makes string order match numeric order, and therefore time order, since the IDs are Snowflakes.

8.1The opposite shape

Kabir sees Nova's announcement and searches for the album's name. Search has the opposite shape to the home timeline, and Krikorian's talk made the contrast explicit. For the home timeline, a tweet is written to every follower's list, and a read touches one list. For search, a tweet is written to exactly one search machine (plus its replicas), and a read asks every machine. Krikorian called search and pull "inverses" of each other.

Search works through an inverted index: for every word, a list of the tweets containing it. Chapter 62 explains how inverted indexes are built and queried, with postings lists, compression and ranking. Twitter's version had an extra demand that web search engines don't: a tweet had to be searchable within seconds of being posted, while the index served queries at the same time.

Twitter's real-time search engine is called Earlybird, introduced in October 2010 and described in a paper at the ICDE conference in 2012. It's built on Lucene, the open-source Java search library. As of autumn 2011, the paper reported over two billion queries a day, an average query latency of 50 milliseconds, and tweets typically searchable within 10 seconds of creation.

8.2An index that's written while it's read

Earlybird's paper is unusually concrete about its data structures, and three choices stand out.

First, the index is cut into segments of a fixed number of tweets: 2²³, about 8.4 million, with 12 segments per server in 2011. New tweets fill one segment at a time, so at any moment only one segment is being written; the others are read-only, and once a segment is full it's rewritten into a compact, compressed form in the background.

Second, each posting, one occurrence of a word in a tweet, is a single 32-bit integer: 24 bits for the tweet's number within its segment and 8 bits for the word's position in the tweet. Eight bits are enough for a position because a tweet was at most 140 characters. A postings list is then just an array of integers. Searches want newest first, and the active segment appends new postings at the end, so a search reads each array backwards.

Third, the active segment has one writer thread and many reader threads, and they don't lock each other. The writer adds all of a tweet's postings, then increments a counter, maxDoc, holding the highest fully indexed tweet. That counter is declared volatile in Java, which places a memory barrier at the increment: everything the writer did before it is guaranteed to be visible to any thread that reads the counter afterwards (chapter 03 covers memory barriers). A reader starts each query by reading maxDoc and ignores any posting with a higher number, so it never sees half of a tweet.

What does this buy? The paper measured it on 2011 hardware, a server with two quad-core processors and 72 GB of memory: about 17,000 queries a second on one 16-million-tweet segment, and indexing at 7,000 tweets a second under heavy query load, with the 99th-percentile latency under 200 milliseconds. Twitter's search had come a long way: in 2008 it ran on MySQL, inherited from Summize, the search start-up Twitter bought, and handled about 20 tweets and 200 queries a second.

8.3Earlybird in 2023

When Twitter published its recommendation code in 2023, the Earlybird README described the system's later shape. The index is split into three clusters: a realtime cluster holding public tweets from about the last seven days, a protected cluster for protected accounts' tweets over the same window, and an archive cluster holding every tweet ever posted, up to about two days ago. Ingesters read tweets from Kafka, a distributed log (chapter 23), and a root service fans each query out to the partitions and merges the answers.

That README holds a surprise for this chapter. It describes Earlybird's main use case as two things: search, and "Timeline In-network Tweet retrieval": given the list of accounts a user follows, find their recent tweets. By 2023, the search index had become the main source of tweets from the people you follow, for both the Following and For You tabs. That's fan-out on read, done by a search engine. To see why Twitter went back to reading at read time, we need to look at what the timeline had become: a ranked feed.

9.1The funnel

Kabir's timeline in 2012 was every tweet from the accounts he follows, newest first. From 2016 Twitter ranked it, and by 2023 the default tab, For You, mixed tweets from accounts Kabir follows with tweets from accounts he doesn't. On 31 March 2023 Twitter published the code behind For You on GitHub as twitter/the-algorithm, with a blog post explaining it, and the published numbers are a good picture of what ranking costs.

Ranking has to narrow about 500 million tweets a day down to the few dozen Kabir will look at. Chapter 58, on Netflix's recommendations, explains the general shape of the answer: a funnel, where cheap methods cut millions of items down to a few thousand, and an expensive model scores only those few thousand. Twitter's funnel had three stages: fetch candidates, rank them, then filter and mix.

Candidate sources. For each request, the pipeline tries to pull the best 1,500 tweets out of hundreds of millions. On average half come from accounts Kabir follows, called in-network, and half from accounts he doesn't, called out-of-network. In-network candidates come from the search index, Earlybird, which ranks the followed accounts' recent tweets with a light model, a logistic regression (a simple weighted sum of features turned into a probability); the most important signal is Real Graph, a model that predicts how likely Kabir is to interact with each account he follows. Out-of-network candidates come from two kinds of source. One walks the graph of who engaged with what ("what did the people Kabir follows recently like?") on an in-memory engine called GraphJet; in 2023 that supplied about 15% of home timeline tweets. The other uses embeddings, vectors that place users and tweets so that similar ones are close: SimClusters groups accounts into 145,000 communities, recomputed every three weeks, and describes each user and tweet by the communities they belong to.

Ranking. All 1,500 candidates are then scored by the same model, the heavy ranker, a neural network of about 48 million parameters. Its input is about 6,000 features describing the tweet, Kabir and the relationship between them. Its output is ten probabilities, one for each way Kabir might engage: like, retweet, reply, open the author's profile, watch half a video, reply and get a reply back from the author, and so on, including negative ones such as "show less often" and reporting the tweet.

Filters and mixing. Finally, rules adjust the ranked list: remove tweets from accounts Kabir blocks or mutes, avoid too many consecutive tweets from one author, keep the in-network and out-of-network balance, require an out-of-network tweet to have a connection to someone Kabir follows, and thread replies with their originals. Then ads and account suggestions are mixed in, and the timeline is sent.

9.2Turning ten probabilities into one score

The heavy ranker's README in the companion repository, twitter/the-algorithm-ml, gives the formula that turns ten probabilities into one score: a weighted sum, each probability times a weight from a configuration file. It also lists the weights as of 5 April 2023:

Why would a report count 738 times as much as a like? The weights look extreme until you remember what they multiply. A like is far more likely than a reply that the author answers, so its probability is much bigger, and the README says the weights were originally set so that each term contributes about equally on average, then adjusted over time to move platform metrics. Read that way, the table seems to say the 2023 ranker prized conversations, especially ones the author joins, and punished tweets a user was likely to report.

All of this is expensive. Twitter's 2023 post says it ran about 5 billion times a day, finished in under 1.5 seconds on average, and needed 220 seconds of CPU time per run, nearly 150 times the time a user waits. That's possible because the 1,500 candidates are scored in parallel across many machines.

9.3Why ranking undoes fan-out

Here's the connection to the rest of the chapter. Fan-out on write precomputes one thing: the newest 800 tweets from the accounts you follow, in time order. A ranked feed doesn't want that list. It wants a pool of recent candidates from everyone you follow, which it then reorders by predicted engagement, plus candidates from accounts you don't follow, which fan-out could never provide. Once ranking runs on every request anyway, the precomputed list saves less, and a source that can answer "recent tweets from these 400 accounts" quickly does the job. A real-time search index can do exactly that.

Twitter's 2023 post says it in one sentence: "We recently stopped using Fanout Service, a 12-year old service that was previously used to provide In-Network Tweets from a cache of Tweets for each user." The code shows the rest. Home Mixer's README lists the Following tab, the strictly reverse-chronological one, as built from an Earlybird candidate pipeline too. Twitter didn't publish a cost comparison or a full explanation for the switch, so the reasoning above is probably the closest we can get: it's what the published architecture implies.

10.1What's documented

Elon Musk bought Twitter in October 2022, and in 2023 the company renamed itself X. Engineering publication mostly stopped, so much of what happened inside since then is unpublished: traffic figures, hardware, how timelines are stored. Three things are documented in public code and posts.

One is the 2023 release covered in section 9, which documented the retirement of the Fanout Service and the move of in-network retrieval to the search index.

Another came in January 2026, when X published a new repository, xai-org/x-algorithm, under the Apache 2.0 licence, with the code that builds the For You feed. It replaces much of the 2023 system. In-network posts now come from a service called Thunder, which, in the repository's words, "holds recent posts in memory as they are published, and returns those from the accounts a viewer follows". Its code shows the data structure: a concurrent hash map from each author's ID to a double-ended queue of that author's recent posts (post ID and creation time), fed from Kafka, with a cap per author and old posts trimmed after a retention window. A request passes in the viewer's following list, up to 10,000 accounts, and Thunder returns up to 1,200 recent posts from them, which are then filtered to the last 48 hours.

Look at the shape of that. It's a user timeline per author, kept in memory, and a home timeline assembled at read time by reading the user timelines of everyone the viewer follows: fan-out on read, the version 1 design, now affordable because everything lives in memory and the result is going to be ranked anyway. The 2012 hybrid treated a few celebrities this way; Thunder treats every author this way.

A third is the ranking model. X's 2026 repository describes Phoenix, a transformer, the kind of neural network behind today's language models, written in JAX (a Python library for numerical computing) with a Rust serving layer, used both to retrieve out-of-network candidates and to rank all of them. (The January release shipped a sample transformer ported from xAI's open Grok-1 model; an August 2026 update replaced it with the production code.) Phoenix reads the viewer's recent engagement history and predicts a probability for each of about 25 actions, positive (like, reply, repost, dwell time) and negative (mute, block, report), and, as in 2023, a weighted sum turns them into one score. Two of its documented design decisions matter for a system designer. When the model scores a batch of posts, each post is compared with the viewer's history but never with the other posts in the batch, so a post's score doesn't depend on what else happens to be scored alongside it, which makes scores consistent and cacheable. And embeddings are looked up by hashing, with no fixed vocabulary, so a post is representable the moment it's created.

10.2Reading the pendulum

Put the three eras side by side and Twitter's home timeline has swung from one end of the fan-out spectrum to the other and back:

Each of these choices was probably right for the product and hardware of its time. Fan-out on write wins when the answer to a read can be computed in advance and reads dominate. Once the answer depends on a model run at read time, there's little left to precompute, and the cost of a hundred-million-insert fan-out buys nothing.

11.1Every box, and why it's there

Here's the design as it stood at its most documented point, around 2012 to 2017, with the hybrid fan-out, followed through one refresh.

11.2From top to bottom

12.1Failures this design has to survive

12.2The tradeoffs, in one table