Designing LeetCode

1.1What it has to do

An online judge, the system behind LeetCode, Codeforces or the IOI, comes down to a short list:

1. Accept a submission: source code, a language and a problem.
2. Compile it, if the language needs compiling, and report a compile error if it fails.
3. Run it against hidden tests: inputs the contestant never sees, each with a time and a memory limit.
4. Decide a verdict: did every test produce a correct answer, in time, within memory?
5. Show the verdict in the browser within seconds.
6. Rank contestants on a live leaderboard, with penalties for wrong attempts.
7. Catch cheating, above all people submitting copies of each other's code.

Every competitive programmer knows the verdicts by their short names:

And the qualities it needs while doing all that:

• Safe: nothing a submission does can harm the judge, read the answers, reach the network, or affect other submissions.
• Fair: the same program on the same test gets the same verdict, whenever it's submitted and whatever else is running.
• Fast: a verdict in a few seconds, even in the last minutes of a contest.
• Correct: the checker that compares outputs must accept every right answer and reject every wrong one.

This list is unusual among system designs. Most systems protect themselves from bad requests; a judge's whole job is to execute bad programs, on purpose, thousands of times a minute. Most of this case study is about doing that safely and fairly.

1.2How big is it?

LeetCode doesn't publish how many people take a weekly contest, how many submissions it judges, or what machines it judges them on. So we'll reason with round numbers and say plainly that they're assumptions. Suppose 30,000 people take the contest, which has four problems and lasts ninety minutes; the real number is probably of that order, but LeetCode doesn't say.

Two things stand out. First, the compute is small by the standards of large web services: a few hundred cores at the peak. A judge is not hard because of its size. Second, each unit of work is unusual: it's somebody else's program, and we have to run it with less trust than we'd give a web request, but measure it with more precision than we'd give almost anything else.

2.1The obvious design

Our first design takes Kavya's code in the web request, writes it to a file, and runs the compiler and then the program right there, on the web server:

# Version 1: inside the HTTP handler. Please don't.
open("/tmp/sol.cpp", "w").write(code)
subprocess.run(["g++", "-O2", "/tmp/sol.cpp", "-o", "/tmp/sol"])
for test in problem.tests:
    out = subprocess.run(["/tmp/sol"], input=test.input, capture_output=True).stdout
    if out != test.expected:
        return "Wrong Answer"
return "Accepted"

This breaks in two quite different ways, and the rest of the design is about fixing them.

One is about time. Compiling C++ takes roughly a second, and running the tests another second or more, so each submission ties up a web server thread for several seconds. An infinite loop ties it up forever, since nothing stops it. During the last ten minutes of the contest, with eighty-odd submissions arriving a second, the web servers would spend all their time compiling and running strangers' code, and nobody could even load the problem page.

The other is far worse. The program runs as the web server's user, on the web server's machine, with the web server's permissions. It can open /srv/problems/4/expected_output.txt and print it. It can open a socket and send the test inputs to a friend. It can call fork() in a loop until the machine can't start another process, which takes down the web server along with it. It can delete files. It can read the database password from the environment. None of that needs cleverness; each one is a few lines of C++.

So we need two things: a way to run submissions somewhere other than the web server, at their own pace, and a box to run each one in that it can't get out of. Running submissions elsewhere is a familiar pattern. Building the box is the heart of this case study, and it starts in section 3.

2.2A queue and judge workers

Our first fix is the pattern that chapter 56 builds in detail: the web server doesn't do the work, it records that the work needs doing. When Kavya submits, the API stores her code with status "In Queue", puts the submission's ID on a queue, and replies straight away with that ID. A fleet of separate machines, the judge workers, take submissions from the queue one at a time, judge them, and write back the verdict.

This solves the time problem. Web servers stay fast whatever the judges are doing, because submitting is now just a database write and a queue push. A slow or looping program ties up one worker, and only until its time limit kills it. When submissions arrive faster than workers can judge them, the queue grows, and verdicts get slower, but nothing falls over. And because workers are separate machines, we can add more of them before a contest starts; section 6 comes back to that.

Kavya's browser needs to learn the verdict somehow. It can poll ("is submission 81723 done yet?") every second, which is simple and costs a request a second per waiting contestant, or keep a WebSocket open and be told. Codeforces's status page shows verdicts changing live in the same way. Either works at this scale; what matters is that the browser never waits on an open HTTP request for the whole judging time.

These queues and workers are exactly chapter 56's design, with leases so that a worker that dies mid-judgement doesn't lose the submission, and retries for the same reason. Judging is naturally safe to retry: running the same code on the same tests again gives the same verdict (if timing is fair, which is section 4's problem), so at-least-once delivery from the queue is fine as long as the verdict write is idempotent, meaning that writing it twice leaves the same result as writing it once.

2.3Inside one judge worker

Now let's follow Kavya's submission inside the worker, because everything that follows happens here.

Three details in that picture are deliberate.

Compiling happens in a sandbox. A compiler is a big program processing input written by a stranger, and that input can attack it. A classic trick is #include "/etc/passwd" or #include "/srv/problems/4/answer.txt": the compiler obligingly reads the file and prints its first lines in the compile error. Templates can also be written to make a compiler use gigabytes of memory or run for minutes. So compiling gets the same box, with limits on time, memory and output size, as running does. Mareš and Blackham point this out about their sandbox in 2012: it "can be easily used to isolate execution of compilers and graders", and Judge0 compiles inside the same sandbox it runs submissions in.

Sandboxes see inputs, never answers. Expected outputs live where only the checker, a trusted program outside the box, can read them.

Judging stops at the first failing test. If test 7 of 60 gives a wrong answer, the verdict is WA whatever the other 53 would do, so there's no point running them. LeetCode's result page shows exactly this when you practise: "37 / 60 testcases passed" and the failing input. During contests its rules say some test cases stay hidden, and a wrong submission doesn't reveal them, so nobody can collect the hidden tests by failing on purpose.

LeetCode adds one twist that most judges don't have. You don't write a whole program; you write a method in a class, such as Solution::maxSum(vector<int>&, int). So the judge must wrap that method in a driver: hidden code that reads each test's input, builds the vector, calls the method and prints its return value. LeetCode's drivers aren't published, but they're probably much like this, and the consequence is clear: the driver is compiled into the same binary as Kavya's code and runs in the same process, so it's no more trustworthy than her code is. Her method can overwrite the driver's memory or call exit(0) after printing an answer of its choosing. Only what runs outside the sandbox can be trusted, and that's the checker, so the comparison happens there.

So now the shape is settled: untrusted code only ever runs inside a sandbox, on a worker, fed from a queue. Next we need to work out what that sandbox has to be.

3.1What a submission can try

Every way a program can affect the world outside itself goes through the kernel, by making a system call (or syscall): a request to the operating system to open a file, allocate memory, start a process, send a packet. Chapter 7 shows what one looks like up close. That gives us a clean way to list the dangers, and it's exactly how Mareš and Blackham's 2012 paper on contest sandboxes, "A New Contest Sandbox", lists them: go through the groups of syscalls and ask what each could be used for.

Starting processes in a loop has a famous form, the fork bomb: a program whose only job is to copy itself, so that each copy copies itself, and the number of processes doubles every step. In bash it fits in thirteen characters, :(){ :|:& };:. A machine with no limit on processes per user locks up within seconds.

From the table we can read off what the sandbox has to do. Some rows need things hidden (the answers, the network, other processes). Some need things limited (memory, CPU time, number of processes, disk). And the last row says that hiding and limiting are only as strong as the kernel doing them.

3.2The old way: watch every system call

Early contest sandboxes took the most direct approach. They used ptrace, the system call debuggers use, to ask the kernel to stop the submitted program every time it was about to make a syscall. A supervising process looked at the call and its arguments, and either let it continue or killed the program. Time and memory were limited with rlimits, per-process resource limits the kernel enforces, set with setrlimit. Mareš and Blackham describe this "tracing sandbox" as by far the most common kind in 2012.

It had three problems, and their paper measures the first. Every syscall now takes two trips to the supervisor, so a program making many of them slows down. They measured the overhead at 5.6 µs per syscall: a program making a million syscalls took 9.26 seconds under ptrace against 3.56 seconds without, a 160% slowdown. Most batch tasks make at most around 10,000 syscalls, for which that's negligible, but interactive tasks, where the program talks to a judge process line by line, could easily make a million. The second problem was threads. A ptrace sandbox follows one process; it can't reliably follow several. That rules out Java, C# and Erlang, whose runtimes start service threads before your code runs. And the third was that the supervisor needs a table of allowed syscalls for each CPU architecture, and on 64-bit x86 a program can even use 32-bit syscall numbers, so the table has to cover both.

3.3isolate: namespaces and cgroups

Mareš and Blackham's answer, a sandbox called isolate, stopped watching syscalls and used two kernel features instead. Both are the subject of chapter 11, so here we only need what each one does for a judge.

Namespaces change what a process can see. Isolate starts the submission in a new, empty set of them:

• a PID namespace, where it is process 1 and can see and signal no other process. A useful extra property, in the paper's words: when the first process of a namespace exits, all the others in it are killed, and "without this feature, reliably terminating a hierarchy of possibly malicious processes is almost impossible."
• a network namespace with no network devices, apart from a loopback interface private to the box. The program can create a socket, but there's nowhere for it to go.
• a mount namespace whose root directory is a small RAM disk containing only what the run needs: read-only copies of /bin, /lib and /usr so the program can load its libraries, and one writable working directory, limited by a disk quota.
• an IPC namespace, so shared memory and message queues created inside can't be found from outside.

Control groups, cgroups, change what a group of processes can use. Isolate puts the submission's processes in one cgroup and sets limits on it: total memory, counted across every process and thread and including memory the kernel allocates on their behalf; total CPU time, summed across all processes, so splitting the work over cores doesn't escape the limit; and, through its -p option, the number of processes, the limit that defuses a fork bomb. (Isolate enforces that one with a per-box process rlimit, since each box runs as its own user; container tools use the cgroup's pids.max setting for the same job.)

Because all of this is enforced inside the kernel by its normal mechanisms, it adds nothing to each syscall. Their measurement: the same million-syscall program took 3.56 seconds in isolate, exactly as long as with no sandbox, with standard deviations under 0.5% of the mean. And it works for any number of threads and on any CPU architecture Linux supports.

Isolate is still the IOI's sandbox; its code lives in the IOI's GitHub organisation. It's what CMS, the contest system used at recent IOIs, runs submissions in, and it's what Judge0, an open-source code execution service in use since 2017 and described in a 2020 MIPRO paper by Došilović and Mekterović, wraps for every compile and run. A Judge0 run is one isolate --run command, and its flags read like the threat table: -t and -w for CPU and wall-clock time, -p for the process limit, --cg-mem for memory, -f for the largest file the program may write, -d /etc:noexec for a read-only directory nothing can be executed from, and no --share-net, so no network.

Isolate's manual also lists the places where namespaces have gaps. A few kernel objects aren't namespaced at all, so two sandboxes running side by side can use them to pass data: kernel keyrings, AF_VSOCK sockets, file locks on files both can see (a shared read-only /usr is enough), and io_uring, an asynchronous I/O interface that can create sockets of any kind. Isolate blocks the system calls for each of these. That's a small sign of the bigger problem in section 3.5: the kernel is huge, and "everything is namespaced" is never quite true.

3.4seccomp-bpf: a filter on every system call

Our second tool is the one ptrace sandboxes wanted to be. seccomp-bpf lets a process install a small filter program in the kernel, written in the classic BPF instruction set (a tiny virtual machine inside the kernel, the ancestor of chapter 48's eBPF), that runs on every system call the process makes from then on. The filter sees the syscall number and its arguments, and returns a decision: allow it, make it fail with an error, or kill the process. Chapter 11 installs a one-rule filter by hand. The difference from ptrace is that the check runs inside the kernel, in a few instructions, with no trip to another process, and that the filter is inherited by every child and thread and can never be removed.

A filter like this shrinks how much of the kernel a submission can reach. A judge can allow only what a compiled C++ program needs to read stdin, write stdout, allocate memory and exit, perhaps a few dozen syscalls, and refuse the other few hundred. That matters because a bug in a syscall nobody calls is still a bug an attacker can call. Firecracker's paper makes the same argument, citing work showing that the Linux kernel has more bugs in less-used syscalls.

nsjail, an open-source sandboxing tool published on Google's GitHub (though not an official Google product), puts all three ideas together: namespaces, resource limits and cgroups, and seccomp-bpf filters written in a small policy language called Kafel. Its documented uses are hosting capture-the-flag security challenges, where the people connecting are trying to break out, re-running crashing programs, and isolating untrusted GUI applications. For a judge, nsjail and isolate do much the same job from slightly different starting points.

Seccomp has a cost, and it's compatibility. The narrower the allowed list, the more legitimate programs break. Firecracker's paper notes that a trivial Linux program needs about 15 distinct syscalls, while a typical Ubuntu installation, according to a study it cites, uses 224 syscalls and 52 distinct ioctl calls. A judge that supports twenty languages, each runtime with its own habits, ends up with a long allowlist, and every entry is more kernel exposed.

3.5One shared kernel

Everything in sections 3.3 and 3.4 shares one weakness: the submission's system calls are still handled by the same Linux kernel as the judge's. Namespaces, cgroups and seccomp are all features of that kernel. If the submission can find a bug in it, it can get past all of them at once.

That isn't hypothetical. Dirty COW, a race condition in how Linux handled copy-on-write memory, disclosed in 2016 as CVE-2016-5195, let an unprivileged process write to files it could only read, including system binaries, and so become root. It had been in the kernel for about nine years. A program that triggered it from inside a namespace sandbox was running exploit code against the very kernel that drew the sandbox's walls. Mareš and Blackham were candid about this in 2012: isolate "is not completely secure with respect to new kernel versions", since new kernels add new kinds of kernel object that may not belong to any namespace.

How much this matters depends on what's at stake. For a contest judge, the prize for escaping is the test data of a weekly contest, and perhaps a foothold in the judge fleet. For a company that runs untrusted code from strangers all day, as LeetCode does even outside contests, a foothold in the fleet is something to defend against seriously. There are two well-known ways to stop sharing a kernel with the submission.

gVisor puts a second kernel in between, written in Go and running as an ordinary process. Google calls it an application kernel. Its main component, the Sentry, implements Linux's system calls itself: when the sandboxed program makes a syscall, gVisor's platform redirects it to the Sentry instead of the host kernel. The default platform since mid-2023, systrap, does this with seccomp's SECCOMP_RET_TRAP action; another uses the CPU's virtualisation support through KVM. The Sentry in turn may only make a small, explicitly listed set of host syscalls: per gVisor's security docs, things like duplicating and closing file descriptors, synchronisation, timers and signals. It "is not permitted to open new files, create new sockets" on the host. File access goes through a separate process, the Gofer.

A Dirty COW exploit inside gVisor attacks the Sentry's implementation of memory, which is different code from Linux's; to reach the host kernel it would first need a bug in the Sentry, then a bug reachable through the Sentry's short list of host syscalls. Its price, in gVisor's own words, is "higher per-system call overhead" and possible "poor performance for system call heavy workloads", plus reduced compatibility, because gVisor doesn't implement every syscall, /proc file or /sys file.

Firecracker goes the rest of the way, to a real virtual machine. Each sandbox gets its own guest Linux kernel, and the CPU's hardware virtualisation keeps that kernel apart from the host (chapter 47 covers how). Firecracker is a virtual machine monitor, the program on the host that creates the VM and emulates its few devices, written by AWS in Rust and described in an NSDI 2020 paper by Agache and colleagues. Its trick is to be small enough that VMs become cheap: with a minimal guest kernel, the paper reports under 5 MB of memory overhead per VM, booting to application code in under 125 ms, and up to 150 new microVMs per second per host. For comparison, the paper notes that QEMU, the usual VMM, is over 1.4 million lines of code.

Firecracker doesn't trust itself either. Its jailer starts each Firecracker process in a chroot, in its own PID and network namespaces, with privileges dropped and a seccomp-bpf profile that, in the 2020 paper, allows 24 syscalls, each with argument filtering, and 30 ioctls, 22 of which KVM's API needs. So a guest that broke out of the VM would land in a process that can do almost nothing. That's the same layering a judge wants: one wall the attacker must break, then another.

4.1CPU time and wall-clock time

Kavya's program is now safely boxed. The judge runs it on test 1 and must decide whether it ran within the problem's one-second limit. Which clock should it read?

There are two candidates. Wall-clock time is what a stopwatch would show, from start to exit. CPU time is how long a processor spent running the program's instructions, which leaves out any time the program spent waiting: for the disk, for a lock, or for its turn on a CPU while something else ran. On an idle machine, for a program that only computes, the two are nearly equal. On a busy one, wall time grows while CPU time ideally doesn't, because the program is only waiting more.

So judges limit CPU time: it's the one that measures the program and not its neighbours. But they keep a wall-clock limit too, set much higher, because a program can use no CPU at all: one that calls sleep(1000000) or waits forever to read input that never comes would sit in its slot indefinitely under a CPU-only limit. Isolate's 2012 paper lists "sleeping" syscalls among the dangers for exactly this reason, and isolate's manual recommends using --time as the main limit, with --wall-time set "to a much higher value as a precaution against sleeping programs". It also has --extra-time: when the CPU limit is exceeded, keep the program running a little longer before killing it, so the judge can report how long it would have taken, "1.03 s", and not just "over".

Isolate gets CPU time from the cgroup's CPU accounting, which the 2012 paper notes works from the scheduler's own nanosecond timestamps of context switches, more precise than the older sampled per-process timers. That precision raises a better question: if the measurement is exact, does the same program always take the same time?

4.2Why the same program takes different times

It doesn't. In a 2011 study, "Fairness of Time Constraints", Mareš took real contestants' submissions from IOI 2009 and ran them over and over on four different machines. Even with only one program running, the times spread out like a bell curve, and the spread grew in proportion to the running time: a little error added throughout the whole run. He listed where it comes from, and every item is a way the hardware or kernel makes the common case faster at the cost of predictability:

• Context switches. Even on an idle machine, the kernel briefly runs its own background work on the same core, and short interruptions are charged to whoever was running.
• Caches. A CPU keeps recently used data in small fast memories, the caches. How well a program's data fits depends on exactly where it lands in memory, and anything else that runs on the core evicts some of it.
• Several cores. When the scheduler moves a program to another core, it leaves its warm cache behind.
• Power management. An idle core runs at a lower frequency, or sleeps, and takes time to speed up again.
• System management mode, where the machine's firmware quietly takes over the CPU, and the time is billed to whatever program was running.

On a judge fleet there's a bigger source than any of these: other submissions. Two programs on two cores of one chip share roughly everything outside the cores: the last level of cache and the path to memory. Two programs on the two hardware threads of one core, with simultaneous multithreading (Intel calls it Hyper-Threading), share far more: the core's execution units themselves, so each one's instructions wait for slots the other is using. Both programs' CPU time goes up, even though neither is waiting in the usual sense.

Frequency is the last piece. Modern chips raise their clock speed above the rated base when only a few cores are busy and there's thermal headroom; Intel calls this Turbo Boost. A program judged on a quiet machine runs at the boosted frequency and finishes sooner; the same program judged during the contest rush, with every core busy, runs at a lower one. Nothing about the program changed.

This program shows the effect. It times a fixed piece of work eight times on a quiet machine, then again while one busy-looping neighbour process runs for every CPU on the machine, and reports both clocks each time. time.perf_counter() reads the wall clock and time.process_time() reads this process's CPU time. Then it sets a limit 10% above the quiet run's CPU time, as a strict judge might, and counts how many runs exceed it.

import multiprocessing as mp, os, statistics, time
 
def solution():                      # stands in for a contestant's program
    total = 0
    for i in range(3_000_000):
        total += i * i % 7
    return total
 
def judge(runs=8):
    walls, cpus = [], []
    for _ in range(runs):
        w0, c0 = time.perf_counter(), time.process_time()
        solution()
        walls.append(time.perf_counter() - w0)
        cpus.append(time.process_time() - c0)
    return walls, cpus
 
def spin():                          # a noisy neighbour: burns CPU forever
    while True:
        pass
 
def show(label, walls, cpus):
    ms = lambda xs: f"{min(xs)*1000:4.0f} / {statistics.median(xs)*1000:4.0f} / {max(xs)*1000:4.0f} ms"
    print(f"{label:22} wall {ms(walls)}   cpu {ms(cpus)}")
 
if __name__ == "__main__":
    alone = judge()
    show("alone", *alone)
 
    hogs = [mp.Process(target=spin, daemon=True) for _ in range(os.cpu_count())]
    for h in hogs: h.start()
    busy = judge()
    for h in hogs: h.terminate()
    show(f"{len(hogs)} busy neighbours", *busy)
 
    limit = statistics.median(alone[1]) * 1.10   # a limit 10% above the quiet run
    print(f"limit {limit*1000:.0f} ms of CPU time")
    for label, (walls, cpus) in [("alone", alone), ("busy", busy)]:
        tle = sum(c > limit for c in cpus)
        print(f"{label:5}: {tle} of {len(cpus)} runs over the limit")

alone                  wall   88 /   89 /   90 ms   cpu   88 /   89 /   90 ms
10 busy neighbours     wall  151 /  180 /  204 ms   cpu  118 /  148 /  168 ms
limit 98 ms of CPU time
alone: 0 of 8 runs over the limit
busy : 8 of 8 runs over the limit

Each row shows the minimum, median and maximum of eight runs. Alone, the two clocks agree and barely move: 88 to 90 ms. With ten busy neighbours, wall time roughly doubles, which is expected, since the program now waits for its turn on a CPU. CPU time is the surprise. CPU time was supposed to leave the waiting out, and it does, but it still rose by about 60%, and it now varies by 50 ms from run to run. That seems to be the shared caches, shared memory bandwidth, and on this kind of laptop chip, the scheduler placing the program on a slower efficiency core (many laptop chips mix fast cores with power-saving ones) or on a core running at a lower frequency. Every one of the eight busy runs would have been Time Limit Exceeded against a limit the quiet runs passed easily. (The exact numbers change from run to run and machine to machine; the gap doesn't.)

That's the unfairness in its plainest form: the verdict depends on when you submitted. Kavya, at minute 88, is submitting at the busiest moment of the contest.

4.3Making timing fair

Each fix follows from a cause, and isolate's manual lists them under "Reproducibility" (as of 2026):

• Fix the frequency. Set the CPU frequency governor to performance so cores don't slow down when idle, and turn off boost (intel_pstate/no_turbo on Intel, cpufreq/boost elsewhere), so a run's speed doesn't depend on how many other cores are busy.
• Use identical cores. On chips with a mix of fast and efficient cores, pin the sandbox to one kind only.
• Pin each run to a CPU, so the scheduler doesn't move it between cores mid-run.
• Turn off address space randomisation, which can change timing, memory use and even the behaviour of programs with memory bugs. Mareš and Blackham's paper adds this one for a fairness reason: a buggy program shouldn't pass on one run and crash on the next.
• Disable transparent huge pages and swap, both of which make memory access times vary with the system's state.

And beyond the manual, the fleet design follows. Run one submission per physical core and leave its sibling hardware thread idle, or turn simultaneous multithreading off. Mareš's 2011 measurements show why: four submissions judged in parallel on a four-core machine had a mean time almost 20% higher than one at a time, with much larger variance. Keep the whole fleet on one hardware model for the whole contest, so a time limit means the same thing on every worker. Codeforces is a worked example: in October 2018 Mike Mirzayanov announced that all its judging had moved to new servers with Intel i3-8100 processors, more of them "which means fewer queues during rounds", and noted that single-core performance was close enough to the old servers that "the time limits in all problems remain the same." The i3-8100 is a four-core chip without Turbo Boost or Hyper-Threading, so two of the noise sources above don't exist on it at all.

One recommendation changed over time, and it's instructive to see why. In 2011, Mareš found that pinning processes to cores and giving them real-time priority made variance worse on the kernels he tested, and concluded the default scheduler did its job well. Isolate's manual today recommends pinning to a single CPU. Both are probably right: the 2011 tests ran one submission at a time on otherwise idle machines, where the scheduler had nothing to fight over, while a busy judge runs many at once.

Even with all of this, timing remains a measurement with an error. So the last fix is in how the judge uses the measurement.

This is where the verdict for Kavya's fourth problem is decided. Suppose her heap-based solution runs test 41 in 0.97 s on a busy worker. Under the rules above, the run was pinned to a quiet core at a fixed frequency, so 0.97 s is close to its true time, and it passes. If a noisy measurement had said 1.02 s, the judge would have rerun it and taken the fastest time. What we want is a verdict that doesn't depend on whether she submitted at minute 10 or minute 88.

4.4Memory, and the other limits

Memory has its own measurement question. Isolate offers two kinds of limit. Without cgroups, --mem limits the program's address space, the total virtual memory it has reserved, so malloc fails when it's reached. With cgroups, --cg-mem limits the memory in use by every process in the box, including what the kernel allocated on their behalf; exceeding it usually gets the program killed by the kernel's out-of-memory killer, which isolate reports as cg-oom-killed. The address-space limit is a poor fit for languages like Java and Go, whose runtimes reserve large ranges of virtual memory they never touch, so for them only the cgroup limit is meaningful.

Each of the remaining limits is small but closes a hole from the threat table: a maximum file size (-f), so a program can't fill the disk with output; a disk quota on the working directory; a stack limit; and no core dumps, so a crashing program doesn't write a copy of its memory to disk. The sandbox reports what happened in a short metadata file: time, time-wall, max-rss or cg-mem, an exitcode or the exitsig that killed it, and a two-letter status, TO (timed out), SG (died on a signal), RE (non-zero exit) or XX (the sandbox itself failed). From those, the worker derives the verdicts from section 1. Judge0's determine_status is a small function that does exactly this mapping.

So now the judge knows, fairly, whether Kavya's program finished in time and within memory. Whether its answer was right is a separate question, and a surprisingly fiddly one.

5.1Comparing bytes isn't enough

Version 1 compared the program's output with the expected output byte for byte. On test 1 of Kavya's problem the expected output is 42\n, and her program prints 42 \n, with a trailing space left over from a loop that prints a space after every number. Byte for byte, that's Wrong Answer. Almost no contestant would call it wrong.

So the usual comparison works on tokens, the runs of non-space characters, and ignores how much whitespace sits between them, including trailing spaces, blank lines and Windows-style \r\n line endings. Codeforces's checker library, testlib, written by Mike Mirzayanov and open-sourced under the MIT licence, has this as its standard checker, wcmp, "compare sequences of tokens". It reads one token from the expected answer and one from the contestant's output, compares them, and repeats; it reports Wrong Answer with the position of the first difference, or if either side has tokens left over at the end.

LeetCode sidesteps part of this, since your method returns a value and the driver prints it, so whitespace is the driver's problem, not yours. But the next two problems apply to every judge.

5.2Floating point

Suppose the answer is a distance, and the expected output is 0.3. Kavya computes it as 0.1 + 0.2 and prints 0.30000000000000004. Her answer is right, and the bytes are wrong.

That happens because a computer stores a real number in a floating-point format: a fixed number of bits for the digits and a few more for where the decimal point goes. The standard 64-bit format, IEEE 754 double precision, has 52 bits for the digits, about 16 significant decimal digits. Most decimal fractions, 0.1 among them, have no exact binary representation, so each is rounded to the nearest one available, and the rounding errors add up as you calculate.

So problems with real-valued answers say how close is close enough. LeetCode's "Average of Levels in Binary Tree", for example, says "Answers within 10^-5 of the actual answer will be accepted." The checker then reads both values as numbers and accepts the contestant's if it's within a tolerance either absolutely (the difference is at most ε) or relatively (the difference is at most ε times the expected value). testlib's doubleCompare does both: it accepts if |result - expected| <= ε, or if the result lies between expected × (1 - ε) and expected × (1 + ε). It also handles the special values explicitly: an expected NaN matches only a NaN, an expected infinity only an infinity of the same sign, and a NaN or infinity where a number was expected is always wrong. Absolute error suits small answers, where relative error would be too strict near zero; relative error suits huge ones, where an absolute 10^-6 would demand more digits than a double holds.

5.3Many right answers: special checkers

LeetCode's first problem, Two Sum, says "You can return the answer in any order." So [0, 1] and [1, 0] are both correct, and no comparison against one stored answer can accept both. Plenty of problems are like this: print any shortest path, any valid arrangement, any subset with the given sum.

For these, the problem setter writes a special checker: a small program that reads the test's input, the contestant's output and, if useful, the reference answer, and decides whether the output is a correct answer. For "any shortest path", it checks that the path is a real path in the graph, that it starts and ends in the right places, and that its length equals the reference answer's length. testlib exists largely to make these checkers easy to write and hard to get wrong.

A checker is trusted code, since it runs outside the sandbox, but it reads untrusted input: the contestant's output can be anything. It might be 2 GB of digits, a number with a thousand digits where an int was expected, or binary garbage. A checker that crashes or misreads that input gives a wrong verdict, and one with a bug might accept wrong answers. That's why testlib's readers fail cleanly on malformed output, with a message like Expected integer, but "abc" found, and why the sandbox's file-size limit also protects the checker.

The hardest variant is the interactive problem, where the program and a judge process talk to each other: the judge asks a question, the program answers, the judge asks the next. Here the timing problem from section 4 comes back sharply. Every exchange is a pair of context switches between two processes, and Mareš found in 2011 that interactive runs were much slower than the same work done in batch, whatever the sandbox. His advice was to avoid interactive tasks with a separate judge process whenever the number of queries is large.

6.1The queue at minute 88

Now we can come back to the moment Kavya pressed Submit, and ask what the whole fleet was doing. In section 1's illustration, submissions in the last ten minutes arrive at about 83 a second, three times the contest's average, and the judge needs about 170 busy cores just to keep pace. Suppose it has 120 in use. Then the queue grows by roughly 25 submissions every second, and after ten minutes it's 15,000 deep. Kavya's submission, near the back, waits several minutes for a worker.

The queue grows that fast because of a simple fact about queues, which chapter 42 works through: once work arrives faster than it's served, the queue doesn't settle at a longer wait; it grows without limit for as long as the overload lasts. Near 100% utilisation, even a small burst makes waits explode. And this isn't hypothetical either. In July 2017, Codeforces's judging stopped for over 13 hours and more than 5,000 submissions piled up "In Queue", as a contestant recorded at the time. In May 2020 Mirzayanov postponed Codeforces Round 639 by three days, because a database problem had appeared that "can dramatically increase judging time (and leads to a huge queue)", and he didn't want to risk the round.

LeetCode's contest rules (last updated in September 2024) set out what happens when the site struggles: if it's slow or down, or Submit stops working, for 15 minutes or less, no action is taken; for longer, the contest is unrated. So the judge has a concrete target: a burst must never turn into a quarter-hour outage.

There's a subtler point in LeetCode's rules that makes a slow queue less harmful than it looks. Finish time is measured to the moment you submit the accepted solution, "the time it took you to first correctly submit", not to the moment the verdict comes back. So if Kavya's submission waits three minutes in the queue and is judged after the contest has ended, it still counts at 88:00. A slow queue costs contestants information (they don't know whether to keep trying) but not rank. Any judge should copy that decision: never let the judge's load leak into the scores.

6.2Scaling before the crowd arrives

Rising load is usually met with an autoscaler: measure queue depth or CPU use, and add workers when it climbs. For a judge, that's too slow to help at the moment it matters. A new worker has to boot, start the judging service, and fetch the test data for the contest's problems before it can judge anything, which probably takes minutes, and the end-of-contest burst lasts about ten. A reactive autoscaler would add most of its capacity just as the burst ends.

But a contest is the easiest load spike there is to predict: its start and end times are published days in advance. So the judge should scale on the calendar. Before the contest starts, bring the fleet up to the size the last few contests needed at their peak, plus headroom; warm each worker by loading every contest problem's tests and compiling each language's driver once; then shrink back after the queue drains, well after the contest ends. Reactive autoscaling stays on as a backstop for surprises.

Inside the contest queue there's one more fairness question. If one contestant submits forty times in a minute, should their submissions sit ahead of Kavya's single one? A plain first-in, first-out queue says yes. Taking submissions round-robin by contestant, one from each contestant with work waiting in turn, means a heavy submitter only delays themselves. Codeforces has a related habit that shows the trade-off: during a round, it judges submissions on a small set of "pretests" quickly, and runs the full test set on every final submission only after the round ends, in the "system testing" phase. That moves most of the judging work to after the burst, at the cost of contestants not knowing their final verdict until later.

7.1How LeetCode ranks

Kavya's Accepted verdict moves her up the leaderboard. To build that leaderboard we need the exact rule, and LeetCode's help centre spells it out (as of 2026). Each problem has a score, harder problems more. Contestants are ranked by total score, and ties are broken by finish time: the time from the start of the contest to your first accepted submission of the last problem you solved, plus penalty time, 5 minutes for every incorrect submission on a problem you eventually solved. Wrong answers on problems you never solve don't count, and nor do Compile Error, Internal Error or "Timeout" verdicts. Resubmitting a problem you've already solved changes nothing.

The ICPC, the long-running university contest, uses the same idea with different numbers: its 2013 World Finals rules gave 20 penalty minutes per rejected run, and added up the time of every solved problem, where LeetCode takes only the time of the last one. That difference changes strategy. Under ICPC rules, solving an easy problem late costs you; under LeetCode's, only your final solve time matters.

Penalties decide a lot under this rule. Two contestants who solve the same problems are ordered by finish time, and one wrong attempt costs five minutes, which in a big contest is hundreds of places.

7.2Ranking with one sorted number

The leaderboard needs two reads to be fast: "show me the top 25" on the contest page, and "what's my rank?" for each of 30,000 people. And it needs one write: when a verdict arrives, update that contestant. Sorting 30,000 rows on every page view would be wasteful when only a few rows change per second.

Our trick is to turn the two-part rule, score high first and then finish time low, into a single number that sorts the same way. If finish times are always under 10,000,000 seconds, then score × 10,000,000 − finish_seconds does it: any extra point outweighs any difference in time, and among equal scores, an earlier finish gives a larger number. Then the leaderboard is a structure that keeps contestants ordered by one number and can answer "how many are above this one?" quickly.

This program computes LeetCode's standings from a log of submissions, builds that key for each contestant, and then does a rank lookup by binary search over the sorted keys:

import bisect
 
POINTS = {"A": 3, "B": 4, "C": 5, "D": 6}
PENALTY = 5 * 60                      # 5 minutes per wrong try, in seconds
 
# (user, problem, seconds from the start, verdict)
log = [
    ("asha",  "A",  310, "AC"), ("asha",  "B", 1100, "WA"), ("asha",  "B", 1580, "AC"),
    ("ben",   "A",  280, "AC"), ("ben",   "B", 1300, "AC"), ("ben",   "C", 4100, "WA"),
    ("chen",  "A",  610, "AC"), ("chen",  "B", 1700, "AC"),
    ("dev",   "A",  200, "AC"), ("dev",   "C", 3000, "TLE"), ("dev",  "C", 4900, "AC"),
    ("kavya", "A",  400, "AC"), ("kavya", "B", 2100, "AC"), ("kavya", "D", 5280, "AC"),
]
 
def standings(log):
    best = {}
    for user, prob, t, verdict in sorted(log, key=lambda s: s[2]):
        u = best.setdefault(user, {"solved": {}, "wrong": {}})
        if prob in u["solved"]:
            continue                                  # resubmits after AC don't count
        if verdict == "AC":
            u["solved"][prob] = t
        elif verdict != "CE":
            u["wrong"][prob] = u["wrong"].get(prob, 0) + 1
    rows = []
    for user, u in best.items():
        score = sum(POINTS[p] for p in u["solved"])
        last = max(u["solved"].values(), default=0)
        pen = sum(u["wrong"].get(p, 0) for p in u["solved"]) * PENALTY
        rows.append((user, score, last + pen))
    return rows
 
def key(score, finish):               # one number that sorts like (score desc, finish asc)
    return score * 10**7 - finish
 
rows = standings(log)
rows.sort(key=lambda r: -key(r[1], r[2]))
for rank, (user, score, finish) in enumerate(rows, 1):
    print(f"{rank}. {user:5} score {score:2}  finish {finish//60:3}m{finish%60:02}s  key {key(score, finish)}")
 
# what a sorted set does for a rank query: binary search over keys kept in order
keys = sorted(key(s, f) for _, s, f in rows)
mine = key(7, 1500)                   # 7 points, finishing at 25m00s
print("7 points at 25m00s would rank", len(keys) - bisect.bisect_right(keys, mine) + 1)

1. kavya score 13  finish  88m00s  key 129994720
2. dev   score  8  finish  86m40s  key 79994800
3. ben   score  7  finish  21m40s  key 69998700
4. chen  score  7  finish  28m20s  key 69998300
5. asha  score  7  finish  31m20s  key 69998120
7 points at 25m00s would rank 4

Look at Asha and Chen. Both solved A and B for 7 points. Asha's last accepted submission came at 26m20s and Chen's at 28m20s, so Asha was faster, but her one wrong answer on B adds 5 minutes, putting her at 31m20s and below Chen. Ben's wrong answer on C costs him nothing, because he never solved C. Dev's TLE on C does count, because he solved C later. And Kavya's last-minute submission at 88m00s is what puts her first: 6 points for problem D outweighs any amount of time.

Its last line is the rank query. Because the keys are kept in sorted order, finding how many contestants are above a given key is a binary search, about log₂ 30,000 ≈ 15 comparisons, with no sorting at query time.

7.3A sorted set in Redis

A Python list kept sorted by bisect works until there are many updates, because inserting into the middle of a list moves everything after it. The structure built for exactly this job is the sorted set in Redis, which chapter 22 covers: members, each with a numeric score, kept in score order. ZADD contest:412 129994720 kavya sets Kavya's key, replacing any previous one. ZREVRANK contest:412 kavya returns her position counting from the top, and ZREVRANGE contest:412 0 24 returns the top 25. Each takes time proportional to log n.

Inside, a large sorted set is a hash table from member to score, plus a skip list: a linked list in score order with extra "express lane" pointers on a random subset of nodes, so a search can skip most of the list. Redis's skip list also stores, on each pointer, how many nodes it jumps over, and adding those up along the search path is how it computes a rank without walking the whole list.

Two details make it correct in practice. The first is that the sorted set isn't the source of truth. The submission log is. LeetCode's rules describe rejudging in detail: if weak tests let wrong solutions through, it adds test cases and rejudges every Accepted submission to that problem; if a wrong test rejected correct ones, it rejudges the Wrong Answers; after a rejudge, the first Accepted submission counts and earlier Wrong Answer or TLE submissions count towards penalty time; and if more than 10% of submissions change result, the contest is unrated. Every one of those outcomes is computed by replaying the log through the ranking rule, exactly as the TryIt does, and writing fresh keys. The sorted set is a fast view of a computation we can always redo. The second is ties. Two contestants with the same score and the same finish time to the second have equal keys, and Redis then orders them by member name (in reverse alphabetical order, for ZREVRANGE and ZREVRANK). If the rules say equal contestants share a rank, the page has to compute that from the scores and not just print positions.

8.1Why comparing every pair doesn't work

After the contest, there's one more job. LeetCode's rules list as a violation "multiple accounts submitting similar code for the same problem", alongside one user submitting from several accounts and posting solutions publicly before the end. Since September 2024 the same list also bans code-generation tools and any outside help. The penalties are serious: on a first violation, the user's contest score and LeetCoins are reset to zero and they're banned from contests and discussion for a month; a second means permanent deactivation without appeal. So somewhere, a program has to find submissions that are copies of each other, even when the copier has renamed every variable and added a few lines.

Suppose 20,000 people solved problem 2. An obvious approach compares every pair, with a diff tool. That's 20,000 × 19,999 ÷ 2, about 200 million pairs, for one problem in one contest. Worse, a diff is fooled by trivial edits: rename nums to a and every line differs.

We need two things: a representation of a program that survives renaming and reformatting, and a way to find similar programs without comparing all pairs. The best-known answer to both is the algorithm inside MOSS, "Measure Of Software Similarity", a plagiarism detection system Alex Aiken developed in 1994 and still runs as a service for programming courses. Schleimer, Wilkerson and Aiken described its core, winnowing, at SIGMOD 2003.

8.2Normalise, then hash every k-gram

Step one removes differences that don't matter. MOSS's architecture has a front end for each language that strips them out before any matching happens: in the paper's words, "whitespace and punctuation are removed, all letters are converted to lower case, or all variable names are replaced by the identifier 'V'." For C++ we drop comments and whitespace, and replace every identifier that isn't a keyword with V. Kavya's loop header for (int i = 0; i < nums.size(); i++) becomes for(intV=0;V<V.V();V++), and so does a copier's for (int idx = 0; idx < a.size(); idx++).

Step two cuts the normalised text into overlapping pieces. A k-gram is a run of k consecutive characters; a text of length n has n − k + 1 of them, starting at every position. Each k-gram is hashed to a number. Two programs that share a stretch of at least k characters share the hashes of the k-grams inside it, and k is the noise threshold: matches shorter than k aren't counted, because short matches like for(intV=0; happen by coincidence in every program.

Keeping every hash would work, but it's a lot: one per character. Picking a subset is the obvious fix, and the obvious subsets break. Keep every hash that's 0 mod 4, say, and a long run of shared code might happen to contain none of them, so a real copy is missed.

8.3Winnowing

Winnowing picks the subset with a guarantee. Slide a window of w consecutive hashes along the sequence. In each window, select the minimum hash, and if the minimum appears more than once, the rightmost one. The selected hashes are the document's fingerprints.

Two properties follow. First, every window contributes at least one fingerprint, so any shared stretch long enough to contain a whole window of shared hashes, w + k − 1 characters, is guaranteed to share a fingerprint. That length is the guarantee threshold, t. Choose k and t, and w = t − k + 1 follows. Second, neighbouring windows overlap in all but one hash, so the minimum of one window is usually still the minimum of the next, and many windows select the same fingerprint. The paper proves that on random data, winnowing keeps a fraction of about 2/(w + 1) of the hashes, and that no algorithm of this kind, one that decides from each window's contents alone, can keep fewer than 1.5/(w + 1). So winnowing is within 33% of the best possible.

Similarity between two programs is then a question about their fingerprint sets. A natural measure is the Jaccard index: the number of fingerprints they share divided by the number in either one.

This program does all three steps on Kavya's sliding-window solution to problem 2, a copy with every name changed and a comment added, a copy that also has a pointless variable woven in, and an independent solution that uses prefix sums. It uses k = 12 and w = 8, so any shared stretch of 19 normalised characters is guaranteed to be caught. zlib.crc32 is a quick standard hash.

import re, zlib
 
KEYWORDS = {"int", "long", "for", "while", "if", "else", "return", "vector",
            "auto", "const", "using", "namespace", "std", "include", "cin", "cout"}
 
def normalise(src):
    src = re.sub(r"//[^\n]*|/\*.*?\*/", "", src, flags=re.S)        # drop comments
    tokens = re.findall(r"[A-Za-z_]\w*|\d+|\S", src)            # split into tokens
    # every identifier that isn't a keyword becomes "V", so renaming doesn't help
    return "".join(t if t in KEYWORDS or not t[0].isalpha() and t[0] != "_" else "V"
                   for t in tokens)
 
def fingerprints(src, k=12, w=8):
    text = normalise(src)
    hashes = [zlib.crc32(text[i:i + k].encode()) for i in range(len(text) - k + 1)]
    picked = set()
    for i in range(len(hashes) - w + 1):                        # every window of w hashes
        window = hashes[i:i + w]
        m = min(window)
        j = max(p for p, h in enumerate(window) if h == m)      # rightmost minimum
        picked.add((i + j, m))
    return {h for _, h in picked}, len(hashes)
 
def similarity(a, b):
    fa, _ = fingerprints(a); fb, _ = fingerprints(b)
    return len(fa & fb) / len(fa | fb)
 
original = """
long long maxSum(vector<int>& nums, int k) {
    long long best = 0, cur = 0;
    for (int i = 0; i < nums.size(); i++) {
        cur += nums[i];
        if (i >= k) cur -= nums[i - k];
        if (i >= k - 1) best = max(best, cur);
    }
    return best;
}"""
 
renamed = """
// my own solution!!
long long maxSum(vector<int>& a, int len) {
    long long answer = 0, window = 0;
    for (int idx = 0; idx < a.size(); idx++) {
        window += a[idx];
        if (idx >= len) window -= a[idx - len];
        if (idx >= len - 1) answer = max(answer, window);
    }
    return answer;
}"""
 
padded = """
long long maxSum(vector<int>& a, int len) {
    long long answer = 0, window = 0;
    int unused = 42;
    for (int idx = 0; idx < a.size(); idx++) {
        window += a[idx];
        if (idx >= len) window -= a[idx - len];
        unused++;
        if (idx >= len - 1) answer = max(answer, window);
    }
    return answer;
}"""
 
independent = """
long long maxSum(vector<int>& nums, int k) {
    vector<long long> pre(nums.size() + 1, 0);
    for (int i = 0; i < nums.size(); i++) pre[i + 1] = pre[i] + nums[i];
    long long best = 0;
    for (int r = k; r <= nums.size(); r++) best = max(best, pre[r] - pre[r - k]);
    return best;
}"""
 
print("normalised:", normalise(original)[:60] + "...")
fp, n = fingerprints(original)
print(f"original: {n} hashed 12-grams, {len(fp)} kept as fingerprints")
print(f"original vs renamed copy:     {similarity(original, renamed):.2f}")
print(f"original vs padded copy:      {similarity(original, padded):.2f}")
print(f"original vs independent work: {similarity(original, independent):.2f}")

normalised: longlongV(vector<int>&V,intV){longlongV=0,V=0;for(intV=0;V<V...
original: 114 hashed 12-grams, 22 kept as fingerprints
original vs renamed copy:     1.00
original vs padded copy:      0.57
original vs independent work: 0.10

Line one shows the normalised text: the names are gone, and only the shape of the code is left. Line two shows winnowing's thinning: 114 hashes, 22 kept, about 19%, close to the 2/(w + 1) = 22% the paper predicts. Then the scores. The renamed copy, comment and all, scores 1.00: after normalisation it's the same text. The padded copy scores 0.57, because the inserted lines break up some windows, but every stretch of 19 unchanged characters still shares a fingerprint, so most of the copy is still caught. An independent solution scores 0.10, from fragments every solution to this problem shares, such as the function signature.

8.4From pairs to an index

Fingerprints also solve the all-pairs problem. Build an inverted index, a map from each fingerprint to the list of submissions that contain it, the same structure a search engine uses for words (chapter 62). Then, for each submission, look up its fingerprints and count how often each other submission appears in their lists. Only pairs that share at least one fingerprint are ever compared, and for most problems that's a tiny fraction of the 200 million.

That 0.10 in the TryIt hints at the real difficulty, and it would probably be higher on an easier problem. A short problem has few ways to solve it, and many correct solutions look alike after normalisation: the same function signature, the same for loop, the same standard algorithm. So a judge's plagiarism check has to discount common code. Fingerprints that appear in a large share of all submissions to a problem carry no evidence and are best ignored, as is anything in the provided starter code. What's left, a long run of uncommon fingerprints shared by two submissions from two accounts, is strong evidence. The 2003 paper reports that across MOSS's several thousand users, false positives "appear to be non-existent". In MOSS's setting, though, a person reads every flagged pair before anything happens, and a contest judge should do the same: the algorithm ranks suspicious pairs, and people decide.

9.1Every box, and why it's there

9.2From top to bottom

10.1Failures this design has to survive

10.2The tradeoffs, in one table