You type grep error app.log | sort | uniq -c > counts.txt and hit enter. Three
processes start, get wired together by two pipes, one of them writes to a file,
and your prompt comes back when the last one exits. It looks like one command.
Your shell turned it into probably two dozen system calls.
A shell is the thinnest possible layer over the Unix process API, which makes it
the best place to learn that API. By the end you'll have a program that reads a
line, parses it, forks children, wires their file descriptors together, and
manages them as jobs you can suspend with Ctrl-Z and resume with fg. There's
no magic left in it: each feature is one or two syscalls used carefully.
01Why build this
Every engineer uses a shell daily, and most have a vague model of what it does. Writing one fixes that model:
- Process lifecycles stop being fuzzy. You'll know what
forkcopies, whatexecreplaces, and why a parent mustwaitor leave zombies behind. - File descriptors click. Redirection, pipes and
2>&1are all justdup2on small integers. After this, a leaked descriptor in a server is something you know how to look for. - Signals make sense. You'll know why Ctrl-C kills the pipeline but not the shell, and why a daemon has to handle SIGTERM itself.
- Containers and CI runners get less mysterious. Every process supervisor, from systemd to supervisord, does a harder version of what your shell does on each command.
It's also a quick win. A useful shell, maybe without job control, with pipes and redirection fits in a few hundred lines, and job control is the part that teaches you the most.
02What you're building
At its centre the shell is one loop. Here's what it does for a two-stage pipeline:
?Why doesn't the shell just run the program itself?
Because Unix splits "start a process" into two calls. fork makes a copy of
the shell, and exec replaces that copy's program with a new one. The gap
between them is where the child, still running your shell's code, can change its
own file descriptors, process group and signal handlers before becoming ls.
Pipes, redirection and job control all live in that gap.
03Before you start
| You need | Why | Where to get it |
|---|---|---|
| Linux or macOS | You'll call POSIX process and terminal APIs directly | Any Unix; WSL works on Windows |
A language with raw fork and exec | Higher-level spawn APIs hide the gap you need | C is simplest; Rust via libc or nix; Go needs syscall and some care |
| Basic knowledge of processes and syscalls | Every milestone is built on them | Chapter 06, Chapter 07 |
strace or dtruss | To compare your syscalls with bash's | Your package manager |
| A second terminal | For when you break the first one | Already there |
04The roadmap
Seven milestones. After milestone 4 you have a shell you could use for simple work; milestones 5 to 7 make it behave like the one you use now.
A REPL that runs one command
1 eveningPrint a prompt, read a line, split it on whitespace, then fork. In the child,
execvp the first word with the rest as arguments. In the parent, waitpid for
that child and decode its status with the WIFEXITED family of macros.
If exec fails, the child must _exit right there. Otherwise you now have two
shells reading from the same terminal, and the confusion that follows is a
rite of passage.
ls -l /tmp runs, the prompt returns after it exits, a missing command prints an error, and Ctrl-D exits the shell.PATH lookup and builtins
1 eveningReplace execvp with your own PATH search: split $PATH on colons, try each
directory, and execv the first match that's executable. Now you know what
bash's hash builtin is caching.
Then add cd, exit and export. Try cd as an external command first and
watch it do nothing: chdir changes the directory of the child, which then
exits. Anything that changes the shell's own state has to run inside the shell.
cd /tmp then pwd prints /tmp, exit 3 leaves with status 3, and ls runs the same binary that command -v ls reports in bash.A real tokenizer
1 weekendSplitting on spaces breaks the first time someone types a quoted filename.
Write a small state machine: outside quotes, whitespace ends a word; inside
single quotes, everything is literal; inside double quotes, $ still expands
and backslash escapes a few characters.
Expand variables and $? during tokenizing, and keep the result as a list of
words, never a re-joined string. Re-splitting expanded text is where quoting bugs
come from.
echo 'a b' $HOME x\ y prints three words with the double space kept, and false; echo $? prints 1.Redirection
1 eveningParse <, >, >> and 2>&1 into the command structure. In the child,
after fork and before exec, open each file and dup2 it onto 0, 1 or 2,
then close the original descriptor.
Order matters. cmd > f 2>&1 sends both streams to the file, while
cmd 2>&1 > f sends stderr to the terminal. Apply redirections left to right
and both cases fall out correctly.
sort < in.txt > out.txt, echo hi >> log and cmd 2>&1 | less all behave exactly as in bash.Pipelines
1 weekendFor N commands, create N−1 pipes. Fork every command, wire each one's stdout to
the next one's stdin with dup2, and in every process close every pipe end
it isn't using, including in the shell itself. Then wait for all of them.
That done test is chosen on purpose. When head exits after three lines, yes
gets SIGPIPE on its next write and dies. If you've leaked a read end anywhere,
yes blocks on a full pipe and the pipeline hangs forever instead.
yes | head -3 prints three lines and returns, and seq 100000 | grep 7 | sort -n | head -3 prints 7, 17 and 27.Signals
1 eveningThe terminal sends SIGINT to every process in the foreground group, which at
this point includes your shell. Make the shell ignore SIGINT and SIGQUIT, and
have each child restore the default disposition before exec.
Watch for interrupted system calls. A signal arriving during read or
waitpid can make it return EINTR, and a shell that treats that as a real
error exits unexpectedly.
sleep 100 kills the sleep and returns to the prompt; Ctrl-C at an empty prompt just prints a new prompt.Job control
1–2 weekendsPut every pipeline in its own process group, calling setpgid in both the
parent and the child to avoid a race. Before waiting on a foreground job, hand
it the terminal with tcsetpgrp; take the terminal back when it stops or exits.
Wait with WUNTRACED so you notice when a job is stopped, not just when it dies.
Keep a job table. fg and bg send SIGCONT to the whole group with a negative
PID. Reap background jobs from a SIGCHLD handler, or by polling with WNOHANG
before each prompt. The glibc manual's chapter on this is the clearest guide
there is, and you'll probably read it twice.
sleep 100 then Ctrl-Z, jobs lists it as stopped, bg resumes it, fg brings it back, and vim suspends and resumes with its screen intact.05Traps that catch everyone
| Symptom | Cause | Fix |
|---|---|---|
| A pipeline never finishes | Some process, often the shell, still holds the pipe's write end | Close every unused pipe end in every process |
| Two prompts appear after a typo | exec failed and the child fell back into the REPL loop | _exit(127) immediately after a failed exec |
cd does nothing | It ran in a forked child | Implement it as a builtin in the shell process |
| Ctrl-C kills the shell | The shell didn't ignore SIGINT | Ignore it in the shell; restore the default in each child before exec |
The shell stops itself on fg | A background process group called tcsetpgrp and got SIGTTOU | Ignore SIGTTOU in the shell |
ps fills with defunct processes | Finished background jobs are never waited on | Reap with waitpid(-1, ..., WNOHANG) |
| Output appears twice after a fork | Buffered stdio was copied into the child unflushed | Flush stdout before fork |
06Stretch goals
- Line editing and history. Put the terminal in raw mode and handle arrow keys, a history file and tab completion yourself, without readline.
- Control flow. Add
&&,||,;, subshells in parentheses, and thenifandwhile. You're now writing a small language parser. - Command substitution and here-documents.
$(...)means running a pipeline and capturing its output into a word. - Scripts. Run a file of commands, with
$1and friends, and try a few real POSIX scripts to see how far you get. - Compare with dash. Run the same commands under
strace -fin your shell and in dash, and account for every difference.
07References worth your time
A short tutorial that gets you from nothing to a REPL with builtins in an evening. A good starting point for milestones 1 and 2.
A complete worked example of process groups, terminal ownership and job tables. The reference for milestone 7.
The Open Group's specification of tokenizing, quoting, expansion and redirection. Dense, but it settles every argument.
The chapters on processes, signals, pipes and job control are the best long-form explanation of the APIs you're calling.
The classic on process relationships, sessions and signals, with the historical reasons behind each odd rule.
MIT's teaching OS includes a tiny shell with pipes and redirection, and the book's first chapter explains it alongside the kernel side.