Build a TCP/IP stack in userspace

Open /dev/net/tun, create a TAP interface with ioctl(TUNSETIFF), bring it up with an address on the host, and read from the file descriptor in a loop. Each read returns one Ethernet frame.

A TUN device would hand you IP packets and skip Ethernet. Use TAP anyway: ARP only exists at layer 2, and the host won't send you any IP until you answer it.

When the host wants to reach your IP, it broadcasts "who has this address?". Parse the ARP packet, and if the target is you, reply with your MAC. Keep a small ARP cache of senders you've seen, because you'll need their MACs to send anything back.

Byte order bites here first. Every multi-byte field is big-endian on the wire. Wrap the conversions in one place, so you don't scatter byte swaps through the code.

Parse the IPv4 header, check its checksum, and drop anything not addressed to you. For ICMP echo requests, swap source and destination, change the type to echo reply, recompute both checksums, and send it back inside a new Ethernet frame.

The Internet checksum from RFC 1071 is a one's-complement sum of 16-bit words. It's a few lines, and it's used again in UDP and TCP, so write it once and test it against a captured packet.

UDP adds only ports, a length and a checksum. The checksum covers a pseudo-header made from the IP addresses as well as the UDP header, which is the first time a layer reaches down into the one below it.

Build a tiny API here: bind a port, receive a datagram, send one. You'll reuse the shape for TCP, and it keeps the protocol code apart from the application.

Keep a transmission control block per connection, keyed by the four-tuple, holding the send and receive sequence variables from RFC 9293 (SND.UNA, SND.NXT, RCV.NXT and friends). On a SYN to a listening port, pick an initial sequence number and reply SYN-ACK. On the final ACK, move to ESTABLISHED.

Send RST for segments to ports nobody is listening on. Compare sequence numbers with wraparound arithmetic from the start: they're 32-bit and they wrap.

Accept in-order data into a receive buffer, advance RCV.NXT, and ACK it. Queue out-of-order segments until the gap fills, instead of dropping them. On the send side, split application data into segments no larger than the peer's MSS and within its advertised window.

A tiny HTTP responder on top gives you a real client to test against. Once curl works, capture the exchange and compare it with the same request to a kernel socket.

Implement FIN in both directions: the side that closes first goes through FIN_WAIT_1, FIN_WAIT_2 and TIME_WAIT, the other through CLOSE_WAIT and LAST_ACK. Each direction closes independently, so one side can keep sending after the other is done.

Draw the state diagram from RFC 9293 and put it next to your code. Most TCP bugs at this stage are a transition the diagram has and your match or switch doesn't.

Keep unacknowledged segments and a retransmission timer. Estimate round-trip time the way RFC 6298 describes, with a smoothed average and a variance term, and set the timeout from both. Don't sample RTT from retransmitted segments, since you can't tell which copy the ACK is for.

Double the timeout on every consecutive retransmission. Then use tc qdisc ... netem to add loss, delay and reordering, and watch your stack recover.

Respect the receiver's window, including a zero window: stop sending and probe periodically until it opens. Then add a congestion window from RFC 5681. Start small and grow it quickly in slow start, grow linearly after a threshold, cut it on loss.

On three duplicate ACKs, retransmit right away instead of waiting for the timer. Log the congestion window on every ACK and plot it. That sawtooth is a better picture of how TCP behaves than any diagram.