ARP, and the MTU problem

The missing link between layer 3 and layer 2

Your host knows it wants to reach IP 192.168.1.1. To build a frame it needs that host's MAC address, and nothing it has so far provides one. ARP (address resolution protocol) is the small protocol that fills the gap. It answers exactly one question: which MAC address owns this IPv4 address on this link?

The exchange is two frames.

1. ARP request, sent as a layer 2 broadcast so it reaches everyone: "who has 192.168.1.1? Tell 192.168.1.42 at 3c:22:fb:8a:19:04." 2. ARP reply, sent as a normal unicast back to the asker: "192.168.1.1 is at aa:bb:cc:00:00:01."

ARP frames are not carried inside IP. They ride directly in Ethernet with EtherType 0x0806, which makes sense, because ARP exists to make IP possible in the first place.

The answer is cached in the ARP table, typically for 60 seconds to a few minutes, so the broadcast happens once per neighbor rather than once per packet.

ip neigh show

192.168.1.1 dev eth0 lladdr aa:bb:cc:00:00:01 REACHABLE
192.168.1.77 dev eth0 lladdr 8c:85:90:04:2b:3e STALE
192.168.1.90 dev eth0  FAILED

REACHABLE was confirmed recently. STALE is believed but unverified, and will be re-checked on next use. FAILED means nobody answered, and that single line is often the whole diagnosis when a host "cannot reach" a machine that is actually powered off.

The rule a host follows before every send

Every outgoing packet passes through the same short decision, and it is worth memorizing because it explains most "works locally, fails remotely" bugs.

1. Compare the destination IP with my own address and prefix. Is it on my network? Unit 4 shows the arithmetic that answers this. 2. If yes, ARP for the destination itself and send the frame straight to it. 3. If no, ARP for the default gateway and send the frame to the router's MAC, while leaving the destination IP untouched.

So a host ARPs for at most one address per off network conversation, the router, and it never learns a remote host's MAC because that would be meaningless.

Two ARP behaviors you will see in the wild:

- Gratuitous ARP is an unsolicited reply announcing "this IP is now at this MAC". Failover systems send it when a virtual IP moves to a standby machine, so every switch and host updates its tables immediately. - ARP spoofing is the attack that follows from ARP having no authentication at all. Any host can claim any IP. The defenses are switch features (dynamic ARP inspection, port security) rather than fixes to ARP.

MTU, fragmentation, and the classic silent failure

The MTU is the largest payload a link will carry, 1500 bytes on standard Ethernet. A path is made of many links, and the smallest MTU along it is the path MTU.

What happens when a 1500 byte packet must cross a link whose MTU is 1400? Two possibilities, and they behave very differently.

DF is the do not fragment flag in the IP header. Modern TCP sets it on purpose, because fragmentation is bad: losing any fragment loses the whole packet, the receiver must buffer pieces, and firewalls often drop fragments.

Instead, senders use path MTU discovery. Send with DF set, and if an ICMP "fragmentation needed" message comes back saying the next link only allows 1400, shrink and resend. It works well when ICMP is allowed through.

It fails when somebody blocks all ICMP at a firewall, which is depressingly common. Then the sender never learns the limit, keeps sending 1500 byte packets that vanish, and gets the signature symptom: a TCP connection that opens fine and then hangs the moment real data flows. Handshake packets are tiny and fit. The first full sized data packet does not.

ping -M do -s 1472 -c 1 example.com

ping: local error: message too long, mtu=1400

That is the whole test. 1472 bytes of payload plus 8 bytes of ICMP header plus 20 bytes of IP header is exactly 1500. Shrink the number until it passes and you have measured the path MTU by hand. Tunnels are the usual culprit, because a VPN wraps your packet in another header and steals 40 to 80 bytes from the budget.