Reading a capture, and the diagnostic ladder

Capture first, guess later

Everything in this course is visible on the wire. tcpdump writes what a NIC sees, and knowing four flags is enough for most work.

sudo tcpdump -n -i eth0 -c 20 'host 93.184.216.34 and tcp port 443'

The expression at the end is a BPF filter, and its vocabulary is the fields from this course: host, net 10.0.0.0/8, port, tcp, udp, icmp, and tcp[tcpflags] & tcp-syn != 0. Filter at capture time on a busy host, because capturing everything and searching afterwards drops packets you needed.

One capture caution. tcpdump on the sending host shows what the kernel handed to the NIC, which is not proof anything left the building. When the two ends disagree, capture at both, and if possible in the middle, because the difference between the captures is the answer.

One packet, field by field

Here is a single TCP SYN, printed as bytes with the layers marked. This is the whole course in 54 bytes.

0000  aa bb cc 00 00 01 3c 22 fb 8a 19 04 08 00
0014  45 00 00 3c 1c 46 40 00 40 06 b1 e6
      c0 a8 01 2a 5d b8 d8 22
0034  c9 3a 01 bb 8f 3a 21 7d 00 00 00 00
      a0 02 fa f0 91 7c 00 00

Decode it in three groups.

Ethernet, bytes 0 to 13 from Lesson 3-1:

IPv4, the next 20 bytes from Lesson 4-1 and Lesson 5-2:

TCP, the next 20 bytes from Lesson 7-2:

Total length says 60, and IP plus TCP headers account for all 60, so this packet carries no data. Which is exactly right for a SYN.

Being able to do this once is worth more than memorizing any tool's interface, because every tool is showing you these same bytes with nicer labels.

The ladder, with commands

When something is broken, the temptation is to jump to the application, because that is where the error appeared. Resist it and climb from the bottom, because a failure at any rung makes everything above it look broken.

Each rung's result rules something out, and the exact wording of a failure is the most valuable clue you get:

- ping to the gateway fails and ip neigh says FAILED. Layer 2 problem, from Lesson 3-3. Wrong VLAN, wrong subnet mask, or the gateway is down. - A local address works but nothing outside does. Missing default route, from Lesson 5-1. - dig works but curl cannot resolve. The application is using a different resolver path, so check the stub configuration rather than the network. - nc says "connection refused". You reached the host and nothing is listening. From Lesson 7-1, check whether the service is bound to loopback only. - nc says "timed out". Nothing came back at all. A firewall dropped it or the route is wrong, from Lesson 9-2. - The connection opens then hangs on the first large response. Path MTU, from Lesson 3-3. Test with ping -M do -s 1472. - It works from one host and not another. Compare the two rung by rung instead of theorizing. The difference is the answer.

A worked diagnosis

Report: "the internal dashboard at dash.corp.example is down for me, it works for everyone else." Climb the ladder.

ip link show eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP

Rung 1 passes.

ip -4 addr show dev eth0
ip route get 10.20.30.40

inet 10.20.5.88/24 scope global dynamic eth0
10.20.30.40 via 10.20.5.1 dev eth0 src 10.20.5.88

Rungs 2 and 3 pass, with a real address and a next hop.

dig +short dash.corp.example

10.20.30.40

Rung 4 passes, and it matches what a colleague sees. Now rung 5.

nc -vz 10.20.30.40 443

nc: connect to 10.20.30.40 port 443 (tcp) failed: Connection timed out

Timed out, not refused, so nothing answered at all. Two candidates from the list above: a filter is dropping the traffic, or the return path is wrong. Capture to find out which.

sudo tcpdump -n -i eth0 'host 10.20.30.40 and tcp port 443'

14:31:02.114 IP 10.20.5.88.53318 > 10.20.30.40.443: Flags [S], seq 91...
14:31:03.118 IP 10.20.5.88.53318 > 10.20.30.40.443: Flags [S], seq 91...
14:31:05.126 IP 10.20.5.88.53318 > 10.20.30.40.443: Flags [S], seq 91...

SYNs leaving, retried with doubling waits from Lesson 7-3, and no SYN-ACK ever. The packets are going out and nothing returns. Since a colleague on 10.20.6.x succeeds, the difference is the source network, which points at a rule that permits 10.20.6.0/24 and not 10.20.5.0/24. The firewall's default deny from Lesson 9-2 is doing exactly what it was told.

The ladder found it in five commands without touching the application, and it produced a specific, checkable claim rather than a theory.

Where to go from here

You now have the full stack: frames and MAC addresses on the link, addresses and subnets at layer 3, the routing decision, translation at the boundary, connections and windows at layer 4, names and configuration above that, and the security and diagnostic layers around all of it. The two habits worth keeping are the ones this last lesson used. Ask which layer, because that turns a vague problem into a specific one. And look at the packets, because the network always tells you the truth when you capture it.