Traceroute: discovering the junctions along the route
It's another ordinary day on the Packet Highway. Cars carry data, routers inspect their tables, and the NAT toll-booth manager is once again looking for his notebook of number plates.
A report reaches the administrator.
— The Internet is broken.
— All of it?
— I don't know. The website won't open.
— Which website?
— The one that worked yesterday.
The administrator sends a packet. Silence. Another packet. Nothing.
— If the car isn't arriving, let's find out which road it's taking.
— How? We don't have cameras at every junction!
— No cameras. But we have traceroute.
Mapping a road without having a map
In our earlier story about TTL, packets were given a limited number of router hops. Forwarding routers decrement the counter, and when it runs out they can return an ICMP Time Exceeded response.
The administrator decides to exploit this in a way no ordinary courier would approve of.
— Give the first car TTL = 1.
— It won't make it!
— Exactly.
The first router stops the car and may send a notice. The administrator records its address. The second car gets TTL = 2 and reaches the next router. Then TTL = 3, TTL = 4… Soon the replies reveal successive junctions.
Traceroute does not read other people's routing tables. It discovers hops by sending probes with increasing hop limits and examining the replies.
Three cars, three tickets
Traditional traceroute often sends three probes per TTL. One reply may go missing, another arrive late, and a third take a different road.
A simplified example using addresses reserved for documentation:
traceroute to example.net (203.0.113.80) 1 192.0.2.1 0.8 ms 0.7 ms 0.9 ms 2 198.51.100.1 4.2 ms 4.1 ms 4.5 ms 3 198.51.100.2 9.8 ms 10.1 ms 10.0 ms 4 * * * 5 203.0.113.80 18.3 ms 18.6 ms 18.4 ms
— First hop, good. Second, good. Third, good… Who stole the fourth one?!
Asterisks: the checkpoint that won't answer the phone
An asterisk * means no response arrived before the timeout. It does not automatically mean that the router is down or that packets cannot continue.
— Could I have confirmation of passage?
— We don't issue those.
— But cars still drive through?
— Certainly.
— Then why won't you issue one?
— Because we don't issue those.
ICMP replies may be filtered or rate-limited. If hop four is silent but hop five replies, there's no reason to declare a disaster at junction four. Although administrators do enjoy a little drama. 🤣
Windows and Linux: two departments of the same office
Windows tracert typically uses ICMP Echo. Classic Linux traceroute often sends UDP probes to high-numbered ports by default. ICMP and TCP variants exist too. The goal is similar, but firewalls and routers may treat different probes differently.
— UDP paperwork?
— Yes.
— And you?
— ICMP.
— The third gentleman?
— TCP.
— You're all asking about the same road!
— Yes, but each of us has a different form.
Milliseconds are not times between checkpoints
Traceroute displays RTTs — the interval between sending a probe and receiving its reply. These are not times between adjacent routers. Each measurement is a separate round trip to an answering hop and back.
A router can forward ordinary traffic quickly while processing diagnostic replies slowly. A single hop showing 300 ms, followed by later hops showing 20 ms, isn't automatically proof of a forwarding problem there.
— Boss, the cars are on time, but issuing certificates takes three hundred milliseconds!
— And the cars?
— Moving.
— Then stop interrupting the clerks.
We'll return to ping, jitter and packet loss in another story.
Three probes and three different roads
With load balancing such as ECMP, different flows may take different paths. Traceroute sends several probes, so its output may combine hops from different routes rather than showing one car's complete journey. Variants like Paris traceroute try to reduce that effect.
— How did it go through Warsaw, then Poznań, then back to Warsaw?!
— That wasn't the same car.
— What?
— You sent three.
ICMP replies may also take a completely different return route. Internet routing isn't necessarily symmetric, and traceroute does not automatically reconstruct that return path.
When traceroute reaches a galaxy far, far away
Finally, the administrator checks one more address. Not because something is broken. Someone mentioned a route that tells a story.
traceroute -6 -I -m 100 obiwan.qwt.ch
For eleven hops everything looks normal: IPv6 addresses, routers, milliseconds. And then…
1 2001:db8::1 (2001:db8::1) 0.798 ms * * 2 2001:db8::2 (2001:db8::2) 0.174 ms 0.241 ms 0.244 ms 3 2001:db8::3 (2001:db8::3) 0.650 ms 0.719 ms 0.831 ms 4 2001:db8::4 (2001:db8::4) 0.132 ms 0.219 ms 0.221 ms 5 * * * 6 2001:db8::6 (2001:db8::6) 1.102 ms * * 7 2001:db8::7 (2001:db8::7) 1.302 ms * * 8 * * * 9 2001:db8::9 (2001:db8::9) 22.028 ms * * 10 2001:db8::a (2001:db8::a) 20.438 ms * * 11 2001:db8::b (2001:db8::b) 20.561 ms * * 12 Episode.iv (2a03:4000:5a:cd:dead:beef:cafe:0) 21.518 ms * * 13 A.NEW.hope (2a03:4000:5a:cd:dead:beef:cafe:1) 23.054 ms * * 14 It.is.a.period.of.civil.war (2a03:4000:5a:cd:dead:beef:cafe:2) 23.005 ms * * 15 Rebel.spaceships (2a03:4000:5a:cd:dead:beef:cafe:3) 25.158 ms * * 16 striking.from.a.hidden.base (2a03:4000:5a:cd:dead:beef:cafe:4) 24.145 ms * * 17 have.won.their.first.victory (2a03:4000:5a:cd:dead:beef:cafe:5) 23.145 ms * * 18 against.the.evil.Galactic.empire (2a03:4000:5a:cd:dead:beef:cafe:6) 23.372 ms * * 19 During.the.battle (2a03:4000:5a:cd:dead:beef:cafe:7) 25.517 ms * * 20 Rebel.spies.managed (2a03:4000:5a:cd:dead:beef:cafe:8) 25.843 ms * * 21 to.steal.secret.plans (2a03:4000:5a:cd:dead:beef:cafe:9) 24.117 ms * * 22 to.the.Empires.ultimate.weapon (2a03:4000:5a:cd:dead:beef:cafe:a) 24.458 ms * * 23 the.DEATH.star (2a03:4000:5a:cd:dead:beef:cafe:b) 22.937 ms * * 24 an.armored.space.station (2a03:4000:5a:cd:dead:beef:cafe:c) 25.089 ms * * 25 with.enough.power.to (2a03:4000:5a:cd:dead:beef:cafe:d) 22.930 ms * * 26 destroy.an.entire.planet (2a03:4000:5a:cd:dead:beef:cafe:e) 23.784 ms * * 27 Pursued.by.the.empires (2a03:4000:5a:cd:dead:beef:cafe:f) 25.727 ms * * 28 sinister.agents (2a03:4000:5a:cd:dead:beef:cafe:10) 23.341 ms * * 29 Princess.Leia.races.home (2a03:4000:5a:cd:dead:beef:cafe:11) 25.539 ms * * 30 aboard.her.starship (2a03:4000:5a:cd:dead:beef:cafe:12) 23.941 ms * * 31 custodian.of.the.stolen.plans (2a03:4000:5a:cd:dead:beef:cafe:13) 24.389 ms * * 32 that.can.save.her (2a03:4000:5a:cd:dead:beef:cafe:14) 22.814 ms * * 33 people.and.restore (2a03:4000:5a:cd:dead:beef:cafe:15) 25.616 ms * * 34 freedom.to.the.galaxy (2a03:4000:5a:cd:dead:beef:cafe:16) 26.575 ms * * 35 0-----I-------I-----0.ll (2a03:4000:5a:cd:dead:beef:cafe:17) 22.838 ms * * 36 0------------------0.ll (2a03:4000:5a:cd:dead:beef:cafe:18) 24.788 ms * * 37 0-----------------0.ll (2a03:4000:5a:cd:dead:beef:cafe:19) 23.205 ms * * 38 0----------------0.ll (2a03:4000:5a:cd:dead:beef:cafe:1a) 22.882 ms * * 39 0---------------0.ll (2a03:4000:5a:cd:dead:beef:cafe:1b) 24.455 ms * * 40 0--------------0.ll (2a03:4000:5a:cd:dead:beef:cafe:1c) 23.004 ms * * 41 0-------------0.ll (2a03:4000:5a:cd:dead:beef:cafe:1d) 24.749 ms * * 42 0------------0.ll (2a03:4000:5a:cd:dead:beef:cafe:1e) 22.795 ms * * 43 0-----------0.ll (2a03:4000:5a:cd:dead:beef:cafe:1f) 23.324 ms * * 44 0----------0.ll (2a03:4000:5a:cd:dead:beef:cafe:20) 25.684 ms * * 45 0---------0.ll (2a03:4000:5a:cd:dead:beef:cafe:21) 23.523 ms * * 46 0--------0.ll (2a03:4000:5a:cd:dead:beef:cafe:22) 22.794 ms * * 47 0-------0.ll (2a03:4000:5a:cd:dead:beef:cafe:23) 25.269 ms * * 48 0------0.ll (2a03:4000:5a:cd:dead:beef:cafe:24) 23.342 ms * * 49 0-----0.ll (2a03:4000:5a:cd:dead:beef:cafe:25) 22.880 ms * * 50 0----0.ll (2a03:4000:5a:cd:dead:beef:cafe:26) 24.875 ms * * 51 0---0.ll (2a03:4000:5a:cd:dead:beef:cafe:27) 24.452 ms * * 52 0--0.ll (2a03:4000:5a:cd:dead:beef:cafe:28) 23.143 ms * * 53 0-0.ll (2a03:4000:5a:cd:dead:beef:cafe:29) 24.602 ms * * 54 00.ll (2a03:4000:5a:cd:dead:beef:cafe:2a) 23.148 ms * * 55 0.ll (2a03:4000:5a:cd:dead:beef:cafe:2b) 22.966 ms * * 56 By.Felix (2a03:4000:5a:cd:dead:beef:cafe:2c) 23.204 ms * * 57 Read.more.at.blog.qwt.ch (2a03:4000:5a:cd:dead:beef:cafe:2d) 20.339 ms * *
The administrator rubs his eyes.
— Wait. What is this?!
— Star Wars.
— But I ran traceroute!
— Correct.
— Why is a router telling me a story about spaceships?!
— Because it can. 🤣
This is a real demonstration of IPv6, Hop Limit and reverse DNS (PTR records). Its creator arranged addresses and responses so that the names of successive hops form a story. It does not mean the packet passed through 57 separate physical routers — part of the route was deliberately engineered. We discussed PTR records in our DNS article.
You can try it yourself with working IPv6. Add -n afterwards: without name resolution, you'll see addresses instead of a story. Technology can be fun.
— Shall we return to diagnosing the outage?
— In a minute. First I'll check whether anyone has done The Lord of the Rings.
The final checkpoint
The administrator returns to the support ticket. The screen shows addresses, asterisks, and plenty of milliseconds.
— So do we know why the website won't open?
— Not yet.
— Then why send all those cars?!
— To find out where they were seen.
— So the sixth router is broken?
— Not necessarily.
— Then where's the fault?!
— That's why we have jobs.
The phone rings.
— Hello, IT? The website works again.
— What changed?
— Nothing. It fixed itself.
The administrator closes the terminal and writes: “Issue resolved before diagnostics could be completed.”
Somewhere on the Packet Highway, six officers are still wondering why they spent the morning stopping cars with permits for one, two and three crossings. One has reportedly requested hazard pay for dealing with administrators. 🤣