* * * * *

                                A routing mess

Theoretically, TCP/IP packets don't have to take the same route but in real
life all the packets travel the same path. Do two traceroutes and you'll see
the same path.

But today, something seriously odd is going on with the Telocity [1] network.
I noticed some annoying network lags between my home system and my colocated
server and did some traceroutes. And I got something I've never seen before:
no two traceroutes (done seconds after each other) where the same!

>
> [spc]linus:/usr/src/linux-2.0.39>/usr/sbin/traceroute tower.conman.org
> traceroute to tower.conman.org (66.33.1.143), 30 hops max, 40 byte packets
>  1  janet (10.0.0.1)  1.814 ms  1.737 ms  1.755 ms
>  2  dsl-64-128-156-186.telocity.com (64.128.156.186)  3.196 ms  3.244 ms  3.321 ms
>  3  route-64-131-188-129.telocity.com (64.131.188.129)  14.046 ms  14.042 ms  14.335 ms
>  4  fe1-2-core1.mia.tlct.net (216.227.80.65)  14.353 ms  15.842 ms  15.055 ms
>  5  63.209.144.5 (63.209.144.5)  28.034 ms  15.807 ms  15.295 ms
>  6  63.212.176.30 (63.212.176.30)  64.815 ms  64.130 ms  64.096 ms
>  7  tower.conman.org (66.33.1.143)  63.686 ms  64.580 ms  65.116 ms
> [spc]linus:/usr/src/linux-2.0.39>/usr/sbin/traceroute tower.conman.org
> traceroute to tower.conman.org (66.33.1.143), 30 hops max, 40 byte packets
>  1  janet (10.0.0.1)  1.811 ms  1.739 ms  1.814 ms
>  2  dsl-64-128-156-186.telocity.com (64.128.156.186)  3.203 ms  3.250 ms  3.166 ms
>  3  route-64-131-188-129.telocity.com (64.131.188.129)  14.206 ms  16.701 ms  14.037 ms
>  4  fe1-2-core1.mia.tlct.net (216.227.80.65)  55.927 ms  14.870 ms  15.310 ms
>  5  pos8-1-hfr2.atl.tlct.net (216.227.126.17)  29.778 ms  27.010 ms  28.602 ms
>  6  ge-v5-hfr1.atl.tlct.net (216.227.126.57)  26.929 ms  27.705 ms  26.858 ms
>  7  pos8-1-hfr2.dfw.tlct.net (216.227.126.22)  72.251 ms  75.729 ms  73.677 ms
>  8  ge-v5-hfr1.dfw.tlct.net (216.227.126.53)  74.153 ms  73.005 ms  75.141 ms
>  9  pos8-1-hfr2.lax.tlct.net (216.227.126.161)  105.344 ms  107.666 ms  106.208 ms
> 10  ge-v5-hfr1.lax.tlct.net (216.227.126.49)  105.677 ms  105.910 ms  106.452 ms
> 11  pos8-1-hfr2.sjc.tlct.net (216.227.126.30)  132.903 ms  130.575 ms  125.659 ms
> 12  fe1-0-pbnap.sjc.tlct.net (216.227.126.169)  130.631 ms  124.767 ms  125.355 ms
> 13  ge-v5-hfr1.sjc.tlct.net (216.227.126.45)  119.255 ms  117.430 ms  125.996 ms
> 14  fe0-0-pbnap.sjc.tlct.net (216.227.126.165)  128.272 ms  130.919 ms  307.509 ms
> 15  * tower.conman.org (66.33.1.143)  64.129 ms  65.449 ms
> [spc]linus:/usr/src/linux-2.0.39>/usr/sbin/traceroute tower.conman.org
> traceroute to tower.conman.org (66.33.1.143), 30 hops max, 40 byte packets
>  1  janet (10.0.0.1)  1.800 ms  1.755 ms  1.731 ms
>  2  dsl-64-128-156-186.telocity.com (64.128.156.186)  3.212 ms  3.170 ms  3.230 ms
>  3  route-64-131-188-129.telocity.com (64.131.188.129)  13.544 ms  14.038 ms  12.598 ms
>  4  fe1-2-core1.mia.tlct.net (216.227.80.65)  15.749 ms  14.805 ms  15.291 ms
>  5  pos8-1-hfr2.atl.tlct.net (216.227.126.17)  31.635 ms  28.690 ms  28.623 ms
>  6  ge-v5-hfr1.atl.tlct.net (216.227.126.57)  27.187 ms  28.049 ms  28.120 ms
>  7  pos8-1-hfr2.dfw.tlct.net (216.227.126.22)  74.507 ms  73.200 ms  72.905 ms
>  8  ge-v5-hfr1.dfw.tlct.net (216.227.126.53)  72.751 ms  73.105 ms  75.415 ms
>  9  pos8-1-hfr2.lax.tlct.net (216.227.126.161)  106.234 ms  106.130 ms  106.299 ms
> 10  ge-v5-hfr1.lax.tlct.net (216.227.126.49)  105.595 ms  106.418 ms  106.217 ms
> 11  pos8-1-hfr2.sjc.tlct.net (216.227.126.30)  135.297 ms  128.321 ms  126.335 ms
> 12  fe1-0-pbnap.sjc.tlct.net (216.227.126.169)  118.552 ms  130.986 ms  126.928 ms
> 13  ge-v5-hfr1.sjc.tlct.net (216.227.126.45)  114.866 ms  117.073 ms  124.424 ms
> 14  fe0-0-pbnap.sjc.tlct.net (216.227.126.165)  127.696 ms  130.745 ms  125.977 ms
> 15  * pos8-1-hfr2.atl.tlct.net (216.227.126.17)  191.308 ms *
> 16  * tower.conman.org (66.33.1.143)  101.322 ms  100.327 ms
> [spc]linus:/usr/src/linux-2.0.39>/usr/sbin/traceroute tower.conman.org
> traceroute to tower.conman.org (66.33.1.143), 30 hops max, 40 byte packets
>  1  janet (10.0.0.1)  2.492 ms  1.769 ms  1.748 ms
>  2  dsl-64-128-156-186.telocity.com (64.128.156.186)  4.209 ms  3.150 ms  3.165 ms
>  3  route-64-131-188-129.telocity.com (64.131.188.129)  14.623 ms  13.612 ms  15.036 ms
>  4  fe1-2-core1.mia.tlct.net (216.227.80.65)  31.694 ms  16.130 ms  15.292 ms
>  5  vlan20-core2.mia.tlct.net (216.227.80.34)  15.440 ms  15.476 ms  16.583 ms
>  6  pos8-2-hfr1.was.tlct.net (216.227.126.217)  41.190 ms  62.272 ms  41.618 ms
>  7  fe0-0-mae-east.was.tlct.net (216.227.126.109)  47.128 ms  40.831 ms  40.425 ms
>  8  at0-0-0-170.br1.WDC2.gblx.net (208.49.231.89)  51.271 ms  40.997 ms  41.915 ms
>  9  pos2-0-155M.cr2.WDC2.gblx.net (208.178.174.61)  41.269 ms  41.624 ms  41.947 ms
> 10  pos1-0-622M.cr1.ATL1.gblx.net (206.132.115.94)  57.903 ms  57.344 ms  58.573 ms
> 11  pos5-0-0-155M.ar1.TPA1.gblx.net (208.178.116.34)  91.555 ms  93.025 ms  93.167 ms
> 12  Customer.s4-0-0.ar1.TPA1.gblx.net (64.212.109.138)  99.760 ms  98.111 ms  216.351 ms
> 13  tower.conman.org (66.33.1.143)  101.450 ms  102.368 ms  101.038 ms
> [spc]linus:/usr/src/linux-2.0.39>/usr/sbin/traceroute tower.conman.org
> traceroute to tower.conman.org (66.33.1.143), 30 hops max, 40 byte packets
>  1  janet (10.0.0.1)  1.770 ms  1.852 ms  1.705 ms
>  2  dsl-64-128-156-186.telocity.com (64.128.156.186)  3.175 ms  3.220 ms  3.290 ms
>  3  route-64-131-188-129.telocity.com (64.131.188.129)  12.757 ms  13.936 ms  14.244 ms
>  4  fe1-2-core1.mia.tlct.net (216.227.80.65)  15.600 ms  16.065 ms  13.803 ms
>  5  pos8-1-hfr2.atl.tlct.net (216.227.126.17)  27.430 ms  28.691 ms  28.694 ms
>  6  ge-v5-hfr1.atl.tlct.net (216.227.126.57)  27.660 ms  26.413 ms  26.573 ms
>  7  pos8-1-hfr2.dfw.tlct.net (216.227.126.22)  74.026 ms  82.005 ms  73.880 ms
>  8  ge-v5-hfr1.dfw.tlct.net (216.227.126.53)  74.488 ms  73.386 ms  73.193 ms
>  9  pos8-1-hfr2.lax.tlct.net (216.227.126.161)  105.209 ms  106.414 ms  106.708 ms
> 10  ge-v5-hfr1.lax.tlct.net (216.227.126.49)  107.248 ms  106.171 ms  127.723 ms
> 11  pos8-1-hfr2.sjc.tlct.net (216.227.126.30)  135.017 ms  127.588 ms  126.325 ms
> 12  fe1-0-pbnap.sjc.tlct.net (216.227.126.169)  133.066 ms  125.410 ms  128.867 ms
> 13  ge-v5-hfr1.sjc.tlct.net (216.227.126.45)  115.226 ms  121.554 ms  119.980 ms
> 14  fe0-0-pbnap.sjc.tlct.net (216.227.126.165)  130.603 ms  134.301 ms  130.845 ms
> 15  ge-v5-hfr1.sjc.tlct.net (216.227.126.45)  119.819 ms  130.060 ms  142.445 ms
> 16  fe0-0-pbnap.sjc.tlct.net (216.227.126.165)  139.018 ms  133.283 ms *
> 17  tower.conman.org (66.33.1.143)  102.361 ms  101.642 ms  100.291 ms
> [spc]linus:/usr/src/linux-2.0.39>/usr/sbin/traceroute tower.conman.org
> traceroute to tower.conman.org (66.33.1.143), 30 hops max, 40 byte packets
>  1  janet (10.0.0.1)  1.740 ms  1.757 ms  1.728 ms
>  2  dsl-64-128-156-186.telocity.com (64.128.156.186)  3.201 ms  3.244 ms  3.170 ms
>  3  route-64-131-188-129.telocity.com (64.131.188.129)  13.510 ms  13.748 ms  14.303 ms
>  4  fe1-2-core1.mia.tlct.net (216.227.80.65)  14.592 ms  14.997 ms  14.759 ms
>  5  vlan20-core2.mia.tlct.net (216.227.80.34)  17.374 ms  16.767 ms  16.539 ms
>  6  fe1-2-core1.mia.tlct.net (216.227.80.65)  16.462 ms  15.245 ms  14.954 ms
>  7  vlan20-core2.mia.tlct.net (216.227.80.34)  14.788 ms  16.020 ms  16.619 ms
>  8  fe1-2-core1.mia.tlct.net (216.227.80.65)  15.897 ms  15.401 ms  16.995 ms
>  9  vlan20-core2.mia.tlct.net (216.227.80.34)  16.266 ms  16.682 ms  15.344 ms
> 10  fe1-2-core1.mia.tlct.net (216.227.80.65)  15.385 ms  16.295 ms  15.085 ms
> 11  vlan20-core2.mia.tlct.net (216.227.80.34)  16.568 ms  17.000 ms  16.552 ms
> 12  fe1-2-core1.mia.tlct.net (216.227.80.65)  16.599 ms  17.155 ms  14.814 ms
> 13  vlan20-core2.mia.tlct.net (216.227.80.34)  14.799 ms  40.861 ms  18.034 ms
> 14  fe1-2-core1.mia.tlct.net (216.227.80.65)  17.572 ms  91.453 ms  15.259 ms
> 15  vlan20-core2.mia.tlct.net (216.227.80.34)  23.225 ms  16.968 ms  16.886 ms
> 16  fe1-2-core1.mia.tlct.net (216.227.80.65)  16.060 ms  17.160 ms  15.896 ms
> 17  vlan20-core2.mia.tlct.net (216.227.80.34)  15.333 ms  16.403 ms  25.721 ms
> 18  fe1-2-core1.mia.tlct.net (216.227.80.65)  17.314 ms  15.686 ms  16.859 ms
> 19  vlan20-core2.mia.tlct.net (216.227.80.34)  23.482 ms  15.241 ms  15.339 ms
> 20  fe1-2-core1.mia.tlct.net (216.227.80.65)  16.103 ms  16.605 ms  17.546 ms
> 21  vlan20-core2.mia.tlct.net (216.227.80.34)  15.845 ms  16.959 ms  18.241 ms
> 22  fe1-2-core1.mia.tlct.net (216.227.80.65)  16.839 ms  15.091 ms  15.442 ms
> 23  vlan20-core2.mia.tlct.net (216.227.80.34)  15.910 ms  16.546 ms  16.501 ms
> 24  fe1-2-core1.mia.tlct.net (216.227.80.65)  16.934 ms  16.800 ms  17.549 ms
> 25  vlan20-core2.mia.tlct.net (216.227.80.34)  17.931 ms  16.327 ms  16.082 ms
> 26  fe1-2-core1.mia.tlct.net (216.227.80.65)  15.973 ms  16.469 ms  16.467 ms
> 27  vlan20-core2.mia.tlct.net (216.227.80.34)  52.760 ms  17.235 ms  15.509 ms
> 28  fe1-2-core1.mia.tlct.net (216.227.80.65)  16.039 ms  16.770 ms  16.421 ms
> 29  vlan20-core2.mia.tlct.net (216.227.80.34)  15.857 ms  16.527 ms  17.723 ms
> 30  fe1-2-core1.mia.tlct.net (216.227.80.65)  17.253 ms  18.240 ms  16.017 ms
> [spc]linus:/usr/src/linux-2.0.39>/usr/sbin/traceroute tower.conman.org
> traceroute to tower.conman.org (66.33.1.143), 30 hops max, 40 byte packets
>  1  janet (10.0.0.1)  43.666 ms  1.827 ms  1.813 ms
>  2  dsl-64-128-156-186.telocity.com (64.128.156.186)  3.306 ms  3.232 ms  3.102 ms
>  3  route-64-131-188-129.telocity.com (64.131.188.129)  14.623 ms  14.539 ms  14.481 ms
>  4  fe1-2-core1.mia.tlct.net (216.227.80.65)  65.600 ms  81.921 ms  58.918 ms
>  5  vlan20-core2.mia.tlct.net (216.227.80.34)  85.022 ms  14.867 ms  14.560 ms
>  6  fe1-2-core1.mia.tlct.net (216.227.80.65)  119.265 ms * ge-v5-hfr1.atl.tlct.net (216.227.126.57)  46.996 ms
>  7  pos8-1-hfr2.dfw.tlct.net (216.227.126.22)  73.053 ms  81.261 ms  94.668 ms
>  8  ge-v5-hfr1.dfw.tlct.net (216.227.126.53)  105.210 ms  75.482 ms *
>  9  209.246.152.61 (209.246.152.61)  80.207 ms  71.852 ms  72.419 ms
> 10  gigabitethernet5-0.core1.Dallas1.Level3.net (209.244.15.21)  71.978 ms  74.689 ms
>     ge-v5-hfr1.lax.tlct.net (216.227.126.49)  106.497 ms
> 11  pos8-1-hfr2.sjc.tlct.net (216.227.126.30)  133.156 ms  141.849 ms  134.666 ms
> 12  fe1-0-pbnap.sjc.tlct.net (216.227.126.169)  125.158 ms  117.098 ms  115.600 ms
> 13  pos2-1-paix.pao.tlct.net (216.227.126.210)  115.742 ms  121.505 ms  162.642 ms
> 14  gblx-px.pao.tlct.net (64.192.16.14)  124.839 ms  117.256 ms  124.204 ms
> 15  so1-0-0-622M.cr1.pao2.gblx.net (208.50.169.141)  122.668 ms  126.308 ms  138.030 ms
> 16  pos1-0-622M.cr1.HOU1.gblx.net (206.132.116.126)  121.666 ms  132.773 ms  124.458 ms
> 17  * pos0-0-155M.ar1.TPA1.gblx.net (206.132.116.98)  151.364 ms *
> 18  Customer.s4-0-0.ar1.TPA1.gblx.net (64.212.109.138)  158.682 ms  246.846 ms  158.229 ms
> 19  tower.conman.org (66.33.1.143)  161.658 ms  163.065 ms  161.680 ms
> [spc]linus:/usr/src/linux-2.0.39>
>
>

Got to see major network rerouting at work here. From 7 to 19 hops no less!
Sheesh!

[1] http://www.telocity.net/

Email author at [email protected]