* * * * *

                       The dreaded “Early Morning Call”

Ah, nothing like getting a call at the ungodly hour of 10:30 to fix a munged
firewall. For reasons I'm still trying to figure out it apparently crashed
last night and the fix was a power cycle.

But in looking over the log files, I kept seeing:

> Jul  2 08:48:07 localhost kernel: SPOOF-SRC-172 IN=eth0 OUT=eth1
>       SRC=172.21.159.36 DST=XXX.XXX.XXX.XXX LEN=40  TOS=0x00
>       PREC=0x00 TTL=238 ID=0 DF PROTO=TCP SPT=11100 DPT=1516
>       WINDOW=0 RES=0x00 RST URGP=0
>

I had programmed the firewall to reject any private IP (Internet Protocol)
address (since you're not supposed to see private IP addresses on the
Internet at large) yet here I was, seeing a private, albeit a less known
private, IP address. Only the source address kept changing for each packet
logged, but always within the 172.21.0.0/24 block.

This meant that the address was originating somewhere on our network, but
where?

Pinging one of the private IP addresses from the router directly connected to
the firewall didn't show anything. It seems that the ping packets were sent
to the main core router, which just dumps private IP addresses to the bit
bucket.

But why was that router throwing anything from 172.21.0.0 to the firewall?

Now, the rule I have for filtering these packets in iptables looks like:

> iptables --table filter --append FORWARD --source 172.16.0.0/12
>       --jump LOG --log-prefix "SPOOF-SRC-172 "
>
> iptables --table filter --append FORWARD  --source 172.16.0.0/12
>       --jump REJECT
>
> iptables --table filter --append FORWARD --destination 172.16.0.0/12
>       --jump LOG --log-prefix "SPOOF-DST-172 "
>
> iptables --table filter --append FORWARD --destination 172.16.0.0/12
>       --jump REJECT
>

which doesn't record the MAC (Media Access Control) address (which uniquely
identifies an Ethernet interface on a network). However, by adding:

> iptables --table mangle --append PREROUTING --source 172.16.0.0/12
>       --jump LOG --log-prefix "WHAT "
>

the MAC address is logged.

Cool!

> Jul  3 17:48:24 localhost kernel: WHAT IN=eth0
>       OUT=MAC=00:10:e0:02:09:dd:00:02:85:12:a7:c0:08:00
>       SRC=172.21.117.49 DST=XXX.XXX.XXX.XXX LEN=40
>       TOS=0x00 PREC=0x00 TTL=239 ID=0 DF PROTO=TCP
>       SPT=11112 DPT=3716 WINDOW=0 RES=0x00 RST URGP=0
>

What's recorded is the MAC address of the interface (the first six octets),
followed by the MAC address the packet came from (the next six octets)
followed by the protocol (the final two octets, and in this case, it's an IP
packet).

Now, the first three octets of a MAC address are the vendor code—who makes
that particular Ethernet device. Easy enough to look up [1] and—

A Riverstone?

It's the router?

There's nothing on our network that uses the 172.16.0.0/12 private IP space.
And yet, here's one of our routers, sprewing forth garbage packets.

Sigh.

Time to get with Dan the Network Engineer on this …

Update on Thursday, July 5^th, 2007

Between Dan the Network Engineer and I, the problem was solved [2].


[1] http://coffer.com/mac_find/
[2] gopher://gopher.conman.org/0Phlog:2007/07/05.1

Email author at [email protected]