From netramet-owner Tue May 5 13:45:18 1998

From netramet-owner Tue May 5 13:45:18 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id NAA09488 for netramet-outgoing; Tue, 5 May 1998 13:40:52 +1200 (NZST)
Received: from n.browlee5.itss.auckland.ac.nz (n.brownlee5.itss.auckland.ac.nz [130.216.4.79]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with SMTP id NAA09483; Tue, 5 May 1998 13:40:49 +1200 (NZST)
From: Nevil Brownlee <[email protected]>
To: [email protected], [email protected]
Subject: NetFlow and RTFM: Architecture questions
Message-ID: <[email protected]>
Date: Tue, 5 May 1998 13:40:54 +1200 (New Zealand Standard Time)
Priority: NORMAL
X-Mailer: Simeon for Win32 Version 4.1.4 Build (40)
X-Authentication: none
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
Sender: [email protected]
Precedence: bulk

Hello all:

I've sent this to the RTFM and NeTraMet mailing lists - I apologise if
you get two copies :-(

NetFlow and RTFM: Architecture questions
----------------------------------------

A Cisco Router uses NetFlow to collect data about the traffic flows
passing through it. This flow data is sent via UDP datagrams to a
user-specified destination (a port number on a host) for further
processing.

I have been working on NetFlowMet, a version of the NeTraMet meter
which uses NetFlow data as input rather than watching the network
directly. This allows flows to be observed in a router, then
collected and analysed using NeTraMet, NeMaC, etc. The development
work has raised a number of questions about the RTFM Architecture;
I'd like some feedback about them from the list, please.

1) 'Routing' Attributes

1.1) SourceASN / DestASN
NetFlow provides Source and Destination ASNs (Autonomous System
Numbers). ASN data can be 'peer AS' or 'origin AS,' and the
user must specify which when starting NetFlow on the router.

These names are confusing within RTFM:
- 'peer AS' describes the adjacent router, making it part of
a flow's RTFM Adjacent address
- 'origin AS' describes the originating host, making it part
of a flow's RTFM Peer address
A further problem is that the NetFlow data packet doesn't
indicate which form of AS is being used!

To avoid confusion, it seems better to have just one pair of
new RTFM attributes for ASN, i.e. to add 'routing' address as
a new kind of address attribute.

+ SourceASN is the source AS } As configured on
+ DestASN is the destination AS } the router

1.2) SourcePrefix / DestPrefix
These are both provided for every flow, not just flows routed via
BGP. They indicate the routing prefix for the source/dest networks
of a flow. For example, a class B network has a prefix of 16 bits.

+ SourcePrefix is the routing prefix of the source network
+ DestPrefix is the routing prefic of the destination network

1.3) MeterId
Several different routers can send data to a single NetFlow meter,
which means we need an attribute to indicate which router a flow's
data came from. I propose to call it MeterId.

I have implemented this by allowing NetFlowMet to receive data on
several different UDP ports, specified as a list.

+ MeterId is the router's (1-origin) index in the NetFlowMet
UDP ports list

(For a NeTraMet meter MeterId will always have the value 1.)

2) 'Basic' Attributes
Basic attributes are the ones described in RFC 2063 and 2064. Most
of them (Source/Dest PeerAddress, Source/Dest TransAddress, etc.)
are provided in the NetFlow data.

SourceInterface and DestInterface indicate which interfaces the
flow came in and went out on.

Adjacent Addresses, alas, can not be implemented for NetFlow.
NetFlow does provide a 'next-hop' address, which is the IP address
which the packet is sent to. In RTFM terms it's the
DestAdjacentAddress. I tried implementing this with an
AdjacentAddressType of IP4, but realised for this to work as
expected the meter needs both the source and destination Adjacent
Addresses in every packet - without this the meter can't match
the flow's S->D packets with its D->S packets.

3) What next?

NetFlowMet will be available as part of the 4.2 release of NeTraMet,
probably towards the end of May. If you're interested in testing
it please send me an email note. A beta version will be in the
beta-versions directory of the NeTraMet ftp site shortly.

Comments / suggestions please on anything mentioned above,
in particular:

* Does this group of 5 new attributes sound OK?

* Are they part of the 'Basic' architecture (in which case they
should go into the I-Ds revising RFCs 2063 and 2064)?

* Or are they really 'New' attributes (belonging in the 'New
Attributes' I-D?

* NetFlow only reports on IPv4 flows. Do the 'routing' attributes
have sensible meanings for other protocols (IPX, AT, CLNS, DECNET)?

Cheers, Nevil

+---------------------------------------------------------------------+
| Nevil Brownlee Director, Technology Development |
| Phone: +64 9 373 7599 x8941 ITSS, The University of Auckland |
| FAX: +64 9 373 7425 Private Bag 92019, Auckland, New Zealand |
+---------------------------------------------------------------------P

From netramet-owner Tue May 12 09:09:29 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id JAA22106 for netramet-outgoing; Tue, 12 May 1998 09:03:10 +1200 (NZST)
Received: from thor.iconz.co.nz ([email protected] [202.14.100.50]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with ESMTP id JAA22085 for <[email protected]>; Tue, 12 May 1998 09:03:03 +1200 (NZST)
Received: from localhost (yoper@localhost)
by thor.iconz.co.nz (8.8.8/8.8.5) with SMTP id JAA23696
for <[email protected]>; Tue, 12 May 1998 09:02:03 +1200
Date: Tue, 12 May 1998 09:02:02 +1200 (NZST)
From: "Andreas P. Hamberger" <[email protected]>
To: [email protected]
Subject: Another question
Message-ID: <[email protected]>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sender: [email protected]
Precedence: bulk

Hi Neil

Thanks for you fast reply. we didn't think about the ethernet Headers. The
-l option might help a bit ( I can't explain 20% though ).
In order to make it work I need to upgrade to
version 4.1. The problem is that Nemac doesn't seem to like the ruleset I
posted you yesterday . This ruleset worked in 3.4 and gives me now the
following output:

>>> No SET statement in rule file /opt/netramet/etc/iconz.rules
No meters to monitor !!!

We are running debian Linux 1.3 (last stable version) with the following
rule set:

#################################################################
SET 2
#
RULES
SourcePeerType & 255 = IP: PushtoAct, count_IP;
Null & 0 = 0: Ignore, 0;
#
count_IP:
SourcePeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next; # addr
DestPeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next;
SourceTransType & 255 = 0: PushPkttoAct, Next; # proto
DestTransType & 255 = 0: PushPkttoAct, Next;
SourceTransAddress & 255.255 = 0: PushPkttoAct, Next; # port
DestTransAddress & 255.255 = 0: CountPkt, 0;
#
FORMAT FlowRuleSet FlowIndex FirstTime " "
SourcePeerType SourceTransType
SourcePeerAddress SourceTransAddress
DestPeerAddress DestTransAddress " "
ToPDUs FromPDUs " " ToOctets FromOctets;
#
STATISTICS
#########################################################################

I found the same question in the mailing-list arcive, but unfortunately no
answer. Any help would be appreciated.

Andreas P. Hamberger, MA

System Administrator

##########################################################
## ##
## ICONZ - The Internet Company of New Zealand ##
## ##
## http://www.iconz.co.nz ##
## ##
## +64-9-3581186 ##
##########################################################

in business never say you are too busy
or you will miss out on the best opportunities

business has nothing to do with busy - ness
Why oper?
yoper

From netramet-owner Tue May 12 10:23:04 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id KAA05181 for netramet-outgoing; Tue, 12 May 1998 10:21:14 +1200 (NZST)
Received: from nosc.ja.net (nosc.ja.net [128.86.16.20]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with SMTP id KAA05163 for <[email protected]>; Tue, 12 May 1998 10:21:05 +1200 (NZST)
Received: from nosc.ja.net by nosc.ja.net with Internet SMTP
id <[email protected]>; Mon, 11 May 1998 23:11:15 +0100
To: "Andreas P. Hamberger" <[email protected]>
cc: [email protected]
Subject: Re: Another question
In-reply-to: Your message of "Tue, 12 May 1998 09:02:02 +1200." <[email protected]>
Date: Mon, 11 May 1998 23:11:12 +0100
Message-ID: <[email protected]>
From: Kevin Hoadley <[email protected]>
Sender: [email protected]
Precedence: bulk

> posted you yesterday . This ruleset worked in 3.4 and gives me now the
> following output:
>
> >>> No SET statement in rule file /opt/netramet/etc/iconz.rules
> No meters to monitor !!!

'No meter to monitor' often means something is wrong with the
communication between the manager and the meter. btw you're not trying a
3.x manager with a 4.x meter are you ? Ruleset number semantics changes with
version 4.x, such that though a 4.x manager can talk to a 3.x meter, the
opposite isn't true, which could explain both the errors you are seeing.

One trivial comment on the ruleset ...

> RULES
> SourcePeerType & 255 = IP: PushtoAct, count_IP;
> Null & 0 = 0: Ignore, 0;

If you are ignoring everything bar IP, is there any need to push the peer
type ? You could just gotoAct count_IP

Kevin Hoadley, JANET.

From netramet-owner Tue May 12 10:41:46 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id KAA08114 for netramet-outgoing; Tue, 12 May 1998 10:40:13 +1200 (NZST)
Received: from thor.iconz.co.nz ([email protected] [202.14.100.50]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with ESMTP id KAA08102 for <[email protected]>; Tue, 12 May 1998 10:40:09 +1200 (NZST)
Received: from localhost (yoper@localhost)
by thor.iconz.co.nz (8.8.8/8.8.5) with SMTP id KAA24112;
Tue, 12 May 1998 10:39:03 +1200
Date: Tue, 12 May 1998 10:39:03 +1200 (NZST)
From: "Andreas P. Hamberger" <[email protected]>
To: Kevin Hoadley <[email protected]>
cc: [email protected]
Subject: Re: Another question
In-Reply-To: <[email protected]>
Message-ID: <[email protected]>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sender: [email protected]
Precedence: bulk

Hi Kevin!

I am using both Nemac and Netramet 4.1 binary distribution with the new
MIB files.

Andreas P. Hamberger, MA

System Administrator

##########################################################
## ##
## ICONZ - The Internet Company of New Zealand ##
## ##
## http://www.iconz.co.nz ##
## ##
## +64-9-3581186 ##
##########################################################

in business never say you are too busy
or you will miss out on the best opportunities

business has nothing to do with busy - ness
Why oper?
yoper

On Mon, 11 May 1998, Kevin Hoadley wrote:

> > posted you yesterday . This ruleset worked in 3.4 and gives me now the
> > following output:
> >
> > >>> No SET statement in rule file /opt/netramet/etc/iconz.rules
> > No meters to monitor !!!
>
> 'No meter to monitor' often means something is wrong with the
> communication between the manager and the meter. btw you're not trying a
> 3.x manager with a 4.x meter are you ? Ruleset number semantics changes with
> version 4.x, such that though a 4.x manager can talk to a 3.x meter, the
> opposite isn't true, which could explain both the errors you are seeing.
>
> One trivial comment on the ruleset ...
>
> > RULES
> > SourcePeerType & 255 = IP: PushtoAct, count_IP;
> > Null & 0 = 0: Ignore, 0;
>
> If you are ignoring everything bar IP, is there any need to push the peer
> type ? You could just gotoAct count_IP
>
> Kevin Hoadley, JANET.
>

From netramet-owner Wed May 13 12:35:37 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id MAA02696 for netramet-outgoing; Wed, 13 May 1998 12:29:04 +1200 (NZST)
Received: from n.browlee5.itss.auckland.ac.nz (n.brownlee5.itss.auckland.ac.nz [130.216.4.79]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with SMTP id MAA02664; Wed, 13 May 1998 12:28:51 +1200 (NZST)
From: Nevil Brownlee <[email protected]>
To: Kevin Hoadley <[email protected]>
Cc: [email protected], "Andreas P. Hamberger" <[email protected]>
Subject: Re: Another question
In-Reply-To: <[email protected]>
Message-ID: <[email protected]>
Date: Wed, 13 May 1998 12:31:15 +1200 (New Zealand Standard Time)
Priority: NORMAL
X-Mailer: Simeon for Win32 Version 4.1.4 Build (40)
X-Authentication: none
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
Sender: [email protected]
Precedence: bulk

Hello all:

On Mon, 11 May 1998 23:11:12 +0100 Kevin Hoadley <[email protected]>
wrote:

> > RULES
> > SourcePeerType & 255 = IP: PushtoAct, count_IP;
> > Null & 0 = 0: Ignore, 0;
>
> If you are ignoring everything bar IP, is there any need to push the peer
> type ? You could just gotoAct count_IP

There's a good reason to always push SourcePeerType, and to include it
in every FORMAT statement - NeMaC uses its value to decide how to
display address values. If it doesn't know the PeerType is IP it will
display it as the default address type, i.e. either MAC (adjacent)
address (6 bytes hex, separated by hyphens) or CLNS (20 bytes hex).

Cheers, Nevil

+---------------------------------------------------------------------+
| Nevil Brownlee Director, Technology Development |
| Phone: +64 9 373 7599 x8941 ITSS, The University of Auckland |
| FAX: +64 9 373 7425 Private Bag 92019, Auckland, New Zealand |
+---------------------------------------------------------------------P

From netramet-owner Wed May 13 12:56:21 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id MAA06141 for netramet-outgoing; Wed, 13 May 1998 12:54:42 +1200 (NZST)
Received: from n.browlee5.itss.auckland.ac.nz (n.brownlee5.itss.auckland.ac.nz [130.216.4.79]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with SMTP id MAA06116 for <netramet@auckland>; Wed, 13 May 1998 12:54:34 +1200 (NZST)
From: Nevil Brownlee <[email protected]>
To: [email protected]
Subject: Re: Another question
Message-ID: <[email protected]>
Date: Wed, 13 May 1998 12:56:58 +1200 (New Zealand Standard Time)
Priority: NORMAL
X-Mailer: Simeon for Win32 Version 4.1.4 Build (40)
X-Authentication: none
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
Sender: [email protected]
Precedence: bulk

--- Begin Forwarded Message ---
Date: Mon, 11 May 1998 15:58:32 -0800 (PST)
From: Andy Davenport <[email protected]>
Subject: Re: Another question
To: [email protected]
Message-id: <[email protected]>
X-VMS-To: in%"[email protected]"
MIME-version: 1.0
Content-type: TEXT/PLAIN; CHARSET=US-ASCII

> One trivial comment on the ruleset ...
>
> > RULES
> > SourcePeerType & 255 = IP: PushtoAct, count_IP;
> > Null & 0 = 0: Ignore, 0;
>
> If you are ignoring everything bar IP, is there any need to push the peer
> type ? You could just gotoAct count_IP
>
> Kevin Hoadley, JANET.

In my experience, yes there is. I found that when I failed to push the
SourcePeerType, NeMaC didn't know how to format IP addresses in the
customary way and instead printed a long string of hex (albeit with
the correct IP address encoded as hex with leading zeroes.) NeMaC
apparently uses SourcePeerType (when present) to properly display
SourcePeerAddress.

Andy Davenport
Harvey Mudd College
--- End Forwarded Message ---

+---------------------------------------------------------------------+
| Nevil Brownlee Director, Technology Development |
| Phone: +64 9 373 7599 x8941 ITSS, The University of Auckland |
| FAX: +64 9 373 7425 Private Bag 92019, Auckland, New Zealand |
+---------------------------------------------------------------------P

From netramet-owner Wed May 13 16:11:39 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id QAA03918 for netramet-outgoing; Wed, 13 May 1998 16:09:29 +1200 (NZST)
Received: from char.ntnet.nt.ca (whitefish.ntnet.nt.ca [199.247.2.8]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with ESMTP id QAA03901; Wed, 13 May 1998 16:09:21 +1200 (NZST)
Received: by whitefish.ntnet.nt.ca with Internet Mail Service (5.5.1960.3)
id <2YAPNFKS>; Tue, 12 May 1998 22:09:37 -0600
Message-ID: <[email protected]>
From: Byron Hynes <[email protected]>
To: "'[email protected]'" <[email protected]>
Cc: "'Nevil Brownlee'" <[email protected]>
Subject: FW: Netramet Rule Help
Date: Tue, 12 May 1998 22:09:35 -0600
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.1960.3)
Content-Type: multipart/mixed;
boundary="---- =_NextPart_000_01BD7E24.F0EF9420"
Sender: [email protected]
Precedence: bulk

This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.

------ =_NextPart_000_01BD7E24.F0EF9420
Content-Type: text/plain;
charset="iso-8859-1"

About 3 weeks ago, I posted the following. We since discovered that our
doubling problem was primarily caused by a RAS router having the wrong
default gateway. (That is, the packets were on the wire twice, so
Netramet was doing EXACTLY what it was supposed to.)

I particularly appreciate the efforts of Mark Newman at KIRK/NTnet and
Nevil for the time they put into testing my rules.

One question remains. I thought it was possible to use a "retry" to
force the "internal" address (that is those that return a "1" from the
gosub) to always be the "source". This would make for easier processing
in the resulting database.

I thought that putting a retry in where I put the "**" below (instead of
the pushpkt) would do what I wanted, but then I stopped getting
measurements at all! I feel so out my league here, but... Is this easy
to do? Did I miss it? Or, do I just not understand the logic?

Thanks for any comments.
--
Byron Hynes, MCSE
Network Support Engineer
NTnet Operations Ltd.

NTnet Operations is a subsidiary of the NTnet Society, a non-profit
community agency.

-----Original Message-----
From: Byron Hynes [mailto:[email protected]]
Sent: Friday, April 17, 1998 4:47 PM
To: '[email protected]'
Subject: Netramet Rule Help

I have a rule-set that I thought was working, but apparently is not.

Want I wanted to capture was different levels of detail, depending on
the address: if the address was within our LAN, save all four octets. If
within the NTnet Blocks, save the "class C" address, with .0, and if it
was from anywhere else, save only "0.0.0.0"

The resulting flow files LOOK correct to me, but when I total up the
results they are vastly inflated (when compared to either our suppliers
counts [also from Netramet, but limited to class C level], or to RAS
logs for dedicated RAS networks). So, obviously, something is wrong.

I'll admit that I don't always understand the flow and logic of rules,
so I might be missing something really obvious. Any help will be
appreciated.

Byron Hynes, MCSE
Tamarack Computers Ltd.
[email protected]

#
# TAMARACK RULES FILE - For V3.2 NeTraMet.
#
SET 2

RULES
# Step 1: Consider IP packets only.
#
StartAllPacketsHere:
SourcePeerType & 255 = IP : GotoAct, StartIPHere;
Null & 0 = 0 : Ignore, 0; # Ignore all other types

# Step 2: Examine and push the packet's source
#
StartIPHere:
v1 & 0 = SourcePeerAddress : AssignAct, Next;
Null & 0 = 0 : Gosub, Classify;
SourcePeerAddress & 255.255.255.255 = 0 : PushPktToAct, LookAtDest;
SourcePeerAddress & 255.255.255.0 = 0 : PushPktToAct, LookAtDest; #
Return 2 - Free NTnet **
SourcePeerAddress & 255.255.255.0 = 0 : PushPktToAct, LookAtDest; #
Return 3 - Billable NTnet **
SourcePeerAddress & 0.0.0.0 = 0 : PushPktToAct, LookAtDest; # Return 4
- South **

# Step 3: Examine and push the packet's destination
#
LookAtDest:
v1 & 0 = DestPeerAddress : AssignAct, Next;
Null & 0 = 0 : Gosub, Classify;
DestPeerAddress & 255.255.255.255 = 0 : PushPktToAct, CountIt;
DestPeerAddress & 255.255.255.0 = 0 : PushPktToAct, CountIt; # Return
2 - Free NTnet
DestPeerAddress & 255.255.255.0 = 0 : PushPktToAct, CountIt; # Return
3 - Billable NTnet
DestPeerAddress & 0.0.0.0 = 0 : PushPktToAct, CountIt; # Return 4 -
South

# Step 4: Count the Packet
#
CountIt:
Null & 0 = 0 : Count, 0;

# Classify looks at both source and dest (in turn)
# Return 1 - Within Tamarack
# Return 2 - Non-Billable NTnet
# Return 3 - Billable NTnet
# Return 4 - South
#
Classify:
v1 & 255.255.255.0 = 199.247.52.0 : Return, 1;
v1 & 255.255.255.0 = 199.247.53.0 : Return, 1;
v1 & 255.255.255.0 = 199.247.54.0 : Return, 1;
v1 & 255.255.255.0 = 199.247.55.0 : Return, 1;
v1 & 255.255.255.0 = 199.247.76.0 : Return, 1;
v1 & 255.255.255.0 = 199.247.89.0 : Return, 2;
v1 & 255.255.248.0 = 207.148.48.0 : Return, 1;
v1 & 255.255.255.0 = 199.247.2.0 : Return, 3; # NTnet Server Net
v1 & 255.255.255.0 = 199.247.60.0 : Return, 4; # Auroranet via
Istar/cancom
v1 & 255.255.255.0 = 199.247.67.0 : Return, 4; # Auroranet via
Istar/cancom
v1 & 255.255.255.0 = 199.247.77.0 : Return, 4; # Yukon
v1 & 255.255.255.0 = 199.247.125.0 : Return, 4; # Yukon
v1 & 255.255.255.0 = 199.247.126.0 : Return, 4; # Yukon
v1 & 255.255.252.0 = 198.161.24.0 : Return, 2; # 24 - 27
v1 & 255.255.255.0 = 198.161.126.0 : Return, 2; # 126
v1 & 255.255.128.0 = 199.247.0.0 : Return, 2; # 0 - 127
v1 & 255.255.255.0 = 206.172.213.0 : Return, 2; # 213
v1 & 255.255.128.0 = 207.148.0.0 : Return, 2; # 0 - 127
Null & 0 = 0 : Return, 4; # Network is not routed by NTnet, belongs
down south

Format
FlowRuleSet FlowIndex FirstTime LastTime
SourcePeerAddress DestPeerAddress
ToPDUs FromPDUs
ToOctets FromOctets;

------ =_NextPart_000_01BD7E24.F0EF9420
Content-Type: application/octet-stream;
name="Byron P Hynes (E-mail).vcf"
Content-Disposition: attachment;
filename="Byron P Hynes (E-mail).vcf"

BEGIN:VCARD
VERSION:2.1
N:Hynes;Byron;;;
FN:Byron P Hynes (E-mail)
ORG:NTnet Operations Ltd.;NTnet Technical Support
TITLE:Lead Network Support Engineer
TEL;WORK;VOICE:(867) 669-6815
TEL;HOME;VOICE:(867) 873-4939
TEL;PAGER;VOICE:(867) 920-5551
TEL;WORK;FAX:(867) 669-7286
ADR;WORK:;Kirk Computer Systems, Ltd.;Box 613;Yellowknife;NT;X1A 2P1;Canada
LABEL;WORK;ENCODING=QUOTED-PRINTABLE:Kirk Computer Systems, Ltd.=0D=0ABox 613=0D=0AYellowknife, NT X1A 2P1=0D=0AC=
anada
EMAIL;PREF;EX:/o=KIRK Computer Systems Limited/ou=NTNET/cn=Recipients/cn=bph
REV:19980513T032541Z
END:VCARD

------ =_NextPart_000_01BD7E24.F0EF9420--

From netramet-owner Thu May 14 13:53:00 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id NAA15980 for netramet-outgoing; Thu, 14 May 1998 13:48:32 +1200 (NZST)
Received: from mail.telstra.com.au (mail.telstra.com.au [192.148.160.16]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with ESMTP id NAA15962 for <[email protected]>; Thu, 14 May 1998 13:48:25 +1200 (NZST)
Received: (from uucp@localhost) by mail.telstra.com.au (8.8.2/8.6.9) id LAA20045 for <[email protected]>; Thu, 14 May 1998 11:47:41 +1000 (EST)
Received: from mail-gw.fwall.telstra.com.au(192.148.147.16)
via SMTP by mail.telstra.com.au, id smtpd019666; Thu May 14 11:46:34 1998
Received: (from uucp@localhost) by mail-gw.fwall.telstra.com.au (8.8.2/8.6.9) id LAA20511 for <[email protected]>; Thu, 14 May 1998 11:46:32 +1000 (EST)
Received: from cdn-mail.dn.itg.telecom.com.au(144.135.109.134), claiming to be "cdn-mail.telecom.com.au"
via SMTP by mail-gw.fwall.telstra.com.au, id smtpd020292; Thu May 14 11:46:03 1998
Received: from shiva.trl.OZ.AU (shiva.trl.oz.au [137.147.20.34]) by cdn-mail.telecom.com.au (8.8.2/8.6.9) with ESMTP id LAA08533 for <[email protected]>; Thu, 14 May 1998 11:46:02 +1000 (EST)
Received: (from kowalski@localhost) by shiva.trl.OZ.AU (8.8.8/8.6.12) id LAA06075 for [email protected]; Thu, 14 May 1998 11:46:01 +1000 (EST)
Date: Thu, 14 May 1998 11:46:01 +1000 (EST)
From: Jacek Kowalski <[email protected]>
Message-Id: <[email protected]>
To: [email protected]
Subject: Improved NeTraMet performance
Sender: [email protected]
Precedence: bulk

As part of our research we've been investigating tools for Internet
performance monitoring and NeTraMet is one of them. We run it on
FreeBSD and monitor a busy fast ethernet segment (up to around 25K pps
on the average and 50K pps in bursts). The box that we use is a
Pentium II 333MHz with 128MB RAM and ultra wide fast SCSI HD. We run
both the meter and the manager on the same PC.

Following our experience of massive packet loss by NeTraMet (up to
80%) in some cases, I took a closer look at its performance. The
first thing that I noticed was that under similar traffic conditions
packet loss was strongly correlated with "fiu" ie., the number of
flows in use (I generated different number of flows by changing length
of peer address masks). I also noticed that "fiu" had very little
impact on "tsu" (number of hash entries) which seemed to converge to a
value well below HASHMOD (hash table size) and obviously big impact on
"cpt" (hash compares/hash searches). All of that was a clear
indication of not very efficient hashing. In order to get an idea
about the impact of the above on the NeTraMet performance I compiled
it with -pg option and run gprof which showed that the function
memcmp() was consuming up to 48% of the CPU time when called by
current(). It was now obvious that the meter was very busy doing
linear searches of long flow lists and lagged behind the BPF that
consequently had to drop packets.

In order to fix this problem I changed the bits of code that
corresponds to the peer address hashing by introducing weights for the
octets (which unclumped the IP addresses having the same sum of
octets) and divided hash_search modulo HASHMOD (now I could get rid of
bitwise & with HASHMOD in current()). The result was a dramatic
improvement in "aps" (average packet loss), it had gone down by more
than an order of magnitude. "cpt" had also declined to 1.1 - 1.4. At
the same time the CPU usage by NeTraMet had decreased from over 99% to
around 50%. In order to further improve the performance I increased
HASHMOD to 65536 which further reduced "aps" to up to 4-5% in heavy
traffic.

In order to obtain stable flow queuing conditions I had to decrease
the garbage collection interval from default 10 to 3 sec. and increase
the number of flow indexes to test during garbage collection from 4 to
12 (but this was not a problem since I had plenty of CPU headroom).

I am still in the process of testing different hash functions and
optimising their parameters and will post my improved code as soon as
I'm finished. I would appreciate any suggestion for good functions for
hashing IP addresses.

DISCLAIMER: The above comments do not necessarily represent the
official view of the Telstra Corporation and do not imply endorsement
by Telstra of the NeTraMet software for the purpose of Internet
monitoring.

The comments above are provided to assist the implementation of the
RTFM Group standard architecture for traffic flow measurement
software.

---------------------------------------------------------------------------
Jacek Kowalski, Ph.D.,Project Leader * e-mail: [email protected]
Telstra Research Laboratories * phone: +61-3-92536322
770 Blackburn Rd., Clayton, VIC 3168 * fax: +61-3-92536777
AUSTRALIA *

From netramet-owner Thu May 14 14:41:33 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id OAA23154 for netramet-outgoing; Thu, 14 May 1998 14:41:19 +1200 (NZST)
Received: from mail.telstra.com.au (mail.telstra.com.au [192.148.160.16]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with ESMTP id OAA23131 for <[email protected]>; Thu, 14 May 1998 14:41:12 +1200 (NZST)
Received: (from uucp@localhost) by mail.telstra.com.au (8.8.2/8.6.9) id MAA07654 for <[email protected]>; Thu, 14 May 1998 12:40:27 +1000 (EST)
Received: from mail-gw.fwall.telstra.com.au(192.148.147.16)
via SMTP by mail.telstra.com.au, id smtpd006726; Thu May 14 12:37:58 1998
Received: (from uucp@localhost) by mail-gw.fwall.telstra.com.au (8.8.2/8.6.9) id MAA07820 for <[email protected]>; Thu, 14 May 1998 12:37:55 +1000 (EST)
Received: from cdn-mail.dn.itg.telecom.com.au(144.135.109.134), claiming to be "cdn-mail.telecom.com.au"
via SMTP by mail-gw.fwall.telstra.com.au, id smtpd007682; Thu May 14 12:37:16 1998
Received: from shiva.trl.OZ.AU (shiva.trl.oz.au [137.147.20.34]) by cdn-mail.telecom.com.au (8.8.2/8.6.9) with ESMTP id MAA22825 for <[email protected]>; Thu, 14 May 1998 12:37:15 +1000 (EST)
Received: (from kowalski@localhost) by shiva.trl.OZ.AU (8.8.8/8.6.12) id MAA13719 for [email protected]; Thu, 14 May 1998 12:37:14 +1000 (EST)
Date: Thu, 14 May 1998 12:37:14 +1000 (EST)
From: Jacek Kowalski <[email protected]>
Message-Id: <[email protected]>
To: [email protected]
Subject: Re: Improved NeTraMet performance
Sender: [email protected]
Precedence: bulk

In my previous posting I incorrectly used "aps" (average pps) instead
of "apl" the average packet loss which is the acronym that I use.

Sorry,
Jacek

From netramet-owner Thu May 14 18:26:12 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id SAA19762 for netramet-outgoing; Thu, 14 May 1998 18:25:21 +1200 (NZST)
Received: from zeus.centaur.de (zeus.centaur.de [194.120.119.100]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with ESMTP id SAA19752 for <[email protected]>; Thu, 14 May 1998 18:25:16 +1200 (NZST)
Received: from localhost (erhardt@localhost)
by zeus.centaur.de (8.8.7/8.8.7) with SMTP id KAA06600;
Thu, 14 May 1998 10:19:53 +0200
X-Authentication-Warning: zeus.centaur.de: erhardt owned process doing -bs
Date: Thu, 14 May 1998 10:19:53 +0200 (MEST)
From: Thomas Erhardt <[email protected]>
To: Byron Hynes <[email protected]>
cc: "'[email protected]'" <[email protected]>
Subject: Re: FW: Netramet Rule Help
In-Reply-To: <[email protected]>
Message-ID: <[email protected]>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sender: [email protected]
Precedence: bulk

> About 3 weeks ago, I posted the following. We since discovered that our
> doubling problem was primarily caused by a RAS router having the wrong
> default gateway. (That is, the packets were on the wire twice, so
> Netramet was doing EXACTLY what it was supposed to.)

If I understand you, then I can't use netramet with the following setup:

Internet----Router1-----------------------------------------------
| | |
| | |
DialUp- Router2 Meter
Router |
| |
| Company's
| WWW-
DialUp- net
Clients

Router1, Router2, DialUp-Router and Meter are in the (ether)net, on that
traffic has to be accounted,
Meter is running Netramet,
default-route of Router2 points to Router1:

packets from DialUp-Clients destined to WWW-net go to DialUp-Router, then
to Router2, then to a host in WWW-net,
the answers go from the host in WWW-net to Router2, then to Router1, then
to DialUp-Router, then to the Client
so the packets are on the wire twice...

Thanks for your help,
Thomas Erhardt

From netramet-owner Thu May 14 21:29:55 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id VAA00403 for netramet-outgoing; Thu, 14 May 1998 21:29:11 +1200 (NZST)
Received: from nosc.ja.net (nosc.ja.net [128.86.16.20]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with SMTP id VAA00394; Thu, 14 May 1998 21:29:02 +1200 (NZST)
Received: from nosc.ja.net by nosc.ja.net with Internet SMTP
id <[email protected]>; Thu, 14 May 1998 10:28:45 +0100
To: Nevil Brownlee <[email protected]>
cc: [email protected]
Subject: Re: Another question
In-reply-to: Your message of "Wed, 13 May 1998 12:56:58 +1200." <[email protected]>
Date: Thu, 14 May 1998 10:28:37 +0100
Message-ID: <[email protected]>
From: Kevin Hoadley <[email protected]>
Sender: [email protected]
Precedence: bulk

> > If you are ignoring everything bar IP, is there any need to push the peer
> > type ? You could just gotoAct count_IP
>
> In my experience, yes there is. I found that when I failed to push the
> SourcePeerType, NeMaC didn't know how to format IP addresses in the
> customary way and instead printed a long string of hex (albeit with
> the correct IP address encoded as hex with leading zeroes.) NeMaC
> apparently uses SourcePeerType (when present) to properly display
> SourcePeerAddress.

Interesting ... despite both this reply and Nevil's, in 2.5 years of using
NeTraMet in a pure IP enviroment I've never had to push the PeerType.

I think the actual behaviour may be a little more bizarre. When we coded our
NetFlow variant of NeTraMet, we found that rulesets that had worked on the
conventional version of NeTraMet failed to work on NetFlowMet. Further
investigation seemed to suggest that the meter wasn't recognising the
traffic as IP. Looking at the code for the meter, it seems that if you
load a ruleset that matches adjacent address, the meter then looks for all
peer types, otherwise it only looks for the peer types requested (see
open_rule_set in meter/flowhash.c). Our normal rulesets had always matched
(but not pushed) adjacent addresses, thus the meter was looking for IP (and
all other peer types), which meant that the handle_pkt routine successfully
parsed IP packets, even though there was no check or push of the peer type
(we could get away with this because we were monitoring an IP only
router interconnect, with odd non-IP protocols like CDP turned off). With
the NetFlow version of NeTraMet however there is no concept of adjacent
address, so the rules needed to be modified to check (though again not push)
against peer type = IP to persuade the meter to pick up IP traffic.

However all this is on the meter side, persuading it to pick up IP traffic.
We've never had any problems on the NeMaC side with it not being able to
format IP addresses once collected by the meter, even though we never
push the peer type.

Odd.

Kevin Hoadley, JANET.

PS. I wonder whether this depends on whether the software was compiled with
CLNS support, which we always turn off ? (I've had my fill of NSAPs, be it
CLNS or CONS ...)

From netramet-owner Thu May 14 21:45:20 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id VAA01196 for netramet-outgoing; Thu, 14 May 1998 21:44:59 +1200 (NZST)
Received: from zeus.centaur.de (zeus.centaur.de [194.120.119.100]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with ESMTP id VAA01180 for <[email protected]>; Thu, 14 May 1998 21:44:53 +1200 (NZST)
Received: from localhost (erhardt@localhost)
by zeus.centaur.de (8.8.7/8.8.7) with SMTP id NAA09394;
Thu, 14 May 1998 13:39:26 +0200
X-Authentication-Warning: zeus.centaur.de: erhardt owned process doing -bs
Date: Thu, 14 May 1998 13:39:25 +0200 (MEST)
From: Thomas Erhardt <[email protected]>
To: Torsten Naumann <[email protected]>
cc: Byron Hynes <[email protected]>,
"'[email protected]'" <[email protected]>
Subject: Re: packets twice on the net (was: Netramet Rule Help)
In-Reply-To: <[email protected]>
Message-ID: <[email protected]>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sender: [email protected]
Precedence: bulk

On Thu, 14 May 1998, Torsten Naumann wrote:

> On Thu, 14 May 1998, Thomas Erhardt wrote:
>
> > Internet----Router1-----------------------------------------------
> > | | |
> > | | |
> > DialUp- Router2 Meter
> > Router |
> > | |
> > | Company's
> > | WWW-
> > DialUp- net
> > Clients
> >
> >
[schnipp]
> > the answers go from the host in WWW-net to Router2, then to Router1, then
> > to DialUp-Router, then to the Client
> > so the packets are on the wire twice...
>
> Not if your routers understand ICMP Redirects. If your routers
> understand this then only the first packet goes twice over the wire
> and then Router1 send an ICMP Redirect to Router2 that there is an
> better route. So the packets go from WWW-net to Router2 from there to
> DialUpRouter. The router holds this new route til reset/power off or
> something else of this kind. After such a learning you should see the
> new route on your router if you do "route -n" or a related command.
>

True, but...
Router2 is a Firewall and does not accept ICMP redirects for security
reasons. Maybe it should accept redirects from the other Routers
in the net.

> # Wenn wir UNIX von dieser Welt entfernen, steht sie morgen still.
> # Wenn wir dasselbe mit Microsoft Office tun, wird sich die
> # Produktivitaet raketengleich steigern. Scott McNealy

:-)

Thomas Erhardt

From netramet-owner Fri May 15 00:29:46 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id AAA10077 for netramet-outgoing; Fri, 15 May 1998 00:27:33 +1200 (NZST)
Received: from zeus.centaur.de (zeus.centaur.de [194.120.119.100]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with ESMTP id AAA10068 for <[email protected]>; Fri, 15 May 1998 00:27:29 +1200 (NZST)
Received: from localhost (erhardt@localhost)
by zeus.centaur.de (8.8.7/8.8.7) with SMTP id QAA11854;
Thu, 14 May 1998 16:20:59 +0200
X-Authentication-Warning: zeus.centaur.de: erhardt owned process doing -bs
Date: Thu, 14 May 1998 16:20:58 +0200 (MEST)
From: Thomas Erhardt <[email protected]>
To: Alan Brown <[email protected]>
cc: Byron Hynes <[email protected]>,
"'[email protected]'" <[email protected]>
Subject: Re: FW: Netramet Rule Help
In-Reply-To: <Pine.LNX.3.96.980514235418.15843G-100000@mailhost.manawatu.net.nz>
Message-ID: <[email protected]>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sender: [email protected]
Precedence: bulk

On Thu, 14 May 1998, Alan Brown wrote:

> Why not run OSPF or set the routers to use ICMP discovery?

Yes, I'll have to setup something like that...

thanks,
Thomas Erhardt

From netramet-owner Sun May 17 00:16:38 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id AAA28309 for netramet-outgoing; Sun, 17 May 1998 00:13:11 +1200 (NZST)
Received: from kaukau.mcs.vuw.ac.nz (kaukau.mcs.vuw.ac.nz [130.195.5.20]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with ESMTP id AAA28304 for <[email protected]>; Sun, 17 May 1998 00:13:09 +1200 (NZST)
Received: from rialto.mcs.vuw.ac.nz (rialto.mcs.vuw.ac.nz [130.195.5.15])
by kaukau.mcs.vuw.ac.nz (8.8.6/8.8.6) with ESMTP id AAA01482
for <[email protected]>; Sun, 17 May 1998 00:13:08 +1200 (NZST)
From: Aaron Roydhouse <[email protected]>
Received: from localhost (aaron@localhost) by rialto.mcs.vuw.ac.nz (8.8.6/8.7.3) with SMTP id AAA08756 for <[email protected]>; Sun, 17 May 1998 00:13:07 +1200 (NZST)
Message-Id: <[email protected]>
To: [email protected]
Subject: NeMaC binary for AIX 4?
Date: Sun, 17 May 1998 00:13:07 +1200
Sender: [email protected]
Precedence: bulk

I would like to use NeMaC on an AIX box, but it only has a horrid
run-time installation of AIX 4.1.3, thus severely limiting my chances
of building anything. I wondered if someone on the list who has built
NeMaC for AIX has a (PowerPC) binary I can use?

Aaron.

From netramet-owner Tue May 19 04:57:48 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id EAA12242 for netramet-outgoing; Tue, 19 May 1998 04:53:55 +1200 (NZST)
Received: from igw3.watson.ibm.com (igw3.watson.ibm.com [198.81.209.18]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with ESMTP id EAA12235 for <[email protected]>; Tue, 19 May 1998 04:53:51 +1200 (NZST)
From: [email protected]
Received: from mailhub.watson.ibm.com (mailhub.watson.ibm.com [9.2.250.97]) by igw3.watson.ibm.com (8.8.7/07-11-97) with ESMTP id MAA08076 for <[email protected]>; Mon, 18 May 1998 12:53:48 -0400
Received: from yktvmv.watson.ibm.com (yktvmv.watson.ibm.com [9.117.33.29]) by mailhub.watson.ibm.com (8.8.7/Feb-20-98) with SMTP id MAA10624 for <netramet%[email protected]>; Mon, 18 May 1998 12:53:46 -0400
Message-Id: <[email protected]>
Received: from YKTVMV by yktvmv.watson.ibm.com (IBM VM SMTP V2R4)
with BSMTP id 0898; Mon, 18 May 98 12:53:20 EDT
Date: Mon, 18 May 98 12:34:26 EDT
To: [email protected]
cc: [email protected]
Subject: Bug in nmc.c (Revision 1.9)
Sender: [email protected]
Precedence: bulk

Hi,

There is an array indexing error in the latest beta version of nmc.c, near
line 600. In the snippet shown just below:

else if (community[0] == '\0') {
strncpy(community,argv[a], NAME_LN);
community[ NAME_LN] = '\0';
}

I believe that the last line should be changed to:

community[NAME_LN-1] = '\0';

To keep from clobbering the first byte of the adjacent variable.

Stephen Stibler

From netramet-owner Tue May 19 12:53:35 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id MAA02814 for netramet-outgoing; Tue, 19 May 1998 12:51:40 +1200 (NZST)
Received: from n.browlee5.itss.auckland.ac.nz (n.brownlee5.itss.auckland.ac.nz [130.216.4.79]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with SMTP id MAA02807 for <[email protected]>; Tue, 19 May 1998 12:51:35 +1200 (NZST)
From: Nevil Brownlee <[email protected]>
To: [email protected]
Subject: fwd: BOUNCE [email protected]: Non-member submission from [Alan Brown <[email protected]>]
Message-ID: <[email protected]>
Date: Tue, 19 May 1998 12:49:19 +1200 (New Zealand Standard Time)
Priority: NORMAL
X-Mailer: Simeon for Win32 Version 4.1.4 Build (40)
X-Authentication: none
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
Sender: [email protected]
Precedence: bulk

--- Begin Forwarded Message ---
Date: Fri, 15 May 1998 00:52:49 +1200 (NZST)
From: Alan Brown <[email protected]>
X-Sender: [email protected]
Reply-To: Alan Brown <[email protected]>
To: Thomas Erhardt <[email protected]>
cc: Torsten Naumann <[email protected]>,
Byron Hynes <[email protected]>,
"'[email protected]'" <[email protected]>
Subject: Re: packets twice on the net (was: Netramet Rule Help)
In-Reply-To: <[email protected]>
Message-ID: <Pine.LNX.3.96.980515003802.15843M-100000@mailhost.manawatu.net.nz>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII

On Thu, 14 May 1998, Thomas Erhardt wrote:

> True, but...
> Router2 is a Firewall and does not accept ICMP redirects for security
> reasons. Maybe it should accept redirects from the other Routers
> in the net.

All admins should be firewalling all local and "martian" IP source
addresses from coming in externally. You should also be firewalling
your routers so that only the internal source IP addresses they
gateway for can get out - inbound and outbound filters should be
applied at _all_ routers inside an organisation as well as at
external gateways.

Doing this will solve a number of potential security problems and
reduce complexity tracing rogue traffic internally as well as
being a "good neighbour" policy inasmuch as forged IP source
addresses can't escape your own network.

In combination, these should reduce authorised ICMP redirects
seen by your routers to trceable and LARTable sources.

Refer: RFC2267 ("Network Ingress Filtering") and
http://www.cert.org/advisories/CA-96.21.tcp_syn_flooding.html

AB

--- End Forwarded Message ---

From netramet-owner Tue May 19 14:42:17 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id OAA18040 for netramet-outgoing; Tue, 19 May 1998 14:41:15 +1200 (NZST)
Received: from n.browlee5.itss.auckland.ac.nz (n.brownlee5.itss.auckland.ac.nz [130.216.4.79]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with SMTP id OAA18020; Tue, 19 May 1998 14:41:03 +1200 (NZST)
From: Nevil Brownlee <[email protected]>
To: Jacek Kowalski <[email protected]>
Cc: [email protected]
Subject: Re: Improved NeTraMet performance
In-Reply-To: <[email protected]>
Message-ID: <[email protected]>
Date: Tue, 19 May 1998 14:38:47 +1200 (New Zealand Standard Time)
Priority: NORMAL
X-Mailer: Simeon for Win32 Version 4.1.4 Build (40)
X-Authentication: none
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
Sender: [email protected]
Precedence: bulk

Hello Jacek:

I'm delighted to see your note on NeTraMet hashing performance. What's
in the implementation now is a 'keep it simple, make sure it works'
approach, with instrumentation to help tune it 'one day soon ..'

> In order to fix this problem I changed the bits of code that
> corresponds to the peer address hashing by introducing weights for the
> octets (which unclumped the IP addresses having the same sum of
> octets) and divided hash_search modulo HASHMOD (now I could get rid of
> bitwise & with HASHMOD in current()). The result was a dramatic
> improvement in "aps" (average packet loss), it had gone down by more
> than an order of magnitude. "cpt" had also declined to 1.1 - 1.4. At
> the same time the CPU usage by NeTraMet had decreased from over 99% to
> around 50%. In order to further improve the performance I increased
> HASHMOD to 65536 which further reduced "aps" to up to 4-5% in heavy
> traffic.
>
> In order to obtain stable flow queuing conditions I had to decrease
> the garbage collection interval from default 10 to 3 sec. and increase
> the number of flow indexes to test during garbage collection from 4 to
> 12 (but this was not a problem since I had plenty of CPU headroom).

Thinking about your comments, I have some further thoughts ..

* The hash for each flow includes all the attributes which were pushed
in the ruleset, AFTER they have been masked. When you have lots
of attributes pushed the simple sum works reasonably well, but it
certainly isn't optimal!

* Source and Dest Peer Address will certainly be pushed for nearly all
rulesets, so its a very good idea to weight the lower bytes; my guess
would be left shifts of say 6 bits for the fourth byte and 4 bits
for the third byte - what have you been using? One problem with
the fourth byte is that many rulesets will mask it out ..

* Using a larger hash table will certainly help. The Unix
meter defaults are 10,000 flows and an 8192-byte hash table; this
sound about right. If the user specifies more flows, it would be a
good idea to increas the hash table size, e.g. to the minimum of
64K and the first power of two below the number of flows.

* Source and Dest Trans Address are also very often used in rulesets.
For IP these are the port numbers, which means that well-known ports
will appear often. It might be interesting to weight the low-order
byte of these too, but I haven't any feeling for suitable wieghts.

I'll be very interested to see what algorithm seems to work best for
you, and will certainly put it into the NeTraMet release files.

Cheers (and thanks), Nevil

+---------------------------------------------------------------------+
| Nevil Brownlee Director, Technology Development |
| Phone: +64 9 373 7599 x8941 ITSS, The University of Auckland |
| FAX: +64 9 373 7425 Private Bag 92019, Auckland, New Zealand |
+---------------------------------------------------------------------P

From netramet-owner Wed May 20 20:46:20 1998
Received: (from majordom@localhost) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) id UAA00323 for netramet-outgoing; Wed, 20 May 1998 20:39:47 +1200 (NZST)
Received: from mail.telstra.com.au (mail.telstra.com.au [192.148.160.16]) by mailhost.auckland.ac.nz (8.8.5/8.7.3-ua) with ESMTP id UAA00316; Wed, 20 May 1998 20:39:42 +1200 (NZST)
Received: (from uucp@localhost) by mail.telstra.com.au (8.8.2/8.6.9) id SAA10536; Wed, 20 May 1998 18:39:08 +1000 (EST)
Received: from mail-gw.fwall.telstra.com.au(192.148.147.16)
via SMTP by mail.telstra.com.au, id smtpd010350; Wed May 20 18:38:20 1998
Received: (from uucp@localhost) by mail-gw.fwall.telstra.com.au (8.8.2/8.6.9) id SAA05374; Wed, 20 May 1998 18:38:12 +1000 (EST)
Received: from cdn-mail.dn.itg.telecom.com.au(144.135.109.134)
via SMTP by mail-gw.fwall.telstra.com.au, id smtpd025630; Wed May 20 17:17:01 1998
Received: from shiva.trl.OZ.AU (shiva.trl.oz.au [137.147.20.34]) by cdn-mail.dn.itg.telecom.com.au (8.8.2/8.6.9) with ESMTP id RAA15369; Wed, 20 May 1998 17:16:34 +1000 (EST)
Received: (from kowalski@localhost) by shiva.trl.OZ.AU (8.8.8/8.6.12) id OAA28752; Wed, 20 May 1998 14:51:26 +1000 (EST)
Date: Wed, 20 May 1998 14:51:26 +1000 (EST)
From: Jacek Kowalski <[email protected]>
Message-Id: <[email protected]>
To: [email protected]
Subject: Re: Improved NeTraMet performance
Cc: [email protected]
Sender: [email protected]
Precedence: bulk

To: [email protected]
Subject: Re: Improved NeTraMet performance

Nevil,

> Thinking about your comments, I have some further thoughts ..
>
> * The hash for each flow includes all the attributes which were pushed
> in the ruleset, AFTER they have been masked. When you have lots
> of attributes pushed the simple sum works reasonably well, but it
> certainly isn't optimal!

I'm doing some off-line investigations using a sample of over 150K IP
addresses external to our WAN. The rule file used to collect them was
very simple: check the mac addresses to see if the traffic goes
outside or comes from the outside of our WAN; if it goes outside then
push DestPeerAddress; if it comes from the outside then push
SourcePeerAddress (ie., our WAN is a single src/dest). This gives
'pure' single IP address hashing. When I used the simple sum with the
default Unix HASHMOD (8192) I only obtained 970 different search_hash
values, ie., 11.84% hash table utilisation. When I used weighting I
got 100% hash table utilisation. Below are some stats for the
conditional distributions of linked list lengths (ie. collisions)
given a non-NULL list (ie. non-empty bucket):

Simple sum:

Min. 1st Qu. Median Mean 3rd Qu. Max.
1 26 136 154.8 288.5 392

Weighted sum:

Min. 1st Qu. Median Mean 3rd Qu. Max.
5 15 18 18.33 21 35

(Qu. stands for quartile).
As you can see the average number of collisions went down from 154.8 to
18.33 with the maximum down from 392 to 35.
When I increased HASHMOD to 65536 the hash table utilisation for the simple
sum obviously went down to 1.48% but the utilisation for the weighted sum was
84.8% with (what's more important) the corresponding stats:

Min. 1st Qu. Median Mean 3rd Qu. Max.
1 1 2 2.702 4 11

> * Source and Dest Peer Address will certainly be pushed for nearly all
> rulesets, so its a very good idea to weight the lower bytes; my guess
> would be left shifts of say 6 bits for the fourth byte and 4 bits
> for the third byte - what have you been using? One problem with
> the fourth byte is that many rulesets will mask it out ..

Unfortunately simple shifting which is equivalent to weighting with
powers of 2 and HASHMOD being a power of 2 is less efficient than
using prime numbers for weighting (it's a pity since that would be a
nice hack :-( ). It shows a cyclic behaviour with quite heavy clusters
of collisions. Here are some example stats for this case:

Min. 1st Qu. Median Mean 3rd Qu. Max.
3 11 14 18.33 24 296

However, shifting for weighting and making HASHMOD a prime number
improves this quite a bit (so part of the hack may be preserved :-) ).
Here's an example

Min. 1st Qu. Median Mean 3rd Qu. Max.
6 15 18 18.36 21 37

The mean here is a bit higher due to smaller HASHMOD 8179. In the
former case there were 395 buckets with more than 37 collision which
is max for the latter.

I weigh bytes 1-3. This way I don't have to do anything to byte 4
which may be chopped off. Note that masking out byte 4 is not really a
problem since it reduces the number of flows in use and therefore
we have many fewer flows to hash.

> * Using a larger hash table will certainly help. The Unix
> meter defaults are 10,000 flows and an 8192-byte hash table; this
> sound about right. If the user specifies more flows, it would be a
> good idea to increas the hash table size, e.g. to the minimum of
> 64K and the first power of two below the number of flows.

Sure it does. See above.

> * Source and Dest Trans Address are also very often used in rulesets.
> For IP these are the port numbers, which means that well-known ports
> will appear often. It might be interesting to weight the low-order
> byte of these too, but I haven't any feeling for suitable wieghts.
>
> I'll be very interested to see what algorithm seems to work best for
> you, and will certainly put it into the NeTraMet release files.

Unfortunately I have to keep you in suspense as far as my final
algorithm is concerned :-(. I do my off-line testing using S-plus
which runs a bit slow. I may cut a few lines of C code to speed up the
testing.

I suppose that adding more functionality to the garbage collector
could further improve NeTraMet's performance. Since some flows carry
more packets (and therefore are more frequently searched for) than the
others, they could possibly be pushed up closer to the top of the list
during garbage collection, if they are not retrieved obviously.
However, I don't have a clear idea of a simple and efficient algorithm
for doing this. Maybe pushing up 2 or 3 largest flows would do.

Cheers,
Jacek

The disclaimer from my first posting in this thread still applies :-(