IP Provider Metrics BOF (IPPM)
Reported by Stan Barber/Academ Consulting Services
Administrative Items
The IPPM BOF was chaired by Matt Mathis. The minutes were recorded and
reported by Stan Barber with a final review by Matt Mathis. The session
was broadcast live on the MBONE.
The group's mailing list will be ready soon. Subscribe by sending mail
to
[email protected]. Attendees of this meeting will not
automatically be added to the list.
Most of the slides presented at the session were text only and have been
imbedded in the minutes.
Consider a Possible Working Group
o Is the working group needed?
o Does it conflict/interact with any other current or past groups?
o What do we want to cover?
o Chair issues and creating an editor/secretary.
o What should the charter say?
The primary topic for this BOF was the formation of a working group,
however, this was not covered until the end of the session. The group
first discussed several of the potential technical work items.
General Requirements for Metrics
What makes a good metric? How do we design a metric that works for all
the users and providers?
o It must be fair for uniform technology. It should be as fair as
possible for non-uniform technology. Note that properly accounting
for intrinsic differences in technology is difficult.
o It should be non-pejorative in the sense that is should not imply
quality. It should support providers who are selling services that
are ``inexpensive yet good enough'' as well as ``the best
possible.''
o It should be a true metric, providing concrete, repeatable
measurements without subjective interpretation (e.g., a tape
measure).
o It should be useful for publication in data sheets. This means
that it is repeatable by anyone. It should have a capability to
measure one provider independent of all others. This is also hard
to do.
o It should be useful for diagnostics. It should be able to
sectionalize a multi-provider path.
o It should be useful for procurement, contracts and RFPs.
o It must be owned ``by the community,'' or more specifically, not
encountered by some company or individual.
What Metrics Are Needed? (A Wish List)
o Path Performance
- Bulk data transfer performance (FTP, etc.)
This is most important to the SuperComputing Centers (see treno
below for a possible diagnostics tool)
- Interactive performance (Telnet, X11, etc.)
- Real-time and multicast performance (delay and loss statistics)
- Packet delay and loss should be considered
- Reliability -- should be considered on its own and not as a
factor that is a second order effect of something else
o Routing stability and robustness
- End-to-end availability
- Time to recover (e.g., switch to backup paths)
- Route stability (spurious route transitions, etc.)
- Coverage (Is the entire Internet reachable?)
- Failure isolation (Do failures cascade, say due to insufficient
routing resources?)
- Are routes aggregated appropriately?
A variety of routing metrics were suggested. This should include
route stability, some notion of reachability/coverage as well as
the degree of route aggregation in the routing system. Routing
diameter should also be considered. (There is more on this later
in these minutes.)
o DNS performance and robustness
o Infrastructure security (protection from sniffing and various
spoofing attacks)
o NOC responsiveness
- Trouble reports
- Security incidents
- DNS problems
How long does it take to fix problems that involve DNS caching?
How long does it take to resolve trouble tickets? How long does it
take to resolve security problems?
There is also some concern that some of these are product or business
related. It was suggested that there be a core and secondary sets of
metrics. Matt believes that the core should only be whatever is really
necessary to deliver packets, including IP and the associated routing
system. Others believe that customer perceptions have to be considered
and this includes the NOC and other difficult-to-measure properties.
It has been suggested that the broader list is really a reflection of
user expectations and requirements. Different user expectations exist
and need to be addressed in some way. Matt thinks that some of this
discussion reflects the differences between the roles of an IP
wholesaler and retailer. It makes sense to start working on the core
metrics and work towards a broader set. Therefore, The IPPM charter
should include the broader metrics, but since they are ``fuzzy'' and
subject to non-consensus associated with subjective measures, they
should be ``tabled'' until after the core metrics are assured
convergence.
Existing Performance Tools
o traceroute and its derivatives
o ping and its derivatives
o ttcp
o tcpdump
o Routing logs and analysis tools
o rtpqual -- multicast sequence number counter (holes in the sequence
space)
o off-net from MERIT
o mrt -- the multithreaded routing toolkit from Merit
o Protocol analyzers, hardware tools and other ``test gear''
o Routing registry and analysis tools related to it
o SATAN port scanning
o Many, many SNMP based tools
Existing tools were listed to give the group confidence that they had a
complete metrics wish list.
A New Tool: Treno, A Traceroute/Reno TCP Derivative
It is an example of a bulk transfer metric. It provides a lot of useful
information. It provides a reasonable mimicry of TCP, but does make use
if ICMP just like regular traceroute. Christian Huitema said that ICMP
was not intended for performance measurement, so as long as this is the
case, ICMP does not provide a good mechanism for doing measurement.
There is considerable variety among routing vendors in their ICMP
processing. Since some vendors do respond to ICMP as fast as they
forward packets and some do not, this may cause vendors' equipment to be
at a disadvantage because of choices the ICMP responses architecture. A
routing requirements working group does exists where this can be
discussed. Some folks suggested that the RA machines might be used as
landmark machines. Bill Manning said that should not be done. Other
landmark machines could be identified.
What is the future of treno? Does it really emulate idealized TCP? If
not, vendors may optimize their software to make it work well with the
tool and not really help make the network work better. Sally Floyd
volunteered to work with Matt to see if treno does emulate an
appropriate TCP. How do we use it for testing? There are issues in
creating a mechanism to do testing (a standard procedure). John Boone
volunteered to work on the development of a standard testing procedure
to measure bulk transfer using treno.]
Routing Metrics -- Bill Manning
With the new Internet Architecture, different sites will use different
paths to get to the same places. The RA mission is to insure a
consistent set of routes for all users. By using a RIPE 181-based
database and through an analysis, a consistent routing configuration can
be presented.
There are a number of things that might be measured:
o The level of aggregation
o Route persistence (or route flaps) [It was suggested that there is
a rate of 30 transitions per second in the current Internet.]
o Scalability -- will the current routing architecture scale? Does
this relate to the first two items?
What about reporting? Since this information can only really be
collected at exchange points, what would be done with the data
collected? Defining a scope is really critical to doing this. Concerns
should be sent to the RA at
[email protected] or
[email protected].
The RA will be all NAPs and the two MAEs and other interconnects. The
NAP operators are not willing to do these type measurements since it is
not related to the Level 2 service they are provisioning.
General information about the RA and current North American routing
architecture is available under ``papers'' at:
http://www.isi.edu/div7/ra
Other Issues
A number of other issues were also discussed during this meeting.
Once this work is complete, specific ownership issues should be
considered. Some standards groups assert ownership of the metrics and
the tools that produce them. This may be appropriate here. There was
also a concern that the meaning of the results should be as immutable
over time (some folks called this tool drift). This may be too hard to
do and is definitely hard to consider when the metrics and the tools are
not yet articulated.
Is it possible to create a metric that could be used by an independent
party that would produce a meaningful result? Can the metrics be
defined in an unambiguous way? Can specific issues such as the
interactive performance of a Telnet situation from the user perspective
be quantified?
There was also considerable concern about providers testing themselves
and other providers versus consumers doing the testing and issues
involving the need for disclosure of components involved (Black-box
versus Clear-box testing). One attendee asserted that information in
the current NSFNET has been available publicly for years and folks have
not exploited that to unreasonably bash on anyone. Matt would like
there to be some way for this to be done using RA-type infrastructure.
This data should be sanitized in such a way that any bias is effectively
removed. If this issue is not resolved by the IETF and the providers,
it is believed that the consumer will assert their right-to-know.
There was also an open issue about how any of this work might be related
to IPv6.
Revisit Working Group Issues
There was strong consensus that IPPM work is important and should be
done within IETF.
There was consensus that this work will initially focus on the delivery
and routing datagrams (i.e., not related services, such as NOC
characteristics).
However, IPPM appears to be within the charter of an existing working
group, the Benchmarking Methodology Working Group (BMWG). Their charter
clearly meant to include measurements on real operational networks. It
may be easier to take the IPPM Effort and move it. On the other hand,
the effective orientation of BMWG has clearly been towards the
laboratory environment. There was some concern that the BMWG is a
little too open ended, and that a focused charter for IPPM would be
better. Christian Huitema was particularly encouraging on this issue.
Matt called for volunteers to chair or co-chair the working group.
Nobody volunteered publicly. [But there have since been several private
responses. -- Matt Mathis]
The group decided that a subcommittee would convene to draft a charter.
The following people volunteered: Paul Love, Teunis J. Ott, John Boone
and Matt.
Matt hopes that the charter and the relationship to the other working
groups will be finalized by the NANOG meeting on 21 May 1995.
