Independent Submission                                        C. Huitema
Request for Comments: 8963                          Private Octopus Inc.
Category: Informational                                     January 2021
ISSN: 2070-1721


           Evaluation of a Sample of RFCs Produced in 2018

Abstract

  This document presents the author's effort to understand the delays
  involved in publishing an idea in the IETF or through the Independent
  Stream, from the first individual draft to the publication of the
  RFC.  We analyze a set of randomly chosen RFCs approved in 2018,
  looking for history and delays.  We also use two randomly chosen sets
  of RFCs published in 2008 and 1998 for comparing delays seen in 2018
  to those observed 10 or 20 years ago.  The average RFC in the 2018
  sample was produced in 3 years and 4 months, of which 2 years and 10
  months were spent in the working group, 3 to 4 months for IETF
  consensus and IESG review, and 3 to 4 months in RFC production.  The
  main variation in RFC production delays comes from the AUTH48 phase.

  We also measure the number of citations of the chosen RFC using
  Semantic Scholar, and compare citation counts with what we know about
  deployment.  We show that citation counts indicate academic interest,
  but correlate only loosely with deployment or usage of the
  specifications.  Counting web references could complement that.

Status of This Memo

  This document is not an Internet Standards Track specification; it is
  published for informational purposes.

  This is a contribution to the RFC Series, independently of any other
  RFC stream.  The RFC Editor has chosen to publish this document at
  its discretion and makes no statement about its value for
  implementation or deployment.  Documents approved for publication by
  the RFC Editor are not candidates for any level of Internet Standard;
  see Section 2 of RFC 7841.

  Information about the current status of this document, any errata,
  and how to provide feedback on it may be obtained at
  https://www.rfc-editor.org/info/rfc8963.

Copyright Notice

  Copyright (c) 2021 IETF Trust and the persons identified as the
  document authors.  All rights reserved.

  This document is subject to BCP 78 and the IETF Trust's Legal
  Provisions Relating to IETF Documents
  (https://trustee.ietf.org/license-info) in effect on the date of
  publication of this document.  Please review these documents
  carefully, as they describe your rights and restrictions with respect
  to this document.

Table of Contents

  1.  Introduction
  2.  Methodology
    2.1.  Defining the Important Milestones
    2.2.  Selecting a Random Sample of RFCs
    2.3.  Conventions Used in This Document
  3.  Analysis of 20 Selected RFCs
    3.1.  RFC 8411
    3.2.  RFC 8456
    3.3.  RFC 8446
    3.4.  RFC 8355
    3.5.  RFC 8441
    3.6.  RFC 8324
    3.7.  RFC 8377
    3.8.  RFC 8498
    3.9.  RFC 8479
    3.10. RFC 8453
    3.11. RFC 8429
    3.12. RFC 8312
    3.13. RFC 8492
    3.14. RFC 8378
    3.15. RFC 8361
    3.16. RFC 8472
    3.17. RFC 8471
    3.18. RFC 8466
    3.19. RFC 8362
    3.20. RFC 8468
  4.  Analysis of Process and Delays
    4.1.  Delays from First Draft to RFC
    4.2.  Working Group Processing Time
    4.3.  Preparation and Publication Delays
    4.4.  Copy Editing
    4.5.  Independent Stream
  5.  Citation Counts
    5.1.  Citation Numbers
    5.2.  Comparison to 1998 and 2008
    5.3.  Citations versus Deployments
    5.4.  Citations versus Web References
  6.  Observations and Next Steps
  7.  Security Considerations
  8.  IANA Considerations
  9.  Informative References
  Acknowledgements
  Author's Address

1.  Introduction

  As stated on the organization's web site, "The IETF is a large open
  international community of network designers, operators, vendors, and
  researchers concerned with the evolution of the Internet architecture
  and the smooth operation of the Internet."  The specifications
  produced by the IETF are published in the RFC series, along with
  documents from the IAB, IRTF, and Independent streams (as per RFC
  8729).  In this memo, the author attempts to understand the delays
  involved in publishing an idea in the IETF or through the Independent
  Stream, from the first individual draft to the publication of the
  RFC.  This is an individual effort, and the author's conclusions
  presented here are personal.  There was no attempt to seek IETF
  consensus.

  The IETF keeps records of documents and process actions in the IETF
  Datatracker [TRKR].  The IETF Datatracker provides information about
  RFCs and drafts, from which we can infer statistics about the
  production system.  We can measure how long it takes to drive a
  proposition from initial draft to final publication, and how these
  delays can be split between working group discussions, IETF reviews,
  IESG assessment, RFC Editor delays and final reviews by the authors
  -- or, for Independent Stream RFCs, draft production, reviews by the
  Independent Submissions Editor, conflict reviews, RFC Editor delays
  and final reviews.  Tracker data is available for all RFCs, not just
  IETF Stream RFCs.

  Just measuring production delays may be misleading.  If the IETF or
  the other streams simply rubber-stamped draft proposals and published
  them, the delays would be short but the quality and impact might
  suffer.  We hope that most of the RFCs that are published are useful,
  but we need a way to measure that usefulness.  We try to do that by
  measuring the number of references of the published RFCs in Semantic
  Scholar [SSCH], and also by asking the authors of each RFC in the
  sample whether the protocols and technologies defined in the RFCs
  were implemented and used on the Internet.  The citations measured by
  the Semantic Scholar include citations in other RFCs and in Internet-
  Drafts.  We also measure the number of references on the web, which
  provides some results but would be hard to automate.

  In order to limit the resources required for this study, we selected
  at random 20 RFCs published in 2018, as explained in Section 2.2.
  The statistical sampling picked both IETF Stream and Independent
  Stream documents.  For comparison purposes, we also selected at
  random 20 RFCs published in 1998 and 20 published in 2008.  Limiting
  the sample to 20 out of 209 RFCs published in 2018 allows for in-
  depth analysis of each RFC, but readers should be reminded that the
  this is a small sample.  The sample is too small to apply general
  statistical techniques and quantify specific ratios, and discussions
  of correlation techniques would be inappropriate.  Instead, the
  purpose is to identify trends, spot issues, and document future work.

  The information gathered for every RFC in the sample is presented in
  Section 3.  In Section 4, we analyze the production process and the
  sources of delays, comparing the 2018 sample to the selected samples
  for 1998 and 2018.  In Section 5.1, we present citation counts for
  the RFCs in the samples, and analyze whether citation counts could be
  used to evaluate the quality of RFCs.

  The measurement of delays could be automated by processing dates and
  events recorded in the Datatracker.  The measurement of published
  RFCs could be complemented by statistics on abandoned drafts, which
  would measure the efficiency of the IETF triaging process.  More
  instrumentation would help understanding how large delays happen
  during working group processes.  These potential next steps are
  developed in Section 6.

2.  Methodology

  The study reported here started with a simple idea: take a sample of
  RFCs, and perform an in-depth analysis of the path from the first
  presentation of the idea to its publication, while also trying to
  access the success of the resulting specification.  This requires
  defining the key milestones that we want to track, and drawing a
  random sample using an unbiased process.

2.1.  Defining the Important Milestones

  The IETF Datatracker records a list of events for each document
  processed by IETF working groups.  This has a high granularity, and
  also a high variability.  Most documents start life as an individual
  draft, are adopted by a working group, undergo a Working Group Last
  Call, are submitted to the IESG, undergo an IETF Last Call and an
  IESG review, get eventually approved by the IESG, and are processed
  for publication by the RFC Editor, but there are exceptions.  Some
  documents are first submitted to one working group and then moved to
  another.  Some documents are published through the Independent
  Stream, and are submitted to the Independent Submissions Editor
  instead of the IESG.

  In order to simplify tabulation, we break the period from the
  submission of the first draft to the publication of the RFC into
  three big components:

  *  The working group processing time, from the first draft to the
     start of the IETF last call;

  *  The IETF processing time, which lasts from the beginning of the
     IETF last call to the approval by the IESG, including the reviews
     by various directorates;

  *  The RFC production, from approval by the IESG to publication,
     including the AUTH48 reviews.

  For submissions to the Independent Stream, we don't have a working
  group.  We consider instead the progression of the individual draft
  until the adoption by the Independent Submissions Editor (ISE) as the
  equivalent of the "Working Group" period, and the delay from adoption
  by the ISE until submission to the RFC Editor as the equivalent of
  the IETF processing time.

  We measure the starting point of the process using the date of
  submission of the first draft listed on that RFC page in the IETF
  Datatracker.  In most cases, this first draft is an individual draft
  that then resubmitted as a working group draft, or maybe resubmitted
  with a new name as the draft was searching for a home in an IETF
  working group, or before deciding for submission on the Independent
  Stream.

  The IETF Datatracker entries for RFCs and drafts do not _always_ list
  working group events like Working Group Last Call.  The only
  intermediate event that we list between the first draft and the
  submission to the IESG is the working group adoption, for which we
  use the date of submission of version 00 of the draft eventually
  published as RFC.  We also use that date (of submission of version
  00) for drafts submitted to the Independent Stream.

2.2.  Selecting a Random Sample of RFCs

  Basic production mechanisms could be evaluated by processing data
  from the IETF Datatracker, but subjective data requires manual
  assessment of results, which can be time-consuming.  Since our
  resources are limited, we will only perform this analysis for a small
  sample of RFCs, selected at random from the list of RFCs approved in
  2018.  Specifically, we will pick 20 RFC numbers at random between:

  *  RFC 8307, published in January 2018, and

  *  RFC 8511, published December 2018.

  The list of 20 selected RFCs is: RFC 8411, RFC 8456, RFC 8446, RFC
  8355, RFC 8441, RFC 8324, RFC 8377, RFC 8498, RFC 8479, RFC 8453, RFC
  8429, RFC 8312, RFC 8492 , RFC 8378, RFC 8361, RFC 8472, RFC 8471,
  RFC 8466, RFC 8362, and RFC 8468.

  When evaluating delays and impact, we will compare the year 2018 to
  2008 and 1998, 10 and 20 years ago.  To drive this comparison, we
  pick 20 RFCs at random among those published in 2008, and another 20
  among those published in 1998.

  The list of the 20 randomly selected RFCs from 2008 is: RFC 5227, RFC
  5174, RFC 5172, RFC 5354, RFC 5195, RFC 5236, RFC 5348, RFC 5281, RFC
  5186, RFC 5326, RFC 5277, RFC 5373, RFC 5404, RFC 5329, RFC 5283, RFC
  5358, RFC 5142, RFC 5271, RFC 5349, and RFC 5301.

  The list of the 20 randomly selected RFCs from 1998 is: RFC 2431, RFC
  2381, RFC 2387, RFC 2348, RFC 2391, RFC 2267, RFC 2312, RFC 2448, RFC
  2374, RFC 2398, RFC 2283, RFC 2382, RFC 2289, RFC 2282, RFC 2404, RFC
  2449, RFC 2317, RFC 2394, RFC 2297, and RFC 2323.

2.3.  Conventions Used in This Document

  The following abbreviations are used in the tables:

  BCP   Best Current Practice
  Exp   Experimental
  Info  Informational
  PS    Proposed Standard
  DS    Draft Standard [This maturity level was retired by RFC 6410.]

  In addition, Status is as defined in RFC 2026, and Stream is as
  defined in RFC 8729.

3.  Analysis of 20 Selected RFCs

  We review each of the RFCs listed in Section 2.2 for the year 2018,
  trying both to answer the known questions and to gather insight for
  further analyses.  In many cases, the analysis of the data is
  complemented by direct feedback from the RFC authors.

3.1.  RFC 8411

  "IANA Registration for the Cryptographic Algorithm Object Identifier
  Range" [RFC8411]:

  Status (Length):    Informational (5 pages)
  Overview:           4 individual drafts
  First draft:        2017-05-08
  Last Call start:    2017-10-09
  IESG eval. start:   2017-12-28
  IESG approved:      2018-02-26 (draft 03)
  AUTH48 start:       2018-04-20
  AUTH48 complete:    2018-07-17
  Published:          2018-08-06
  IANA action:        create table

  This RFC was published from the individual draft, which was not
  resubmitted as a working group draft.

  The draft underwent minor copy editing before publication.

  Some but not all of the long delay in AUTH48 is due to clustering
  with [RFC8410].  MISSREF state concluded on 2018-05-09 and the
  document re-entered AUTH48 at once.  AUTH48 lasted over two months
  after that.  (For state definitions, see <https://www.rfc-
  editor.org/about/queue/#state_def>.)

  The time after AUTH48 and before publication (3 weeks) partly
  overlaps with travel for IETF 102 and is partly due to coordinating
  the cluster.

3.2.  RFC 8456

  "Benchmarking Methodology for Software-Defined Networking (SDN)
  Controller Performance" [RFC8456]:

  Status (Length):    Informational (64 pages)
  Overview:           2 individual drafts; 9 WG drafts
  First draft:        2015-03-23
  WG adoption:        2015-10-18
  Last Call start:    2018-01-19
  IESG eval. start:   2018-02-27
  IESG approved:      2018-05-25
  AUTH48 start:       2018-08-31
  AUTH48 complete:    2018-10-16
  Published:          2018-10-30

  The draft underwent extensive copy editing, covering use of articles,
  syntax, and word choice.  The changes are enough to cause pagination
  differences.  The "diff" tool marks pretty much every page as
  changed.  Some diagrams see change in protocol elements like message
  names.

  According to the author, the experience of producing this document
  mirrors a typical one in the Benchmarking Methodologies Working Group
  (BMWG).  There were multiple authors in multiple time zones, which
  slowed down the AUTH48 process somewhat, although the AUTH48 delay of
  46 days is only a bit longer than the average draft.

  The RFC was part of cluster with [RFC8455].

  BMWG publishes Informational RFCs centered around benchmarking, and
  the methodologies in RFC 8456 have been implemented in benchmarking
  products.

3.3.  RFC 8446

  "The Transport Layer Security (TLS) Protocol Version 1.3" [RFC8446],
  as the title indicates, defines the new version of the TLS protocol.
  From the IETF Datatracker, we extract the following:

  Status (Length):    Proposed Standard (160 pages)
  Overview:           29 WG drafts
  First draft:        2014-04-17
  Last Call start:    2018-02-15
  IESG eval. start:   2018-03-02
  IESG approved:      2018-03-21 (draft 28)
  AUTH48 start:       2018-06-14
  AUTH48 complete:    2018-08-10
  Published:          2018-08-10

  This draft started as a WG effort.

  The RFC was a major effort in the IETF.  Working group participants
  developed and tested several implementations.  Researchers analyzed
  the specifications and performed formal verifications.  Deployment
  tests outlined issues that caused extra work when the specification
  was almost ready.  This complexity largely explains the time spent in
  the working group.

  Comparing the final draft to the published version, we find
  relatively light copy editing.  It includes explaining acronyms on
  first use, clarifying some definitions standardizing punctuation and
  capitalization, and spelling out some numbers in text.  This
  generally fall in the category of "style", although some of the
  clarifications go into message definitions.  However, that simple
  analysis does not explain why the AUTH48 phase took almost two
  months.

  This document's AUTH48 process was part of the "GitHub experiment",
  which tried to use GitHub pull requests to track the AUTH48 changes
  and review comments.  The RFC Production Center (RPC) staff had to
  learn using GitHub for that process, and this required more work than
  the usual RFC.  The author and AD thoroughly reviewed each proposed
  edit, accepting some and rejecting some.  The concern there was that
  any change in a complex specification might affect a protocol that
  was extensively reviewed in the working group, but of course these
  reviews added time to the AUTH48 delays.

  There are 21 implementations listed in the Wiki of the TLS 1.3
  project [TLS13IMP].  It has been deployed on major browsers, and is
  already used in a large fraction of TLS connections.

3.4.  RFC 8355

  "Resiliency Use Cases in Source Packet Routing in Networking (SPRING)
  Networks" [RFC8355] is an Informational RFC.  It originated from an
  informational use-case draft; it was mostly used for the BOF creating
  the WG, and then to drive initial work and evolutions from the WG.

  Status (Length):    Informational (13 pages)
  Overview:           2 individual drafts; 13 WG drafts
  First draft:        2014-01-31
  WG adoption:        2014-05-13
  Last Call start:    2017-04-20
  IESG eval. start:   2017-05-04 (draft 09)
  IESG approved:      2017-12-19 (draft 12)
  AUTH48 start:       2018-03-12
  AUTH48 complete:    2018-03-27
  Published:          2018-03-28

  Minor set of copy edits, mostly for style.

  No implementation of the RFC itself, but the technology behind it
  (such as Segment Routing Architecture [RFC8402] and TI-LFA [TI-LFA])
  is widely implemented and deployment is ongoing.

  According to participants in the discussion, the process of adoption
  of the source packet routing standards was very contentious.  The
  establishment of consensus at both the working group level and the
  IETF level was difficult and time-consuming.

3.5.  RFC 8441

  "Bootstrapping WebSockets with HTTP/2" [RFC8441]

  Status (Length):    Proposed Standard (8 pages)
  Overview:           3 individual drafts; 8 WG drafts; Updates RFC
                      6455
  First draft:        2017-10-15
  WG adoption:        2017-12-19
  Last Call start:    2018-05-07 (draft 05)
  IESG eval. start:   2018-05-29 (draft 06)
  IESG approved:      2018-06-18 (draft 07)
  AUTH48 start:       2018-08-13
  AUTH48 complete:    2018-09-15
  Published:          2018-09-18
  IANA action:        table entries

  This RFC defines the support of WebSockets in HTTP/2, which is
  different from the mechanism defined for HTTP/1.1 in [RFC6455].  The
  process was relatively straightforward, involving the usual type of
  discussions, some on details and some on important points.

  Comparing the final draft and published RFC shows a minor set of copy
  edits, mostly for style.  However, the author recalls a painful
  process.  The RFC includes many charts and graphs that were very
  difficult to format correctly in the author's production process that
  involved conversions from markdown to XML, and then from XML to text.
  The author had to get substantial help from the RFC Editor.

  There are several implementations, including Firefox and Chrome,
  making RFC 8441 a very successful specification.

3.6.  RFC 8324

  "DNS Privacy, Authorization, Special Uses, Encoding, Characters,
  Matching, and Root Structure: Time for Another Look?"  [RFC8324].
  This is an opinion piece on DNS development, published on the
  Independent Stream.

  Status (Length):    Informational (29 pages)
  Overview:           5 individual drafts; Independent Stream
  First draft:        2017-06-02
  ISE review start:   2017-07-10 (draft 03)
  IETF conflict review start:  2017-10-29
  Approved:           2017-12-18 (draft 04)
  AUTH48 start:       2018-01-29 (draft 05)
  AUTH48 complete:    2018-02-26
  Published:          2018-02-27

  This RFC took only 9 months from first draft to publication, which is
  the shortest in the 2018 sample set.  In part, this is because the
  text was privately circulated and reviewed by the ISE's selected
  experts before the first draft was published.  The nature of the
  document is another reason for the short delay.  It is an opinion
  piece and does not require the same type of consensus building and
  reviews as a protocol specification.

  Comparing the final draft and the published version shows only minor
  copy edits, mostly for style.  According to the author, this is
  because he knows how to write in RFC style with the result that his
  documents often need a minimum of editing.  He also makes sure that
  the document on which the RFC Production Center starts working
  already has changes discussed and approved during Last Call and IESG
  review incorporated, rather than expecting the Production Center to
  operate off of notes about changes to be made.

3.7.  RFC 8377

  "Transparent Interconnection of Lots of Links (TRILL): Multi-
  Topology" [RFC8377]

  Status (Length):    Proposed Standard (20 pages)
  Overview:           3 individual drafts; 7 WG drafts; Updates RFCs
                      6325 and 7177
  First draft:        2013-09-03
  WG adoption:        2015-09-01
  Last Call start:    2018-02-19 (draft 05)
  IESG eval. start:   2018-03-06 (draft 05)
  IESG approved:      2018-03-12 (draft 06)
  AUTH48 start:       2018-04-20 (draft 06)
  AUTH48 complete:    2018-07-31
  Published:          2018-07-31
  IANA action:        table entries

  Minor set of copy edits, mostly for style, also clarity.

3.8.  RFC 8498

  "A P-Served-User Header Field Parameter for an Originating Call
  Diversion (CDIV) Session Case in the Session Initiation Protocol
  (SIP)" [RFC8498].

  Status (Length):    Informational (15 pages)
  Overview:           5 individual drafts; 9 WG drafts
  First draft:        2016-03-21
  WG adoption:        2017-05-15
  Last Call start:    2018-10-12 (draft 05)
  IESG eval. start:   2018-11-28 (draft 07)
  IESG approved:      2018-12-11 (draft 08)
  AUTH48 start:       2019-01-28
  AUTH48 complete:    2019-02-13
  Published:          2019-02-14
  IANA action:        table rows added.

  Copy edits for style, but also clarification of ambiguous sentences.

3.9.  RFC 8479

  "Storing Validation Parameters in PKCS#8" [RFC8479]

  Status (Length):    Informational (8 pages)
  Overview:           5 individual drafts; Independent Stream
  First draft:        2017-08-08
  ISE review start:   2018-12-10 (draft 00)
  IETF conflict review start:  2018-03-29
  Approved:           2018-08-20 (draft 03)
  AUTH48 start:       2018-09-20 (draft 04)
  AUTH48 complete:    2018-09-25
  Published:          2018-09-26

  The goal of the draft was to document what the gnutls implementation
  was using for storing provably generated RSA keys.  This is a short
  RFC that was published relatively quickly, although discussion
  between the author, the Independent Submissions Editor, and the IESG
  lasted several months.  In the initial conflict review, the IESG
  asked the ISE to not publish this document before IETF working groups
  had an opportunity to pick up the work.  The author met that
  requirement by a presentation to the SECDISPATCH WG during IETF 102.
  Since no WG was interested in picking up the work, the document
  progressed on the Independent Stream.

  Very minor set of copy edits, moving some references from normative
  to informative.

  The author is not aware of other implementations than gnutls relying
  on this RFC.

3.10.  RFC 8453

  "Framework for Abstraction and Control of TE Networks (ACTN)"
  [RFC8453]

  Status (Length):    Informational (42 pages)
  Overview:           3 individual drafts; 16 WG drafts
  First draft:        2015-06-15
  WG adoption:        2016-07-15
  Out of WG:          2018-01-26 (draft 11)
  Expert review requested:  2018-02-13
  Last Call start:    2018-04-16 (draft 13)
  IESG eval. start:   2018-05-16 (draft 14)
  IESG approved:      2018-06-01 (draft 15)
  AUTH48 start:       2018-08-13
  AUTH48 complete:    2018-08-20
  Published:          2018-08-23
  IANA action:        table rows added.

  Minor copy editing.

3.11.  RFC 8429

  "Deprecate Triple-DES (3DES) and RC4 in Kerberos" [RFC8429]

  Status (Length):    BCP (10 pages)
  Overview:           6 WG drafts
  First draft:        2017-05-01
  Last Call start:    2017-07-16 (draft 03)
  IESG eval. start:   2017-08-18 (draft 04)
  IESG approved:      2018-05-25 (draft 05)
  AUTH48 start:       2018-07-24
  AUTH48 complete:    2018-10-31
  Published:          2018-10-31
  IANA action:        table rows added.

  This draft started as a working group effort.

  This RFC recommends deprecating two encryption algorithms that are
  now considered obsolete and possibly broken.  The document was sent
  back to the WG after the first Last Call, edited, and then there was
  a second Last Call.  The delay from first draft to Working Group Last
  Call was relatively short, but the number may be misleading.  The
  initial draft was a replacement of a similar draft in the KITTEN
  Working Group, which stagnated for some time before the CURDLE
  Working Group took up the work.  The deprecation of RC4 was somewhat
  contentious, but the WG had already debated this prior to the
  production of this draft, and the draft was not delayed by this
  debate.

  Most of the 280 days between IETF LC and IESG approval were because
  the IESG had to talk about whether this document should obsolete RFC
  4757 or move it to Historic status, and no one was really actively
  pushing that discussion for a while.

  The 99 days in AUTH48 are mostly because one of the authors was a
  sitting AD, and those duties ended up taking precedence over
  reviewing this document.

  Minor copy editing, for style.

  The implementation of the draft would be the actual removal of
  support for 3DES and RC4 in major implementations.  This is
  happening, but very slowly.

3.12.  RFC 8312

  "CUBIC for Fast Long-Distance Networks" [RFC8312]

  Status (Length):    Informational (18 pages)
  Overview:           2 individual drafts; 8 WG drafts
  First draft:        2014-09-01
  WG adoption:        2015-06-08
  Last Call start:    2017-09-18 (draft 06)
  IESG eval. start:   2017-10-04
  IESG approved:      2017-11-14 (draft 07)
  AUTH48 start:       2018-01-08
  AUTH48 complete:    2018-02-07
  Published:          2018-02-07
  IANA action:        table rows added.

  Minor copy editing, for style.

  The TCP congestion control algorithm Cubic was first defined in 2005,
  was implemented in Linux soon after, and was implemented in major
  OSes after that.  After some debates from 2015 to 2015, the TCPM
  Working Group adopted the draft, with a goal of documenting Cubic in
  the RFC Series.  According to the authors, this was not a high-
  priority effort, as Cubic was already implemented in multiple OSes
  and documented in research papers.  At some point, only one of the
  authors was actively working on the draft.  This may explain why
  another two years was spent progressing the draft after adoption by
  the WG.

  The RFC publication may or may not have triggered further
  implementations.  On the other hand, several OSes picked up bug fixes
  from the draft and the RFC.

3.13.  RFC 8492

  "Secure Password Ciphersuites for Transport Layer Security (TLS)"
  [RFC8492]

  Status (Length):    Informational (40 pages)
  Overview:           10 individual drafts; 8 WG drafts; Independent
                      Stream
  First draft:        2012-09-07
  Targeted to ISE:    2016-08-05
  ISE review start:   2017-05-10 (draft 01)
  IETF conflict review start:  2017-09-04
  Approved:           2017-10-29 (draft 02)
  AUTH48 start:       2018-10-19 (draft 05)
  AUTH48 complete:    2019-02-19
  Published:          2019-02-21
  IANA action:        table rows added.

  This RFC has a complex history.  The first individual draft was
  submitted to the TLS Working Group on September 7, 2012.  It
  progressed there, and was adopted by the WG after 3 revisions.  There
  were then 8 revisions in the TLS WG, until the WG decided to not
  progress it.  The draft was parked in 2013 by the WG chairs after
  failing to get consensus in WG Last Call.  The AD finally pulled the
  plug in 2016, and the draft was then resubmitted to the ISE.

  At that point, the author was busy and was treating this RFC with a
  low priority because, in his words, it would not be a "real RFC".
  There were problems with the draft that only came up late.  In
  particular, it had to wait for a change in registry policy that only
  came about with the publication of TLS 1.3, which caused the draft to
  be published after RFC 8446, and also required adding references to
  TLS 1.3.  The author also got a very late comment while in AUTH48
  that caused some rewriting.  Finally, there was some IANA issue with
  the extension registry where a similar extension was added by someone
  else.  The draft was changed to just use it.

  Changes in AUTH48 include adding a reference to TLS 1.3, copy editing
  for style, some added requirements, added paragraphs, and changes in
  algorithms specification.

3.14.  RFC 8378

  "Signal-Free Locator/ID Separation Protocol (LISP) Multicast"
  [RFC8378] is an Experimental RFC, defining how to implement Multicast
  in the LISP architecture.

  Status (Length):    Experimental (21 pages)
  Overview:           5 individual drafts; 10 WG drafts
  First draft:        2014-02-28
  WG adoption:        2015-12-21
  Last Call start:    2018-02-13 (draft 07)
  IESG eval. start:   2018-02-28 (draft 08)
  IESG approved:      2018-03-12 (draft 09)
  AUTH48 start:       2018-04-23
  AUTH48 complete:    2018-05-02
  Published:          2018-05-02

  Preparing the RFC took more than 4 years.  According to the authors,
  they were not aggressively pushing it and just let the working group
  process decide to pace it.  They also did implementations during that
  time.

  Minor copy editing, for style.

  The RFC was implemented by lispers.net and Cisco, and it was used in
  doing IPv6 multicast over IPv4 unicast/multicast at the Olympics in
  PyeungChang.  The plan is to work on a Proposed Standard once the
  experiment concludes.

3.15.  RFC 8361

  "Transparent Interconnection of Lots of Links (TRILL): Centralized
  Replication for Active-Active Broadcast, Unknown Unicast, and
  Multicast (BUM) Traffic" [RFC8361]

  Status (Length):    Proposed Standard (17 pages)
  Overview:           3 individual drafts; 14 WG drafts
  First draft:        2013-11-12
  WG adoption:        2014-12-16
  Last Call start:    2017-11-28 (draft 10)
  IESG eval. start:   2017-12-18 (draft 11)
  IESG approved:      2018-01-29 (draft 13)
  AUTH48 start:       2018-03-09
  AUTH48 complete:    2018-04-09
  Published:          2018-04-12

  According to the authors, the long delays in producing this RFC were
  due to a slow uptake of the technology in the industry.

  Minor copy editing, for style.

  There was at least one partial implementation.

3.16.  RFC 8472

  "Transport Layer Security (TLS) Extension for Token Binding Protocol
  Negotiation" [RFC8472]

  Status (Length):    Proposed Standard (8 pages)
  Overview:           1 individual draft; 15 WG drafts
  First draft:        2015-05-29
  WG adoption:        2015-09-11
  Last Call start:    2017-11-13 (draft 10)
  IESG eval. start:   2018-03-19
  IESG approved:      2018-07-20 (draft 14)
  AUTH48 start:       2018-09-17
  AUTH48 complete:    2018-09-25
  Published:          2018-10-08

  This is a pretty simple document, but it took over 3 years from
  individual draft to RFC.  According to the authors,the biggest
  setbacks occurred at the start: it took a while to find a home for
  this draft.  It was presented in the TLS WG (because it's a TLS
  extension) and UTA WG (because it has to do with applications using
  TLS).  Then the ADs determined that a new WG was needed, so the
  authors had to work through the WG creation process, including
  running a BOF.

  Minor copy editing, for style, with the addition of a reference to
  TLS 1.3.

  Perhaps partially due to the delays, some of the implementers lost
  interest in supporting this RFC.

3.17.  RFC 8471

  "The Token Binding Protocol Version 1.0" [RFC8471]

  Status (Length):    Proposed Standard (18 pages)
  Overview:           1 individual draft; 19 WG drafts
  First draft:        2014-10-13
  WG adoption:        2015-03-15
  Last Call start:    2017-11-13 (draft 16)
  IESG eval. start:   2018-03-19
  IESG approved:      2018-07-20 (draft 19)
  AUTH48 start:       2018-09-17
  AUTH48 complete:    2018-09-25
  Published:          2018-10-08

  This document presents a Token Binding Protocol for TLS.  We can
  notice a period of 5 months before adoption of the draft by the WG.
  That explains in part the overall time of almost 4 years from first
  draft to publication.

  Minor copy editing, for style.

  The web references indicate adoption in multiple development
  projects.

3.18.  RFC 8466

  "A YANG Data Model for Layer 2 Virtual Private Network (L2VPN)
  Service Delivery" [RFC8466]

  Status (Length):    Proposed Standard (158 pages)
  Overview:           5 individual drafts; 11 WG drafts
  First draft:        2016-09-01
  WG adoption:        2017-02-26
  Last Call start:    2018-02-21 (draft 07)
  IESG eval. start:   2018-03-14 (draft 08)
  IESG approved:      2018-06-25 (draft 10)
  AUTH48 start:       2018-09-17
  AUTH48 complete:    2018-10-09
  Published:          2018-10-12

  Copy editing for style and clarity, with also corrections to the YANG
  model.

3.19.  RFC 8362

  "OSPFv3 Link State Advertisement (LSA) Extensibility" [RFC8362] is a
  major extension to the OSPF protocol.  It makes OSPFv3 fully
  extensible.

  Status (Length):    Proposed Standard (33 pages)
  Overview:           4 individual drafts; 24 WG drafts
  First draft:        2013-02-17
  WG adoption:        2013-10-15
  Last Call start:    2017-12-19 (draft 19)
  IESG eval. start:   2018-01-18 (draft 20)
  IESG approved:      2018-01-29 (draft 23)
  AUTH48 start:       2018-03-19
  AUTH48 complete:    2018-03-30
  Published:          2018-04-03

  The specification was first submitted as an individual draft in the
  IPv6 WG, then moved to the OSPF WG.  The long delay of producing this
  RFC is due to the complexity of the problem, and the need to wait for
  implementations.  It is a very important change to OSPF that makes
  OSPFv3 fully extensible.  Since it was a non-backward compatible
  change, the developers started out with some very complex migration
  scenarios but ended up with either legacy or extended OSPFv3 LSAs
  within an OSPFv3 routing domain.  The initial attempts to have a
  hybrid mode of operation with both legacy and extended LSAs also
  delayed implementation due to the complexity.

  Copy editing for style and clarity.

  This specification either was or will be implemented by all the
  router vendors.

3.20.  RFC 8468

  "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for the IP
  Performance Metrics (IPPM) Framework" [RFC8468].

  Status (Length):    Informational (15 pages)
  Overview:           3 individual drafts; 7 WG drafts
  First draft:        2015-08-06
  WG adoption:        2016-07-04
  Last Call start:    2018-04-11 (draft 04)
  IESG eval. start:   2018-05-24 (draft 05)
  IESG approved:      2018-07-10 (draft 06)
  AUTH48 start:       2018-09-13
  AUTH48 complete:    2018-11-05
  Published:          2018-11-14

  RFC 8468 was somehow special in that there was not a technical reason
  or interest that triggered it, but rather a formal requirement.
  While writing RFC 7312, the IP Performance Metrics (IPPM) Working
  Group realized that RFC 2330, the IP Performance Metrics Framework
  supported IPv4 only and explicitly excluded support for IPv6.
  Nevertheless, people used the metrics that were defined on top of RFC
  2330 (and, therefore, IPv4 only) for IPv6, too.  Although the IPPM WG
  agreed that the work was needed, the interest of IPPM attendees in
  progressing (and reading/reviewing) the IPv6 draft was limited.
  Resolving the IPv6 technical part was straightforward, but
  subsequently some people asked for a broader scope (topics like
  header compression, 6LoWPAN, etc.), and it took some time to figure
  out and later on convince people that these topics are out of scope.
  The group also had to resolve contentious topics, for example, how to
  measure the processing of IPv6 extension headers, which is sometimes
  nonstandard.

  The time in AUTH48 state for this document was longer than average.
  According to the authors, the main reasons include:

  *  Workload and travel caused by busy work periods of all coauthors

  *  Time zone difference between coauthors and editor (at least US,
     Europe, and India, not considering travel)

  *  RFC Production Center proposed and committed some unacceptable
     modifications that needed to be reverted

  *  Lengthy discussions on a new document title (required high effort
     and took a long time, in particular reaching consensus between
     coauthors and editor was time-consuming and involved the AD)

  *  RFC Production Center correctly identified some nits (obsoleted
     personal websites of coauthors) and coauthors attempting to fix
     them.

  The differences between the final draft and the published RFC show
  copy editing for style and clarity, but do not account for the back
  and forth between authors and editors mentioned by the authors.

4.  Analysis of Process and Delays

  We examine the 20 RFCs in the sample, measuring various
  characteristics such as delay and citation counts, in an attempt to
  identify patterns in the IETF processes.

4.1.  Delays from First Draft to RFC

  We look at the distribution of delays between the submission of the
  first draft and the publication of the RFC, using the three
  milestones defined in Section 2.1: processing time in the working
  group, IETF processing time, and RFC production time.  The following
  table shows the number of days in each phase for the 20 RFCs in the
  sample:

      +======+============+=======+=========+======+======+======+
      |  RFC | Status     | Pages | Overall |   WG | IETF | Edit |
      +======+============+=======+=========+======+======+======+
      | 8411 | Info       |     5 |     455 |  154 |  140 |  161 |
      +------+------------+-------+---------+------+------+------+
      | 8456 | Info       |    64 |    1317 | 1033 |  126 |  158 |
      +------+------------+-------+---------+------+------+------+
      | 8446 | PS         |   160 |    1576 | 1400 |   34 |  142 |
      +------+------------+-------+---------+------+------+------+
      | 8355 | Info       |    13 |    1517 | 1175 |  243 |   99 |
      +------+------------+-------+---------+------+------+------+
      | 8441 | PS         |     8 |     327 |  204 |   31 |   92 |
      +------+------------+-------+---------+------+------+------+
      | 8324 | Info (ISE) |    29 |     270 |   38 |  161 |   71 |
      +------+------------+-------+---------+------+------+------+
      | 8377 | PS         |     8 |    1792 | 1630 |   21 |  141 |
      +------+------------+-------+---------+------+------+------+
      | 8498 | Info       |    15 |    1059 |  935 |   59 |   65 |
      +------+------------+-------+---------+------+------+------+
      | 8479 | Info (ISE) |     8 |     414 |  233 |  144 |   37 |
      +------+------------+-------+---------+------+------+------+
      | 8453 | Info       |    42 |    1165 | 1036 |   46 |   83 |
      +------+------------+-------+---------+------+------+------+
      | 8429 | BCP        |    10 |     548 |   76 |  313 |  159 |
      +------+------------+-------+---------+------+------+------+
      | 8312 | Info       |    18 |    1214 | 1113 |   16 |   85 |
      +------+------------+-------+---------+------+------+------+
      | 8492 | Info (ISE) |    40 |    2358 | 1706 |  172 |  480 |
      +------+------------+-------+---------+------+------+------+
      | 8378 | Exp        |    21 |    1524 | 1446 |   27 |   51 |
      +------+------------+-------+---------+------+------+------+
      | 8361 | PS         |    17 |    1612 | 1477 |   62 |   73 |
      +------+------------+-------+---------+------+------+------+
      | 8472 | PS         |     8 |    1228 |  899 |  249 |   80 |
      +------+------------+-------+---------+------+------+------+
      | 8471 | PS         |    18 |    1228 |  899 |  249 |   80 |
      +------+------------+-------+---------+------+------+------+
      | 8466 | PS         |   158 |     771 |  538 |  124 |  109 |
      +------+------------+-------+---------+------+------+------+
      | 8362 | PS         |    33 |    1871 | 1766 |   41 |   64 |
      +------+------------+-------+---------+------+------+------+
      | 8468 | Info       |    15 |    1196 |  979 |   90 |  127 |
      +------+------------+-------+---------+------+------+------+
      | average           |    35 |    1172 |  948 |  117 |  118 |
      +-------------------+-------+---------+------+------+------+
      | average (not ISE) |    36 |    1200 |  999 |  110 |  104 |
      +-------------------+-------+---------+------+------+------+

                                Table 1

  The average delay from first draft to publication is about 3 years
  and 3 months, but this varies widely.  Excluding the RFCs from the
  Independent Stream, the average delay from start to finish is 3 years
  and 4 months, of which on average 2 years and 9 months are spent
  getting consensus in the working group, and 3 to 4 months each for
  IETF consensus and for RFC production.

  The longest delay is found for [RFC8492], 6.5 years from start to
  finish.  This is however a very special case -- a draft that was
  prepared for the TLS Working Group and failed to reach consensus.
  After that, it was resubmitted to the ISE, and incurred atypical
  production delays.

  On average, we see that 80% of the delay is incurred in WG
  processing, 10% in IETF review, and 10% for edition and publication.

  For IETF Stream RFCs, it appears that the delays for Informational
  documents are slightly shorter than those for protocol
  specifications, maybe six months shorter on average.  However, there
  are lots of differences between individual documents.  The delays
  range from less than a year to more than 5 years for protocol
  specifications, and from a year and 3 months to a bit more than 4
  years for Informational documents.

  We can compare the delays in the 2018 samples to those observed 10
  years ago and 20 years before:

               +============+============+=======+=======+
               | RFC (2008) | Status     | Pages | Delay |
               +============+============+=======+=======+
               | 5326       | Exp        |    54 |  1584 |
               +------------+------------+-------+-------+
               | 5348       | PS         |    58 |   823 |
               +------------+------------+-------+-------+
               | 5281       | Info       |    51 |  1308 |
               +------------+------------+-------+-------+
               | 5354       | Exp        |    23 |  2315 |
               +------------+------------+-------+-------+
               | 5227       | PS         |    21 |  2434 |
               +------------+------------+-------+-------+
               | 5329       | PS         |    12 |  1980 |
               +------------+------------+-------+-------+
               | 5277       | PS         |    35 |   912 |
               +------------+------------+-------+-------+
               | 5236       | Info (ISE) |    26 |  1947 |
               +------------+------------+-------+-------+
               | 5358       | BCP        |     7 |   884 |
               +------------+------------+-------+-------+
               | 5271       | Info       |    22 |  1066 |
               +------------+------------+-------+-------+
               | 5195       | PS         |    10 |   974 |
               +------------+------------+-------+-------+
               | 5283       | PS         |    12 |  1096 |
               +------------+------------+-------+-------+
               | 5186       | Info       |     6 |  2253 |
               +------------+------------+-------+-------+
               | 5142       | PS         |    13 |  1005 |
               +------------+------------+-------+-------+
               | 5373       | PS         |    24 |  1249 |
               +------------+------------+-------+-------+
               | 5404       | PS         |    27 |   214 |
               +------------+------------+-------+-------+
               | 5172       | PS         |     7 |   305 |
               +------------+------------+-------+-------+
               | 5349       | Info       |    10 |  1096 |
               +------------+------------+-------+-------+
               | 5301       | PS         |     6 |   396 |
               +------------+------------+-------+-------+
               | 5174       | Info       |     8 |   427 |
               +------------+------------+-------+-------+

                                 Table 2

              +============+============+=======+=========+
              | RFC (1998) | Status     | Pages |   Delay |
              +============+============+=======+=========+
              | 2289       | PS         |    25 |     396 |
              +------------+------------+-------+---------+
              | 2267       | Info       |    10 | unknown |
              +------------+------------+-------+---------+
              | 2317       | BCP        |    10 |     485 |
              +------------+------------+-------+---------+
              | 2404       | PS         |     7 |     488 |
              +------------+------------+-------+---------+
              | 2374       | PS         |    12 |     289 |
              +------------+------------+-------+---------+
              | 2449       | PS         |    19 |     273 |
              +------------+------------+-------+---------+
              | 2283       | PS         |     9 |     153 |
              +------------+------------+-------+---------+
              | 2394       | Info       |     6 |     365 |
              +------------+------------+-------+---------+
              | 2348       | DS         |     5 |     699 |
              +------------+------------+-------+---------+
              | 2382       | Info       |    30 |     396 |
              +------------+------------+-------+---------+
              | 2297       | Info (ISE) |   109 |      28 |
              +------------+------------+-------+---------+
              | 2381       | PS         |    43 |     699 |
              +------------+------------+-------+---------+
              | 2312       | Info       |    20 |     365 |
              +------------+------------+-------+---------+
              | 2387       | PS         |    10 |     122 |
              +------------+------------+-------+---------+
              | 2398       | Info       |    15 |     396 |
              +------------+------------+-------+---------+
              | 2391       | PS         |    10 |     122 |
              +------------+------------+-------+---------+
              | 2431       | PS         |    10 |     457 |
              +------------+------------+-------+---------+
              | 2282       | Info       |    14 |     215 |
              +------------+------------+-------+---------+
              | 2323       | Info (ISE) |     5 | unknown |
              +------------+------------+-------+---------+
              | 2448       | Info (ISE) |     7 |      92 |
              +------------+------------+-------+---------+

                                 Table 3

  We can compare the median delay, and the delays observed by the
  fastest and slowest quartiles in the three years:

              +======+=============+========+=============+
              | Year | Fastest 25% | Median | Slowest 25% |
              +======+=============+========+=============+
              | 2018 |         715 |   1221 |        1537 |
              +------+-------------+--------+-------------+
              | 2008 |         869 |   1081 |        1675 |
              +------+-------------+--------+-------------+
              | 1998 |         169 |    365 |         442 |
              +------+-------------+--------+-------------+

                                 Table 4

  The IETF takes three to four times more to produce an RFC in 2018
  than it did in 1998, but about the same time as it did in 2008.  We
  can get a rough estimate of how this translates in terms of "level of
  attention" per RFC by comparing the number of participants in the
  IETF meetings of 2018, 2008, and 1998 [IETFCOUNT] to the number of
  RFCs published these years [RFCYEAR].

   +======+=========+========+========+======+=========+============+
   | Year | Number  | Spring | Summer | Fall | Average | Attendees/ |
   |      | of RFCs |     P. |     P. |   P. |      P. | RFC        |
   +======+=========+========+========+======+=========+============+
   | 2018 |     208 |   1235 |   1078 |  879 |    1064 | 5.1        |
   +------+---------+--------+--------+------+---------+------------+
   | 2008 |     290 |   1128 |   1181 |  962 |    1090 | 3.8        |
   +------+---------+--------+--------+------+---------+------------+
   | 1998 |     234 |   1775 |   2106 | 1705 |    1862 | 8.0        |
   +------+---------+--------+--------+------+---------+------------+

                                Table 5

  The last column in the table provides the ratio of average number of
  participants to the number of RFCs published.  If the IETF were a
  centralized organization, and if all participants and documents were
  equivalent, this ratio would be the number of participants dedicated
  to produce an RFC on a given year.  This is of course a completely
  abstract figure because none of the hypotheses above are true, but it
  still gives a vague indication of the "level of attention" applied to
  documents.  We see that this ratio has increased from 2008 to 2018,
  as the number of participants was about the same for these two years
  but the number of published RFCs decreased.  However, this ratio was
  much higher in 1998.  The IETF had many more participants, and there
  were probably many more eyes available to review any given draft.  If
  we applied the ratios of 1998, the IETF would be producing 119
  documents in 2018 instead of 208.

4.2.  Working Group Processing Time

  The largest part of the delays is spent in the working groups, before
  the draft is submitted to the IESG for IETF review.  As mentioned in
  Section 2.1, the only intermediate milestone that we can extract from
  the IETF Datatracker is the date at which the document was adopted by
  the working group, or targeted for independent submission.  The
  breakdown of the delays for the documents in our sample is:

     +======+============+======+================+================+
     | RFC  | Status     |   WG | Until adoption | After adoption |
     +======+============+======+================+================+
     | 8411 | Info       |  154 |              0 |            154 |
     +------+------------+------+----------------+----------------+
     | 8456 | Info       | 1033 |            209 |            824 |
     +------+------------+------+----------------+----------------+
     | 8446 | PS         | 1400 |              0 |           1400 |
     +------+------------+------+----------------+----------------+
     | 8355 | Info       | 1175 |            102 |           1073 |
     +------+------------+------+----------------+----------------+
     | 8441 | PS         |  204 |             65 |            139 |
     +------+------------+------+----------------+----------------+
     | 8324 | Info (ISE) |   38 |              0 |             38 |
     +------+------------+------+----------------+----------------+
     | 8377 | PS         | 1630 |            728 |            902 |
     +------+------------+------+----------------+----------------+
     | 8498 | Info       |  935 |            420 |            515 |
     +------+------------+------+----------------+----------------+
     | 8479 | Info (ISE) |  233 |              0 |            233 |
     +------+------------+------+----------------+----------------+
     | 8453 | Info       | 1036 |            396 |            640 |
     +------+------------+------+----------------+----------------+
     | 8429 | BCP        |   76 |              0 |             76 |
     +------+------------+------+----------------+----------------+
     | 8312 | Info       | 1113 |            280 |            833 |
     +------+------------+------+----------------+----------------+
     | 8492 | Info (ISE) | 1706 |           1428 |            278 |
     +------+------------+------+----------------+----------------+
     | 8378 | Exp        | 1446 |            661 |            785 |
     +------+------------+------+----------------+----------------+
     | 8361 | PS         | 1477 |            399 |           1078 |
     +------+------------+------+----------------+----------------+
     | 8472 | PS         |  899 |            105 |            794 |
     +------+------------+------+----------------+----------------+
     | 8471 | PS         | 1127 |            153 |            794 |
     +------+------------+------+----------------+----------------+
     | 8466 | PS         |  538 |            178 |            360 |
     +------+------------+------+----------------+----------------+
     | 8362 | PS         | 1766 |            240 |           1526 |
     +------+------------+------+----------------+----------------+
     | 8468 | Info       |  979 |            333 |            646 |
     +------+------------+------+----------------+----------------+
     | Average           |  948 |            285 |            663 |
     +-------------------+------+----------------+----------------+

                                Table 6

  The time before working group adoption averages to a bit more than 9
  months, compared to 1 year and almost 10 months for processing time
  after adoption.  We see that RFC 8492 stands out, with long delays
  spent attempting publication through a working group before
  submission to the Independent Submissions Editor.  If we remove RFC
  8492 from the list, the average time until adoption drops to just
  over 7 months, and becomes just 25% of the total processing time in
  the WG.

  There are a few documents that started immediately as working group
  efforts, or were immediately targeted for publication in the
  Independent Stream.  Those documents tend to see short processing
  times, with the exception of RFC 8446 on which the TLS Working Group
  spent a long time working.

4.3.  Preparation and Publication Delays

  The preparation and publication delays include three components:

  *  the delay from submission to the RFC Editor to beginning of
     AUTH48, during which the document is prepared (referred to as "RFC
     edit" below);

  *  the AUTH48 delay, during which authors review and eventually
     approve the changes proposed by the editors (referred to as
     "AUTH48" below);

  *  the publication delay, from final agreement by authors and editors
     to actual publication (referred to as "RFC Pub" below).

  The breakdown of the publication delays for each RFC is shown in the
  following table.

   +=======+========+=======+==========+========+=========+=========+
   |   RFC | Status | Pages | RFC edit | AUTH48 | RFC Pub |    Edit |
   |       |        |       |          |        |         | (total) |
   +=======+========+=======+==========+========+=========+=========+
   |  8411 | Info   |     5 |       53 |     88 |      20 |     161 |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8456 | Info   |    64 |       98 |     46 |      14 |     158 |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8446 | PS     |   160 |       85 |     57 |       0 |     142 |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8355 | Info   |    13 |       83 |     15 |       1 |      99 |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8441 | PS     |     8 |       56 |     33 |       3 |      92 |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8324 | Info   |    29 |       42 |     28 |       1 |      71 |
   |       | (ISE)  |       |          |        |         |         |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8377 | PS     |     8 |       39 |    102 |       0 |     141 |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8498 | Info   |    15 |       48 |     16 |       1 |      65 |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8479 | Info   |     8 |       31 |      5 |       1 |      37 |
   |       | (ISE)  |       |          |        |         |         |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8453 | Info   |    42 |       73 |      7 |       3 |      83 |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8429 | BCP    |    10 |       60 |     99 |       0 |     159 |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8312 | Info   |    18 |       55 |     28 |       2 |      85 |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8492 | Info   |    40 |      355 |    123 |       2 |     480 |
   |       | (ISE)  |       |          |        |         |         |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8378 | Exp    |    21 |       42 |      9 |       0 |      51 |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8361 | PS     |    17 |       39 |     31 |       3 |      73 |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8472 | PS     |     8 |       59 |      8 |      13 |      80 |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8471 | PS     |    18 |       59 |      8 |      13 |      80 |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8466 | PS     |   158 |       84 |     22 |       3 |     109 |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8362 | PS     |    33 |       49 |     11 |       4 |      64 |
   +-------+--------+-------+----------+--------+---------+---------+
   |  8468 | Info   |    15 |       65 |     53 |       9 |     127 |
   +-------+--------+-------+----------+--------+---------+---------+
   | Average        |       |       74 |     39 |       5 |     118 |
   +----------------+-------+----------+--------+---------+---------+
   | Average        |       |       59 |     35 |       5 |      99 |
   | (without 8492) |       |          |        |         |         |
   +----------------+-------+----------+--------+---------+---------+

                                Table 7

  On average, the total delay appears to be about four months, but the
  average is skewed by the extreme values encountered for [RFC8492].
  If we exclude that RFC from the computations, the average delay drops
  to a just a bit more than 3 months: about 2 months for the
  preparation, a bit more than one month for the AUTH48 phase, and 5
  days for the publishing.

  Of course, these delays vary from RFC to RFC.  To try explain the
  causes of the delay, we compute the correlation factor between the
  observed delays and several plausible explanation factors:

  *  the number of pages in the document,

  *  the amount of copy editing, as discussed in Section 4.4,

  *  whether or not IANA actions were required,

  *  the number of authors,

  *  the number of draft revisions,

  *  the working group delay.

  We find the following values:

         +===================+==========+========+=============+
         | Correlation       | RFC edit | AUTH48 | Edit(total) |
         +===================+==========+========+=============+
         | Number of pages   |     0.50 |  -0.04 |        0.21 |
         +-------------------+----------+--------+-------------+
         | Copy-Edit         |     0.42 |   0.24 |        0.45 |
         +-------------------+----------+--------+-------------+
         | IANA              |    -0.14 |  -0.21 |        0.12 |
         +-------------------+----------+--------+-------------+
         | Number of authors |     0.39 |  -0.07 |        0.18 |
         +-------------------+----------+--------+-------------+
         | Number of drafts  |     0.18 |  -0.33 |       -0.19 |
         +-------------------+----------+--------+-------------+
         | WG delay          |     0.03 |  -0.16 |       -0.15 |
         +-------------------+----------+--------+-------------+

                                 Table 8

  We see some plausible explanations for the production delay.  It will
  be somewhat longer for longer documents or for documents that require
  a lot of copy editing (see Section 4.4).  Somewhat surprisingly, it
  also tends to increase with the number of authors.  It does not
  appear significantly correlated with the presence or absence of IANA
  action.

  The analysis of RFC 8324 in Section 3.6 explains its short editing
  delays by the experience of the author.  This makes sense: if a
  document needs less editing, the editing delays would be shorter.
  This is partially confirmed by the relation between the amount of
  copy editing and the publication delay.

  We see fewer plausible explanations for the AUTH48 delays.  These
  delays vary much more than the preparation delay, with a standard
  deviation of 20 days for AUTH48 versus 10 days for the preparation
  delay.  In theory, AUTH48 is just a final verification: the authors
  receive the document prepared by the RFC production center, and just
  have to give their approval, or maybe request a last minute
  correction.  The name indicates that this is expected to last just
  two days, but in average it lasts more than a month.

  We often hypothesize that the number of authors influences the AUTH48
  delay, or that authors who have spent a long time working on the
  document in the working group somehow get demotivated and spend even
  longer to answer questions during AUTH48.  This may happen sometimes,
  but our statistics don't show that - if anything, the numerical
  results point in the opposite direction.

  After asking the authors of the RFCs in the sample why the AUTH48
  phase took a long time, we got three explanations:

  1.  Some RFCs have multiple authors in multiple time zones.  This
      slows down the coordination required for approving changes.

  2.  Some authors found some of the proposed changes unnecessary or
      undesirable, and asked that they be reversed.  This required long
      exchanges between authors and editors.

  3.  Some authors were not giving high priority to AUTH48 responses.

  As mentioned above, we were not able to verify these hypotheses by
  looking at the data.  The author's experience with this document
  suggests another potential delay for the Independent Stream RFC:
  processing delay by the Independent Submissions Editor, discussed in
  Section 4.5.

4.4.  Copy Editing

  We can assess the amount of copy editing applied to each published
  RFC by comparing the text of the draft approved for publication and
  the text of the RFC.  We do expect differences in the "boilerplate"
  and in the IANA section, but we will also see differences due to copy
  editing.  Assessing the amount of copy editing is subjective, and we
  do it using a scale of 1 to 4:

  1:  Minor editing

  2:  Editing for style, such as capitalization, hyphens, "that" versus
      "which", and expanding all acronyms at least once.

  3:  Editing for clarity in addition to style, such as rewriting
      ambiguous sentences and clarifying use of internal references.
      For YANG models, that may include model corrections suggested by
      the verifier.

  4:  Extensive editing.

  The following table shows that about half of the RFCs required
  editing for style, and the other half at least some editing for
  clarity.

                    +======+============+===========+
                    |  RFC | Status     | Copy Edit |
                    +======+============+===========+
                    | 8411 | Info       |         2 |
                    +------+------------+-----------+
                    | 8456 | Info       |         4 |
                    +------+------------+-----------+
                    | 8446 | PS         |         3 |
                    +------+------------+-----------+
                    | 8355 | Info       |         2 |
                    +------+------------+-----------+
                    | 8441 | PS         |         2 |
                    +------+------------+-----------+
                    | 8324 | Info (ISE) |         2 |
                    +------+------------+-----------+
                    | 8377 | PS         |         3 |
                    +------+------------+-----------+
                    | 8498 | Info       |         3 |
                    +------+------------+-----------+
                    | 8479 | Info (ISE) |         1 |
                    +------+------------+-----------+
                    | 8453 | Info       |         2 |
                    +------+------------+-----------+
                    | 8429 | BCP        |         2 |
                    +------+------------+-----------+
                    | 8312 | Info       |         2 |
                    +------+------------+-----------+
                    | 8492 | Info (ISE) |         3 |
                    +------+------------+-----------+
                    | 8378 | Exp        |         2 |
                    +------+------------+-----------+
                    | 8361 | PS         |         2 |
                    +------+------------+-----------+
                    | 8472 | PS         |         2 |
                    +------+------------+-----------+
                    | 8471 | PS         |         2 |
                    +------+------------+-----------+
                    | 8466 | PS         |         3 |
                    +------+------------+-----------+
                    | 8362 | PS         |         3 |
                    +------+------------+-----------+
                    | 8468 | Info       |         3 |
                    +------+------------+-----------+

                                 Table 9

  This method of assessment does not take into account the number of
  changes proposed by the editors and eventually rejected by the
  authors, since these changes are not present in either the final
  draft or the published RFC.  It might be possible to get an
  evaluation of these "phantom changes" from the RFC Production Center.

4.5.  Independent Stream

  Out of 20 randomly selected RFCs, 3 were published through the
  Independent Stream.  One is an independent opinion, another a
  description of a non-IETF protocol format, and the third was
  [RFC8492], which is a special case.  Apart from this special case,
  the publication delays were significantly shorter for the Independent
  Stream than for the IETF Stream.

  The authors of these 3 RFCs are regular IETF contributors.  This
  observation motivated a secondary analysis of all the RFCs published
  in the Independent Stream in 2018.  There are 14 such RFCs: 8507,
  8494, 8493, 8492, 8483, 8479, 8433, 8409, 8374, 8369, 8367, 8351,
  8328, and 8324.  (RFCs 8367 and 8369 were published on 1 April 2018.)
  The majority of the documents were published by regular IETF
  participants, but two of them were not.  One describes "The BagIt
  File Packaging Format (V1.0)" [RFC8493], and the other the "Yeti DNS
  Testbed" [RFC8483].  They document a data format and a system
  developed outside the IETF and illustrate the outreach function of
  the Independent Stream.  In both cases, the authors include one
  experienced IETF participant, who presumably helped outsiders
  navigate the publication process.

  The present document experienced some publication delays due to the
  Independent Submissions Editor.  The ISE is a bottleneck and is a
  volunteer resource.  Although the ISE as a lone person operating as a
  volunteer is still roughly adequate resource for the job, the
  delivery will necessarily be best effort with delays caused by spikes
  in ISE load, work commitments, and other life events.  These delays
  may not be fundamentally critical to RFC delivery, but they are
  capable of introducing a significant percentage delay into what might
  otherwise be a smooth process.

5.  Citation Counts

  In this exploration, we want to examine whether citation counts
  provide a meaningful assessment of the popularity of RFCs.  We obtain
  the citation counts through the Semantic Scholar API, using queries
  of the form: <https://api.semanticscholar.org/v1/paper/10.17487/
  rfc8446?include_unknown_references=true>

  In these queries, the RFC is uniquely identified by its DOI
  reference, which is composed of the RFC Series prefix 10.17487 and
  the RFC identifier.  The queries return a series of properties,
  including a list of citations for the RFC.  Based on that list of
  citations, we compute three numbers:

  *  The total number of citations

  *  The number of citations in the year of publication and the year
     after that

  *  For the RFC published in 1998 or 2008 that we use for comparison,
     the number of citations in the years 2018 and 2019.

  All the numbers were retrieved on October 6, 2019.

5.1.  Citation Numbers

  As measured on October 6, 2019, the citation counts for the RFC in
  our sample set were:

             +============+============+=======+===========+
             | RFC (2018) | Status     | Total | 2018-2019 |
             +============+============+=======+===========+
             | 8411       | Info       |     1 |         0 |
             +------------+------------+-------+-----------+
             | 8456       | Info       |     1 |         1 |
             +------------+------------+-------+-----------+
             | 8446       | PS         |   418 |       204 |
             +------------+------------+-------+-----------+
             | 8355       | Info       |     3 |         3 |
             +------------+------------+-------+-----------+
             | 8441       | PS         |     1 |         1 |
             +------------+------------+-------+-----------+
             | 8324       | Info (ISE) |     0 |         0 |
             +------------+------------+-------+-----------+
             | 8377       | PS         |     0 |         0 |
             +------------+------------+-------+-----------+
             | 8498       | Info       |     0 |         0 |
             +------------+------------+-------+-----------+
             | 8479       | Info (ISE) |     0 |         0 |
             +------------+------------+-------+-----------+
             | 8453       | Info       |     3 |         3 |
             +------------+------------+-------+-----------+
             | 8429       | BCP        |     0 |         0 |
             +------------+------------+-------+-----------+
             | 8312       | Info       |    25 |        16 |
             +------------+------------+-------+-----------+
             | 8492       | Info (ISE) |     4 |         4 |
             +------------+------------+-------+-----------+
             | 8378       | Exp        |     1 |         1 |
             +------------+------------+-------+-----------+
             | 8361       | PS         |     0 |         0 |
             +------------+------------+-------+-----------+
             | 8472       | PS         |     1 |         1 |
             +------------+------------+-------+-----------+
             | 8471       | PS         |     1 |         1 |
             +------------+------------+-------+-----------+
             | 8466       | PS         |     0 |         0 |
             +------------+------------+-------+-----------+
             | 8362       | PS         |     1 |         1 |
             +------------+------------+-------+-----------+
             | 8468       | Info       |     1 |         1 |
             +------------+------------+-------+-----------+

                                 Table 10

  The results indicate that [RFC8446] is by far the most cited of the
  20 RFC in our sample.  This is not surprising, since TLS is a key
  Internet Protocol.  The TLS 1.3 protocol was also the subject of
  extensive studies by researchers, and thus was mentioned in a number
  of published papers.  Surprisingly, the Semantic Scholar mentions a
  number of citations that predate the publication date.  These are
  probably citations of the various draft versions of the protocol.

  The next most cited RFC in the sample is [RFC8312] which describes
  the Cubic congestion control algorithm for TCP.  That protocol was
  also the target of a large number of academic publications.  The
  other RFCs in the sample only have a small number of citations.

  There is probably a small bias when measuring citations at a fixed
  date.  An RFC published in January 2018 would have more time to
  accrue citations than one published in December.  That may be true to
  some extent, as the second most cited RFC in the set was published in
  January.  However, the effect has to be limited.  The most cited RFC
  was published in August, and the second most cited was published in
  2019.  (That RFC got an RFC number in 2018, but publication was
  slowed by long AUTH48 delays.)

5.2.  Comparison to 1998 and 2008

  In order to get a baseline, we can look at the number of references
  for the RFCs published in 2008 and 1998.  However, we need to take
  time into account.  Documents published a long time ago are expected
  to have accrued more references.  We try to address this by looking
  at three counts for each document: the overall number of references
  over the document's lifetime, the number of references obtained in
  the year following publication, and the number of references observed
  since 2018:

       +============+============+=======+===========+===========+
       | RFC (2008) | Status     | Total | 2008-2009 | 2018-2019 |
       +============+============+=======+===========+===========+
       | 5326       | Exp        |   138 |        14 |        15 |
       +------------+------------+-------+-----------+-----------+
       | 5348       | PS         |    14 |         3 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 5281       | Info       |    69 |        15 |         7 |
       +------------+------------+-------+-----------+-----------+
       | 5354       | Exp        |    17 |        13 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 5227       | PS         |    19 |         1 |         2 |
       +------------+------------+-------+-----------+-----------+
       | 5329       | PS         |    24 |         6 |         1 |
       +------------+------------+-------+-----------+-----------+
       | 5277       | PS         |    32 |         3 |         2 |
       +------------+------------+-------+-----------+-----------+
       | 5236       | Info (ISE) |    25 |         5 |         4 |
       +------------+------------+-------+-----------+-----------+
       | 5358       | BCP        |    21 |         2 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 5271       | Info       |     7 |         2 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 5195       | PS         |     7 |         4 |         2 |
       +------------+------------+-------+-----------+-----------+
       | 5283       | PS         |     8 |         1 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 5186       | Info       |    14 |         4 |         2 |
       +------------+------------+-------+-----------+-----------+
       | 5142       | PS         |     8 |         4 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 5373       | PS         |     5 |         2 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 5404       | PS         |     1 |         1 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 5172       | PS         |     2 |         0 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 5349       | Info       |     8 |         0 |         2 |
       +------------+------------+-------+-----------+-----------+
       | 5301       | PS         |     5 |         1 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 5174       | Info       |     0 |         0 |         0 |
       +------------+------------+-------+-----------+-----------+

                                 Table 11

       +============+============+=======+===========+===========+
       | RFC (1998) | Status     | Total | 1998-1999 | 2018-2019 |
       +============+============+=======+===========+===========+
       | 2289       | PS         |     2 |         0 |         1 |
       +------------+------------+-------+-----------+-----------+
       | 2267       | Info       |   982 |         5 |        61 |
       +------------+------------+-------+-----------+-----------+
       | 2317       | BCP        |     9 |         1 |         2 |
       +------------+------------+-------+-----------+-----------+
       | 2404       | PS         |   137 |         6 |         1 |
       +------------+------------+-------+-----------+-----------+
       | 2374       | PS         |    42 |         4 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 2449       | PS         |     7 |         2 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 2283       | PS         |    17 |         3 |         2 |
       +------------+------------+-------+-----------+-----------+
       | 2394       | Info       |    13 |         2 |         1 |
       +------------+------------+-------+-----------+-----------+
       | 2348       | DS         |     5 |         0 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 2382       | Info       |    17 |        12 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 2297       | Info (ISE) |    36 |        11 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 2381       | PS         |    39 |        12 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 2312       | Info       |    14 |         3 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 2387       | PS         |     4 |         1 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 2398       | Info       |    17 |         0 |         1 |
       +------------+------------+-------+-----------+-----------+
       | 2391       | PS         |    31 |         3 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 2431       | PS         |     3 |         0 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 2282       | Info       |     8 |         0 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 2323       | Info (ISE) |     1 |         0 |         0 |
       +------------+------------+-------+-----------+-----------+
       | 2448       | Info (ISE) |     0 |         0 |         0 |
       +------------+------------+-------+-----------+-----------+

                                 Table 12

  We can compare the median number of citations and the numbers of
  citations for the least and most popular quartiles in the three
  years:

    +============================+===========+========+============+
    | References                 | Lower 25% | Median | Higher 25% |
    +============================+===========+========+============+
    | RFC (2018)                 |         0 |      1 |          3 |
    +----------------------------+-----------+--------+------------+
    | RFC (2008)                 |       6.5 |     11 |      21.75 |
    +----------------------------+-----------+--------+------------+
    | RFC (2008), until 2009     |         1 |    2.5 |        4.5 |
    +----------------------------+-----------+--------+------------+
    | RFC (2008), 2018 and after |         0 |      0 |          2 |
    +----------------------------+-----------+--------+------------+
    | RFC (1998)                 |      4.75 |   13.5 |      32.25 |
    +----------------------------+-----------+--------+------------+
    | RFC (1998), until 1999     |         0 |      2 |       4.25 |
    +----------------------------+-----------+--------+------------+
    | RFC (1998), 2018 and after |         0 |      0 |          1 |
    +----------------------------+-----------+--------+------------+

                                Table 13

  The total numbers show new documents with fewer citations than the
  older ones.  This can be explained to some degree by the passage of
  time.  If we restrict the analysis to the number of citations accrued
  in the year of publishing and the year after that, we still see about
  the same distribution for the three samples.

  We also see that the number of references to RFCs fades over time.
  Only the most popular of the RFC produced in 1998 are still cited in
  2019.

5.3.  Citations versus Deployments

  The following table shows side by side the number of citations as
  measured in Section 5.1 and the estimation of deployment as indicated
  in Section 3.

          +============+============+===========+============+
          | RFC (2018) | Status     | Citations | Deployment |
          +============+============+===========+============+
          | 8411       | Info       |         1 |     medium |
          +------------+------------+-----------+------------+
          | 8456       | Info       |         1 |     medium |
          +------------+------------+-----------+------------+
          | 8446       | PS         |       418 |       high |
          +------------+------------+-----------+------------+
          | 8355       | Info       |         3 |     medium |
          +------------+------------+-----------+------------+
          | 8441       | PS         |         1 |       high |
          +------------+------------+-----------+------------+
          | 8324       | Info (ISE) |         0 |        N/A |
          +------------+------------+-----------+------------+
          | 8377       | PS         |         0 |    unknown |
          +------------+------------+-----------+------------+
          | 8498       | Info       |         0 |    unknown |
          +------------+------------+-----------+------------+
          | 8479       | Info (ISE) |         0 |        one |
          +------------+------------+-----------+------------+
          | 8453       | Info       |         3 |    unknown |
          +------------+------------+-----------+------------+
          | 8429       | BCP        |         0 |       some |
          +------------+------------+-----------+------------+
          | 8312       | Info       |        25 |       high |
          +------------+------------+-----------+------------+
          | 8492       | Info (ISE) |         4 |        one |
          +------------+------------+-----------+------------+
          | 8378       | Exp        |         1 |       some |
          +------------+------------+-----------+------------+
          | 8361       | PS         |         0 |        one |
          +------------+------------+-----------+------------+
          | 8472       | PS         |         1 |     medium |
          +------------+------------+-----------+------------+
          | 8471       | PS         |         1 |     medium |
          +------------+------------+-----------+------------+
          | 8466       | PS         |         0 |    unknown |
          +------------+------------+-----------+------------+
          | 8362       | PS         |         1 |     medium |
          +------------+------------+-----------+------------+
          | 8468       | Info       |         1 |       some |
          +------------+------------+-----------+------------+

                                Table 14

  From looking at these results, it is fairly obvious that citation
  counts cannot be used as proxies for the "value" of an RFC.  In our
  sample, the two RFCs that have high citation counts were both widely
  deployed, and can certainly be described as successful, but we also
  see many RFCs that saw significant deployment without garnering a
  high level of citations.

  Citation counts are driven by academic interest, but are only loosely
  correlated with actual deployment.  We saw that [RFC8446] was widely
  cited in part because the standardization process involved many
  researchers, and that the high citation count of [RFC8312] is largely
  due to the academic interest in evaluating congestion control
  protocols.  If we look at previous years, the most cited RFC in the
  2008 sample is [RFC5326], an experimental RFC defining security
  extensions to an experimental delay tolerant transport protocol.
  This protocol does not carry a significant proportion of Internet
  traffic, but has been the object of a fair number of academic
  studies.

  The citation process tends to privilege the first expression of a
  concept.  We see that with the most cited RFC in the 1998 set is
  [RFC2267], an informational RFC defining Network Ingress Filtering
  that was obsoleted in May 2000 by [RFC2827].  It is still cited
  frequently in 2018 and 2019, regardless of its formal status in the
  RFC Series.  We see the same effect at work with [RFC8441], which
  garners very few citations although it updates [RFC6455] that has a
  large number of citations.  The same goes for [RFC8468], which is
  sparsely cited while the [RFC2330] is widely cited.  Just counting
  citations will not indicate whether developers still use an old
  specification or have adopted the revised RFC.

5.4.  Citations versus Web References

  Web references might be another indicator of the popularity of an
  RFC.  In order to evaluate these references, we list here the number
  of results returned by searches on Google and Bing, looking for the
  search term "RFCnnnn" (e.g., "RFC8411"), and copying the number of
  results returned by the search engines.  The table below presents the
  results of these searches, performed on April 4, 2020.

        +============+============+===========+========+=======+
        | RFC (2018) | Status     | Citations | Google |  Bing |
        +============+============+===========+========+=======+
        | 8411       | Info       |         1 |    301 |    94 |
        +------------+------------+-----------+--------+-------+
        | 8456       | Info       |         1 |    266 |  8456 |
        +------------+------------+-----------+--------+-------+
        | 8446       | PS         |       418 |  25900 | 47800 |
        +------------+------------+-----------+--------+-------+
        | 8355       | Info       |         3 |    521 |   114 |
        +------------+------------+-----------+--------+-------+
        | 8441       | PS         |         1 |   2430 | 59500 |
        +------------+------------+-----------+--------+-------+
        | 8324       | Info (ISE) |         0 |    393 |   138 |
        +------------+------------+-----------+--------+-------+
        | 8377       | PS         |         0 |    264 | 10900 |
        +------------+------------+-----------+--------+-------+
        | 8498       | Info       |         0 |    335 | 10100 |
        +------------+------------+-----------+--------+-------+
        | 8479       | Info (ISE) |         0 |    564 | 11000 |
        +------------+------------+-----------+--------+-------+
        | 8453       | Info       |         3 |    817 | 11400 |
        +------------+------------+-----------+--------+-------+
        | 8429       | BCP        |         0 |    391 | 41600 |
        +------------+------------+-----------+--------+-------+
        | 8312       | Info       |        25 |   1620 |  2820 |
        +------------+------------+-----------+--------+-------+
        | 8492       | Info (ISE) |         4 |    323 |  9400 |
        +------------+------------+-----------+--------+-------+
        | 8378       | Exp        |         1 |    418 | 11600 |
        +------------+------------+-----------+--------+-------+
        | 8361       | PS         |         0 |    499 |    92 |
        +------------+------------+-----------+--------+-------+
        | 8472       | PS         |         1 |    496 |   169 |
        +------------+------------+-----------+--------+-------+
        | 8471       | PS         |         1 |   1510 | 11600 |
        +------------+------------+-----------+--------+-------+
        | 8466       | PS         |         0 |    766 |   173 |
        +------------+------------+-----------+--------+-------+
        | 8362       | PS         |         1 |     67 |   147 |
        +------------+------------+-----------+--------+-------+
        | 8468       | Info       |         1 |    453 |   127 |
        +------------+------------+-----------+--------+-------+

                                Table 15

  The result counts from Bing are sometimes surprising.  Why would RFC
  8441 gather 59,500 web references?  Looking at the results in detail,
  we find a mix of data.  Some of them are logs of development projects
  implementing Web Sockets, which is exactly what we are looking for,
  but others appear spurious.  For example, a shop selling rugby
  jerseys is listed because its phone number ends with "8441".  Other
  pages were listed because street numbers or product numbers matched
  the RFC number.  The same type of collision may explain the large
  reference counts on Bing for RFCs 8377, 8498, 8479, 8453, 8429, 8378,
  and 8471.  The result counts on Bing do not appear to provide a good
  metric.

  On Google, all RFCs garner at least a 250 references, largely because
  the whole RFC catalog is replicated on a large number of web servers.
  Deviations from that baseline are largely correlated with the number
  of citations in the Semantic Scholar, with a couple of exception: RFC
  8441 and RFC 8471 garner more references than the low citation counts
  would predict.  Looking at the results, we find many references in
  development databases explaining how these protocols are implemented
  in various code bases and open source projects.  This means that
  counting Google results would give some indication about an RFC's
  popularity, complementing the citation counts.

  There are some practical problems in using the counts of Google
  results.  Google searches are personalized, the results depend on the
  source of the queries, and the counts may vary as well.  The search
  results depend on the search algorithm, and there is no guarantee
  that counts will not change when the algorithm changes.  On the other
  hand, the results do indicate that some of the RFCs in our sample are
  being used by developers or in deployments.

6.  Observations and Next Steps

  The author's goal was to get a personal understanding of the "chain
  of production" of the RFCs, and in particular to look at the various
  causes of delays in the process.  As shown in Section 4, the average
  RFC was produced in 3 years and 4 months, which is similar to what
  was found in the 2008 sample, but more than three times larger than
  the delays for the 1998 sample.

  The working group process appears to be the main source of delays.
  Efforts to diminish delays should probably focus there, instead of on
  the IETF and IESG reviews or the RFC production.  For the RFC
  production phase, most of the variability originates in the AUTH48
  process, which is influenced by a variety of factors such as number
  of authors or level of engagement of these authors.

  Most of the delay is spent in the working group, but the IETF
  Datatracker does not hold much information about what happens inside
  the working groups.  For example, events like Working Group Last
  Calls were not recorded in the history of the selected drafts
  available in the Datatracker.  Such information would have been
  interesting.  Of course, requiring that information would create an
  administrative burden, so there is clearly a trade-off between
  requiring more work from working group chairs and providing better
  data for process analysis.  (It appears that this information can be
  available in the Datatracker for more recent drafts, if the WG chairs
  use the Datatracker properly.)

  The Independent Stream operates as expected.  The majority of the
  authors of the Independent Stream RFCs appear to be in IETF insiders,
  but there is significant amount of engagement by outside parties.

  The analysis of citations in Section 5.1 shows that citation numbers
  are a very poor indication of the "value" of an RFC.  Citation
  numbers measure the engagement of academic researchers with specific
  topics, but have little correlation with the level of adoption and
  deployment of a specific RFC.  The result counts of Google searches
  do capture references outside academia, such as logs of development
  projects.  This might be informative, but it is not clear that the
  counts would not change over time due to algorithm changes or
  personalization.

  This document analyses a small sample of RFCs "in depth".  This
  allowed gathering of detailed feedback on the process and the
  deployments.  On the other hand, much of the data on delays is
  available from the IETF Datatracker.  It may be worth considering
  adding an automated reporting of delay metrics in the IETF
  Datatracker.

  This document only considers the RFCs that were published in a given
  year.  This approach can be criticized as introducing a form of
  "survivor bias".  There are many drafts proposed to the IETF, and
  only a fraction of them end up being published as RFCs.  On one hand,
  this is expected, because part of the process is to triage between
  ideas that can gather consensus and those that don't.  On the other
  hand, we don't know whether that triage is too drastic and has
  discouraged progress on good ideas.

  One way to evaluate the triage process would be to look at
  publication attempts that were abandoned -- for example, drafts that
  expired without progressing or being replaced.  The sampling
  methodology could also be used for that purpose.  Pick maybe 20
  drafts at random, among those abandoned in a target year, and
  investigate why they were abandoned.  Was it because better solutions
  emerged in the working group?  Or maybe because the authors
  discovered a flaw in their proposal?  Or was it because some
  factional struggle blocked a good idea?  Was the idea pursued in a
  different venue?  Hopefully, someone will try this kind of
  investigation.

7.  Security Considerations

  This document does not specify any protocol.

  We might want to analyze whether security issues were discovered
  after publication of specific standards.

8.  IANA Considerations

  This document has no IANA actions.

  Preliminary analysis does not indicate that IANA is causing any
  particular delay in the RFC publication process.

9.  Informative References

  [IETFCOUNT]
             IETF, "Past IETF Meetings",
             <https://www.ietf.org/how/meetings/past/>.

  [RFC2267]  Ferguson, P. and D. Senie, "Network Ingress Filtering:
             Defeating Denial of Service Attacks which employ IP Source
             Address Spoofing", RFC 2267, DOI 10.17487/RFC2267, January
             1998, <https://www.rfc-editor.org/info/rfc2267>.

  [RFC2330]  Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
             "Framework for IP Performance Metrics", RFC 2330,
             DOI 10.17487/RFC2330, May 1998,
             <https://www.rfc-editor.org/info/rfc2330>.

  [RFC2827]  Ferguson, P. and D. Senie, "Network Ingress Filtering:
             Defeating Denial of Service Attacks which employ IP Source
             Address Spoofing", BCP 38, RFC 2827, DOI 10.17487/RFC2827,
             May 2000, <https://www.rfc-editor.org/info/rfc2827>.

  [RFC5326]  Ramadas, M., Burleigh, S., and S. Farrell, "Licklider
             Transmission Protocol - Specification", RFC 5326,
             DOI 10.17487/RFC5326, September 2008,
             <https://www.rfc-editor.org/info/rfc5326>.

  [RFC6455]  Fette, I. and A. Melnikov, "The WebSocket Protocol",
             RFC 6455, DOI 10.17487/RFC6455, December 2011,
             <https://www.rfc-editor.org/info/rfc6455>.

  [RFC8312]  Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and
             R. Scheffenegger, "CUBIC for Fast Long-Distance Networks",
             RFC 8312, DOI 10.17487/RFC8312, February 2018,
             <https://www.rfc-editor.org/info/rfc8312>.

  [RFC8324]  Klensin, J., "DNS Privacy, Authorization, Special Uses,
             Encoding, Characters, Matching, and Root Structure: Time
             for Another Look?", RFC 8324, DOI 10.17487/RFC8324,
             February 2018, <https://www.rfc-editor.org/info/rfc8324>.

  [RFC8355]  Filsfils, C., Ed., Previdi, S., Ed., Decraene, B., and R.
             Shakir, "Resiliency Use Cases in Source Packet Routing in
             Networking (SPRING) Networks", RFC 8355,
             DOI 10.17487/RFC8355, March 2018,
             <https://www.rfc-editor.org/info/rfc8355>.

  [RFC8361]  Hao, W., Li, Y., Durrani, M., Gupta, S., and A. Qu,
             "Transparent Interconnection of Lots of Links (TRILL):
             Centralized Replication for Active-Active Broadcast,
             Unknown Unicast, and Multicast (BUM) Traffic", RFC 8361,
             DOI 10.17487/RFC8361, April 2018,
             <https://www.rfc-editor.org/info/rfc8361>.

  [RFC8362]  Lindem, A., Roy, A., Goethals, D., Reddy Vallem, V., and
             F. Baker, "OSPFv3 Link State Advertisement (LSA)
             Extensibility", RFC 8362, DOI 10.17487/RFC8362, April
             2018, <https://www.rfc-editor.org/info/rfc8362>.

  [RFC8377]  Eastlake 3rd, D., Zhang, M., and A. Banerjee, "Transparent
             Interconnection of Lots of Links (TRILL): Multi-Topology",
             RFC 8377, DOI 10.17487/RFC8377, July 2018,
             <https://www.rfc-editor.org/info/rfc8377>.

  [RFC8378]  Moreno, V. and D. Farinacci, "Signal-Free Locator/ID
             Separation Protocol (LISP) Multicast", RFC 8378,
             DOI 10.17487/RFC8378, May 2018,
             <https://www.rfc-editor.org/info/rfc8378>.

  [RFC8402]  Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L.,
             Decraene, B., Litkowski, S., and R. Shakir, "Segment
             Routing Architecture", RFC 8402, DOI 10.17487/RFC8402,
             July 2018, <https://www.rfc-editor.org/info/rfc8402>.

  [RFC8410]  Josefsson, S. and J. Schaad, "Algorithm Identifiers for
             Ed25519, Ed448, X25519, and X448 for Use in the Internet
             X.509 Public Key Infrastructure", RFC 8410,
             DOI 10.17487/RFC8410, August 2018,
             <https://www.rfc-editor.org/info/rfc8410>.

  [RFC8411]  Schaad, J. and R. Andrews, "IANA Registration for the
             Cryptographic Algorithm Object Identifier Range",
             RFC 8411, DOI 10.17487/RFC8411, August 2018,
             <https://www.rfc-editor.org/info/rfc8411>.

  [RFC8429]  Kaduk, B. and M. Short, "Deprecate Triple-DES (3DES) and
             RC4 in Kerberos", BCP 218, RFC 8429, DOI 10.17487/RFC8429,
             October 2018, <https://www.rfc-editor.org/info/rfc8429>.

  [RFC8441]  McManus, P., "Bootstrapping WebSockets with HTTP/2",
             RFC 8441, DOI 10.17487/RFC8441, September 2018,
             <https://www.rfc-editor.org/info/rfc8441>.

  [RFC8446]  Rescorla, E., "The Transport Layer Security (TLS) Protocol
             Version 1.3", RFC 8446, DOI 10.17487/RFC8446, August 2018,
             <https://www.rfc-editor.org/info/rfc8446>.

  [RFC8453]  Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for
             Abstraction and Control of TE Networks (ACTN)", RFC 8453,
             DOI 10.17487/RFC8453, August 2018,
             <https://www.rfc-editor.org/info/rfc8453>.

  [RFC8455]  Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V.,
             and S. Banks, "Terminology for Benchmarking Software-
             Defined Networking (SDN) Controller Performance",
             RFC 8455, DOI 10.17487/RFC8455, October 2018,
             <https://www.rfc-editor.org/info/rfc8455>.

  [RFC8456]  Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V.,
             and S. Banks, "Benchmarking Methodology for Software-
             Defined Networking (SDN) Controller Performance",
             RFC 8456, DOI 10.17487/RFC8456, October 2018,
             <https://www.rfc-editor.org/info/rfc8456>.

  [RFC8466]  Wen, B., Fioccola, G., Ed., Xie, C., and L. Jalil, "A YANG
             Data Model for Layer 2 Virtual Private Network (L2VPN)
             Service Delivery", RFC 8466, DOI 10.17487/RFC8466, October
             2018, <https://www.rfc-editor.org/info/rfc8466>.

  [RFC8468]  Morton, A., Fabini, J., Elkins, N., Ackermann, M., and V.
             Hegde, "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for
             the IP Performance Metrics (IPPM) Framework", RFC 8468,
             DOI 10.17487/RFC8468, November 2018,
             <https://www.rfc-editor.org/info/rfc8468>.

  [RFC8471]  Popov, A., Ed., Nystroem, M., Balfanz, D., and J. Hodges,
             "The Token Binding Protocol Version 1.0", RFC 8471,
             DOI 10.17487/RFC8471, October 2018,
             <https://www.rfc-editor.org/info/rfc8471>.

  [RFC8472]  Popov, A., Ed., Nystroem, M., and D. Balfanz, "Transport
             Layer Security (TLS) Extension for Token Binding Protocol
             Negotiation", RFC 8472, DOI 10.17487/RFC8472, October
             2018, <https://www.rfc-editor.org/info/rfc8472>.

  [RFC8479]  Mavrogiannopoulos, N., "Storing Validation Parameters in
             PKCS#8", RFC 8479, DOI 10.17487/RFC8479, September 2018,
             <https://www.rfc-editor.org/info/rfc8479>.

  [RFC8483]  Song, L., Ed., Liu, D., Vixie, P., Kato, A., and S. Kerr,
             "Yeti DNS Testbed", RFC 8483, DOI 10.17487/RFC8483,
             October 2018, <https://www.rfc-editor.org/info/rfc8483>.

  [RFC8492]  Harkins, D., Ed., "Secure Password Ciphersuites for
             Transport Layer Security (TLS)", RFC 8492,
             DOI 10.17487/RFC8492, February 2019,
             <https://www.rfc-editor.org/info/rfc8492>.

  [RFC8493]  Kunze, J., Littman, J., Madden, E., Scancella, J., and C.
             Adams, "The BagIt File Packaging Format (V1.0)", RFC 8493,
             DOI 10.17487/RFC8493, October 2018,
             <https://www.rfc-editor.org/info/rfc8493>.

  [RFC8498]  Mohali, M., "A P-Served-User Header Field Parameter for an
             Originating Call Diversion (CDIV) Session Case in the
             Session Initiation Protocol (SIP)", RFC 8498,
             DOI 10.17487/RFC8498, February 2019,
             <https://www.rfc-editor.org/info/rfc8498>.

  [RFCYEAR]  RFC Editor, "Number of RFC Published per YEAR",
             <https://www.rfc-editor.org/rfcs-per-year/>.

  [SSCH]     Allen Institute for AI, "Semantic Scholar | AI-Powered
             Research Tool", <https://www.semanticscholar.org/>.

  [TI-LFA]   Litkowski, S., Bashandy, A., Filsfils, C., Decraene, B.,
             and D. Voyer, "Topology Independent Fast Reroute using
             Segment Routing", Work in Progress, Internet-Draft, draft-
             ietf-rtgwg-segment-routing-ti-lfa-05, 15 November 2020,
             <https://tools.ietf.org/html/draft-ietf-rtgwg-segment-
             routing-ti-lfa-05>.

  [TLS13IMP] TLS WG, "TLS 1.3 Implementations", commit dcb7890, 14
             October 2019, <https://github.com/tlswg/tlswg-
             wiki/blob/master/IMPLEMENTATIONS.md>.

  [TRKR]     IETF, "IETF Datatracker", <https://datatracker.ietf.org/>.

Acknowledgements

  Many thanks to the authors of the selected RFCs who were willing to
  provide feedback on the process: Michael Ackermann, Zafar Ali, Sarah
  Banks, Bruno Decraene, Lars Eggert, Nalini Elkins, Joachim Fabini,
  Dino Farinacci, Clarence Filsfils, Sujay Gupta, Dan Harkins, Vinayak
  Hegde, Benjamin Kaduk, John Klensin, Acee Lindem, Nikos
  Mavrogiannopoulos, Patrick McManus, Victor Moreno, Al Morton, Andrei
  Popov, Eric Rescorla, Michiko Short, Bhuvaneswaran Vengainathan, Lao
  Weiguo, and Li Yizhou.  Many thanks to Adrian Farrel for his useful
  advice, to Stephen Farrell and Colin Perkins for their guidance on
  the use of citations, and to Dave Crocker for a comprehensive review.
  Thanks also to Alice Russo and the RFC Editor team for their work
  improving this document and checking the accuracy of the data.

Author's Address

  Christian Huitema
  Private Octopus Inc.
  427 Golfcourse Rd
  Friday Harbor, WA 98250
  United States of America

  Email: [email protected]

You are viewing proxied material from gopher.fnord.one. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.