https://www.ics.uci.edu/~rohit/IEEE-L7-http-gopher.html

Who Killed Gopher?
An Extensible Murder Mystery


   By Rohit Khare // December 23, 1998

  As best our forensics team can reconstruct, a serial killer
  first surfaced in the beginning of 1991, born of an
  academic's midnight hack gone awry. NeXTstep users, you see
  -- precisely the kind of object-oriented revolutionaries
  that would think they could get away with inventing a brand
  new protocol for a mature Internet.

  The first victim was the campus phonebook: it dismembered
  the whole corpus into tiny bits of `hypertext', left
  hanging together by search strings alone. Here we see its
  modus operandi: not content to slice information into neat
  lists or trees, it left behind completely unstructured
  graphs littered with redirects hither and yon across the
  Internet to co-conspirators' servers.

  Hegemonic fantasies drove it to assimilate more and more
  multimedia formats at each node. Crude at first, shoveling
  any old file through an 8-bit clean channel, it grew more
  explicit until it could label text, images, audio, even
  compound documents. By now, it can even masquerade behind
  several renderings, negotiating whatever face -- or
  language! -- pleases the victim.

  As a protocol, it was a blunt weapon: aim a socket at the
  target, send a one-line demand, and slurp down the response
  until the connection died. The crazy way it grew from there
  is proof itself that the killer dropped out of Application
  Layer Protocol design school at an early age it shows no
  understanding of the basics of TCP, botching handshakes,
  slow-start, server-sends-first-byte, even tripping over the
  Nagle timer. Instead, in a stroke of streetwise punk
  genius, it stayed 'simple' enough that any two-bit Telnet
  client could take a hit.

  And what an addictive product it was! Workgroupies were
  seduced by the ease of installing new servers and authoring
  new content -- and pushing free software helped the lock-in
  cycle along. The killer preyed on individual users'
  self-esteem by promising instant fame^*, in the form of a
  backlink from the master list of servers.

    ^*Anyone have firm attribution for the famous quote, "On
    the Internet, everyone will be famous for 15 people"?

  Soon, it was assimilating more than bags of bits: it was a
  gateway drug to harder services. Scripts, database lookups,
  even long-established news, file, and email access
  transactions were reduced to mere documents in a menu. The
  address sent in the 'demand' grew into a miniature RPC.

  All the while, with every new user, every new server, every
  nugget of information it wrested from 'legacy' protocols,
  the killer bled off a little more of the Internet's
  bandwidth -- getting away with murder with every handshake!

  Such profligacy is no idle threat: for a while, the killer
  was the largest share of application packet traffic on the
  national NSFNET backbone.

  Or was that the victim?

  Did the killer come knocking on port 70 or port 80? From
  boombox.micro.umn.edu or info.cern.ch? Bearing the
  imprimatur of campus administrators or physicists? Are we
  hunting gophers or spiders?

Dial 'H' For Murder

  In fact, the profile in Table 1 fits both suspects to a
  'T'. In the early '90s, an interesting drama played out
  between two contenders for the first integrated,
  interactive internet information retrieval protocols --
  neither of which, arguably, had any technical reason to
  exist.

  That's not to say the systems weren't inevitable: everyone
  suspected there'd have to be an easier way to use the
  Internet in the '90s than UNIX shell experience. That
  encompasses innovations in user interface, authoring
  formats, and above all, resource identification. It does
  not, however, motivate the original editions of the Gopher
  and HyperText Transfer Protocols. They were painfully
  trivial hacks that did nothing FTP didn't already handle.

  So here begins our mysterious tale: why either protocol
  ever rose to prominence in the first place; and how the
  fratricidial drama eventually played out. Understanding how
  HTTP killed gopher may lead us to the forces which may, in
  turn, topple HTTP.

  Science proves a blind alley, though. Their fates were
  decided not on technical merits, but on economic and
  psychological advantages. The 'Postellian' school of
  protocol design focused on engineering 'right' solutions
  for core applications (batch file transfer, interactive
  terminals, mail and news relays) anchored in unique
  transport layer adaptations (slow-start, Nagle timers, and
  routing as respective examples). Our two specimens are
  'post-Postel', in their details and in their adoption
  dynamics. They are stateless; they don't have (Gopher) or
  dilute (HTTP) the theory of reply codes; they scale poorly,
  imperiling the health of the Internet; and they are
  'luxuries' for publishing discretionary information, not
  Host Requirements which must be compiled into every node.

Burrowing into Gopher

  While the Web was arguably invented a few months before
  Gopher, the latter was the first to garner public notice.
  By the fall of 1994, RFC 1689's census of information
  retrieval tools estimated there were 4,800 Gophers, and
  1,200 anonymous FTP archives, but only 600 Webs. Early and
  "explosive" adoption had a lot to do with Gopher's
  "fiercely simple" design, and hence its wide availabilty on
  a number of students' (DOS, Windows, Mac, NeXT) and campus
  IS departments' (UNIXes, VMS, VM/CMS, MVS) platforms.

  It was designed by the microcomputer support folks at the
  University of Minnesota for a teletext-like Campus-Wide
  Information System (CWIS). The original prototype was a
  multicolumn tree-browser pioneered for the NeXTstep file
  browser. Each scrolling column had a list of titles;
  clicking on an entry would either display that file in the
  pane below, or load the child directory listing in the
  column to the right.

  Naturally, the messages on the wire follow directly: open a
  connection to the target server on port 70; send a one-line
  'selector' string; get back either the file itself, or a
  list of further selector lines, each prefixed by a
  character indicating the type of file it points to. Table 2
  lists the rudimentary type system Gopher shipped with,
  illustrating on one hand its ambitions to assimilate new
  media types and on the other what a thin skein it was
  around protocol- and platform-specific data formats.

  The flip side of 'thin skein' is 'ease of implementation',
  though. In effect, Gopher was a (very!) poor man's version
  of FTP, sans authentication, navigation, authoring, or
  bulk/restartable transfers. It only used TCP as a
  half-duplex fopen() call. It just happened that some of the
  "files" it transferred were menus, in a Gopher-specific,
  tab-delimited, US-ASCII format. So a Gopher server did
  little more than read files out of a specified directory
  tree; and clients could be hacked out of spare Telnet
  client code.

  A little further up the evolutionary tree, servers allowed
  scripts and other programs to be served as well. Rather
  than copying a file back across the socket, it connected it
  to some process's output. Similarly, multiprotocol clients
  could recognize a non-Gopher selector and switch to another
  language (as for phonebook queries). "Gateways exist to
  seamlessly access a variety of non-Gopher services such as
  ftp, WAIS, USENET news, Archie, Z39.50 (1992 rev), X.500
  directories, Sybase and Oracle SQL servers, etc."

  The base protocol didn't leverage TCP well, though. For
  example, TCP's three-way handshake ends with a packet from
  the server to the client -- so even when the client opens
  the connection, the server can send data first for free.
  Gopher did not have a server-hello message, and thus no
  version number to key upgrades off of. To signal the end of
  a particular transmission, Gopher borrowed the SMTP/NNTP
  convention of <CR><LF>.<CR><LF>. This only worked for its
  own menu format, though -- as soon as an actual file was
  transferred, TCP FIN had to be used as the EOF.

  This left lots of socket buffers in the FIN_WAIT state at
  the server side, just like the Web. At the same time, the
  Gopher Team was planning to scale to Internet-wide use.
  They did plan for redundant backup servers: a '+' line was
  an alternate for the selector on the previous line. They
  maintained a master list of publically-accessible servers,
  which was in turn indexed regularly by Veronica (Very Easy
  Rodent-Oriented Net-wide Index to Computerized Archives, by
  analogy to the Archie FTP server index).

  In the long-run, though, no one ever plastered Gopher
  selectors on shirts and lunch trucks and golf tees...

Spinning the Web

    "The key insight I credit Tim Berners-Lee with, is the
    URL: the idea that there's a Uniform Resource Locator
    that says I can point at any bit of information on the
    Internet." -- some obscure Web scholar in Stephen
    Segaller's new book Nerds 2.01: A Brief History of the
    Internet

  If anything, Tim's original Web manifesto was even more
  ambitious than Gopher's. Its hook for assimilating "the
  entire universe of network-accessible information" was the
  URI, which is able to subsume entire namespaces as merely
  one more scheme (telnet:, mail:, and so on). URIs reduced
  operational instructions on how to access a resource to a
  declarative address. For example, contrast the one-line
  ftp: URL with the MIME message/external-body part for FTP
  access -- logins, passwords, alternate sites, alternate
  directories, alternate formats. As RFC 1630 intended "URIs
  may, if necessary, be passed using pen and ink."

  But while URIs were the innovative essence of the Web, the
  other two legs of the triad were less steady. HTML is
  merely a markup language which makes it natural to embed
  URIs as hyperlinks. URIs could just as well have been
  popularized as a pointer format within Rich Text Format,
  PostScript, LaTeX, or any other language. HTTP is merely a
  lookup protocol which makes it natural to resolve http:
  URLs or gateway to other schemes. URIs could just as well
  have been popularized as a naming interface to existing
  clients for FTP, Gopher, News, etc.

  Somehow, though, HTTP/0.9 took root. It is a famously
  'simple' protocol, arguably even less useful than Gopher.
  Here is its entire grammar:

    Simple-Request = "GET" SP Request-URI CRLF
    Simple-Response = [ Entity-Body ]

  That's right: it only supported one method, GET, and
  required a new TCP connection for every transaction. Even
  the 1.0 edition, which eventually added authentication and
  navigation and simultaneous transfers of several files for
  a single "page", had no excuse to be born when FTP was on
  the shelf. Except that HTTP had broken authentication,
  nonstandard conventions for navigating directories and
  welcome/index/home pages, and ran spectacularly afoul of
  slow-start for each of its typically-small file transfers.

  If HTTP/0.9 was really shipped for no better reason than
  that programming separate FTP-Control and FTP-Data channels
  seemed too hard, why did it survive? Many of the earliest
  "web" sites were, in fact, just ftp: URLs. The saving grace
  was 1.0's new metadata headers. It augmented the one-line
  request with additional fields for authentication,
  arguments modifying the method, and information about the
  user-agent's preferred languages and formats. The entity
  bodies it returned were prefixed by MIME Content-Types,
  last-modified dates, base-URI, and more. This information,
  finally, was what took HTTP beyond the realm of file
  transfer to hypertext transfer.

  The headers are how HTTP's adoption cycle diverged from
  Gopher's. Both servers were simple to install and author
  content for -- there was even a hybrid gn server which
  could vend files over both port 70 and 80. There were kudos
  for new sites, in the form of the Web Virtual Library
  project at CERN, and later W3C, and early search engines
  like Brian Pinkerton's NeXTstep DigitalLibrarian-derived
  WebCrawler. The client was the defining wedge. Graphical
  web browsers leveraged rich HTML text with MIME media type
  headers to display graphical images, to launch helper
  applications, and present multilingual content.

Extensibility: the fingerprint of a killer

  The straw that broke the Gopher's back wasn't even a
  feature of HTTP per se. The synergy of HTML and HTTP in the
  graphical browser began with FORMs, through refresh META
  tags, up through scripting and applet OBJECTs. While one
  protocol stayed lean, the other opened up so much headroom
  that it grew into a beast so complex even today's base 1.1
  specification takes 160+ pages.

  An avalanche of new headers modified GET to only retrieve
  the full entity if-modified-since an existing copy, or only
  a specified byte-range; enhanced security with keyed-digest
  passwords; even reverse-engineered state back into HTTP
  using 'cookies.' HTTP reply codes were added for payments,
  for cache validation, for upgrading to new protocols.

  Entirely new functionality could be grafted on HTTP using
  new methods. WebDAV added LOCK and PUT; collections which
  allowed MOVE and COPY on entire swaths of URLspace; and
  even went beyond new headers to encoded additional
  parameters in XML. Conversely, HTTP was borrowed wholesale
  for new protocols for printing (IPP), multiparty call setup
  (SIP), and event notification (RVP).

  Politically, any server or client could inaugurate a new
  method; third-parties could even thread themselves into the
  proxy chain and filter messages in transit. Proxies exist
  to add public annotations (Crit-Link Mediator), anonymize
  access (Lucent Personal Web Assistant), convert file
  formats (ProxiNet for PalmIIIs), language translation
  (AltaVista's BabelFish can be so hacked), and content
  filtering (PICS). Decentralized extensiblity was key to
  HTTP's adoption. Rather than waiting for the IETF standards
  process to revise a canonical table of one-letter content
  codes, for example, MIME headers could deploy new
  content-types on the spot.

  At the same time, uncontrolled divergence reduces the value
  of the Web platform as the probability of compatible
  extensions matching up declines. New versions of HTTP would
  appear to be one solution, but there is little political
  will to empanel a standing working group to review a
  grab-bag of extensions for 1.2, 1.3, ... 39.7 and so on.
  What is neccessary? Some hook for identifying extensions,
  the level of support required, and which parts of URLspace
  are so extended.

  W3C has been promoting such mechanisms for years -- three
  calendar years! -- to better "Lead the [Controlled]
  Evolution of the World Wide Web." At first, the motivating
  scenarios were complex economic and social negotiations
  (payments, content selection, privacy, cryptographic
  algorithms, &c). Early revisions of PEP advertised
  preference lists of alternative extensions, applied in
  pipelined sequence. PEP could also transport metadata about
  which extensions applied to other resources and required
  orders.

  Though PEP remained a W3C experiment, the basic ideas of 1)
  packaging an entire extension -- methods, headers,
  encodings and all -- behind a single name and 2) indicating
  clearly if an extensions was required or optional for 3)
  the next hop or end-to-end were reincarnated as the
  Mandatory proposal. The current draft defines four header
  fields enumerating the URIs of the extensions which apply
  (Man: and Opt: for client and server; C-Man: and C-Opt: for
  the next hop). To interoperate with the installed base, any
  request with mandatory extensions prefixes the method name
  with 'M-', which forces 1.0 and 1.1 servers to fail with
  "501 Not Implemented."

Transport: the Achilles' heel

  By April 1995, Web usage passed Gopher's share as well as
  every other Internet application protocol as a fraction of
  NSFnet backbone usage. Even patching its bandwidth leaks
  with 1.1's persistent connections can't mask HTTP's
  inefficient waste of Internet resources. Its content base
  has grown so immense that caching alone can't reduce
  bandwidth demand by a significant margin for the average
  user or ISP. Former strengths have now become liabilities:
  full 'English' headers, stateless transactions, full
  enumeration of client capabilities once critical for
  debugging now inflate request packets while progressive
  rendering of composite web pages places a premium on
  simultaneous downloads (aggravating the share of packets
  wasted on handshakes and delayed by slow-start).

  Individually, these phenomena have appeared in application
  protocols before -- and were nipped in the bud by active
  involvement by transport protocol folks, or coevolved with
  TCP. When Telnet generated a packet per keystroke, even on
  links where you could type faster than the ACKs returned,
  the Nagle timer throttled TCP down to batching keystrokes
  into a single packet per round-trip. When file transfers
  immediately swamped LANs, slow-start eased up to limit
  congestion.

  So when HTTP presents its demand for many small transfers
  in parallel immediately, there are solutions on the shelf.
  Some vendors are already marketing "accelerators" which
  rely on matched encoders at server and client, like
  Sitara's SpeedServer. Unfortunately, these approaches
  aren't backward compatible with HTTP as-is -- nor as simple
  to code.

  W3C's HTTP-NG (Next Generation) project sees it as a
  three-layer problem: message transport, remote invocation,
  and "The Classic Web Application" (TCWA). At the bottom,
  they propose a multiplexing protocol which combines all the
  traffic destined for a single web server into a single TCP
  connection. This allows the conversations regarding
  individual transfers to be diced up independently: an
  intelligent server might respond with the first few
  critical bytes of all the images in the first packet. A
  fully specified W3Mux layer still needs to address flow
  control (gating the relative priority of the subchannels
  without starving them), simultaneous open (which TCP does
  allow), and lots of interoperability testing.

  The middle layer 'compresses' the data to be transferred by
  marshalling arguments in native data types rather than
  English. So numbers would be sent as integers; dates as
  binary fields; and strings as reusable tokens. The tip of
  their proposal (reaching into the cloudy heights of
  improbability, some critics might say) is reengineering the
  Web as a distributed object system, modeled as operations
  on data structures rather than documents.

Building the Perfect Beast

  Having slain Gopher (and a host of other competitors), the
  Web appears to be expanding in all directions today. Just
  as the SMTP state machine defined a style or protocol
  design for a generation, every new application these days
  seems to ape HTTP's state-less style -- if only to
  piggyback a free ride through firewalls at port 80. The
  Internet Architecture Board is on the verge of issuing
  guidelines to rein in this tendency. First, the HTTP port
  should be reserved for protocols which manipulate the kind
  of information already in web servers. WebDAV is OK, since
  it extends documents; IPP would not, since it uses POST as
  a barn door for its printing procedure calls. Second, HTTP
  should not be referenced by-copy. Lots of other efforts
  want to 'borrow' a few headers for authentication or
  navigation and end up copying text into other specs,
  raising the scepter of divergent versions. Third, HTTP is
  not a valid precedent for MIME users -- it plays fast and
  loose with certain rules which enable absolute
  interoperablity through mail gateways. HTTP allows bare
  <CR> and <LF> as well as inventing a new Content-Encoding:
  process.

  The IAB guidelines are a reminder that HTTP still occupies
  a limited niche in the space of possible Transfer
  Protocols. True, its message format can encode any MIME
  entity, even live data streams -- but only from the server
  to the client. True, its address format can point at
  anything -- but even then it can't encode whether the
  resource is expected to be secured, paid for, or otherwise
  reprocessed. And it is most certainly restricted to
  one-to-one, synchronous, request-response (client-pull)
  interaction. All the various hacks to 'push' data to groups
  of subscribers are, well, hacks: automatic polling, shared
  caches, and "we will mail your response within two working
  days".

  The moral, then, for would-be sucessors to HTTP is to pick
  another niche and follow the network effects.
  Internet-scale event notification, for example, requires
  true push and subscriber management. Current efforts to
  shove presence information through HTTP may succeed, but
  any larger shift to event-based integration could catalyze
  a new, symmetric protocol which allows servers to initiate
  connections back to clients. Simply compressing existing
  HTTP traffic without adding new functionality faces a
  powerful entrenched competitor. Adding programmability,
  whether betting on HTTP-NG or the Mandatory scheme, is an
  attempt to harness network effects between all the
  communities that have been angling to extend HTTP to their
  application. Yet, the rewards for cooperation are spread
  thin: only the server which simultaneously supports HTTP,
  WebDAV, and Internet Printing Protocol would gain from a
  common syntax for those extensions.

  And now we are left back where we started: the mystery of
  protocol adoption. Reuse, extensiblity, simplicity are all
  tricks to reduce the cost of implementing a new protocol --
  but that doesn't mean the market as a whole will make the
  most efficient decision. If you're a Gopher, it may just
  bury you :-)

  The morbid conceit of this article was inspired by the
  first-rate book Aramis by French historian of technology
  Bruno Latour. It told the tale of two decades of R&D on a
  Parisian peoplemover system in the first person -- the
  train itself speaking of its struggle to become, and its
  death.
    ______________________________________________________

 Table 1. Fingerprints of a murderer -- Spider or Gopher?

  Vitals

  Academic NeXTstep hack, about 8 years old; birthmark at
  port 70 or 80

  First Victim

  Notorious primary use was for mere phonebooks; gatewayed
  queries to existing campus database and represented results
  as hypertext nodes

  Disguise

  Began without any meaningful type system at all, but soon
  grew past file extensions to vend multimedia information,
  even negotiating formats

  Modus Operandi

  Open a new connection, send a one-line demand for the
  information at a given address, then receive the response
  until connection dies

  Come-on

  Infiltrate small workgroups with small, easy, and free
  tools, promising acceptance with a simple `backlink' from a
  master list of servers

  Serial Victims

  Reaching beyond the degenerate case of static information,
  its addresses became entry points to scripts, databases,
  and other `stateless' servers

  Motive

  Even its first manifesto laid out hegemonic plans: to
  incorporate, by reference or by proxy, every other Internet
  information service.
    ______________________________________________________

 Table 2. Gopher's rudimentary type system used character codes to
 prefix selector lines. Gopher+ later added content negotiation.

  0

   File

  1

   Directory

  2

   CSO phone-book server

  3

   Error

  4

   BinHexed Mac file

  5

   DOS binary file

  6

   Uuencoded Unix file

  7

   Index-Search server

  8

   Text-based telnet session

  9

   Binary file

  +

   Redundant server

  T

   Text-based tn3270 session

  g

   GIF image file

  I

   Any image file
    ______________________________________________________

 Table 3. RFCs and Internet-Drafts discussed in this issue.

  RFC

  Date

  Title and Comments

  1436

  Mar 1993

  The Internet Gopher Protocol
  Described a `fiercely simple' request-response protocol and
  menu format

  1630

  June 1994

  Universal Resource Identifiers
  Raised the stakes from Gopher's own selectors to any
  namespace

  1689

  Aug 1994

  Status Report on Networked Information Retrieval: Tools and
  Groups
  A wonderful archaeological census of competing protocols,
  many now extinct

  1945

  May 1996

  HyperText Transfer Protocol 1.0
  Also defined the more primitive 0.9 GET. Even 1.0 only
  added HEAD and POST

  2068

  Jan 1997

  HyperText Transfer Protocol 1.1
  A vastly expanded contender with better caching, security,
  and extensibility

  2227
  P. Std.

  Oct 1997

  Simple Hit-Metering and Usage-Limiting for HTTP
  Entire RFC defined a single Meter:

  2396
  D. Std.

  Aug 1998

  Uniform Resource Identifiers: Generic Syntax
  The deceptively small definitive grammar took years to
  hammer out

  TBA
  P. Std

  Dec 1998

  Extensions to HTTP for Distributed Authoring
  Adding locks, collection-resources, and namespace
  operations took major reengineering

  I-Draft

  Nov 1997

  PEP -- an Extension Mechanism for HTTP
  `Protocol Extension Protocol' had negotiation policies for
  pipelining message processors

  I-Draft

  July 1998

  HTTP State Management Mechanism
  Cookies took three years from a quick, risky hack to a
  reasonably secure mechanism

  I-Draft

  Aug 1998

  On the use of HTTP as a Substrate for Other Protocols
  Official -- conservative -- IESG and IAB guidance on port
  assignment, URI schemes, and HTTP security

  I-Draft

  Sep 1998

  HTTP Authentication: Basic and Digest
  A companion specification for nonexistent (UUencoded
  passwords!) and new security.

  I-Draft

  Nov 1998

  HTTP Extension Framework
  Pared down to headers which label `Mandatory' and optional
  extensions of each message

  I-Draft

  Nov 1998

  Upgrading to TLS Within HTTP/1.1
  Using the Upgrade: for Transport Layer Security, in lieu of
  a dedicated port like https: at 443