Internet Gopher Project: The Rise and Fall of Gopher
1
Internet Gopher Project
The Rise and Fall of Gopher
Robert Alberti
Program for Individualized Learning 3254 Spring Semester 2011 Associate Professor Michael Whalen, reviewer
June 1, 2011
Internet Gopher Project: The Rise and Fall of Gopher Internet Gopher: The Rise and Fall of Gopher Bob Alberti
2
Introduction
In April of 1990 I took a job as a programmer/help desk operator with the University of Minnesotaâs Microcomputer Workstation and Network Center (MWNC), then part of th
e Academic Computing Systems and Services department (which has since gone through half a dozen name changes to become the Office of Information Technology.) I heard fr
om an acquaintance that this was quiet, interesting work, and the Regentâs Scholarship employee benefit program offered free classes that might allow me to complete my
undergraduate degree (yes, the same undergraduate degree for which this paper is written, twenty years later!). After only a few months of the low-stress, low-demand ta
sks I had been led to expect by my acquaintance I was assigned to a new team writing something called âInternet Gopherâ (Gopher).
Rapid Growth
The Internet had by 1991 reached a point of information supersaturation: there were approximately 375,000 Internet hosts in the world, up from 80,000 only two years pri
or, with the number of hosts increasing logarithmically (Lottor, 1992). Already rapid, commercial involvement in the Internet would further increase its growth. However
, the Internet was still at this time running primarily across a backbone funded by the National Science Foundation (the NSFnet backbone), and commercial exploitation o
f this public infrastructure was prohibited in most cases (NSFnet, 1992). The quarter million hosts with domain names ending in â.eduâ (indicating colleges, universitie
s, and other educational institutions) still outnumbered all others including commercial hosts ending in â.com.â Some method was needed to index and organize the data o
n these systems, both on the macro scale of the Internet and on the scale of individual institutions. The University of Minnesota was among many exploring the usefulnes
s of computers as means to convey information to the campus â the campus-wide information system, or CWIS.
Internet Gopher Project: The Rise and Fall of Gopher
3
Enter the Gopher
The project to which I had been assigned employed new âclient-serverâ software architecture, and I was responsible for a lot of C-language UNIX coding, including the Un
ix Gopher client. Working with client-server architecture was fascinating, although its overall impact was by no means immediately clear to me. We were writing Gopher i
n response to the University of Minnesotaâs Campus-Wide Information Services (CWIS) committee, about which my boss Mark McCahill had very little good to say. Apparently
this committee had been meeting for several years, and while it had assembled a phonebook-thick collection of functional requirements specifying to a very fine degree
of detail what the application should do, no actual code had yet been written. McCahillâs plan, as explained after I attended my first confusing meeting of the CWIS com
mittee in November of 1990, was to create a working prototype CWIS application in the month before the next scheduled meeting, and to debut it there. The committee that
had not produced a line of code in years would be treated to a complete client-server application created in-between meetings. I was naïve enough to think such an acc
omplishment would be lauded by the committee. Development was intense, including 36-hour programming sessions (Romenesko, 1996). [McCahill] with assistance from the new
ly formed Gopher Team, fine-tuned the prototype as music from Nirvana and Mudhoney blared in the background. "It was a fun time,'' recalls team member Torrey.âIt was a
lot of late nights and weekends and a lot of beer, pizza and speed metal.'' A month later we had working examples of a Gopher server, and Mac and UNIX clients. I attend
ed my second and last CWIS committee meeting, where we demonstrated the capabilities of the newly-created âGopherâ program. While I naïvely expected that our hard work
would be well-regarded, the reception was instead what might be expected when a number of civil service bureaucrats are presented with the surprise of an accomplishmen
t that bypasses all of their efforts, and threatens to render all their work moot. Gopher team member Farhad Anklesaria called the meeting a âtotal disasterâ (Frana, 20
04). I remember one committee member literally jumping up and down in fury. At the end of the meeting, the CWIS committee prohibited the Gopher programming team from an
y further development on the application.
Internet Gopher Project: The Rise and Fall of Gopher Since we were prohibited from working on Gopher, we put the source code up on the FTP server on boombox.micro.umn.e
du (Boombox) and informed colleagues at other institutions about its
4
availability. Remember in those days the only way to retrieve something from the Internet was to know its address in advance, so the only way for information about the
availability of Gopherâs source code to spread was via Usenet conferencing (Anklesaria F. , 2011), e-mail discussion lists or verbally.
Outsourced
Despite the challenge of getting word out, the prototype Gopher client and server proved compelling enough to attract attention. In part this was due to the fact that o
ne needed almost nothing to run a Gopher client. By providing a public login over Telnet (Figure 1) individuals who were curious about Gopher could immediately try out
the UNIX client, and this demonstration was often sufficient to persuade them to download and install the Gopher software via Xmodem or FTP transfer.
Figure 1 Telnet to Gopher Client on Boombox
Distribution was quite rapid once people were aware Gopher existed. By facilitating its own distribution, Gopher may have been the Internetâs first âviral application.
â The Gopher team members were not shy about explaining to those who wrote with suggestions for improvements that we were prohibited from working on Gopher by the campu
s CWIS group, and we happily provided the names of University administrators to whom comments might be addressed. Senior University personnel began receiving calls and
e-mails from other institutions complimenting them on Gopher and asking about the Gopher development prohibition. After a couple of months, Gopherâs popularity and unde
niable utility became an embarrassment to the CWIS committee, and we were able to formally resume the work we had continued doing on our own time.
Internet Gopher Project: The Rise and Fall of Gopher
5
History
When I entered the University of Minnesota in 1980 computers were already employed for registration, although the method was crude by modern standards. Each seat in a c
ourse was represented by a computer punch-card encoded with the appropriate registration information. To register one proceeded through the established paper-based regi
stration processes making numerous trips on foot between the financial offices, the university registration offices, and the individual college and program offices to g
ain the appropriate approvals. Once approved the student was handed the computer card that represented their seat in the course, and made a final trip over to the unive
rsity registration office, where the card was handed to a clerk to be added to the dayâs registration run. When the stack of the dayâs cards was batch-read into the mai
nframe registration computer, oneâs registration was finally complete. This mainframe legacy was part of the cultural divide that made my visits to the CWIS committee s
o very exciting. The committee was primarily composed of those whose whole experience of computers was limited to mainframes, and who were accustomed to the slow pace o
f technologic change in mainframe technology. McCahillâs team was younger: programmer Paul Lindner had only just graduated from college, I was 29, and McCahill was 34.
All of us had experienced the rapid development of personal computers (PCs) during the 1970âs and 1980âs. My first computer had
Figure 2 PDP 8L loading 0201 with 6622 (Skip)
been a PDP 8/L (Figure 2) that booted from toggle switches in 1977. My first laptop was a NEC-PC8201a (Figure 3) which I carried to class in 1983 to looks of sheer asto
nishment at its 8-line, 40-character display. In six years Iâd gone from wire-wrapped core memory to something not unlike a Neo Dana (Figure 4)
Figure 3 NEC PC8201a
available today.
McCahill, Anklesaria and Lindner were particularly interested in clientserver architecture, which distributed the work of a given computational task between two compute
rs: the server responds tersely to many requests for data; the
Figure 4 Neo Dana
client issues requests, receives responses, and handles the computationally-large work of displaying the
Internet Gopher Project: The Rise and Fall of Gopher results to the user. Nowadays client-server architecture is ubiquitous, but in 1991 the growth of the Internet (e.g
, servers) and the increase in power of the personal computer (clients) had developed to
6
the point where client-server architecture was increasingly feasible. Client server architecture leveraged this growing power of PCs to distribute the computing load ac
ross servers and clients, allowing for a much richer user experience than could possibly be managed by a mainframe somehow running all displays. Unfortunately the Unive
rsity of Minnesota campus CWIS committee was unfamiliar with and skeptical of client-server architecture. From their point of view the mainframe was an established mean
s of computing, and they had little reason to believe personal computers would be useful as anything other than toys. The Gopher team, on the other hand, had developed
a feel for Mooreâs Law (Moore, 1965), and understood (from such experiences as mine with my laptop) that personal computers would soon outstrip the power of established
mainframe systems. While it is easy to criticize the campus CWIS committee for its failure of vision, itâs important to note that if they had not restricted the develo
pment of Gopher by its U of MN authors, Gopher may not have been so rapidly distributed and popularized across the Internet. Interestingly, this paralleled my experienc
e a few years prior with my company GamBit MultiSystems (Alberti, 2010): if a former employee had not cut us off from our original market (p. 37), we would not have res
ponded by franchising to thirteen cities in the U.S. and Canada. Being âforced from the nestâ under unfortunate circumstances led to greater eventual success in both ca
ses.
Setting
In order to understand Gopherâs significance and its impact on contemporary computing in 1991, it is important to understand the environment from which it emerged. In 1
991, computers were the realm of academics and hobbyists, and the landscape of services and connections was much more fractured and difficult to navigate than it is tod
ay. Connectivity was primarily provided by modems with speeds ranging from 300 to 2400 baud (Daxial Communications, 2003). E-mail, if it was available, was managed by i
ndividual departmental mail servers â you couldnât write to â
[email protected],â â and there were no on-line directories.
Internet Gopher Project: The Rise and Fall of Gopher Most interpersonal contact was accomplished through e-mail lists or by reading Usenet, which was an increasingly un
wieldy1 database of interest-based forums distributed via NNTP protocol (Kantor & Lapsley, 1986) to a growing number of servers around the Internet.
7
The primary means of moving files was over File Transfer Protocol (FTP) (Postel, 1980). FTPâs stateful architecture and unusual two-port communications protocol is an a
rtifact of its antiquity. FTP was developed back when there were no personal computers, only mainframe computers and dumb terminals, or maxi-Hosts and TIPs respectively
in the original ARPANET design (EdmondsonYurkanan, 2002). The Dumb Terminal was used to tell a Source Mainframe to exchange files with Target Mainframe (Figure 5). The
Source Mainframe had to talk to the Dumb Terminal and the Target Mainframe on separate communications lines, since it had to maintain connection state with both.
Figure 5 Original FTP architecture
When PCâs emerged, FTP had to be adapted accommodate the new paradigm. The âTarget Mainframeâ was now actually the hard drive on the PC (Figure 6), resulting in the FTP
architecture most commonplace today.
Figure 6 FTP on PC architecture
1
I was a Usenet administrator from 1994-1996 and the network load imposed by this service was ridiculous. Since there was no subscription mechanism only a tiny fraction
of Usenet data was ever read from a given server.
Internet Gopher Project: The Rise and Fall of Gopher As the Internet grew, firewalls were used to protect private networks, rendering the FTP architecture even more pro
blematic. The Firewall, a device designed to exclude arbitrary inbound connections, had to be taught how to permit Port 20 FTP data inbound connections, and how to rout
e those to the various client PCs which had requested the connection (Figure 7). So the Firewall must note outbound FTP requests, and maintain a table of internal clien
t PCâs to which inbound connection attempts might be expected at some future time. Such âstateful inspectionâ firewalls had not been invented when Gopher was being writ
ten (Higgins, 2008).
8
Port 21 Commands (Outbound)
Port 20 Data (Inbound) Server
Firewall Client
Figure 7 FTP through Firewall
To be sure, this is a simplification that illustrates FTP in âActiveâ mode. There is also a âPassiveâ mode which allows limited bidirectional communications on Port 21.
There are also issues regarding the transmission of binary versus ASCII data. Active and Passive, Binary and ASCII, all of these were additional complications that mad
e FTP a challenge to many early Internet adopters. In order to download a file from a server through a firewall, you had to put your computer in Binary mode, then Passi
ve mode, then initiate the transfer all while maintaining a connected session state. Maintaining the connected session state could be a problem when downloading large f
iles, as some FTP servers would time out if no new commands were received in a certain period. Depending on the transmission speed (remember, there were still 300 baud
modems in use), the time required to transfer a file could exceed the maximum connection time! Modern FTP software has addressed these challenges by writing smarter, mo
re powerful client software that would have been impossible back when we were developing Gopher. All of these settings had to be carefully and deliberately configured,
and lists of useful FTP sites and their particular settings had to be maintained by hand.
Internet Gopher Project: The Rise and Fall of Gopher
9
Finally, FTP service required a login. By 1991, an informal but well-understood convention had been adopted of anonymous FTP login using one of the accounts âftp,â âgue
st,â or âanonymous,â with the individualâs e-mail address as an untested password (the authenticity of the e-mail address was unverified). A limit on the number of simu
ltaneous anonymous logins meant that popular FTP sites were frequently full and not immediately accessible.
Structure of the Gopher Protocol
Notable about Gopher is its extremely terse communication protocol, written to support the âlowest common denominatorâ protocols, which in 1991 included 300-baud modems
Gopher provides acceptable performance over slow connections with the protocol described in Figure 8.
Figure 8 Basic Gopher Protocol
The protocol is described in the gray boxes in Figure 8 (which are not part of the transmission): Type Display String (tab) Selector String (tab) hostname (tab) port (e
nd of line) Unusual among protocols, which usually employ a consistent style of delimitation, Gopher mixes column-delimited (the single initial type character) and tab-
delimited fields (all the others). The selection of the tab delimiter was I believe unfortunate, as some editors would replace tabs with multiple space characters, invi
sibly breaking Gopher links. Developed before the invention of the Universal Resource Locator (URL) which has since become a mainstay of Web communications, the Gopher
protocol can be seen to contain the elements that became part of the URL specification in 1994. In fact both Gopherâs McCahill and the Webâs Berners-Lee are co-authors
of the URL definition RFC 1738 (Berners-Lee, Masinter, & McCahill, 1994). It is now possible to use a single-line URL such as
Gopher://sdf.org:70/users/alberti/welcome.
txt to reach the same document. The initial TYPE character notifies the client as to the type of data available at the location specified by the other fields.
Internet Gopher Project: The Rise and Fall of Gopher
10
The Display String is the user-friendly label behind which the technical information leading to the item will be hidden. This innovation was an important contribution t
owards Gopherâs initial popularity, as it facilitated the use of the Internet by ânormalâ users. The Selector String is the system-specific path to the individual item.
This is the element most responsible for making the Gopher link transparent to the end user. By properly structuring the Selector String, resources could be obtained f
rom a variety of different kinds of servers without the user needing to know anything about those servers. The Hostname and Port specify the TCP/IP addressable location
of the Internet service that can respond to the request specified in the Selector String. Some of the characteristics that emerge from this protocol are explored below
