14-Apr-86 23:07:17-PST,21304;000000000000
Mail-From: NEUMANN created at 14-Apr-86 23:05:42
Date: Mon 14 Apr 86 23:05:42-PST
From: RISKS FORUM (Peter G. Neumann, Coordinator) <
[email protected]>
Subject: RISKS-2.42
Sender:
[email protected]
To:
[email protected]
RISKS-LIST: RISKS-FORUM Digest, Monday, 14 Apr 1986 Volume 2 : Issue 42
FORUM ON RISKS TO THE PUBLIC IN COMPUTER SYSTEMS
ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator
Contents:
Robot safety (Ron Cain via Bill Park)
Use of computer files as evidence (Rob Horn)
Review of *Softwar* (Gary Chapman)
Computerized Voting -- No Standards and a Lot of Questions
(Summary of Eva Waskell's talk by Ron Newman)
The RISKS Forum is moderated. Contributions should be relevant, sound, in good
taste, objective, coherent, concise, nonrepetitious. Diversity is welcome.
(Contributions to
[email protected], Requests to
[email protected].)
(Back issues Vol i Issue j stored in SRI-CSL:<RISKS>RISKS-i.j. Vol 1: MAXj=45)
----------------------------------------------------------------------
Date: Mon 14 Apr 86 13:22:55-PST
From: Bill Park <
[email protected]>
Subject: [Ron Cain <
[email protected]>: robot safety]
To:
[email protected]
Mail-From: CAIN created at 14-Apr-86 09:19:46
Date: Mon 14 Apr 86 09:19:46-PST
From: Ron Cain <
[email protected]>
Subject: robot safety
To: IA.STAFF: ;
For those who hadn't heard, I thought I'd mention two close calls
we had out in the welding lab a week or so ago. It is worth keeping them
in mind the next time you stand near a robot.
In the first incident, a 68000 board in our system failed and
caused the processor to jump to (of all places) a robot move routine.
We were all standing around the emergency stop button looking at a
terminal, and Jeff and Talia got to the button within a few milliseconds of
hearing the crunching noise which marked the premature demise of a small
jack belonging to the lab. With our sensor mounted on the end-effector as
it was, it could have been alot worse if we had been further from a kill
button.
The second incident was even more sobering. Some drive motor
cards in the Cincinati-Milacon box failed and joints 5 and 6 began
jerking around randomly. Again, the kill button was nearby, and a
potentially disastrous situation (at least for the sensor) was avoided.
It could have been any other joint -- including the base or the shoulder.
And someone could have been standing next to it. We do all the time.
The point is just this: it can and does happen.
Watch yerselves around robots.
... ron
------------------------------
Date: Mon, 14 Apr 86 13:28:33 est
From:
[email protected] (Rob Horn)
To: wanginst!sri-csl.arpa!RISKS
Subject: Re: Use of computer files as evidence (RISKS-2.39)
The use of computerized data as evidence has been treated carefully in
the environmental field (a litigious arena which includes acid rain,
toxic wastes, etc.). The basic rule is:
Computer-based data is NOT evidence unless ALL parties involved
agree to treat it as evidence.
Yet, almost all of the data acquisition and processing is performed by
computers. The route around this that is used by the legal process is
a dual PAPER or (only recently) MICROFILM evidence trail. Using this
trail the following must be shown:
1). All instrumentation calibrations are traceable to NBS standards, with
logs that are properly documented and signed by humans,
in non-erasable ink on paper. (Also on numbered sheets in bound
notebooks only, with countersigned dates and occasion Q/A checks).
2). The computer processing includes the processing of routine calibration
so that the computer is part of the calibration loop.
3). All reports are provided in both computerized and hardcopy form. The
hardcopy version is certified and signed by a Q/C person.
4). All equipment logs and records are duly signed and archived.
In fact, the computer records are generally trusted and used, but all
significant evidence is verified against the paper trail. This does not
prevent tampering, but it does introduce several levels of human
verification and record keeping on top of the computer. The legal system
is comfortable with its ability to deal with human error and dishonesty, so
they switch to the human trail when in doubt.
These rules posed quite a problem in automating some of the data acquisition
processes, because the people involved would NOT SIGN reports that they could
not verify. (They had significant personal liability). Most of the reports
had to be generated on the spot (so that the signer could verify that the
equipment was behaving correctly), and include a hardcopy printout that
showed all of the equations and intermediate computations used (so that
the signer could double check whenever the numbers looked unusual or the
value looked like it might have legal significance). Then from these
individual data items computerized reports could be generated, but again
the signers of those reports insisted on hardcopy for intermediate
terms and double checked all the suspicious or signficant numbers.
Did mistakes get through? Probably. But the error levels were low and
bad reports had a decent chance of being corrected. Disputed reports could
be re-created by hand from "raw" data if necessary. The "raw" data being
computerized instrumentation reports that were paper logged and signed.
Was the computerization complete? Definitely not. The people involved
refused to sign reports from a program where they were unable to perform
independent validation on a spot check basis, nor where they could not
find a totally hardcopy re-creation path.
My experience in this is now four years old, but this area changes
slowly and the rules are probably still the same. The people involved
are very unwilling to abandon their independent audit path. They were
only willing to trust computers for the general case, not the oddball
or legally significant items. For things like averages, etc. they were
willing to trust computers after verifying 5% (selected at random) by hand.
Rob Horn
UUCP: ...{decvax, seismo!harvard}!wanginst!infinet!rhorn
Snail: Infinet, 40 High St., North Andover, MA
------------------------------
Date: Fri, 11 Apr 86 09:19:00 pst
From: Gary Chapman <
[email protected]>
Subject: Review of *Softwar*
To:
[email protected]
I thought participants of Risks might be interested in a recently released
book called *Softwar*, by Thierry Breton and Denis Beneich, two French
computer professionals. The book is a computer science thriller, so for all
of you out there who have longed for computer scientist heroes and heroines
who resemble Indiana Jones or Mata Hari, this book is for you. (*Softwar is
published by Holt, Rinehart Winston, and is available only in hardcover right
now, at $15.95.)
The two principal characters in the book are computer scientists, one male
and one female, one American and one Russian, who happen to have been lovers,
too, of course. The American is Assistant Professor of Computer Science at
MIT Brendan Barnes, who is an expert on software reliability and debugging.
The Russian, who was a grad student at MIT, is Yulya Voronkov, a beautiful
Soviet computer scientist who is one of the department heads at the main
Soviet computing center in Krasnoyarsk in Siberia.
Barnes writes a piece for *Computers and Society* that talks about the
potential of using software as a weapon in the ideological war with the
Soviets. This piece naturally attracts the attention of the CIA, and Barnes
is gently (and without much resistance) coaxed into becoming a member of a
team of military officers, CIA agents and technical experts who plan to use
software bugs to plague the Soviet effort to computerize their economy. They
call these "softbombs," in a "softwar" with the Soviets. As one character
puts it in one of the many extemporaneous speeches about the role of
computers in national security:
...any sector of society can be destabilized, even completely
paralyzed--industry and defense, civil and military communications, logistics
and transport, public administration, the entire economy--simply by a couple
of keystrokes on a computer terminal, anywhere in the world. We do
definitely see this as the electronic battleground of the future, and we
definitely see ourselves of being in the process of seizing the high ground
for ourselves before the other side can get there.
Barnes and his colleagues start by sabotaging a piece of software bought by
the Soviets from the French. It runs on a newly purchased "Craig 1" that the
Soviets bought from the United States. The software is programmed to spit
out garbage when the U.S. Naval Weather Station in the Virgin Islands reports
a barometric pressure of 1230 millibars. Then it is programmed to restore
all the data in perfect shape when the Weather Station reports that same
figure again. Of course, the Naval Weather Station is instructed not to
report that figure unless specifically told to do so, so the "softbomb" is
detonated at the choosing of the CIA. They pick a detonation time about an
hour before the "Craig 1" is to be demonstrated to a visiting delegation of
the Soviet Academy of Sciences.
But, aha! There is a clever programmer at the console of the "Craig 1" who
is bound and determined to find out why the machine went crazy at such an
embarrassing time. He eventually discovers the programming trick, and is on
to how this is the product of deliberate tampering by someone outside the
Soviet Union. The KGB zeroes in on Professor Barnes, and he nearly catches a
hand grenade in a Paris bar.
From there on out, it's a battle of wits between the American computer
scientist and his Soviet counterparts, and of course gradually that becomes
the gorgeous and brilliant Yulya, his former grad student and former lover.
The book is a fun read most of the time, especially for those intrigued by
MIT trivia, Soviet trivia and computer trivia. There are a few too many
spots where some character gives a speech about the importance of computers
to some such thing or other (Barnes gives a long speech to his wife about why
he's mixed up with the CIA and catching hand grenades in Paris and having an
affair with a beautiful Carribbean journalist, and it turns out that he's a
radical democrat who wants computers used to increase the democratic process
in the West). But on the whole, it's a fairly conventional thriller spiced
up for computer professionals with lots of jargon and speculation, and of
course, dashing, sexy and adventurous computer scientists.
-- Gary Chapman
------------------------------
Subject: "Computerized Voting -- No Standards and a Lot of Questions"
To: risks@sri-csl
Date: Mon, 14 Apr 86 21:50:29 -0500
From: Ron Newman <
[email protected]>
The following is a slightly edited version of an article I wrote for the
April, 1985 issue of the Computer Professionals for Social Responsibility
Boston Chapter newsletter.
~~~~~~~~~~~~~~~
Our guest at CPSR/Boston's March 19 meeting was Eva Waskell, an independent
science writer, former computer programmer, and current stringer for The
Economist. She spoke with considerable alarm about the rapid and
unregulated spread of computerized vote-counting systems in American
elections.
Waskell became interested in computerized vote-counting when Severo Ornstein
of CPSR National suggested that she look into several lawsuits pending against
Computer Election Systems (CES) of California. CES is the leading vendor of
such software; it estimates that approximately 25% of the U.S. popular vote is
cast on its equipment. Losing candidates in three states have sued the
company, claiming that its system produced inaccurate or fraudulent results.
While investigating, Waskell was appalled to find out that only one person
outside of CES, a consultant for one of the plaintiffs, had ever examined the
code. Waskell's investigation resulted in several New York Times
articles last summer.
To use a computerized ballot system, a voter inserts a punch card into a book
containing the names of each candidate for office. The voter casts a vote by
pushing a stylus through a hole in the book next to the name of the candidate.
thus punching out the appropriate hole in the punch card. When the polls
close, punch cards from all the precincts are trucked to a central location
and tabulated on a mainframe, using software provided by CES or a competitor.
The first such system was developed by IBM in 1964, for use in Los Angeles
elections. In 1969, there were accusations of fraud in LA's elections.
Fearing unfavorable publicity, IBM got out of the election business. Four of
IBM's employees left IBM to form CES.
Waskell pointed out four problems with this type of system:
1) A single central computer, in a single location, is counting all the votes.
This takes control away from precinct poll workers, who formerly counted the
votes and could recognize deviations from traditional voting patterns in their
precincts. It also makes rigging the election much easier: instead of having
to buy off many individual precinct workers, who are known to the community,
one need bribe only a single computer operator, who is known by almost none of
the voters.
2) Election officials must now be much more than clerical workers -- they must
have technical skills. Frequently, new people are hired from the outside to
learn and operate the computer equipment. Officials often do not know what
the new people are doing. In one state, workers rubber-stamped computer
printouts without examining them. A Minnesota election official commented:
"It's kind of like black magic -- we really don't know what's going on."
3) There are no standards for election software, so anyone can write a
vote-counting program. Vendors often talk state legislators into writing
enabling legislation which is vague and favors their company. When a state
Board of Elections certifies a computer system, the board often fails to
consult any computer experts, and when it does consult experts, it may ignore
their advice. The state of Pennsylvania certified a computerized election
system despite strong objections from two CMU professors. (One of the
CMU professors, Michael Shamos, wrote a report called "The Votomatic
Election System: An Evaluation" in November 1980.)
4) Vendors consider their software to be proprietary. As a result, in the last
20 years, almost nobody has examined any of the software. Compare this to
accounting software, which is subjected to audit by third parties. It is hard
to have confidence that software is performing accurately when you cannot look
at the code.
Waskell said that states and municipalities have ignored four clear warnings
against adopting these systems. In 1970, a Los Angeles blue-ribbon committee
recommended that all vote-counting software be independently audited. Similar
recommendations have been issued by the National Bureau of Standards (1975),
CMU computer science professor Michael Shamos (1980), and the independent
auditing firm of Coopers & Lybrand (1982). Nevertheless, none of the programs
has been audited.
According to Waskell, vote-counting programs are typically 4,000-5,000 lines
of COBOL "spaghetti code." Earlier this year, an Indiana consulting firm
analyzed CES's program on behalf of one of the losing candidates who is suing
CES. They found numerous problems, including the following:
The translation between the Hollerith punch card code and characters was
nonstandard. The 1971 NCR system which the software ran on did not use
standard EBCDIC.
The contents of memory were continually being redefined. Numerous variables
and fields were overlaid in memory.
The same memory locations were re-used for the vote counts of different
races.
There was a total lack of structure. The program contained no PERFORM UNTIL
(DO-loop) statements but had numerous undocumented GOTOs.
COBOL's ALTER verb was used, producing self-modifying code.
A call was made to an undocumented, unknown subroutine.
The program interacted heavily with the operator, who can operate the console
switches to examine and modify any part of the memory or program after each
set of data is tallied.
The program made it easy for the operator to turn off error logging and audit
trails, without leaving any trace.
There was heavy use of control cards in the data deck to redefine data
fields, raising the possibility that a "knowledgable" voter could punch a
control card and drop it into the ballot envelope to change the program's
processing of election results.
CES sends undocumented "updates" to election personnel before each election.
The program used a time card to set the time and required that the computer's
clock be disabled. This makes it impossible to determine how long the
program runs or to accurately determine when logs or printouts are produced.
The program did not correctly count "crossover votes," in which, for example,
a voter punches a vote for a straight Democratic ticket and punches votes for
several individual Republicans. Before an election in West Virginia,
newspaper publicity specifically said that such votes were allowed, yet the
program failed to count them.
The program failed to keep a count of invalid ballots.
A report to the Illinois Board of Elections in September, 1985, revealed that
of the voting systems that the state tested before elections, 28% contained
errors. Although those errors were corrected, such a high error rate suggests
that many errors are never detected or corrected. Waskell said that other
states' election officials are unaware of the Illinois findings. It disturbed
her that Illinois failed to keep a record of the errors it found, but simply
sent them back to the vendors for correction.
Suits against CES have been filed in Indiana, West Virginia, and Florida, but
judges have dismissed several of the cases for lack of evidence, saying that
computer experts' testimony is mere "speculation" and "suspicion." It is hard
to successfully prosecute such a case when the computer system itself is
designed to ensure that no evidence exists.
In the Indiana case, the plaintiff charged that a CES representative was in
the counting room on election night, turned off the program's logging, and
added two extra control cards after the last votes were counted. In the West
Virginia case, a CES representative allegedly connected a modem to the
computer and was down on his hands and knees around the compter on election
night, claiming that "a screw was loose." In addition, the West Virginia
candidate alleged that the county clerk's husband manipulated the computer's
switches during the count. Evidence in this case is difficult to obtain
because the county clerk destroyed all the ballots 61 days after the election
and returned the program deck to the vendor.
According to Waskell, a company called Cronus recently purchased both CES and
two of its competitors, Thornber and Governmental Data. Together, these three
companies market 60-80% of the voting systems in use. Cronus is financially
tied to the Tyler Corp., whose chief executive officer is Fred Meyer, the
Republican Chairman of Dallas County, Texas. Meyer announced his candidacy
for Mayor of Dallas one month after the city bought a CES voting system.
Ms. Waskell closed her presentation with a series of recommendations. The
Federal Government, using election powers outlined in the constitution, should
mandate that all vendors conform to NBS standards. State election laws should
be changed to show a greater understanding of the technologies. Local
election officials must ensure that audit trails are always turned on and that
they are continuous and unbroken. Also, local officials should count a random
5% sample of the vote by a different method, thoroughly test computer systems
before adopting them, and be accountable and responsible for their use.
People interested in more information about this subject may want to
read New York Times articles by David Burnham, published on 7/29/85, 7/30/85,
8/4/85, 8/21/85, 9/24/85, and 12/18/85, and a letter to the editor, 8/6/85.
Ms. Waskell was the source for much of the information in these
articles. If you write to me (newman@mit-athena), I can tell you how
to reach Ms. Waskell--I'm uncertain whether she wants her address & phone
number posted on the net.
------------------------------
End of RISKS-FORUM Digest
************************
-------
