TELECOM Digest Sun, 7 Nov 93 10:27:30 CST Volume 13 : Issue 739
Inside This Issue: Moderator: Patrick A. Townson
Orange County DACS Outage (Urban Surfer)
Dialup by Modem Bank to Ethernet (Scott M. Pfeffer)
Fire Update 11-5-93 9:00 AM PST (Pete Tompkins)
"Fake Switch" Box or Tester (Karl Bunch)
Canada Goes 1+ 10D For All Long Distance, Sept '94 (Dave Leibold)
Skokie, IL, and Telephone History (Dave Levenson)
----------------------------------------------------------------------
Date: Fri, 05 Nov 1993 13:20:02 PST
From: Urban Surfer <
[email protected]>
Subject: Orange County DACS Outage
Reply-To:
[email protected]
Organization: Pacificare Health Systems
About six weeks ago, I posted in the Digest an account of the DACS
outage in Orange County, CA. I received several queries for more
information. It seems that a lot of people were disturbed to learn
about the potentioal points of failure on a DACS as well as the bug we
experienced.
I recently took a tour of the affected CO and met with the switch and
DACS administrators to ask further questions. At this point, they
believe that they have fully addressed all software and procedural
issues with the DACS IV. They also stated that the software patches
they applied have been propagated throughout the entire Bell network.
The following is the public disclosure report sent to the FCC from
Pacific Bell. This report was retyped from a fax, so any errors are
mine or my secretary's.
FINAL SERVICE DISRUPTION REPORT
CATEGORY: 50,000+
REPORTING COMPANY: Pacific Bell
REPORT CONTACT/TELEPHONE: Eva Low (510) 823-2910
LOCATION OF DISRUPTION: Anaheim, California (ANHMCA#11)
1. DATE AND TIME AND INCIDENT:
9/15/93 0752 HRS.
2. GEOGRAPHIC AREA AFFECTED:
The failure of this Digital Crossconnect System (DCS)
affected a portion of the city of Anaheim, California. This
geographic area is located in the Los Angeles, California
LATA 730.
3. ESTIMATED NUMBER OF CUSTOMERS AFFECTED:
Potentially, 67,528 customers could have been affected by
this failure. This estimate was derived based on the number
and type of working circuits on the DCS.
4. TYPES OF SERVICES AFFECTED (e.g., INTEREXCHANGE, LOCAL,
CELLULAR, 911 EMERGENCY SERVICES.):
All services using the interoffice transport network, into
and out of, the Anaheim 11 central office building were
affected. This included two local switching entities which
were isolated from the interoffice network (intraoffice
call was not affected.) Operator and directory assistance
services were adversely affected and the Anaheim Public Safety
Answering Point (PSAP) was without Automatic Location
Identification (ALI) during this failure.
5. DURATION OF THE INCIDENT:
Date and time of disruption: 9/15/93 at 0752
Date and time of full service restoral: 9/15/93 at 1557
Duration of incident (minutes): 485
6. ESTIMATED NUMBER OF BLOCKED CALLS:
Approximately 746,950 calls were blocked during this
incident. This estimate is based on data from the switches
using the same day and time of the week prior to the incident.
7A. CAUSE OF THE INCIDENT:
This service outage was caused by a software defect in the
Digital Access Cross-Connect System IV-2000 (DACS IV-2000).
The software defect caused information in an area of the
database called "Frame Data Page" to become corrupted. This
corruption did not have an immediate impact on service. The
Frame Data Page contains critical information related to the
systems software program identity which is used by the DACS
IV-2000 during system recovery. This corruption went
undetected and was propagated from active memory to the hard
disk and system backup tapes.
Prior to the outage, most input commands issued to the DACS
IV-2000 were responded to with "Retry Later" (RL) messages.
In accordance with standard procedures, a system reset was
activated to clear the system of RL responses in order to
reestablish communications with the DACS IV-2000.
The system design is such that when a system reset is
activated, data resident on the hard disk is loaded onto
active memory. On this occasion, the aforementioned
corrupted Frame Data Page caused the DACS IV-2000 to
reinitialize the cross-connect map and drop all active cross
connects in the system. A total system outage ensued.
Attempts to recover the system by rebooting from system backup
tapes failed because the corrupted Frame Data Page also
existed on these tapes.
AT&T determined that the database corruption resulted from
improper software process interactions involving the
preemption of a lower priority process by a higher priority
process during a very specific small window of time when
the program was manipulating internal data pointers. The use
of these data pointers by the higher priority process
resulted in corruption of the Frame Data Page described above.
7B. NAME AND TYPE OF EQUIPMENT/VENDOR NAME:
Name: Digital Access Crossconnect System IV-2000 (DACS IV)
Type: Digital Crossconnect System (DCS)
Vendor: American Telephone & Telegraph (AT&T)
7C. SPECIFIC PART OF THE NETWORK INVOLVED
(e.g..LOOP SWITCH, INTEROFFICE):
This disruption involved the interoffice transport portions
of the network.
8. METHOD USED TO RESTORE SERVICE:
Standard emergency action recovery procedures were executed
by Pacific Bell field personnel under the direction of
Pacific Bell Electronic Systems Assistance Center (ESAC), in
consultation with AT&T RTAC, TSO, and Bell Laboratories (Bell
Labs).
Multiple attempts to recover system operation from storage
media failed since the corruption was present on the hard
drive as well as all backup tapes maintained in the office.
A special software debugging tool called "DACSmate: was
thereupon attached to the DACS IV-2000. Through the use of
DACSmate, AT&T Bell Labs performed an intensive analysis of
the database on the backup tape and determined that the
extent of the corruption was confined to the Frame Data Page
only: the cross-connect map itself maintained integrity.
Bell Labs used DACSmate to copy the existent cross-connect
map from taped and reloaded the DCS hardware, thereby
restoring transmission for customer service, however the
system controller remained in an out-of-service state.
Subsequently, Bell Labs prepared a "new" database containing
both a valid Frame Data page and the cross-connect map
information. Standard procedures were used to load this
database from tape into the system, and full functionality was
restored to the DACS IV-2000.
9. STEPS TAKEN TO PREVENT REOCCURRENCE OF THE OUTAGE:
1. AT&T issued/reissued the following bulletins, called
either COACH or Urgent Problem Notification (UPN)
Bulletins, as a result of this incident:
a) UPN Bulletin 9309171.1 was issued on September 17,
1993, alerting the industry that system resets can
result in service interruptions, and hence, are not
to be performed as part of normal troubleshooting of
main controller problems.
The UPN also recommends that the next level of
support be contacted because it may be necessary to
use special debugging tools to ensure that data
corruption is not present.
b) UPN Bulletin 9309171.2, issued on October 13, 1993,
amends UPN Bulletin 9309171.1. The amended version
describes the cause of the data corruption and also
identifies the correcting software program releases
(see item 2 below).
c) COACH Bulletin #050393.2, issued on September 17,
1993, cancels the use of resets (recommended in
Bulletin, #050393.1). This issue recommends that
the next level of support be contacted and the next
level of support be contacted and that special
debugging tools may be needed to ensure that data
corruption is not present.
2. The correction for the process interaction problem
is available in the following software releases:
Redundant Controller: 2.3drc (avail 11/7/93)
3.0drc (avail now)
2.3d (avail 10/25/93)
Additional defensiveness measures have been
developed to have the system automatically validate
the database for integrity and to prevent the
inadvertent propagation of corrupted data. These
changes (tracked via MR CS 93-26601) will be
available as follows:
Redundant Controller: 2.3drc (avail 11/7/93)
3.01drc(avail now)
Simplex Controller: 2.3d (avail 10/25/93)
A software patch (overwrite) for MR# CS 93-26601 was
developed for Simplex Controller 2.2d and Redundant
Controller 2.2drc and is now available.
Pacific Bell is planning to deploy Release 2.3 or
later in all DACS IV-2000 offices by 1994. Patch
application prior to this will be determined on a
site-by-site basis.
3. Pacific Bell issued an ESAC Flash, 93-010F
prohibiting the use of system resets without ESAC
involvement. Moreover, ESAC will use DACSmate to
verify that the database is not corrupted prior to
initiating a reset. Deployment of additional
defensive measures (2.2 patch, Release 2.3 and
3.0.2 software) provides this data validation
internal to the DACS IV-2000 software. (See also
item 2 and 4 in this section).
4. The DACSmate software debugging tool currently does
not have remote access capability; however,
enhancements for remote access via an x.25 wide area
network are under development. This remote
capability requires development of a companion
called DACSlink, which AT&T will be jointly testing
with Pacific Bell in December 1993. Pending
successful completion of testing, Pacific
Bell will implement DACSlink in all of its DACS
IV-2000 offices during 1994.
On October 8, 1993, AT&T provided Pacific Bell with
additional portable DACSmate units, pending the
deployment of DACSlink.
Matt Holdrege
[email protected] MH235
------------------------------
From:
[email protected] (Scott M. Pfeffer)
Subject: Dialup by Modem Bank to Ethernet
Date: 6 Nov 93 18:44:05 GMT
Organization: Southwestern Bell Telephone Company
Dialup to modem bank. Investigating options such as Trailblazer
series, etc.
Need information on availability, pricing, recommendations, caveats,
or experiences with implementing dialup to a host which can route
TCP/IP traffic across modems.
Nice picture:
--------
| OFC/ | _____ _______
| HOME | | | /| |
| CPTR |---|Modem|---SWITCHED TELCO----| Modem |
| | |_____| \| Pool |
-------- |_______|
|||||||
--------- ________ \_____/
| Network\ _________ | Linkup | ||
| of |=====|Sun SPARC| <======| Device |======>//
| Suns / |_________| |________|
---------
Specifically,
I am looking for information on the following pieces:
1. The Modem Pool. (What vendors, advice...)
2. The Linkup Device between the modem pool and the Sun (can be PC
bridge, or an interface card in the Sun for the modem pool device,
or anything else...) (Trailblazer?).
3. I have no questions about the the OFC/Home cptr,
the Modem attached to the home computer, or the network of suns.
Those details have already been worked out...
Requirements:
1. Support 2400, 9600 baud. Autobaud detection would be
nice. Initially 8 or 16 modems at least. Must be easily
expandable.
2. Support compression, error control, but allow non-compression
and/or non-error-control modems at the OFC/HOME to work
(through automatic negotiation).
3. Total transparency between Sun and OFC/HOME CPTR. That is,
once connected, ALL data from OFC to Sun will be
delivered untouched, and from Sun to OFC, too.
This means the linkup device as well as the modem pool device
must not interfere or attempt to interpret the data coming
across once the connection is established.
4. Cost reduction information valuable...
Why:
1. Have client application on OFC/HOME computers that I want to
talk to a server application on the Sun SPARC via DIALUP.
2. Client talks TCP over phone line using SLIP/PPP for serial IP.
Thanks in advance. Please reply to:
Scott Pfeffer
[email protected] or call direct: (314) 235-7213
Information Services, Southwestern Bell Telephone
18-N-22, One Bell Center,St. Louis, Missouri 63101
------------------------------
From:
[email protected] (Tompkins)
Subject: Fire Update 11-5-93 9: 00 AM PST
Reply-To:
[email protected] (Tompkins)
Organization: Transaction Technology, Inc.
Date: Sat, 6 Nov 1993 17:35:30 GMT
I am thoroughly impressed with the quality of the telephone service
throughout Malibu the last three days. Outbound calls were totally
unrestricted at all times; inbound volume was limited intentionally to
insure our ability (and more importantly, emergency personnel's
ability) to call out as necessary. A number of areas in east Mailbu
(310-456 exchange) still get a busy. I suspect cables going to some
of the fire areas have been welded together by the fire.
Away from phones for a minute: I just drove into work for the first
time since Monday. Pacific Coast Highway is open to Malibu residents
(acutally, its open to anyone, but none of the canyon accesses are
open to non-residents). The scene was one of total devastation -- but
it was also a scene of many successes. For ten miles along PCH, the
fire burned right up to the Highway where there wasn't a structure.
And for that same stretch, it burned up to the back wall of the
structures that fronted PCH. La Costa (one of the areas that has
gotton a lot of news coverage) looked like a bomb hit it -- but the La
Costa houses on PCH are untouched even though they are a mere 50' from
their neighbors to the rear, which, in most cases, were destroyed!
In Carbon Canyon, the fire raced through in seconds. The residents
all assumed their houses were gone -- but the vast majority were
missed -- partly the luck of the draw, and partly the hard work of
fire fighters, partly aided by the large separation between houses
(these are two to ten acre lots). In one case (a close personal
friend), the fire fighters apparently cut out a piece of burning roof;
broke in through a window to extinguish some burning furniture. They
actually went out of their way to cover the other furniture, obviously
taking pains to minimize the damage they caused -- they hauled the
smoldering furniture a hundred yards down the canyon, and went on to
the next house. This is one family who left KNOWING their house was
gone and returning to find really minor damage!
Further west (PCH DOES run east and west through Malibu, in spite of
what the TV newspeople might have you believe!), a ring of burned out
brush encircles the Civic Center and also the Malibu Knolls
residential area, but nothing was burned (at least as far as you see
from the Highway). Anyone who has visited Malibu has probably noticed
the castle overlooking the Civic Center. Its walls are singed, as are
the walls of many of its neighbors. Hughes Research, Pepperdine and
the neighboring residential area, Malibu Country Estates: same
stroy -- fire right up to the edge of the buildings.
Four houses at Puerco Canyon (a little further west) have burnt brush
all the way around them, with no apparant damage!
Some hot spots still exist in Corral Canyon on the west flank and also
in and around Topanga on the east flank, but there is no further
threat to structures. We all thank God for the cool, damp sea breezes
that returned to the area Wednesday.
I don't want to minimize the devastation and the loss of hundreds of
people, but it is obvious that in many areas the skillful and hard
work of thousands of fire fighters from all over the western U.S.
saved many hundreds more homes. The hearts of all of Malibu goes out
to these hard working people.
Pete Tompkins
------------------------------
Date: Sat, 06 Nov 1993 19:37:02 GMT
From:
[email protected] (Karl Bunch)
Subject: "Fake Switch" Box or Tester
Organization: Think Tank Software, Norwalk, CA
I'm looking for a circuit or "magic box" that would allow me to
basicly plug to phones back-to-back. Given Phone A & B if phone A
were picked up Phone B would ring, and when phone B is picked up they
could converse as normal until one of them hangs-up. The same could
be true in the reverse (Phone B would ring A if it's on-hook etc.)
I want to hook up a phone to a voice-mail board and allow the board to
ring the phone or the phone to "call into" the board without using up
phone lines etc.
I'm extremely ignorant as to how phones even ring. So be very complete
in any reply you may make.
Please reply by e-mail.
Thanks for any help,
Karl Bunch UUCP: ..!uunet!cerritos.edu!ttank!karl
Think Tank Software INTERNET:
[email protected]
------------------------------
Date: Sat, 06 Nov 1993 23:06:58 -0400
From:
[email protected] (Dave Leibold)
Subject: Canada Goes 1 + 10D For All Long Distance, Sept '94
[from Bell News, Bell Ontario, 25 Oct 93. Text is Bell Canada's.]
Canadian dialing patterns to change in 1994.
Get ready for the next big change in dialing patterns.
The way callers make long distance calls within their own area code
will change for everyone in Canada (and most of North America) on
September 4, 1994.
Currently, to dial long distance within your own area code, you dial 1
or 0 and then the seven-digit number. Area code 905 is the only
exception to this.
But population growth, as well as the boom in new technologies such as
cellular and fax machines, have used up almost all of the available
area codes under the present North American Numbering Plan (NANP).
To solve the problem, calling long distance within your own area code
will require you to drop in your area code and dial 1 or 0 + area code
+ xxx-xxxx (just as you do when making long distance calls to other
area codes).
No change will take place in the way that local calls or long distance
calls to other area codes are dialed.
Dave Leibold - via FidoNet node 1:250/98
INTERNET:
[email protected]
------------------------------
From:
[email protected] (Dave Levenson)
Subject: Skokie, IL, and Telephone History
Organization: Westmark, Inc.
Date: Sat, 6 Nov 1993 17:59:11 GMT
In light of our Moderator's recent move from Chicago to Skokie, IL, I
thought I'd share a bit of telephone trivia from the mid 1960's.
It was reported in that year that the largest number of telephones per
capita (86 per 100 population) anywhere in the world was in the
District of Columbia, USA. The second largest value of this number (I
think the number was around 70 or so) was in Skokie, Illinois.
(Now that Pat lives there, the number is probably higher!)
Dave Levenson Internet:
[email protected]
Westmark, Inc. UUCP: {uunet | rutgers | att}!westmark!dave
Stirling, NJ, USA Voice: 908 647 0900 Fax: 908 647 6857
[Moderator's Note: Actually, I have but three lines: one for voice,
one for data and one for fax. I'll make do somehow. The fax is now
on a full time dedicated line and available to anyone who wants to
use it: 708-329-0572. The Skokie area was also the home of Teletype
Corporation as some old-timers may recall. I am just just hoping
very desparately that things will work out financially for me and
the family. :( PAT]
------------------------------
End of TELECOM Digest V13 #739
******************************
******************************************************************************
Downloaded From P-80 International Information Systems 304-744-2253
