Subj : ARRL Systems Service Disruption
To   : QST
From : ARRL de WD1CKS
Date : Wed Oct 23 2024 10:22 pm

10/23/2024

Updated 10/23/2024

The ARRL DXCC� System has been returned to service, and our staff is again
processing applications for credit toward DXCC awards. The queue includes
nearly 3,000 award applications submitted via Logbook of The World� (LoTW�)
accounts and mailed paper applications. We are processing the backlog as
quickly as possible, and will provide additional progress updates.

For now, the online DXCC application will remain offline. Any new award
applications we receive will experience significant delays as we work through
the backlog.

As previously reported, LoTW� was returned to service on Monday, July 1, 2024.
Since then, the system has been processing log uploads with little or no
processing delay.

We are grateful for your continued patience and understanding as we continue to
make progress on the few remaining services that have required additional
attention to bring back online.

This story will be updated with new developments.

Updated 9/25/24

Two major systems that have not been returned to service include DXCC and our
accounting system. Most other systems have returned to full service, and we are
working to clean up the stragglers.

DXCC��System

Our DXCC software was written some 20 years ago.� Although an attempt was made
8 years ago to reprogram it to function in a newer environment, that effort was
unsuccessful, so the old system remains running today.� The complication with
bringing it back to service is that it runs on an older version of Windows that
is not supported, and more importantly, we cannot purchase adequate protection
to return it to service on an internet-based network.

We have created an air-gapped network to run the system on and have it
operating in a test, using a different method for connecting it to Logbook of
The World� and online DXCC -- the web service that runs the online DXCC
application.� This configuration is being tested and we expect it can go into
production soon.� We will work to process the backlog as quickly as possible.

New DXCC?

There were statements made by an ARRL Director about work on a new DXCC system.
That is not factual. While we have ideas for merging DXCC and LoTW� into a
single ARRL Radiosport platform, this concept has not been fully spec'd out or
put into development.

ARRL Financials

The system we use for accounting was hit hard by the cyber-attack.� Currently,
the system is fully operational and the ARRL Foundation books are in place.
ARRL's corporate books are current as of 6/30/23, and we are working to load
the balance of the data through 12/31/23. Once we are certain that the books
tie to the audited 2023 financials, we will move to load the 2024 transactions,
which are in digital and paper formats. That will take some time, but we will
have all the data current in time for the 2025 Annual Meeting and the annual
audit.

Backups

Another statement made was about ARRL's backups of our IT environment.� There
was an effort last year to move all of our backups into the cloud.� That was
completed mid-2023.� There was a routine discipline of backups that went
through mid-May 2024 when the cyber attack occurred.� Unfortunately, the entire
cloud infrastructure was penetrated, and all of the environments were deleted.
Not encrypted.� This caused us to revert to other backups that had been made
and stored in other places.� As a result, we were largely able to continue all
of our major services especially publishing our 4 magazines and keeping book
projects, like ARRL Handbook 101, on schedule.

Today, ARRL is using a proprietary system to backup systems and data locally
and to the cloud.� We are also continuing to use air gapped, off-site backups
in specific areas and are expanding that for disaster recovery purposes.� The
new IT environment is currently being rolled out, so the assertion that there
is something wrong or staff are incompetent is not factual.

We appreciate your continued patience and understanding.

This story will be updated with new developments.


Updated 9/3/2024

As previously reported, the ARRL Volunteer Examiner Coordinator (VEC) Program
continues to operate during the systems service disruption.

ARRL Volunteer Examiners (VEs) should continue to submit exam registrations and
material requests. We are able to post new or revised exam session dates and
details to the website, and ship out exam materials. International Amateur
Radio Permits and License Class Certificates are being created and shipped.�

We are unable to create Volunteer Examiner (VE) badges, certificates, and
stickers. New ARRL VE applications and renewals are unable to be processed at
this time. Additionally, the VE session counts webpage data entry programming
has been unavailable since May 12th. It will be updated with new data as soon
as we are able.

A comprehensive update on the status of all ARRL VEC services is available
here[1].

This story will be updated with new developments.


Updated 08/22/2024 �

Work is continuing to return the DXCC systems to service. DXCC award
processing, including the "Online DXCC" application system, is unavailable at
this time. While all DXCC user data is secure and unaffected, we have taken the
precautionary measure of keeping the service offline until we can ensure the
security and integrity of our networks.

As previously reported, Worked All States (WAS) applications are being
processed. WAS certificates and endorsement stickers are being mailed. �

VUCC applications are being processed, and VUCC certificates and endorsement
stickers are being mailed.

Thank you for your continued patience and understanding.

This story will be updated with new developments.

Updated 07/09/2024

As previously reported, ARRL� Logbook of The World� (LoTW�) returned to service
on Monday, July 1, 2024. As anticipated, there was a significant rise in use to
catch up on processing logs following the outage. It took 4 days for normal
processing times to return.

The ARRL Radiosport Department has resumed processing applications for most
awards. The systems for Worked All States (WAS), VUCC, and issuing credits for
CQ WAZ and CQ WPX awards have returned to service. These award applications are
being processed on a "first in/first out" basis.

WAS applications submitted through July 4 and VUCC applications submitted
through July 1 have been processed. CQ awards applications are processed by CQ
and not ARRL.

While we can process these ARRL awards and update records, the ability to print
and deliver certificates, endorsements, and other material has not yet
returned.

The online DXCC� application is unavailable at this time and DXCC awards are
not able to be processed. Work continues to return the DXCC systems to service.
All DXCC data is secure.

This story will be updated with new developments.

Updated 07/03/2024�

The ARRL Volunteer Examiner Coordinator Program continues to operate during the
systems service disruption.

ARRL Volunteer Examiners (VEs) should continue to submit exam registrations and
material requests. While we are unable to post new or revised exam session
dates and details to the website, we can ship out exam materials. Please
remember that most exam materials are available on our website
(www.arrl.org/resources-for-ves[2]). We are processing Amateur Radio License
applications to the FCC. The VEC exam session upload webpage was not affected
by the incident. We are currently unable to create International Amateur Radio
Permits, License Class Certificates, and Volunteer Examiner (VE) badges,
certificates, and stickers. New ARRL VE applications and renewals are unable to
be processed at this time.

ARRL is continuing to accept reimbursement forms to cover the one-time $35
application fee for new license candidates younger than 18-years of age for
tests administered under the auspices of the ARRL VEC. Reimbursement checks may
take longer than normal to be processed at this time.�

This story will be updated with new developments.

Updated 07/02/2024

A comprehensive update on the status of ARRL VEC services is available here[3].

Updated 07/01/2024

Effective 12:00pm ET / 16:00 UTC we will be returning Logbook of The World�
(LoTW�)[4] to service.�

As work progressed on the network, some users encountered LoTW opening briefly
during which some 6600 logs were uploaded. The logs were not processed until
this weekend as we tested that the interfaces to LoTW were functioning
properly.

We are taking steps to help manage what will likely be a huge influx of logs.
We are requesting that if you have large uploads, perhaps from contests or from
a DXpedition, please wait a week or two before uploading to give LoTW a chance
to catch up. We have also implemented a process to reject logs with excessive
duplicates. Please do not upload your entire log to "ensure" your contacts are
in LoTW as they will be rejected. Lastly, please do not call ARRL Headquarters
to report issues you are having with LoTW. You can contact support at
[email protected][5].

Through the end of the year, you may experience planned times when LoTW will be
unavailable. We have been using this time to evaluate operational and
infrastructure improvements we would like to make to LoTW. Those times will be
announced.

We appreciate your patience as we worked through the challenges keeping LoTW
from returning to service. We know the importance of LoTW to our members, and
to the tens of thousands of LoTW users who are not ARRL members. LoTW, just
behind QST, is our second most popular ARRL benefit.��

This story will be updated with new developments.

Updated 6/21/2024

ARRL staff continue to work with outside industry experts to ensure that all
servers have been remediated, tested, and all data has been confirmed. We have
also been working to ensure that the network is secure.

As noted in previous updates, Logbook of The World� servers and data are fine
and working. LoTW� has dependencies on other servers, for example membership
data, which have not yet been returned to service. We will announce when LoTW
is open for user access. We anticipate that the queue will grow quite large
once live, and we are evaluating ways to manage that potential issue.

As a reminder, 2024 ARRL Field Day[6] is this weekend. We note that some groups
have scaled back or cancelled their outdoor plans due to excessive heat or
severe weather threats. Please use caution this weekend (visit Heat.gov[7] for
heat and health information). Enjoy Field Day. The Field Day bulletin will be
transmitted from W1AW and K6KPH as scheduled (see PDF schedule[8]). Your Field
Day results may be uploaded to https://field-day.arrl.org/fdentry.php[9] or
mailed to ARRL after the event.�

This story will be updated with new developments.�

Updated 6/14/2024

We are continuing to make progress on restoring the ARRL network and many
related systems and databases. Thank you for your continued patience and
understanding as we carefully address and restore affected services.

While the Logbook of The World� server, Online DXCC, and related user data are
secure and unaffected, we have taken the precautionary measure of keeping the
services offline until we can ensure the security and integrity of our
networks.

The ARRL Learning Center (learn.arrl.org[10]), our online course and training
hub, is accessible. The Learning Center includes content from across a variety
of amateur radio topics and interests. The Learning Center will require a
slightly different login experience while we continue to restore interconnected
databases. Users who have logged into the Learning Center before will need to
use their EMAIL ADDRESS and use the 'forgot my password' function to set a new
Learning Center password. Users who are new to the system will sign up/register
using their email address. When ARRL systems are fully restored, user accounts
will be reconciled with membership accounts again.

ARRL Field Day is June 22 - 23, 2024. Visit www.arrl.org/field-day[11] for
rules and other resources to help prepare your participation. Following the
event, online entries will be supported as usual at
https://field-day.arrl.org/fdentry.php[12].

The ARRL Volunteer Examiner Coordinator is continuing to process Amateur Radio
License applications to the FCC. We have processed all applications from exam
sessions uploaded by Volunteer Examiners through June 12, 2024. Sessions mailed
to ARRL VEC and received by us are processed within 1 - 2 business days.

The new Amateur Extra-class Question Pool will be effective beginning July 1,
2024. New Extra-class exam booklet designs�will be supplied to VE teams as soon
as possible. Teams may also contact the ARRL VEC after July 1 for an emailed
version in the interim.

This story will be updated with new developments.

Updated 6/6/2024

We are aware that certain members believe we should be openly communicating
everything associated with this incident. We are working with industry experts,
including cyber crime attorneys and the authorities, who have directed us to be
conservative and cautious with our communications while restoring the ARRL
network.

ARRL's association membership system was not impacted by this attack. Members
can renew online at www.arrl.org/join or by phoning ARRL. Additionally, the
online ARRL Store is operating as normal. ARRL Field Day merchandise is still
available for immediate shipping. Please see this latest news[13] story about
ARRL Field Day.

The digital edition[14] of the July 2024 issue of QST will be available as
scheduled, on Friday, June 7.�The print edition will go to press on Monday,
June 10. Members may also access digital editions of QST dating back to 2012,
and current and previous issues of On the Air, QEX, and NCJ magazines back to
2020.

As previously reported, the ARRL Volunteer Examiner Coordinator is processing
Amateur Radio License applications received from Volunteer Examiners who
completed candidate exam sessions. As of June 6, our staff are processing
applications from exams given on May 22.

As previously reported, W1AW, the Hiram Percy Maxim Memorial Station is fully
operational[15] with voice, CW, and digital bulletins, code practice, and
qualifying runs[16]. The station continues to welcome visiting operators[17]
during normal visiting hours.

The ARRL June VHF Contest[18] is June 8 - 10, 2024. Online log submission will
be supported as usual. The log submission function was similarly supported for
the recent ARRL Digital Contest[19], held June 1 - 2.

The publication of most ARRL e-newsletters, including the weekly ARRL Letter,
has resumed with minimal disruption to regular schedules. A new platform for
emailing newsletters has been implemented, leading to some variations in
layout.

Updated 6/4/2024

On or around May 12, 2024, ARRL was the victim of a sophisticated network
attack by a malicious international cyber group. ARRL immediately involved the
FBI and engaged with third party experts to investigate.

This serious incident was extensive and categorized by the FBI as "unique,"
compromising network devices, servers, cloud-based systems, and PCs.

ARRL management quickly established an incident response team. This has led to
an extensive effort to contain and remediate the networks, restore servers, and
staff are beginning the testing of applications and interfaces to ensure proper
operation.

Thank you for your patience and understanding as our staff continue to work
through this with an outstanding team of experts to restore full functionality
to our systems and services.

We will continue to update members as advised and to the extent we are able.

This story will be updated with new developments.

Updated 5/31/2024

Our phone system has been returned to service. ARRL may be reached at
1-860-594-0200 or�1-888-277-5289�(toll-free in the US), Monday through Thursday
from 8 AM to 7 PM Eastern Time, and until 5 PM on Fridays. Members can also
email general inquiries to [email protected][20].

The ARRL International Digital Contest is this weekend, June 1 - 2, 2024, and
the ARRL June VHF Contest is June 8 - 10, 2024. The ARRL Contest Portal will be
available with limited functionality at contests.arrl.org[21]. Log submission
pages will continue to be available at contest-log-submission.arrl.org[22], and
logs received can be viewed at contests.arrl.org/logsreceived.php[23]. The log
submission links are also available on the contest landing pages at
www.arrl.org/arrl-digital-contest[24] and www.arrl.org/june-vhf[25].

Members are reminded that the June 2024 issue of QST magazine, the special ARRL
Field Day edition, can be viewed online[26]. Members may also access digital
editions of QST dating back to 2012. Access to archives for QST issues older
than 2012, which are accessed through the ARRL website, is not presently
available. For On the Air, QEX, and NCJ magazines, members may access the
current issues and previous issues back to 2020. Visit
www.arrl.org/arrl-magazines[27] for links to all four magazines and to download
the ARRL Magazines app.

Work is continuing to publish the July 2024 issue of QST magazine. We
anticipate an on-time delivery of the digital edition, and delayed delivery for
those who subscribe to the printed magazine.

We have heard from members asking if they can renew their membership at this
time. Members can renew online at www.arrl.org/join[28] or by phoning ARRL.

As reported earlier this week, ARRL Store orders (www.arrl.org/shop[29]) have
resumed shipping.

This story will be updated with new developments.

Updated 5/29/2024

This update includes information about the status of several services as we
continue to respond to a serious incident involving access to our network and
systems.

The ARRL Volunteer Examiner Coordinator (ARRL VEC) has resumed the processing
of Amateur Radio License applications with the FCC. A more comprehensive update
on the status of ARRL VEC services is available here[30].

There has been no interruption to visitor operating at W1AW, the Hiram Percy
Maxim Memorial Station. The station resumed voice bulletins on Thursday, May
23. All other scheduled transmissions, including Morse code practice, and code
and digital bulletins, will resume on Thursday, May 30. Please refer to the
regular operating schedule at www.arrl.org/w1aw-operating-schedule[31].

After last week's distribution of the ARRL�Letter, our e-newsletter service has
resumed. Current editions of ARRL�Club News and The ARES��Letter have also been
distributed.

ARRL�Store orders have resumed shipping. Orders are being fulfilled from
earliest order dates to the latest. Please allow additional time for our
processing.

There has been no disruption to the @arrl.net email forwarding service, though
forwarding email addresses and aliases cannot be modified at this time.

Our telephone system is unavailable at this time.

We appreciate your patience as we continue working on restoring access to
affected systems and services.

This story will be updated with new developments.

Updated 5/22/2024

We are continuing to address a serious incident involving access to our network
and systems. Several services, such as Logbook of The World� and the ARRL
Learning Center, are affected.

We have heard from many LoTW� users, asking about the status of the service and
its data. This is not an LoTW server issue, and LoTW data is secure.

Our editorial and production team is preparing the July issue of QST magazine,
which is still going to press. It may be delivered a few days late to members
who receive print subscriptions. The digital edition should be published on
time.

We appreciate your continued patience as our staff and others work tirelessly
to restore affected systems.

This story will be updated with new developments.

Updated 5/17/2024

Some members have asked whether their personal information has been compromised
in some way. ARRL does not store credit card information anywhere on our
systems, and we do not collect social security numbers. Our member database
only contains publicly available information like name, address, and call sign
along with ARRL specific data like email preferences and membership dates.

Original story below:
5/16/2024

We are in the process of responding to a serious incident involving access to
our network and headquarters-based systems. Several services, such as Logbook
of The World� and the ARRL Learning Center, are affected. Please know that
restoring access is our highest priority, and we are expeditiously working with
outside industry experts to address the issue. We appreciate your patience.

This story will be updated with new developments.


[1] http://www.arrl.org/news/arrl-vec-services-update-during-systems-disruption
[2] http://www.arrl.org/resources-for-ves
[3] http://www.arrl.org/news/arrl-vec-services-update-during-systems-disruption
[4] https://lotw.arrl.org/lotwuser/default
[5] mailto:[email protected]?subject=LoTW%20Help
[6] http://www.arrl.org/field-day
[7] https://www.heat.gov/
[8] http://www.arrl.org/files/file/Field-Day/2024/4_37-202420Sked.pdf
[9] https://field-day.arrl.org/fdentry.php
[10] https://learn.arrl.org/
[11] http://www.arrl.org/field-day
[12] https://field-day.arrl.org/fdentry.php
[13] https://www.arrl.org/news/2024-arrl-field-day-2-weeks-away-free-shipping-on-fd-merch
[14] http://www.arrl.org/magazines
[15] https://www.arrl.org/w1aw-operating-schedule
[16] https://www.arrl.org/qualifying-run-schedule
[17] https://www.arrl.org/visit-us
[18] https://www.arrl.org/june-vhf
[19] https://www.arrl.org/arrl-digital-contest
[20] mailto:[email protected]
[21] https://contests.arrl.org/
[22] https://contest-log-submission.arrl.org/
[23] https://contests.arrl.org/logsreceived.php
[24] https://www.arrl.org/arrl-digital-contest
[25] https://www.arrl.org/june-vhf
[26] https://edition.pagesuite-professional.co.uk/Launch.aspx?PBID=604160a2-6442-4ed0-9695-2e753da4eb98
[27] http://www.arrl.org/arrl-magazines
[28] http://www.arrl.org/join
[29] http://www.arrl.org/shop
[30] http://www.arrl.org/news/arrl-vec-services-update-during-systems-disruption
[31] http://www.arrl.org/w1aw-operating-schedule

---
� Synchronet � Whiskey Lover's Amateur Radio BBS

You are viewing proxied material from bbs.kd3.us. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.