Are You Prepared? Laboratory Downtime in the Ransomware Era

By Toby C Cornish, MD, PhD , David S McClintock, MD

At 11:30 AM on October 28, 2020, the electronic health record (EHR) and integrated laboratory information system (LIS) at the University of Vermont Medical Center (UVMMC) suddenly and without warning became unreachable. It soon became apparent that it was not just the Epic EHR system and Beaker LIS module that was down—the UVMMC computer network was down in its entirety. Of course, information system downtimes and isolated network outages occasionally happen, and when they do, they are typically fixed in a few hours or less. With these assumptions in mind, the UVMMC’s clinical laboratory downtime procedures were put into action.

Initially, the full impact of the network outage was not readily apparent. Some deficiencies were immediately both obvious and familiar for the clinical laboratories—no LIS, no EHR system, no interfaces. Unlike outages restricted to a single information system (eg, the LIS), however, the network outage had severed lines of communication throughout the enterprise. Paging was down, as was the voice over internet protocol telephone system. To make matters worse, Microsoft 365 cloud-based services (eg, Microsoft Teams, Outlook, SharePoint) became unavailable on cellular mobile devices because the medical center’s authentication services were down. Network-based fax machines, critical to results delivery during typical downtimes, stopped functioning in both the lab and the clinics. Although analytical instruments continued to function in the clinical labs, they were unable to download test orders or upload results, resulting in manual entry and reporting. Finally, as workflows shifted heavily to paper, printing became complicated as networked printers were rendered useless by the outage.¹ A total loss of the network brought with it a full awareness of the connected nature of the lab.

The clinical laboratories remained somewhat in the dark as the downtime stretched from hours to days. Two days into the downtime, the outage’s root cause was finally revealed: ransomware had been detected on the UVMMC network, the network administrators had shut down the network to limit its spread, and the perpetrators of the cyberattack had contacted the medical center with a ransom demand.

RANSOMWARE AND THE RISE OF CYBERATTACKS IN HEALTH CARE

Ransomware is now recognized as a major mainstream problem, but it is not a new phenomenon. The first recorded instance of ransomware predates the commercial internet and was spread by an unlikely vector: mailed floppy disks. In 1989, Dr Joseph L. Popp, an evolutionary biologist, mailed 20,000 floppy disks to attendees of the World Health Organization’s AIDS conference in Stockholm. The AIDS Information Introductory Diskette purported to be a prognostic tool, but upon inserting the disk, the AIDS/PC Cyborg Trojan hid folders and encrypted file names, rendering the computer unusable. A message then appeared instructing the user to send $189 in an envelope to a PO box in Panama.²

With the rise of the internet, ransomware reemerged in the 2000s. It was not until the 2010s and the emergence of cryptocurrency and the dark Web, however, that ransomware attacks became frequent and widespread. The first documented ransomware attack on a health care facility was at Surgeons of Lake County in 2012, but this incident was not believed to be directed toward health care itself and was instead of the indiscriminate type seen in early ransomware attacks.³

In the past few years, the modus operandi of cybercriminals has changed from a shotgun approach to targeted attacks on large organizations. Targeting of health care facilities began with a ransomware attack in 2014 at Clay County Hospital in rural Illinois.³ Unfortunately, hospitals and health systems have several qualities that make them particularly attractive as ransomware targets: (1) modern health care systems are highly dependent on networked information systems; (2) they possess an enticing bounty of personal demographic, financial, and health care–associated information about patients; (3) they typically rely on older systems with limited vendor competition; and (4) they have been late to put an emphasis on cybersecurity practices. Ultimately, these qualities render health care facilities softer targets than other, commercial enterprises.

Cybercrime continues to be a growing problem, with the annual global cost of ransomware attacks doubling from 2019 ($11.5 billion) to 2020 ($20 billion).⁴ Of this, health care data breaches in the United States total about $6.2 billion per year, with an average cost of $3.62 million per cyberattack.⁴ A recent survey⁵ suggests that ransomware attacks hit 34% of health care organizations in 2021. Cybercriminals successfully encrypted data in 65% of these organizations, and of these, 34% elected to pay the ransom to get their data back.⁵ Notably, the survey respondents reported that paying the ransom did not guarantee that all data were unencrypted.

In addition to the extreme costs incurred, ransomware attacks represent a significant threat to both patient safety and personal data protection. Although there are early anecdotes of cybercriminals avoiding or even canceling attacks on hospitals, they appear to be increasingly less concerned about the potentially life-threatening impact of their ransomware attacks. While it is impossible to quantify the total impact of cybercrime on patient health and safety, we do know that the first deaths allegedly linked to ransomware attacks were reported in 2019 and 2020.^6,7 Furthermore, in 2021, multiple lawsuits were filed against health care institutions in relation to cyberattacks.^8,9 The claims in the lawsuits range from negligence by the hospital for not dutifully notifying patients about the patient safety risks during cyberattack-induced hospital downtimes to the increased costs patients incurred from their personal data being stolen.

CYBERATTACKS AND DOWNTIME IN THE LABORATORY

By and large, clinical laboratories are collateral damage in cyberattack campaigns waged against larger health care organizations; attacks directly targeting labs have occurred recently, however, and more may follow.¹⁰ Clinical labs are particularly sensitive to the damage inflicted by cyberattacks because of their heavy reliance on connected information systems. UVMMC discovered this fact firsthand during its extended downtime.^1,11 It also discovered that somewhat counterintuitively, direct patient care areas, such as the operating rooms, procedure areas, and clinics, were considerably less affected by the network outage, with a reduction in specimen volume of only about 30%.^1,11 Given the laboratory’s role as a service provider for practically every clinical specialty, it relies on the continuous flow of data from the many different health information systems within the health care enterprise. Therefore, preparation for extended downtimes is absolutely critical to a lab’s business continuity.

Laboratories are required to have downtime procedures. The College of American Pathologists (CAP) Laboratory Accreditation Program makes several references to downtimes in its regulatory checklists, the most notable being GEN.43837 (“Downtime Result Reporting”).¹² Compliance with this phase II item (ie, seriously affecting patient care) requires that “There are written procedures to ensure reporting of patient results in a prompt and useful fashion during partial or complete downtime and recovery of the system.” ¹⁰ Other checklist items reference the fact that lab procedure manuals, temperature charts, and other critical requirements must still be completed during downtimes. Of note, laboratory downtime procedure requirements should not be confused with laboratory computing requirements related to downtimes, such as GEN.43946 (“Data Preservation/Destructive Event”) that address the need for disaster recovery planning.¹⁰ The former is the lab’s domain, while the latter is that of information technology (IT) services.

At first, UVMMC followed a conventional approach to downtime procedures. It quickly found these procedures inadequate for such an extended downtime event, however.^1,11 For example, the anatomic pathology (AP) lab initially held specimens in anticipation of the system being restored, and the clinical pathology (CP) lab pivoted to preprinted downtime labels with barcodes. Both labs were forced to fall back to paper requisitions. These are sound procedures for brief outages (perhaps up to 48 hours), with most laboratories employing similar downtime procedures.

Interestingly, while the CAP requires disaster recovery procedures be tested periodically for effectiveness, it makes no similar pronouncements about testing downtime procedures.¹² For most labs and IT groups, however, brief downtimes—scheduled and unscheduled—occur at least once per quarter, which provides sufficient opportunity to exercise downtime procedures. As UVMMC experienced, however, these brief downtimes do not adequately simulate extended outages, nor do they fully test disaster recovery planning for institutional processes. Active testing of downtimes has been proposed for clinical laboratories, but such events are difficult to model and implement, which is precisely why studying real-world experiences like those of UVMMC is so important.¹³

An old Yiddish proverb states, “man plans, and God laughs,” reflecting the inevitable breakdown of even the best plans in the face of unexpected circumstances. Planning is essential, but it is impossible to predict the impact of a downtime that has yet to occur. Furthermore, as the information systems affected by an outage become more widespread and the length of the outage increases, downtime procedures meant to cope with, at most, a 48-hour downtime become increasingly less applicable. In these situations, adaptability, communication, and innovation become more important than rigid planning, and this is probably the most critical lesson to be learned from the UVMMC experience.¹⁴

Uptime and the UVMMC Experience

In the end, UVMMC declined to pay the ransom. Instead, it contacted the US Federal Bureau of Investigation and began the arduous task of disinfecting and restoring thousands of workstations, servers, and network devices. Eventually, on November 22, after a 25-day outage, UVMMC started to bring its network back online. In the lab, though, the work of reconciling paper downtime results with the LIS and EHR system would continue for months.¹⁵

In a special series of 4 articles, authors from the UVMMC Department of Pathology and Laboratory Medicine describe their experiences dealing with a 25-day downtime resulting from this ransomware attack.^1,11,14,15 The first article in this series appears in this issue of the American Journal of Clinical Pathology.¹¹ Subsequent articles in the series will follow in future issues. Part 1 of the series focuses on the downtime response in the AP lab; part 2 focuses on the downtime response in the CP lab; part 3 discusses the importance of communication, coordination, and use of an “incident command team” for managing operations in crisis; and part 4 details the transition to uptime, including postdowntime reconciliation, billing, and quality assurance. Given the increased importance of cybersecurity and cyberattacks within health care, this series of articles is exceedingly topical and should be considered required reading for laboratory professionals, laboratory IT staff, and administrators alike.

References