Tennessee and Nashville cybersecurity called into question following bombing

The blast that rocked downtown Nashville on Christmas morning had cybersecurity ramifications across the country, leaving millions of people digitally stranded without phone or internet connections, knocking 911 communications completely offline for entire cities, damaging hospital computer systems, silencing some government communications and blocking countless businesses from processing electronic payments.

With a single blast, an apparent suicide bomber brought much of the United States’ digital world to its knees.

The attack on the Second Avenue AT&T data center accentuated how reliant society can be on a single corporate telecommunications system.

Among the known impacts of the blast were:

• Many cities outside of Nashville lost 911 capabilities, forcing citizens to use local phone numbers instead. AT&T provides the emergency communications system to the state, which is overseen by the Tennessee Emergency Communications Board.
• While Nashville’s own 911 system was not affected, according to an Office of Emergency Management spokesman, the city is an AT&T subscriber for its internet and cell phone services, including for the Metro Nashville Police Department. Additionally, state employees temporarily lost data access, including to email, in the aftermath of the blast.
• Multiple hospital workers reported mass confusion following the blast because their computer and phone systems relied on AT&T networks. Frontline hospital staff switched to paper records, withstood long waits for lab results and faced major complications writing prescriptions and handling medical records.
• Major corporations and small businesses alike lost access to credit card processing equipment that relied on AT&T.
• Federal Aviation Administration communications at the Nashville airport were also affected, temporarily delaying flights on Christmas day and prompting air traffic control to use different air routes for some flights.
• The most obvious impact was regular citizens and small business owners cut off from family, friends and work because of the blast at a time when the pandemic has left most people isolated at home and reliant on their digital connections.

AT&T was still scrambling to repair infrastructure damaged by the blast. The company began posting updates on Christmas, painting the picture of a harrowing and complicated corporate wide response. At one point overnight on Friday into Saturday, a fire at the facility reignited, forcing the evacuation of repair workers.

AT&T sends disaster response crews to address repairs

The biggest issue, according to AT&T, was electricity, as the building’s power source was knocked out by the RV explosion. Burrowing through walls, repair workers connected gas powered generators, pumped out more than 3 feet of water and assessed wider damage to the building.

The company said on Sunday that power had been restored to four floors inside the building and that 65% of the mobility sites affected by the explosion had been fixed. On social media, customers complained of outages in Kentucky, Alabama, Georgia and Florida. 

“Today our teams are working on adding additional cabling and more generators to provide more power into the building for our equipment and the restoration efforts,” AT&T’s Sunday morning update read. “We continue to assess the damage to the facility and have confirmed that the building’s commercial power connections are damaged and offline. We are working with the local power utility on repairs. In addition, there is other significant damage to the building from the blast, including to the elevators, some beams/columns, and the building’s façade.”

By Sunday afternoon much of the internet and cell phone service had returned in the Nashville area.

TN board oversees 911 systems

State oversight of emergency communications falls under the purview of the Tennessee Emergency Communications Board. Tennessee is part of AT&T’s emergency services internet protocol network, providing emergency communications systems through the internet to local governments.

The board has a mission of “ensuring that every citizen can effectively access the life-saving power of 911.” According to the board’s strategic plan, most recently updated in 2019, the board was collaborating with AT&T on transitioning to the company’s nationwide emergency communications network.

“AT&T supports neighboring states on its nationwide ESInet; As Tennessee transitions to this network, TECB plans to have interoperability discussions with AT&T,” the emergency communication board’s 2019 strategic plan update said.

In its annual report filed last year, the emergency communications board described the process and advantages of switching to AT&T’s new digital emergency communications network.

“The project, referred to as NG911, involves utilization of the State’s secure, private, outsourced Multiprotocol Label Switching network called NetTN, provided by AT&T and managed by Strategic Technology Solutions in the Tennessee Department of Finance and Administration,” the annual report stated. “The new network improves redundancy, reliability, and 911 call delivery. It enhances interoperability and increases the ease of communication between ECDs, allowing immediate transfer of 911 calls, caller information, and other data on a statewide level. NG911 will also provide alternate paths to process emergency calls in the event of an outage, providing life saving capabilities in the event of an emergency that would have been unachievable on the outdated analog network.”

State Sen. Jeff Yarbro said that 911 centers and cellular service are “critical safety infrastructure,”

“We need to rethink our system and approach to make it far more resilient,” said Yarbro, D-Nashville. “Especially now that the vulnerability is apparent, Tennessee cannot afford to leave itself open to a single attack that could cascade into a series of crises.”

A spokesman for the board said on Sunday that sStaff members of the Tennessee Emergency Communications Board have been in regular contact with regional public safety answering points (PSAPS) to assess outages and offer any assistance we can provide.  We continue to share alternative contact information for 911 centers and are in regular contact with AT&T about restoration of services.”

Metro regularly audits cybersecurity

While Metro was fortunate not to lose its 911 access, the bombing called into question the city’s emergency communications system. The Metro Audit Committee regularly examines the city’s emergency communications systems and publishes its findings in reports that are largely redacted for security reasons.

Metro officials have questioned over the years if the AT&T data center was in an appropriate location on a highly trafficked urban street instead of a secluded remote area.

Metro Councilman Bob Mendes, who has served on the audit committee and on the Nashville Electric Service board of directors, said his review of those reports showed  “areas that need improvement” from time to time but not indicators that the city’s emergency communications is “on fire” or deeply troubled.

“My sense from being on the NES board and the Metro audit committee from 2015 to 2019 was our emergency communications were being stress-tested pretty regularly and pretty thoroughly and the city was doing a good job of staying on top of it,” Mendes said.

Hospital spokespeople with Vanderbilt University Medical Center and HCA, two of the region’s three major hospital systems, did not immediately respond to questions about how their facilities were impacted. 911 systems in suburban counties and as far away as Knox County were impacted by the data loss. 

Metro Councilman Freddie O’Connell whose district includes the historic Second Avenue block that was the epicenter of the attack said he hopes the attack prompts the city to take a broader examination of its cybersecurity and infrastructure security.

O’Connell pointed out that Nashville has experienced loss of electricity and phone service multiple times in 2020. Besides the blast, residents were left in the dark following the tornado and derecho storms earlier this year.

“What you learned yesterday is our emergency communications apparatus is highly sensitive, not just for Nashville but for the region. Even the airport was down on Friday,” O’Connell said. “What does someone who doesn’t have any cell service whatsoever do if you don’t have a landline and your only provider is AT&T? I do think it raises important questions of the redundancy of our telecommunications.”