A four-plus-hour halt for all US airline flights on Nov. 13, apparently caused by a botched update of telecom equipment, is a high-profile reminder to IT leaders that modernization effects often don’t go as expected and the impact can be severely consequential.
Updating any IT or telecom component without a backup system in place can be risky business, as can scheduling updates during peak system use. In the FAA’s case, the ground stop order came just before 4 p.m. Eastern Time, with about 3,500 flights in the air.
“Peak demand is not the time to discover whether a change was effectively tested,” says Michael Jenkins, CTO at cybersecurity provider ThreatLocker. “Unless waiting creates a greater immediate risk, test, stage, and schedule critical updates for a period when the system can tolerate disruption. Base a maintenance window on operational reality, not convenience.”
IT leaders should consider upgrades during times of lower demand, with enough time to validate the change and restore service before activity increases, agrees Alex Perry, a partner at IT research firm ISG.
“The timing should reflect the potential service impact, even when the task itself is routine,” he says. “Even work that is considered routine can have a significant impact when other systems and services depend on the equipment being changed. A remote location does not necessarily mean the impact will stay local.”
Never a good time
Other IT leaders note that there’s never an optimal time for an organization like the FAA to upgrade its network, with flights happening around the clock. In cases like that, especially with always-on critical systems, a more complex risk calculus is needed,
“You obviously want to avoid introducing unnecessary change during periods of peak business activity,” says Richard Amos, CIO at IT solutions and managed services provider Blue Mantis. “But at some point, delaying change can create more risk than executing the change.”
Eliminating all risk during an IT upgrade is unrealistic, he adds. Instead, IT leaders should focus on understanding and managing the risk, and making deliberate decisions about the level of risk the organization is willing to accept, he says.
Details are sketchy
Nearly a week after the FAA incident, the exact cause of the ground stop ordered remains unknown, with officials from President Donald Trump’s administration alternatively saying it was related to a replacement of copper telecom wiring, replacement of a router, or a software upgrade.
Supposedly, when a new router was connected to the FAA network at a “remote” location in Arkansas, it began to flood the national air traffic communications network with about 1,000 connection attempts every four to five seconds, effectively creating an internal DDoS attack.
Despite the ongoing confusion about the cause of the incident, most IT leaders see a pressing need for the FAA to update its decades-old, copper-based telecom network. But IT modernization efforts rarely go exactly as planned, they say.
Instead of episodic modernization efforts, organizations should consider lifecycle management plans, Blue Mantis’ Amos advises. IT leaders should maintain accurate asset inventories and understand end-of-life and end-of-support dates, he says.
However, decades-old technology isn’t necessarily bad technology, he adds.
“If something has been engineered for reliability and continues to perform its intended function, its age alone isn’t sufficient reason to replace it,” Amos says. “The problem tends to be security, support, recovery, and dependencies — in other words, the risk accumulating around it.”
That said, organizations may be ignoring significant risks if they ignore IT upgrades because the current system works, he adds.
“‘If it ain’t broke, don’t fix it’ can work for your toaster, but it is not a technology strategy,” Amos says. “At the same time, you shouldn’t modernize simply because something new is available, or bright and shiny. Modernization should be tied to a business outcome like reducing risk, increasing resiliency, improving security, lowering cost, and such.”
In addition to the timing of upgrades, another major lesson from the FAA ground stop is to test and the proposed fixes and to have backup systems in place, says Joe Saunders, founder and CEO of cybersecurity vendor RunSafe Security.
“The biggest lesson is that modernization isn’t just about replacing old hardware or migrating to newer software,” he says. “It’s about building resilience into the entire system.”
Before major upgrades, IT leaders need to understand how different technologies interact, identify potential points of failure, and ensure that a problem in one area doesn’t cascade across operations, Saunders advises.
“Critical infrastructure operators should be planning not only for upgrades, but also for continuity when those upgrades don’t go exactly as expected,” he adds. “Resilience has to be treated as a core requirement, not an afterthought.”
Replacing old systems can become necessary, but IT leaders should focus on incremental upgrades, extensive testing, parallel operations where possible, and architectures that limit the impact of any single failure, Saunders recommends.
“Critical systems can’t remain frozen in time indefinitely,” he says. “Older technology often becomes harder to support, maintain, and secure as expertise, replacement parts, and vendor support diminishes.”
Evaluate risks and rewards
The “ain’t broke” approach to IT systems maintenance can be seductive for the bottom line, and even work for a while, but IT leaders should use the FAA ground stop and other high-profile outages of late as stark reminders to CEOs and their boards that infrastructure upgrades are necessary and must be invested in and planned for well in advance, instead of waiting until a system reaches a crisis point before taking action or rushing through modernization to capitalize quickly on technology advances.
“There’s some truth to the idea that proven systems can be highly reliable, especially in safety-critical environments where stability matters,” Saunders says. “The challenge is that ‘not broken’ doesn’t necessarily mean sustainable or secure over the long term.”
ISG’s Perry notes that modernization often happens incrementally, with new equipment and software introduced alongside older technology. IT leaders should check how the old and new technology will work together. Before work begins, the technical team and contractors should agree on the potential impact, who can authorize or stop the change, and how operations will continue if something goes wrong, he recommends.
“You need the right technical staff, vendor support, and people with authority to make decisions if service is disrupted,” Perry adds. “The window has to allow time for recovery from a failed change, as well as installation. A quiet period that ends just before a critical business deadline may leave very little room to respond.”