switch glitch

Anger Over $230m Computer System Behind Subway Meltdown

Riders tried to pack onto a train at Columbus Circle after major service disruptions during Friday's heatwave, July 19, 2019.
Riders tried to pack onto a train at Columbus Circle after major service disruptions during Friday’s heatwave, July 19, 2019. Photo: Jose Martinez/THE CITY

Some MTA board members bugged out Monday over a computer glitch blamed for Friday night’s subway meltdown as agency documents confirmed the fiasco was no isolated incident.

Board member Larry Schwartz even raised the possibility of a cyber attack or other criminal activity.

But as THE CITY first reported Sunday, the breakdown marked the latest in a string of software problems with the computerized Automatic Train Supervision system that routes trains along all but one of the numbered subway lines and the 42nd Street Shuttle.

“This was a software bug and because that bug… was also in our backup servers, that’s why they went down as well,” New York City Transit President Andy Byford said at a committee meeting Monday. “That bug automatically powered down our servers without notice.”

System Cost $230 Million

Starting in 1997 and running through 2008, the MTA spent close to $230 million to install the ATS system, which brought real-time countdown clocks to subway platforms on most of the numbered lines and gave dispatchers the ability to control train movement remotely.

The system has been flagged for problems at least 13 times since June, including a pair of so-called “major incidents” this month that contributed to more than 180 subway delays.

“We have on occasions had problems with ATS, but nothing of this magnitude,” Byford said.

New York City Transit Authority President Andy Byford attends an MTA board meeting, Jan. 27, 2019.
New York City Transit Authority President Andy Byford attends an MTA board meeting, Jan. 27, 2019. Photo: Ben Fractenberg/THE CITY

The latest bug arrived at the start of a weekend heatwave, crippling service on the 1, 2, 3, 4, 5, and 6 lines, as well as the 42nd Street Shuttle, for nearly 90 minutes. Officials said the cause differed from what was behind a July 6 failure of Automatic Train Supervision, which, according to another incident report, delayed 61 trains.

That breakdown, Byford said, was caused by a single piece of code.

“When we did the downloads and we went through the logs with our supplier, one digit, one illegal character, for whatever reason, caused an illogical command to be given,” he said.

Since then, Byford said, a “patch” has been applied.

“Things often go completely wrong because of just one tiny piece of code,” said Sarah Feinberg, an MTA board member who chairs the transit committee. “Which is why companies are supposed to check and recheck and have multiple people reading every single piece of code, every single line of code, testing, retesting and checking for glitches all the time.”

Past Glitches Snagged Service

Friday’s nightmare was caused by a different computer problem in the same system.

Tim Minton, an MTA spokesperson, said software upgrades and maintenance activities related to Automatic Train Supervision are ongoing.

“Engineers, both at the MTA and our software vendors, are vigilant to test prior to deployment, detect and resolve problem patterns that may develop,” Minton said. “Glitches are addressed promptly, with the goal of providing uninterrupted service.”

That was not the case on July 6 and at other times. Documents obtained by THE CITY show that Automatic Train Supervision failure played a part in several other major incidents on the numbered lines dating to October 2018. Among them:

On March 22, 130 trains were delayed when a “disruption in the connection of the ATS-A system” occurred for 51 minutes just after 9 am.

On March 21, 85 trains ran late after “a software event caused a disruption in the connection of certain Automatic Train Supervision-A consoles.” The disruption lasted 74 minutes.

On October 21, 2018, a “system-wide ATS-A failure” delayed 87 trains.

“Appropriate delay announcements were made,” each incident report noted.

‘No Suspicion’ of Sabotage

THE CITY’s report struck a nerve with City Council Speaker Corey Johnson — who called the outage “unacceptable” and demanded an “explanation on how it won’t happen again.”

The report also prompted Schwartz, an MTA board member and a confidante to Governor Andrew Cuomo, to question the “credibility” of THE CITY. Schwartz called for “outside expertise” to be brought in to examine Sunday’s incident, as well as an investigation by the MTA Inspector General, to see whether foul play was involved.

MTA board member Lawrence Schwartz (left) at a January MTA board meeting.
MTA board member Lawrence Schwartz (left) at a January MTA board meeting. Photo: Ben Fractenberg/THE CITY

“When you mention the word ‘bug’, you know, I want to make sure that, and with law enforcement as well, one, was this a criminal activity that took place?” he said.

Byford ruled out any system sabotage.

“There is no suspicion that this was either a virus and there’s definitely no suspicion or concern that this was in any way a law-enforcement issue, a terrorist issue, some sort of hacking or cyber attack,” he said. “That’s absolutely not the case.”

He apologized again for the subway meltdown.

“The bottom line is, we gave people a pretty horrendous journey on Friday night,” Byford said. “I feel really badly about that.”

Sign up for “THE CITY Scoop,” our daily newsletter where we send you stories like this first thing in the morning.

Want to republish this story? See our republication guidelines.