Severe ACE Outage Prompts CBP to Examine National Downtime Procedures, Agency Officials Say

CBP will use the widespread ACE outage that occurred on the evening of Nov. 14 to inform national downtime procedures and best practices currently under development, agency officials said on a Nov. 15 call with ACE filers and developers. CBP headquarters is communicating with the ports to determine what worked well during the downtime and identify any issues and deficiencies, CBP Deputy Executive Assistant Commissioner Cynthia Whittenburg said. The outreach is part of an effort to “ensure that our downtime procedures are efficient as possible” and incorporate any best practices identified, she said.

The agency effort to glean best practices from the ports will help resolve issues related to discrepancies between local downtime procedures, said Jim Swanson, director of cargo security and controls at CBP’s Office of Field Operations. The agency will take what it hears back and update national downtime procedures. CBP is working with a Commercial Customs Operations Advisory Committee (COAC) working group to develop a “trade-facing best practices document” and a description of CBP national downtime procedures, and has also asked ports to identify their own local procedures, which may vary due to differences in each port’s infrastructure, Swanson said.

The outage, first mentioned in a CSMS message sent by CBP at 4:03 p.m. EST on Nov. 14 and further detailed in a second message sent at 9:03 p.m., was caused by a database issue, said Jim Byram of CBP’s Office of Information Technology. The system was back up and running in the late evening, with cargo release functionalities available at around 10:20 p.m. and the backlog finally cleared by about 1 a.m. Nov. 15, CBP said in a subsequent message. IBM technicians have been “working around the clock,” alongside CBP technical staff, to determine the root cause, though none had yet been identified as of mid-morning Nov. 15, Byram said.

CBP ports of entry quickly moved to downtime procedures as the problems became evident, in some cases even before they were notified by CBP about 45 minutes after the problem started, Swanson said. In some cases that downtime did include heightened enforcement “because we were working with less information,” which did result in some “minor delays,” Swanson said. Some ports stayed open late to clear the backlog, and additional staff was provided to express consignment facilities for manual work “to keep things going,” he said.

CBP considered moving to a backup database during the outage, but ultimately its technicians decided they could bring the primary database back up before they could get the secondary system going, Byram said. In the meantime, the agency could have issued more precise information to the trade on exactly what systems were affected by the outage, he said. CBP will also be working on its ACE dashboard to ensure that it accurately reflects system status, Byram said, after reports that there was no indication on the dashboard that the system was down.

Another “lesson learned” during the outage is the need for improved communication between CBP and the Census Bureau on how the outage affected export filing systems, Swanson said. According to Census’ Paul Newman, the bureau did not issue any downtime notices because it was unaware that ACE issues were occurring. Both Census and CBP need to approve the activation of downtime, but Census officials were under the impression that the system was back up by the time they left the office the evening of Nov. 14, Newman said. Census officials were able to access the system, and received no subsequent messages that the issues were still occurring and affected exports, he said. “We did not implement downtime because we thought the system was operational,” Newman said.