Immigration clearance system disruption lasted 4½ hours, affected 85,000 land and air travellers

A disruption during a trial for an immigration clearance system upgrade on March 31 by the Immigration and Checkpoints Authority (ICA) affected a total of 85,000 air and land passengers.

They included 30,000 passengers on 113 departure and 111 arrival flights; 21 of the passengers missed their flights.

Another 55,000 travellers, who passed through the two land checkpoints at Woodlands and Tuas, were also affected.

On Friday in Parliament, Minister of State for Home Affairs Muhammad Faishal Ibrahim said that following this incident, ICA is reviewing the approach to the system upgrade.

Responding to questions from several MPs about the disruption, he said the pre-scheduled trial was needed for an upgrade of the Multi-Modal Biometrics System (MMBS).

The system facilitates automated immigration clearance using travellers’ biometrics.

But the trial caused the storage systems to overload at about 10.40am, and the process was aborted immediately.

Associate Professor Faishal said vendors were on standby to tackle problems if a disruption happened, and the plan was to recover the system within 30 minutes if the trial did not go well.

But when problems surfaced, the disruption was worse than expected and lasted for about 4½ hours. 

Prof Faishal said: “The extent of the overload was much more severe than anticipated, and the vendors who were on-site had to work with their global support team to diagnose and reboot the servers.”

ICA activated its business continuity plan once the system went down, and off-duty officers were recalled to help operate manual immigration counters and perform crowd control.

Across all checkpoints, the failover process kicked in, and all manual counters, and some automated lanes, switched to backup systems.

However, Prof Faishal said that not all the automated lanes have this failover capability, as different models were procured over the years.

At the land checkpoints, travellers experienced delays of up to 30 minutes at the start of the incident.

At Woodlands Checkpoint, there was another hour of delays for cars as the car arrival zone had to be converted to clear motorbikes manually.

Prof Faishal said the immigration clearance at Changi Airport’s departure halls was significantly affected, with travellers redirected to manual counters for immigration clearance.

He added that ICA stepped up manning of the manual counters by recalling off-duty officers, deploying administrative staff and retaining officers from the outgoing shift.

Changi Airport Group assisted ICA by deploying more Changi youth ambassadors and office staff, along with the group’s duty terminal managers and its customs, immigration and quarantine team.

Said Prof Faishal: “They helped identify travellers whose flights were departing soon so that their departure clearance could be prioritised. Public announcements were also made to appeal for such travellers to step forward.”

He added that the 21 travellers who missed their flights at Terminal 4 were offered the option of booking alternative flights within a week at no extra cost.

Before the incident, Prof Faishal said, ICA had been upgrading its systems progressively and cautiously, with 10 out of 12 systems enhanced. Only the MMBS and one other system had not yet been upgraded.

For the MMBS, the system upgrade would involve replicating the large biometric database to an upgraded system.

Prof Faishal added: “This has to be done continuously over a few days, and the MMBS has to remain operational during this period.”

Prof Faishal said user acceptance tests which are standard procedure for any system upgrade were successful. Subsequent trials in the production environment were also stable.

Following that, he said there were plans to conduct further controlled trials during different times of the day, to ensure the system upgrade would not disrupt operations, before proceeding with the actual upgrade.

An earlier trial conducted on March 15 from 1.30am to 3.30am was successful, said Prof Faishal.

He said ICA is reviewing the approach to the upgrade, following the disruption.

He added the alternative approach that ICA was initially offered by the vendors that involved upgrading the system directly without replicating the database, was not tenable as it may have required five days of system downtime.

Said Prof Faishal: “Even with the implementation of business continuity plans, degradation in service standards was inevitable given the volume of travellers, and especially at the onset of the incident. We apologise to affected travellers for the inconvenience caused and thank them for their understanding.”