Scheduled maintenance
Twilio Services
Issues related to Twilio’s Software
External Connectivity
Issues found outside of Twilio’s Network
Zipwhip Gateway
Issues related to Zipwhip's Services
System Metrics
Connectivity Lost on 10% of Super SIM Devices
Incident
Report for Twilio
Resolved
Connectivity losses in Super SIM Devices have been restored and Super SIM Devices are operating normally at this time.
Monitoring
We are now seeing recovery of the Super SIM connectivity issues. We are continuing to monitor for service stability. We will provide another update in 2 hours or as soon as more information becomes available.
Update
While attempting to restore all Twilio services to normal, we made a configuration change at our datacenter modifying how some traffic between Twilio's Mobile Core and our carrier partners is handled at 2:16 PM PT. This may have created some delays or prevented devices from connecting in a small number of countries. The most notable impact observed by Twilio was in New Zealand. At ~8:00 PM PT, the issue was identified and we temporarily blocked attachments on the impacted connectivity partner in New Zealand. SIMs failing to connect should have failed over to backup IMSIs on the SIMs allowing them to reconnect via an alternative connectivity partner. At ~9:00 PM PT, the configuration change was reverted and we re-enabled the blocked connectivity partner in New Zealand. Impact in other countries not identified by Twilio should have subsided after the configuration change was reverted. We'll provide another update in 16 hours or as soon as more information becomes available.
Update
The transfer of traffic between nodes within our datacenter has completed. During the process, we observed 7% of active data sessions on Super SIM being dropped. Devices that attempted to reconnect immediately after losing connectivity should have been able to successfully reconnect to the new node. Our team is actively working to restore the service completely. We'll provide another update in 16 hours or as soon as more information becomes available.
Update
The transfer of traffic between nodes within our datacenter has completed. During the process, we observed 7% of active data sessions on Super SIM being dropped. Devices that attempted to reconnect immediately after losing connectivity should have been able to successfully reconnect to the new node. Our team is actively working to restore the service completely. We'll provide another update in 8 hours or as soon as more information becomes available.
Update
Twilio engineering is planning to move traffic between nodes within our datacenter as part of the ongoing emergency maintenance at 12:15 PM PT. Super SIM data connectivity is expected to be impacted to a similar degree as the 8:15am PT drop when a similar event occurred. We'll provide another update in 8 hours or as soon as more information becomes available.
Update
Twilio engineering is continuing to work with our datacenter partners to restore all Twilio services. We are monitoring for any changes in traffic patterns in our datacenter that may result in a temporary drop in data session connectivity. We'll provide another update in 4 hours or as soon as more information becomes available.
Identified
During the emergency maintenance we observed 10% of active data sessions on Super SIM being dropped as they might have lost data connectivity at 8:15 AM PT. The drop was caused by moving traffic back to primary datacenter hardware to which the latest update has been applied. Devices that attempted to reconnect immediately after losing connectivity should have been able to successfully reconnect to the new node. Our team is actively working to restore the service completely. We'll provide another update in 2 hours or as soon as more information becomes available.
Update
During the emergency maintenance we observed 10% of active data sessions on Super SIM being dropped as they might have lost data connectivity at 8:15 AM PT. The drop was caused by moving traffic back to primary datacenter hardware to which the latest update has been applied. Devices that attempted to reconnect immediately after losing connectivity should have been able to successfully reconnect to the new node. Our team is actively working to restore the service completely. We'll provide another update in 2 hours or as soon as more information becomes available.
Update
During the emergency maintenance we observed 10% of active data sessions on Super SIM being dropped as they might have lost data connectivity. The drop was due to a failover operation required to re-establish connectivity in one of our data centers. Devices that attempted to reconnect immediately after losing connectivity should have been able to successfully reconnect to the redundant instances while the impacted one recovered. Our team is actively working to restore the service completely. We'll provide another update in 2 hours or as soon as more information becomes available.
Update
During the emergency maintenance we observed 10% of active data sessions on Super SIM being dropped as they might have lost data connectivity. The drop was due to a failover operation required to re-establish connectivity in one of our data centers. Devices that attempted to reconnect immediately after losing connectivity should have been able to successfully reconnect to the redundant instances while the impacted one recovered. Our team is actively working to restore the service completely. We'll provide another update in 2 hours or as soon as more information becomes available.
Update
We have identified the cause of the Super SIM connectivity issues. Our team has determined that the problem stems from a network node malfunction. To address this issue, the emergency maintenance is in progress. During this maintenance, there may be temporary connectivity loss for some devices. We apologise for any inconvenience this may cause. We'll provide another update in 2 hours or as soon as more information becomes available.
Update
We have identified the cause of the Super SIM connectivity issues. Our team has determined that the problem stems from a network node malfunction. To address this issue, we will be conducting an emergency maintenance within the next hour, beginning at 2 am PT. During this maintenance, there may be temporary connectivity loss for some devices. We apologize for any inconvenience this may cause. We'll provide another update in 2 hours or as soon as more information becomes available.
Monitoring
Our engineers have observed 10% of active data sessions on Super SIM being dropped at 1:15 PT. Devices that attempted to reconnect immediately after losing connectivity should have been able to successfully reconnect to the redundant instances while the impacted one recovered. We will continue to monitor for system stability. We'll provide another update in 2 hours or as soon as more information becomes available.