The incident has been resolved. All pending events with the exception of ~1.82% have been processed. Most of which is likely to be duplicates. The system is back to normal. All customers should be able to access the previously missing data via the API and the UI.
Posted Oct 11, 2024 - 09:26 PDT
Update
As of Wednesday, October 9th, 4:00 PM UTC, we have resumed near-real-time processing for the amount billed fields in the Usage API and Console insights for both Programmable Wireless and Super SIM. However, we are still facing issues with missing billed amounts for the following intervals: Monday, October 7th, 8:00 AM UTC – Tuesday, October 8th, 9:30 AM UTC: 30% of usage events are missing. Tuesday, October 8th, 9:30 AM UTC – Wednesday, October 9th, 4:10 PM UTC: 100% of usage events are missing. The team has identified the issue and is working on a fix. The missing usage events and billed amounts for these intervals will be reprocessed and backfilled early next week.
We will post the update in the next 24 hours or sooner if more information will become available.
Posted Oct 10, 2024 - 10:49 PDT
Update
We are still experiencing the issue with the billing pipeline. The bug in software caused the billing information to be delayed on the UI and API. The team is still working to identify the issue. We will post the update in the next 16 hours or sooner if more information will become available.
Posted Oct 10, 2024 - 00:14 PDT
Update
We are still experiencing the issue with the billing pipeline. The bug in software caused the billing information to be delayed on the UI and API. The team is still working to identify the issue. We will post the update in the next 8 hours or sooner if more information will become available.
Posted Oct 09, 2024 - 16:21 PDT
Update
We are still experiencing the issue with the billing pipeline. The bug in software caused the billing information to be delayed on the UI and API. The team is still working to identify the issue. We will post the update in the next 4 hours or sooner if more information will become available.
Posted Oct 09, 2024 - 12:24 PDT
Update
We are still experiencing the issue with the billing pipeline. The bug in software caused the billing information to be delayed on the UI and API. The team is still working to identify the issue. We will post the update in the next 2 hours or sooner if more information will become available.
Posted Oct 09, 2024 - 10:27 PDT
Update
Since 2024-10-08 1:00PM UTC we experience the issue with the billing pipeline. The bug in software caused the billing information to be delayed on the UI and API. The team is working to identify the issue. We will post the update in the next 1 hour or sooner if more information will become available.
Posted Oct 09, 2024 - 09:27 PDT
Investigating
Our monitoring systems have detected a potential issue, billing information for Super SIM and Programmable Wireless is delayed via API and UI. Our engineering team has been alerted and is actively investigating. We will update as soon as we have more information.
Posted Oct 09, 2024 - 09:11 PDT
This incident affected: INTERNET OF THINGS (Programmable Wireless, Super SIM).