Is Cekat AI Down?

**AI replies not sent for some WhatsApp messages** **Incident date:** 31 October 2026 **Started:** 05:05 WIB **Service recovered:** 09:46 WIB **Time until service recovery:** About 4 hours and 41 minutes **Latest status:** Recovered. A permanent fix has been implemented. All times in this report are WIB \(UTC\+7\). 1. Overview On 31 October 2026, some WhatsApp messages from three businesses were not sent. Most of them were AI agent replies, so the end customers did not receive an answer. Before Cekat sends a paid WhatsApp message, it sets aside the message's cost from the business's WABA balance. The problem happened when a business had **exactly** enough balance left for one more message. A small calculation error made the system think the balance was slightly short. It then tried to charge the "missing" amount, which was zero, to the business's postpaid limit, and that request was rejected. The message was not sent. The issue stopped on its own for each business once its balance changed, and the last failure was at 09:46 WIB. The same issue also affected one business earlier, on 24 October. 1. Customer impact * On 31 October, **282 WhatsApp messages** from **3 businesses** were not sent. Almost all of them were AI agent replies. * On 24 October, between 10:17 and 21:59 WIB, **119 messages** from **1 business** were not sent for the same reason. * The affected messages were **not sent later**. They were not delayed or queued; they failed. * Only WhatsApp businesses billed in Indonesian rupiah \(IDR\) were affected, and only while their WABA balance was exactly the price of one message. * Some businesses without a postpaid limit may have seen an "insufficient WABA balance" message in the same situation, even though their balance was exactly enough. * **No incorrect charges.** Failed messages did not reserve or deduct any balance. 1. Why it happened **A tiny calculation error made an exact balance look short** Balances are stored with six decimal places. To work out how much balance a business has free, the system subtracts the amount already set aside from the total balance. The way computers handle decimal numbers can leave a very small error. For example, a calculation that should give 763.521791 can instead give 763.52179099999995. When the free balance was exactly the price of the message, this tiny error made it look slightly smaller than the price. The system then treated the balance as short by an amount so small it rounded to zero. **The zero amount was rejected** When a balance does not fully cover a message, the remainder is charged to the postpaid limit. Our system requires both parts of that split to be more than zero. Because the remainder was zero, the request was rejected and the message was not sent. **Why it kept failing for the same business** A failed message does not change the balance, so every following message at the same price met the same situation and failed the same way, until the balance changed \(for example after a top-up\). **Why only some businesses were affected** This only happens when the free balance equals the message price exactly, which usually happens on the last message a balance can pay for. Even then, it depends on whether the tiny error goes slightly below or slightly above the exact amount. Businesses billed in USD were not affected, because that calculation was already rounded correctly. 1. Timeline * **24 October, 10:17 – 21:59** One business is affected; 119 messages are not sent. * **31 October, 05:05** Messages from the first business start failing. * **05:19 – 06:06** A second business is affected. * **05:56 – 08:50** A third business is affected. * **09:46** Last failed message. * **10:18** Incident posted on the status page; investigation starts. * **11:48** Cause identified. * **15:14** Incident marked resolved. A permanent fix is deployed. 1. How service recovered For each affected business, messages started sending again once its balance changed, for example after a top-up or after earlier reserved amounts were released. No new failures have been seen since 09:46 WIB. The permanent fix rounds the free balance and the free postpaid limit to the same six decimal places used to store them, so an exact balance is always treated as enough. It has been tested with the exact numbers from this incident. 1. Follow-up actions P0 = most urgent · P1 = high priority **P0 — Most urgent** * **Release the permanent fix.** Until it is released, the issue can happen again to a business whose balance reaches exactly one message price. \[Done\] * **Alert on failed balance reservations.** Notify the team immediately when messages fail because their cost could not be reserved, instead of finding out from customer reports. \[On going\] **P1 — High priority** * **Alert when AI replies are not delivered.** Watch for a rise in AI replies that fail to send, whatever the cause. * **Use exact decimal calculations for money.** Move balance calculations to a method that keeps exact decimal values, so this type of rounding error cannot happen. \[Done\] 1. Main lesson Money must be calculated exactly. A rounding error far too small to see was enough to block messages, and because nothing alerted on these failures, it went unnoticed on 24 October and was only caught on 31 October. We have fixed the calculation and adding alerts so failures like this are caught right away. --- [2026-10-31T18:31:04.777+07:00] (resolved) This incident has been resolved. Thank you for your understanding and sorry for the inconvenience. --- [2026-10-31T11:49:37.987+07:00] (identified) We have identified the issue and the fixing is ongoing. Only some message impacted --- [2026-10-31T11:49:01.210+07:00] (investigating) We found out AI is not replying for some messages. Impacting 3 business. Issue start at 05:00 GMT+7. We're investigating and will update the RCA

# Delays in Cekat chat features **Incident date:** 30 October 2026 **Started:** 14:59 WIB **Service recovered:** 18:31 WIB **Time until service recovery:** About 3 hours and 32 minutes **Latest status:** Recovered. All times in this report are WIB \(UTC\+7\). ## Overview On 30 October 2026, some Cekat features became slower or stopped progressing as expected. This affected AI replies, follow-up messages, broadcasts, and automatic conversation resolution. The problem was in **Temporal**, the system Cekat uses to schedule and manage background work. The part of Temporal that tracks each task's progress could not reliably read the information it needed from the database. As a result, it could not reliably pass work to the workers that carry out those tasks. The team increased Temporal's capacity, and service recovered by **18:31 WIB**. Most waiting tasks had cleared by then. About **410 delayed-message tasks**remained and were being processed quickly. Some tasks had stopped retrying after repeated failures and needed separate attention. The investigation did not confirm when all remaining work was completed or cancelled. ## Customer impact Customers experienced delays in: * AI replies * Follow-up messages * Broadcast messages * Automatic resolution of conversations handled by humans * Automatic resolution of conversations handled by AI There were also two risks beyond delays: * **Some tasks stopped completely.** After too many failures, the system set them aside. They would not resume until the team recovered them. * **Some follow-ups could still be sent after they were no longer needed.** When processing was moved back to the previous system, cancellation requests no longer reached follow-ups already scheduled in Temporal. Those follow-ups could remain scheduled even after a customer replied. No permanent data loss was confirmed. The investigation did not confirm whether any outdated messages were sent or what happened to every stopped task. ## Why it happened ### Temporal could not access the database fast enough Before processing a task, Temporal needs to read its saved progress. Each of the two processing instances responsible for this had only four connections available to read from the database at the same time. That limit had been reduced earlier to fit a smaller database setup. Although the database was later upgraded, Temporal's access limit stayed the same. As more work arrived, tasks waited too long for access and failed. They then tried again, adding even more work to an already overloaded system. **More work arrived → tasks waited too long → tasks failed and retried → the waiting work grew.** ### Increased traffic and a broadcast burst pushed it over capacity The team was gradually moving delayed messages and follow-ups from the previous background-processing system to Temporal. The amount of database work was already about **2.5 times the previous day's level**. Around 15:00, Temporal was handling 80% of the work being moved over. A broadcast campaign added another burst of tasks. This appears to have pushed the system beyond what it could handle. The database itself was initially responding quickly. The main problem was Temporal's limited capacity to access it. ## Timeline | Time | What happened | | --- | --- | | Around 14:00–14:10 | The database was upgraded to support more work. Temporal's database-access limit stayed unchanged. | | **14:59** | Temporal began repeatedly failing to read task information. | | 15:00–15:40 | Failed tasks kept retrying, and the problem grew. | | 15:34–15:40 | The team added computing power and restarted the processing instances. This did not fix the problem; errors increased around the restart. | | **16:10** | The team began moving new delayed-message and follow-up work back to the previous system. This was fully in effect by 16:30, but existing Temporal tasks kept running and retrying. | | Around 16:50 | Some tasks stopped retrying and were set aside for later recovery. | | 17:30–17:40 | About 4,400 delayed-message tasks were waiting. Most attempts to read task information were still failing. | | **Around 17:49** | The team increased the number of Temporal instances responsible for tracking task progress from two to four. | | 18:20 | Three of the four instances were working normally, and waiting work had started to clear. | | **18:31** | All four instances were working normally. Other monitored queues were empty, and about 410 delayed-message tasks remained and were clearing quickly. | ## How service recovered **Adding more processing instances helped.** Increasing from two to four gave Temporal more capacity to read task information and spread the work. Recovery took time because the new instances had to catch up on overdue tasks. **Moving new work back to the previous system helped reduce incoming load, but was not enough on its own.**Tasks already in Temporal kept retrying. Broadcasts also continued using Temporal, so moving delayed messages and follow-ups back did not stop all incoming work. **Adding computing power did not solve the problem.**The initial constraint was database access, not computing power. The accompanying restart may have made things worse temporarily because the system had to reload information it previously held in memory. ## Follow-up actions The incident also highlighted problems in how scheduled jobs decide whether an action is still appropriate. A job must check the conversation **when it runs**, rather than assume nothing has changed since it was scheduled. **P0 = most urgent · P1 = high priority** ### P0 — Most urgent * **Account for all remaining work.** Confirm that waiting tasks finish and that stopped tasks are either recovered or cancelled. * **Keep the increased capacity in future updates.** Save the four-instance setting so a later system update does not accidentally reduce capacity back to two. * **Check before sending an AI reply.** The job currently does not check whether the conversation is already resolved. Add that check and skip the reply if it is resolved. * **Make broadcast failures easier to identify.** Include the campaign and batch identifiers in error records, so the team can immediately tell which group of broadcast messages is having a problem. * **Prevent AI auto-resolve from closing active conversations.** The current job can mark a conversation as resolved while the customer and AI agent are still talking. Redesign it to check the conversation's current status and recent messages before deciding whether to resolve it. ### P1 — High priority * **Review database-access capacity.** Make sure Temporal's limits match the upgraded database and allow room for busy periods and system updates. * **Add alerts for customer-facing delays and stopped tasks.** Alert the team when tasks wait too long, repeatedly fail, or stop processing. The delay alerts found during the incident covered test environments only. Owners and completion status for these actions have not been recorded. ## Main lesson We increased the amount of work sent to Temporal without enough capacity to handle busy periods. Once failed tasks started retrying, reducing new traffic was not enough to recover. We need to plan for both busy periods and delayed work. Just as importantly, every scheduled action must check whether it is still appropriate before sending a message or resolving a conversation. --- [2026-10-31T18:30:19.160+07:00] (resolved) Issue has been fully recovered. Thank you for your understanding and sorry for the inconvenience. --- [2026-10-30T17:49:34.306+07:00] (monitoring) We are continuing to monitor for any further issues. --- [2026-10-30T17:27:24.019+07:00] (monitoring) We are continuing to monitor for any further issues. --- [2026-10-30T17:51:04.078+07:00] (monitoring) A fix has been implemented and we are monitoring the results. --- [2026-10-30T17:50:41.874+07:00] (identified) We identified an issue that caused some chats, broadcasts, and AI Agents to delay or fail to send responses between approximately 3:00 PM and 4:15 PM WIB. We have applied a mitigation and are validating service recovery. We are also assessing affected conversations and whether missed responses can be safely restored. We will provide another update once validation is complete.

This incident has been resolved. --- [2026-10-09T17:12:15.026+07:00] (monitoring) We are continuing to monitor for any further issues. --- [2026-10-09T17:11:32.095+07:00] (monitoring) A fix has been implemented and we are monitoring the results. --- [2026-10-09T17:11:00.638+07:00] (identified) The issue has been identified and a fix is being implemented. --- [2026-10-09T17:10:17.652+07:00] (investigating) We are currently investigating this issue.