Engati · Resilience
Auto-retry framework
Meta can’t deliver every message the first time. This framework gives failed deliveries another chance, and 45% of them now get through on the first retry.
(01) The problem
Marketing and CRM platforms like LeadSquared and MoEngage trigger messages through Engati’s API: about 2 million a day. When Meta failed to deliver one, it stayed failed. Around 100K deliveries a day, roughly 5%, were simply lost.
I designed and built the framework from scratch; rollout was with the DevOps team.
(02) How it works
- LeadSquared or MoEngage sends an API trigger.
- It enters through the API gateway and reaches trigger-mvc, the action trigger management service.
- trigger-mvc hands it to the messaging pipeline, which keeps its trackerId in Redis and sends it through Meta to the user.
- Meta sends a webhook back to the webhook receiver: delivered, or failed with a status code.
- The analytics pipeline records why it failed and passes the failure on to trigger-mvc.
- trigger-mvc checks the status code. Retryable failures wait in RabbitMQ, with fixed-interval or exponential back-off.
- When the wait is over, trigger-mvc reads the trackerId from Redis, fetches the original payload from MongoDB and sends it out through messaging again.
(03) Key decisions
trackerId ties a retry to its original
Each retry carries the original message’s trackerId, so trigger-mvc fetches that exact payload from MongoDB and sends the same message again instead of building a new one.
Retry only on failure codes
Only a few Meta failure codes schedule a retry, and Meta sends those only for messages it didn’t deliver. Permanent failures are never retried, so nothing hammers a message that can’t go through.
Back off, don’t pile on
Fixed-interval and exponential back-off spread retries out, so a burst of failure webhooks from Meta doesn’t turn into a retry storm.
(04) The hardest tradeoff
Simple over exactly-once
A message is retried only when Meta reports one of a few failure codes, and Meta doesn’t send a failure for a message it delivered. So I didn’t add a separate “already delivered” check before re-sending: that kept the retry path simple. The cost is that nothing on our side would stop a duplicate if a failure and a delivery ever crossed. I accepted that edge case rather than put a lookup or a lock in front of every retry.
(05) Results
- ~45K failed deliveries a day now reach users on the first retry: about 45% of the ~100K that used to stay failed.
- Across all ~2M daily triggers, the failure rate falls from ~5% to ~2.75% after one retry, and lower after the later ones.
- Absorbed peaks of 300K–400K retries a day during Meta delivery failures.