Shashank H RBackend Engineer
all systems operational--:--:-- IST
← Back to workCase study 01 / 04

Engati · Resilience

Auto-retry framework

Meta can’t deliver every message the first time. This framework gives failed deliveries another chance, and 45% of them now get through on the first retry.

Role
Designed & built it
Stack
Java · Spring Boot · RabbitMQ · MongoDB · Redis
Scale
~2M triggers a day
Status
● in production
45%of failed deliveries recovered on the first retry
~2MAPI triggers a day
~100Kfailed deliveries a day before retries
~45Kof them delivered a day on the first retry

(01) The problem

Marketing and CRM platforms like LeadSquared and MoEngage trigger messages through Engati’s API: about 2 million a day. When Meta failed to deliver one, it stayed failed. Around 100K deliveries a day, roughly 5%, were simply lost.

I designed and built the framework from scratch; rollout was with the DevOps team.

(02) How it works

integrationsLeadSquared · MoEngageapi-gatewayentry pointtrigger-mvcaction trigger mgmtmessagingmessaging pipelinemeta→ the userredistrackerId per messagewebhook-receiverMeta webhooksanalyticsrecords the reasonrabbitmqretry + back-offmongodbpayload by trackerIdAPI triggertrackerIdwebhookfailure reasonback-offpayload1234567
  1. LeadSquared or MoEngage sends an API trigger.
  2. It enters through the API gateway and reaches trigger-mvc, the action trigger management service.
  3. trigger-mvc hands it to the messaging pipeline, which keeps its trackerId in Redis and sends it through Meta to the user.
  4. Meta sends a webhook back to the webhook receiver: delivered, or failed with a status code.
  5. The analytics pipeline records why it failed and passes the failure on to trigger-mvc.
  6. trigger-mvc checks the status code. Retryable failures wait in RabbitMQ, with fixed-interval or exponential back-off.
  7. When the wait is over, trigger-mvc reads the trackerId from Redis, fetches the original payload from MongoDB and sends it out through messaging again.

(03) Key decisions

trackerId ties a retry to its original

Each retry carries the original message’s trackerId, so trigger-mvc fetches that exact payload from MongoDB and sends the same message again instead of building a new one.

Retry only on failure codes

Only a few Meta failure codes schedule a retry, and Meta sends those only for messages it didn’t deliver. Permanent failures are never retried, so nothing hammers a message that can’t go through.

Back off, don’t pile on

Fixed-interval and exponential back-off spread retries out, so a burst of failure webhooks from Meta doesn’t turn into a retry storm.

(04) The hardest tradeoff

Simple over exactly-once

A message is retried only when Meta reports one of a few failure codes, and Meta doesn’t send a failure for a message it delivered. So I didn’t add a separate “already delivered” check before re-sending: that kept the retry path simple. The cost is that nothing on our side would stop a duplicate if a failure and a delivery ever crossed. I accepted that edge case rather than put a lookup or a lock in front of every retry.

(05) Results

  • ~45K failed deliveries a day now reach users on the first retry: about 45% of the ~100K that used to stay failed.
  • Across all ~2M daily triggers, the failure rate falls from ~5% to ~2.75% after one retry, and lower after the later ones.
  • Absorbed peaks of 300K–400K retries a day during Meta delivery failures.
Try the interactive simulation of this flow in the playground →