Shashank H RBackend Engineer
all systems operational--:--:-- IST
← Back to workCase study 02 / 04

Engati · Event-driven data

RCS billing pipeline

Every RCS message event ends up on a customer’s bill. This pipeline counts each one once, even when a job runs again.

Role
Designed & built it
Stack
Apache Kafka · AWS S3 · Apache Spark
Scale
~8M billing events a day
Status
● in production
~4Mconversations a day
~8Mbilling events a day, all in working hours
~500 RPSat peak; about 250 RPS on average
3 keysbotRef · customerId · metric type

(01) The problem

RCS usage is billed from webhook events, so the numbers have to be exact. About 8 million of them arrive across the working day, and a billing job that fails halfway, or runs twice, must never change what a customer pays.

I designed and built the pipeline from scratch; rollout was with the DevOps team.

(02) How it works

rcs webhooksevery message eventkafkaevent streamaws s3raw eventsspark jobsidempotent · replay-safebillingper bot & customerreplay any period ↺12345
  1. Every RCS message event arrives as a webhook and is published to Apache Kafka.
  2. Events are persisted to AWS S3, so the raw data is always kept.
  3. Spark jobs aggregate them by botRef, customerId and metric type.
  4. The totals feed customer billing.
  5. Each run re-reads a window that overlaps the previous day and replaces that range’s totals, so a re-run gives the same result.

(03) Key decisions

Kafka in front

Publishing webhooks to Kafka separates receiving events from processing them, so a burst of traffic doesn’t hold billing up.

Keep the raw events

Everything lands in S3 first, so any period can be recomputed from the source if something goes wrong.

One billing ID per message

Every event for a message (the first send, its updates, repeated packets, a refund) folds into one billing ID. Refunded or superseded messages drop out, and each ID is counted once.

(04) The hardest tradeoff

A daily batch, not real time

With a tight deadline, I chose an async ingestion pipeline and a daily Spark job over real-time counting. Usage appears a day late, but the job is simple to run and to recompute. With more time, I would have built real-time ingestion and reporting.

(05) Results

  • Takes in ~8M billing events a day from ~4M conversations, all inside working hours: about 250 requests a second on average and ~500 at peak.
  • Accurate customer billing for RCS, counted per bot, customer, day and billing unit.
  • Safe to re-run: the day’s totals are deleted, then rewritten with an upsert, so a repeated run gives the same numbers.
  • The delete and rewrite aren’t one transaction, so a job that fails midway is simply run again. Each run records its status, and a failure raises a Slack alert.