Designing an email notification system

I've wanted to create a series on system design for a while now, more as a personal library of reference for myself. I hope this will be useful for others as well.

In this post, I will go through how I would design an email notification system given some functional and non-functional requirements. Let's get started.

Objective

Design an email notification system where users can create, update and delete reminders. The user can create a one-time or a recurring reminder. The system should send email notifications within 5 minutes of the scheduled time. It should be able to handle 1 million reminders per day and scale to 10 million reminders per day in the future. The system should send emails to the users exactly once. There is a user profile service that we can use to get the user profile information, specifically the email address and the timezone of the user.

Functional requirements

• Users can create, update and delete reminders.
• Users can create one-time or recurring reminders.

Non-functional requirements

• Fault tolerant (retries, DLQ)
• Scalable (spikes, future growth)
• Maintainable (monitoring, alerting)
• Exactly once delivery of email notifications

Estimates

Intial: 1 million reminders / day = ~12 reminders / second or ~43,200 reminders / hour
10 million reminders / day = ~116 reminders / second or ~420,000 reminders/ hour

Initial load can be handled by a single server, but we need to design the system to scale to handle future growth at 10 million reminders per day.

Assumptions

• User profile service handles scaling reads (read cache, replicas, DB etc.).
• API gateway handles load balancing, authentication, rate limiting and routing.

Data model

1. Reminders table

   • id
   • user_id
   • content
   • reminder_date
   • reminder_time (nullable)
   • freq (nullable)
   • interval (nullable)
   • next_run_utc (nullable)
   • status (pending, claimed, sent, cancelled, failed)
   • claimed_by
   • claimed_at
   • client_request_id
     partial index on (next_run_utc) where status = 'pending'

2. Delivery (reminder level send log)

   • id
   • reminder_id
   • user_id
   • run_time_utc
   • status (sent, failed, bounced)
   • provider_message_id
   • sent_at
   • created_at
     index(reminder_id, run_time_utc)
     unique(reminder_id, run_time_utc)

3. Outbox

   • id (UUIDv7)
   • payload (jsonb)
   • published
   • created_at
     partial index(created_at) where published = false

High-level design

The high level design of the system would look like this:

I had GPT clean up the diagram to make it a bit cleaner:

1. The user creates a reminder. Request is routed via the API gateway to the reminder service.
2. The reminder service validates the request, fetches the user profile to get the relevant timezone and computes the next_run_utc based on the reminder_date, reminder_time and optionally frequency and interval if the reminder is recurring.
3. It stores the reminder in the reminders table with status = 'pending'.
4. A coordinator/scheduler polls the reminders table for pending reminders that are due to be sent (next_run_utc <= now()), claims them (status = 'claimed', claimed_at = now()) and publishes them to the outbox table with published = false in the same transaction.
5. Optionally, the coordinator can publish the reminders it has claimed to the message queue.
6. The outbox worker or sweeper polls the outbox table for unpublished reminders, pushes the messages to the message queue and marks them as published.
7. A set of workers consumes the messages from the message queue and sends the email notifications to the users via an email provider. The workers update the delivery table with the status of the email (sent), provider_message_id and sent_at. Additionally, it fetches the user profile from the service to get the latest email address and timezone of the user. It recomputes the next_run_utc for recurring reminders and writes this to the reminders table as well with status='pending' and inserts the status=sent of the reminder instance in the delivery log table.
8. If the email bounces, the webhook from the email provider will hit an endpoint on the reminder service to update the delivery log status to bounced and also update the reminders table in a single transaction.

Specific architectural decisions

• The database I've chosen here is PostgreSQL. For this workload, it provides transactional guarantees across multiple tables, provides a mechanism for locking rows for concurrent scheduler/coordinator access, can push down business invariants to the database layer (partial indexes, constraints). Databases like DynamoDB can provide transactions too but they have more overhead and require more application-level coordination.
• My message queue of choice (if on AWS) would be SQS. Other queues would work as well, but some functionality would need to be hand-rolled. SQS comes with a lot of things I need in my system out of the box like retries, dead letter queues, visibility timeouts etc. It is also a managed service so the operational overhead is low.

Addressing the non-functional requirements

Fault tolerance

The system needs to be fault tolerant on the application as well as the infrastructure level.

From the application side, this will primary be retries, dead-letter queues and idempotency.

• If there is an error while creating the reminder in the reminders service, the client will send a client_request_id along with the request. If this id already exists in the reminders table, a duplicate reminder will not be created.
• Fetching user profile data from the service will be retried with exponential backoff in case of failures. The response returned to the client will be mostly a product decision. Either we return an error to the client or we can create the reminder with a default timezone (UTC) and update it when the client is ready. But this adds complexity to the system.
• Coordinator/scheduler can have multiple instances. Since we're using Postgres, we can lock the rows, claim them and publish them to the outbox table in a single transaction.
• If the coordinator fails after claiming but before publishing to the queue, the outbox worker will pick these up and publish them.
• If a worker fails while processing the reminder/notification, it will perform some bounded linear backoff retries (keeping in mind the 5 minute SLA). During retries, to make sure we don't run out of the visiblity timeout (SQS will make the reminder available again), we will change/extend the visibility timeout on SQS for the message. This can be set to the retry time + buffer + jitter.
• For retries, we will only retry in specific recoverable scenarios. For ex. if we get a 500 or 429 from the email provider, we can retry in-process. In case of the 429, if we get a Retry-After header from the email provider, we use that value to set the backoff. If we get a 400/422, then there might be an issue with the reminder itself and retrying will only burn our service SLO. In this case we can update the delivery log and reminders table to failed and ack/delete the message so it isn't redelivered.
• If an error is encountered across multiple instances/messages, then there might be an issue with either the user profile service or the email provider. In this case, we can use a circuit breaker and set it to open for the provider or the service. In case of the provider, we can switch to a backup provider. After some specified time, we can set the circuit breaker to half-open and retry the original provider. If it's working, we check for some more time and then revert back to it while flipping the circuit breaker closed.
• In case we crash during sending the email but before recording the status in our delivery log table and deleting the message, the message will be made visibile again and redelivered to a worker. When the worker processes it, it sends the reminder_id + run_time_utc that uniquely identifies a particular instance of a reminder as the idempotency key to the email provider. This will allow the email provider to check if the email for this specific key has already been sent out. This ofcourse, has to supported by the email provider (ex. Resend).
• For additional reliability on the email provider webhook side, we can also add a small service that pushes webhook payloads into a queue so it can be consumed to update the reminders and delivery log tables.This adds additional complexity but ensures we don't loose any webhook to update the state of a delivery instance and provides a buffer so the reminder service can update at its own pace.

From the infrastructure side,

• All service and consumer instances will be running via service orchestrators like Kubernetes or ECS that will provision new containers/instances on failure/crash.
• SQS will have a visibility timeout = 2 * p99 processing time, DLQ policy after 3 attempts (maxReceiveCount) and 7-14 days of retention on the dead-lettered messages.
• Postgres will need to be deployed to multiple availability zones for reliablity. On AWS, this would be multi-az with 1 standby replica. Depending on the criticality of this service, we can also have a intra-region synchronous replica or a cross-region async read replica as well.
• A connection pooler and load balancer could also be added to our database layer. Reads can be load balanced across multiple read replicas while writes go to the primary. Connection pooler makes sure connections can be reused across transactions.

Scalability

• Reminder service can be scaled depending on the CPU/memory utilisation of the underlying runner (container, VM, machine).
• Coordinator service can be scaled based on the number of due reminders. This can be done using an autoscaler like KEDA that can read off the reminders table, check the number of reminders that are due as the percent delta of the growth compared to the previous tick. We can define step sizes or scale factors based on this delta to make sure we can scale coordinator instances out.
• If the system scales to 10M reminders / day, we can scale the machine, container or VM hosting Postgres vertically. At the scale of 10M reminders/day, we do not need to shard the database across multiple instances to be able to meet the demand.
• If we need to scale the Postgres writes, we can additional read replicas and load balance read requests across them. Writes however would still need to be routed to the primary.
• Workers can also be scaled horizontally based on the queue depth for AWS SQS. We can reuse KEDA here as well to scale the number of workers based on the rate of growth of the number of reminders that are waiting to be sent.

Maintenance

We need to monitor each part of the system so we can identify delays and failures before they affect the 5 minute SLA.

From the monitoring side, we can track the following metrics:

• Reminder service: API request rate, error rate and p99 latency.
• Coordinator/scheduler: tick duration, rows claimed per tick and next_run_lag. This is now() - min(next_run_utc) across pending reminders that are due. If there are no due reminders, the lag is 0.
• Outbox: number of unpublished rows, age of the oldest unpublished row in seconds and number of messages published by the sweeper.
• Queue: queue depth, consumer lag, redelivery rate and oldest message age. For SQS, oldest message age tells us how far behind the workers are.
• Workers: processing duration, success/failure rate and email provider call latency.
• Database: write latency, read and write IOPS, storage usage, database connections, dead to live tuples ratio and replica lag.

We also need to track delivery_lag = delivery_log.sent_at - delivery_log.run_time_utc. Here, run_time_utc stores the reminder's next_run_utc at claim time. This measures the delay against when that specific reminder instance was due.

For recurring reminders, we cannot use the current next_run_utc since the worker updates it for the next occurrence. Also, sent_at measures when the provider accepts the email, not when it reaches the user's inbox.

For alerting, we can start with:

• next_run_lag > 2 minutes: the coordinator is falling behind on claiming due reminders.
• DLQ message count > 0: a message exhausted its retries and needs investigation.
• delivery_lag > 2-3 minutes: emails are taking too long to send. This gives us some time to respond before the 5 minute SLA is breached.

From the database maintenance side,

• We can partition the delivery log by run_time_utc into weekly partitions. Once the retention period expires, we can detach and drop older partitions instead of deleting rows individually.
• Delivery log retention needs to cover the retry and DLQ replay window so we still have the records for those reminder instances.
• Optionally, we can partition the outbox by created_at into hourly partitions if the volume justifies it. We must retain any partition with unpublished rows so we do not lose reminders waiting to enter the queue.
• We need to create partitions ahead of time. Postgres also requires unique constraints on partitioned tables to include the partition key, so we need to account for this in the outbox keys.
• For tables with frequent updates like the outbox and delivery log, we can lower autovacuum_vacuum_scale_factor. This makes vacuum run sooner to reclaim space from dead tuples.
• For insert-heavy delivery log partitions, we can also tune autovacuum_vacuum_insert_scale_factor and autovacuum_analyze_scale_factor based on the workload.
• If indexes become bloated, we can use REINDEX CONCURRENTLY to rebuild them while normal reads and writes continue. This still adds CPU and I/O load, so we need to plan for that.
• We need regular backups with point-in-time recovery and periodic restore tests. Runbooks need to cover database failover, provider failures and safe DLQ replay without duplicate sends.

Data flow

The queries we would need across the entire flow in the system:

• Client --> service (for retries from the client side)

1-- simple insert +
2ON CONFLICT (user_id, client_request_id) DO NOTHING;

• Coordinator claims pending reminders + writes to the outbox

1WITH due AS (
2    SELECT id
3    FROM reminders
4    WHERE status = 'pending'
5      AND next_run_utc <= NOW()
6    ORDER BY next_run_utc, id
7    LIMIT 100
8    FOR UPDATE SKIP LOCKED
9),
10claimed AS (
11    UPDATE reminders AS r
12    SET status = 'claimed',
13        claimed_by = 'coordinator-1',
14        claimed_at = NOW()
15    FROM due
16    WHERE r.id = due.id
17    RETURNING r.id, r.user_id, r.content, r.next_run_utc
18)
19INSERT INTO outbox (id, payload, published, created_at)
20SELECT
21    uuidv7(),
22    jsonb_build_object(
23        'reminder_id', id,
24        'user_id', user_id,
25        'content', content,
26        'run_time_utc', next_run_utc
27    ),
28    FALSE,
29    NOW()
30FROM claimed;

• Coordinator publishes these claimed rows to the message queue in-process.

• Outbox worker/sweeper (every 60 seconds to check for unpublished messages)

1SELECT id, payload:: text AS message_body
2FROM outbox
3WHERE published = false
4ORDER BY created_at, id
5LIMIT 100;

Publish to queue + set published = true

• Optional claim sweeper

1UPDATE reminders
2SET status = 'failed'
3WHERE status = 'claimed' AND claimed_at < NOW() - INTERVAL '15 minutes';

• insert into delivery log in the same transaction

• Worker (fetch --> send --> commit --> ack)

Check if re-delivered i.e. if the worker crashed after processing but before deleting/acking the message from the queue, the message will be redelivered. Here, we check if the delivery log for that message exists.

1SELECT 1
2FROM delivery_log
3WHERE reminder_id = '123' AND run_time_utc = '';

Check cancellation before sending (in case the user deleted the reminder between queueing the reminder and sending it).

1SELECT status
2FROM reminders
3WHERE id = '123';

If status = 'cancelled', don't send.

Record the delivery and update reminders in the same transaction (insert into delivery log + update reminders with status='sent' if one-time reminder and status='pending' + computed next_run_utc if recurring). Delete/ack the message from the queue.

What about extreme scale

What would happen if the system were to scale even further than the estimated 10-100M reminders/day. What about 1B or 10B reminders/day? That would be ~11,600 to ~116,000 reminders / second. Most of the stateless components like the reminder service, coordinator/scheduler, outbox worker, DLQ consumer and queue processor/workers can scale horizontally. At that scale, the bottlenecks would come down to the database and the message queue layers.

For the database layer, each reminder occurrence or instance involves claiming the reminder, inserting the outbox event, marking the outbox as published, inserting the delivery record, updating the reminder's state and next occurrence. At 10B/day it comes to ~600,000 row mutations/second on average. And this is before new reminder creations, retries and other updates. The default way to tackle this intially is to try vertical scaling. Scale the underlying machine with more CPU, memory and faster storage. Then come batched transactions, better connection pooling, read replicas etc. But writes only go to the primary so read replicas will not help much here.

The next step would be to shard the tables across multiple database instances. Most of the transactions are around a single reminder and its occurrences. If we shard the tables across multiple instances, almost all (if not all) of the transactions would be local to that shard so we don't need to worry about distributed transactions in this case. Each shard would then be for ex. Shard A: reminders + outbox + delivery log. We can hash the reminder_id to distribute reminders evenly across the shards.
Every thing else stays the same, scheduler still claim and write to the outbox from their assigned shards. Outbox workers check their assigned shards to publish messages to the queue.

This ofcourse comes with complexity. The routing and shard movement is now something that needs to be managed at the application level or via some distributed Postgres solution.

For the queue, if we use SQS, AWS has a limit of 120,000 in-flight messages per standard queue. Number of in-flight messages is roughly equal to the consumption rate * average processing time (till deletion). At 10B/day, with an average of 10 seconds from receipt to deletion/ack, this would be 116,000 messages/second * 10 seconds = roughly 1.16 million in-flight messages. Although, we can request AWS to increase the quotas, we still need to design around it.

One possible solution is to use a cell-based architecture where each cell has its own Postgres shard, SQS queue and worker pool.
for ex. Cell A: Postgres shard --> SQS queue(s) --> worker pool

Conclusion

The design I've presented here is one way to build an email notification system. The architecture and components we choose will depend on the requirements, expected scale and our own experience with similar systems. We can also build this around DynamoDB, cache idempotency lookups in Redis or use Lambda functions to consume from the DLQ and update the database. Each choice comes with its own trade-offs in reliability, cost and operational complexity.

The goal at the end is to meet the requirements with a design we can operate, scale and most importantly, maintain, as the workload grows.

I'll continue to post more system design solutions here.