Payment Gateway System Design

A Payment Gateway looks simple from the user's point of view.

You select a product, click Pay, enter your payment details, and within a few seconds you see:

Payment Successful

But behind that simple screen, a distributed system is coordinating multiple services, databases, banks, card networks, payment processors, and external APIs.

And payments are different from many other software systems.

If a chat message is delivered twice, it is annoying.

If a payment is charged twice, it is a serious financial problem.

A Payment Gateway must therefore focus heavily on:

• Payment correctness

• Security

• Idempotency

• Reliability

• Transaction tracking

• Duplicate payment prevention

• Retries

• Webhooks

• Reconciliation

• Refunds

• Auditability

• High availability

In this guide, we will design a Payment Gateway System step by step and understand the important architecture decisions and trade-offs commonly discussed in system design interviews.

What Is a Payment Gateway?

A Payment Gateway is a system that helps a business accept and process electronic payments.

Suppose Alice purchases a laptop from an online store.

The basic flow looks like:

Alice
  |
  v
E-commerce Website
  |
  v
Payment Gateway
  |
  v
Payment Processor / Banking Network
  |
  v
Bank

The Payment Gateway coordinates the payment and tells the merchant whether it succeeded, failed, or is still being processed.

Depending on the payment method and region, the actual payment ecosystem can involve several additional participants.

Payment Gateway vs Payment Processor

These terms are sometimes used interchangeably, but conceptually they perform different jobs.

Step 1: Define the Functional Requirements

The Payment Gateway should support the core operations required to process and manage payments safely.

• Create and process payments

• Check payment status

• Prevent duplicate payments

• Send payment-status updates

• Handle failed and pending payments

• Support refunds and partial refunds

• Track payment attempts and transactions

• Support multiple payment methods

The core requirement is to process payments correctly while maintaining a clear transaction history.

Step 2: Define Non-Functional Requirements

A Payment Gateway must prioritize financial correctness, reliability, and security.

• Correctness: Prevent duplicate or incorrect financial operations.

• Reliability: Handle failures, retries, and uncertain payment outcomes safely.

• Security: Protect payment data and access to financial operations.

• Durability: Never lose confirmed payment records.

• High Availability: Keep payment services available during failures.

• Auditability: Maintain a clear history of financial operations.

• Scalability: Handle increasing transaction volume and traffic spikes.

For payment systems, financial correctness is more important than raw speed.

Step 3: Estimate the Scale

Suppose our Payment Gateway handles:

50 million payments per day

Average transactions per second:

50,000,000 / 86,400
≈ 579 payments/second

Peak traffic may be many times higher.

For example, flash sales or major shopping events can generate sudden spikes.

Payment traffic is therefore not just about average TPS.

We should design for peak load.

Step 4: High-Level Payment Gateway Architecture

A simplified architecture could look like:

                    +-------------------+
                    | Customer / Client |
                    +---------+---------+
                              |
                              v
                    +-------------------+
                    | Merchant Backend  |
                    +---------+---------+
                              |
                              v
                    +-------------------+
                    | API Gateway       |
                    +---------+---------+
                              |
                              v
                    +-------------------+
                    | Payment Service   |
                    +----+---------+----+
                         |         |
                         |         +----------------+
                         |                          |
                         v                          v
                 +---------------+          +-------------+
                 | Payment DB    |          | Risk/Fraud  |
                 +---------------+          +------+------+
                                                  |
                                                  v
                                         +-------------------+
                                         | Payment Router    |
                                         +---------+---------+
                                                   |
                                      +------------+------------+
                                      |                         |
                                      v                         v
                               +--------------+          +--------------+
                               | Provider A   |          | Provider B   |
                               +------+-------+          +------+-------+
                                      |                         |
                                      v                         v
                               Banking / Card / Payment Networks

Supporting Components:

+-------------------+
| Ledger Service    |
+-------------------+

+-------------------+
| Message Queue     |
+-------------------+

+-------------------+
| Webhook Service   |
+-------------------+

+-------------------+
| Refund Service    |
+-------------------+

+-------------------+
| Reconciliation    |
+-------------------+

The Payment Service manages the core payment lifecycle, while the Payment Database stores durable state, the Risk Service evaluates fraud signals, and the Payment Router selects the appropriate provider.

The supporting components handle accounting, asynchronous events, merchant notifications, refunds, and reconciliation.

Step 5: Payment Service

The Payment Service manages the payment lifecycle.

Its responsibilities may include:

Create payment
Validate amount and currency
Check idempotency
Maintain payment state
Call payment routing logic
Store provider references
Expose payment status

A payment record may look like:

payment_id
merchant_id
order_id
amount
currency
status
payment_method
provider
provider_payment_id
created_at
updated_at

Step 6: Payment States

Payments should have explicit states to track their lifecycle correctly.

CREATED
   |
   v
PROCESSING
   |
   +--------> SUCCESS
   |
   +--------> FAILED
   |
   +--------> PENDING

Depending on the payment method, additional states may exist. The important point is to allow only valid state transitions.

Why Do We Need a PENDING State?

Suppose the gateway sends a payment request to a provider. The payment succeeds, but the response is lost because of a network timeout.

We cannot mark it FAILED because the customer may already have been charged. We also cannot mark it SUCCESS without confirmation.

The correct state is:

PENDING / UNKNOWN

The system can later determine the final status using the provider's status API, webhook, or reconciliation process.

A timeout does not necessarily mean payment failure.

Step 7: Payment API Design

A simplified API may look like:

POST /payments

Request:

{
  "order_id": "order_5001",
  "amount": 200000,
  "currency": "INR"
}

Response:

{
  "payment_id": "pay_123",
  "status": "CREATED"
}

Another API can retrieve the payment:

GET /payments/pay_123

And another may initiate a refund:

POST /payments/pay_123/refunds

Step 8: Never Trust the Client for the Payment Amount

Suppose the browser sends:

{
  "product": "Laptop",
  "amount": 10
}

Should the Payment Gateway trust it?

No.

A malicious user could modify browser requests.

The merchant backend should determine the authoritative order amount.

A safer flow is:

Client
   |
   v
Merchant Backend
   |
   v
Order Database
   |
Determine Actual Amount
   |
   v
Payment Gateway

Sensitive business decisions should not depend entirely on client-provided values.

Step 9: Idempotency

Idempotency is one of the most important concepts in payment system design.

Suppose Alice clicks:

Pay ₹2,000

The merchant sends the payment request.

The gateway processes it successfully.

But the response is lost because of a network problem.

The merchant retries.

Without protection:

Request 1 → Charge ₹2,000
Request 2 → Charge ₹2,000

Alice may be charged ₹4,000.

That is unacceptable.

Idempotency Key

The merchant sends a unique key:

Idempotency-Key: order-5001-payment

The Payment Gateway stores:

order-5001-payment
        |
        v
payment_123

If the same request arrives again:

Same Idempotency Key
        |
        v
Existing Payment
        |
        v
Return Existing Result

No second payment is created.

Idempotency Database Design

We may maintain something like:

IdempotencyRecord
------------------------
merchant_id
idempotency_key
request_hash
payment_id
response
created_at
expires_at

The unique constraint might be:

(merchant_id, idempotency_key)

This prevents two concurrent requests from successfully creating two different payments with the same key.

Why Store the Request Hash?

Suppose a merchant sends:

Key: abc123
Amount: ₹500

and later accidentally sends:

Key: abc123
Amount: ₹5,000

These are not logically the same request.

The system should detect that the same idempotency key is being reused with different request parameters.

A request fingerprint or hash can help identify this misuse.

Step 10: Handling Concurrent Duplicate Requests

Checking the idempotency key in application code is not enough.

Imagine two identical requests arrive at exactly the same time.

Both servers do:

Does key exist?

No.

Then both create a payment.

This is a race condition.

We need an atomic guarantee.

For example, a database unique constraint can ensure:

Only one row for
(merchant_id, idempotency_key)

One request wins.

The other reads the existing result.

Step 11: Payment Router

A Payment Gateway may connect to multiple providers.

For example:

Provider A
Provider B
Provider C

The Payment Router decides where a transaction should go.

The decision may consider:

Payment method
Country
Currency
Provider availability
Success rate
Cost
Merchant configuration
Transaction size

For example:

UPI
 |
 v
Provider A

Card
 |
 v
Provider B

This keeps provider-specific logic away from the core Payment Service.

Step 12: Provider Adapter

Different payment providers expose different APIs.

Instead of filling the Payment Service with provider-specific code, create adapters.

                 Payment Router
                       |
          +------------+------------+
          |            |            |
          v            v            v
      Adapter A    Adapter B    Adapter C
          |            |            |
          v            v            v
      Provider A   Provider B   Provider C

Each adapter converts our internal payment model into the provider's format.

This gives us a clean abstraction.

Step 13: Payment Processing Flow

Suppose Alice pays ₹2,000.

The flow may be:

Merchant
   |
   v
Payment Gateway
   |
   v
Validate Request
   |
   v
Check Idempotency
   |
   v
Create Payment
   |
   v
Risk Check
   |
   v
Payment Router
   |
   v
Provider
   |
   v
Financial Network

If the provider responds successfully:

Payment DB
   |
status = SUCCESS

The merchant can then be informed.

Step 14: Synchronous vs Asynchronous Payment Processing

Payment processing can be synchronous or asynchronous depending on whether the final result is immediately available.

A Payment Gateway should not assume every payment will reach a final state within a single HTTP request. Webhooks or status checks are used to resolve asynchronous payments.

Step 15: Webhooks

A webhook allows one system to asynchronously notify another system about an event.

Suppose the Payment Gateway learns:

payment_123 → SUCCESS

The merchant needs to know.

The gateway can send:

POST merchant.com/webhooks/payment

with an event such as:

{
  "event_id": "evt_9001",
  "type": "payment.success",
  "payment_id": "pay_123",
  "order_id": "order_5001"
}

The merchant updates its order:

PAID

Step 16: Why Webhooks Need Retries

What if the merchant's server is temporarily unavailable?

Gateway
   |
   v
Merchant
   |
  500

We should not permanently lose the notification.

Instead:

Attempt 1 → Failed

Wait

Attempt 2 → Failed

Wait longer

Attempt 3 → Success

Exponential backoff can be used.

For example:

1 minute
5 minutes
30 minutes
2 hours

The exact retry policy depends on product requirements.

Step 17: Webhook Queue

Webhook delivery should normally not block payment processing.

Instead:

Payment Service
      |
      v
Payment Event
      |
      v
Message Queue
      |
      v
Webhook Worker
      |
      v
Merchant

Now even if the merchant's server is slow, our core payment path remains available.

Step 18: Duplicate Webhooks

Webhook delivery usually follows an at-least-once model.

That means the same event may be delivered multiple times.

For example:

payment.success
payment.success
payment.success

The merchant must handle duplicate events safely.

Each webhook should therefore contain a unique event ID.

event_id = evt_9001

The merchant can remember processed event IDs.

Step 19: Webhook Security

A merchant should not trust any random HTTP request claiming:

Payment Successful

Webhook requests should be authenticated.

One common approach is signing the webhook payload with a secret.

The merchant verifies the signature before processing the event.

We may also include timestamps to reduce replay risk.

Step 20: Payment Ledger

A payment-status table tells us:

payment_123 = SUCCESS

But financial systems often need a more reliable accounting model.

That is where a ledger becomes important.

A ledger records financial movements as entries rather than repeatedly overwriting a balance.

Why Not Just Store a Balance?

Suppose a merchant balance is:

₹10,000

Then we directly update it:

₹10,000 → ₹12,000

Later somebody asks:

Why did the balance increase by ₹2,000?

If we only stored the latest value, answering may be difficult.

A ledger stores the movements.

Opening Balance      ₹10,000
Payment Received     +₹2,000
Refund               -₹500
Fee                  -₹50

Now the final balance can be derived and audited.

Step 21: Double-Entry Ledger

Financial systems commonly use double-entry accounting.

Every movement has corresponding debit and credit entries.

Conceptually, if a customer payment increases the amount owed to a merchant:

Debit  Payment Clearing Account
Credit Merchant Payable Account

The exact accounts depend on the business model.

The important property is that entries balance.

Total Debits = Total Credits

This gives us a strong accounting invariant.

Example Ledger Entry

Suppose a ₹1,000 payment is completed.

Conceptually:

Transaction: txn_1001

Debit:
Payment Clearing    ₹1,000

Credit:
Merchant Payable    ₹1,000

If the entries do not balance, something is wrong.

This helps detect financial inconsistencies.

Step 22: Immutable Ledger

Ledger records should generally be append-only.

Instead of editing:

₹1,000

into:

₹900

we add another entry:

Adjustment: -₹100

This preserves history.

For financial systems, history is extremely valuable.

Step 23: Payment Database Design

A simplified payment table could be:

Payment
-------------------------
payment_id
merchant_id
order_id
amount
currency
status
payment_method
provider
provider_payment_id
idempotency_key
created_at
updated_at

We may also have:

PaymentAttempt
-------------------------
attempt_id
payment_id
provider
provider_reference
status
error_code
created_at

Why separate attempts?

Because one logical payment may involve multiple processing attempts.

Step 24: Payment vs Payment Attempt

A Payment represents the logical transaction, while a Payment Attempt represents each attempt to process it.

Payment: pay_123

Attempt 1 → Provider A → FAILED
Attempt 2 → Provider B → SUCCESS

The payment remains pay_123, while both attempts are preserved for debugging, auditing, and reconciliation.

A new provider should be tried only when the previous attempt is confirmed failed, not when its outcome is unknown.

Step 25: Refund Database Design

A refund should have its own identity.

Refund
----------------------
refund_id
payment_id
amount
status
provider_refund_id
reason
created_at

This also supports partial refunds.

For example:

Original Payment = ₹5,000

Refund 1 = ₹1,000
Refund 2 = ₹500

Remaining captured amount:

₹3,500

The system must prevent total successful refunds from exceeding the refundable amount.

Step 26: Payment State Machine

A state machine prevents invalid transitions.

For example:

CREATED
   |
   v
PROCESSING
   |
   +------> SUCCESS
   |
   +------> FAILED
   |
   +------> PENDING
               |
               +----> SUCCESS
               |
               +----> FAILED

We should reject impossible transitions.

For example:

FAILED → SUCCESS

may or may not be legal depending on what FAILED means.

If FAILED is final, it should not later become successful.

If the outcome is uncertain, call it PENDING rather than incorrectly marking it FAILED.

Precise state semantics matter.

Step 27: The Most Dangerous Payment Failure

Consider this scenario:

Payment Gateway
       |
       | Charge ₹2,000
       v
Provider
       |
       | Payment succeeds
       v
Bank

But the response from the provider never reaches us.

Our server sees:

TIMEOUT

Should we retry the charge?

Not immediately.

The first charge may already have succeeded.

Blindly retrying can create a duplicate payment.

Instead, mark the payment:

PENDING

and query the provider using the original transaction reference.

This is a fundamental payment design rule:

A timeout does not necessarily mean failure.

Step 28: Handling Unknown Payment Status

If a provider times out:

PROCESSING
     |
     v
UNKNOWN / PENDING

A background worker can check:

GET provider/payment/status

If the provider says:

SUCCESS

update our payment.

If it says:

FAILED

mark it failed.

If the outcome is still unknown, retry the status check later according to policy.

Step 29: Reconciliation

Even with retries, webhooks, and status APIs, systems can disagree.

For example:

Our Database:
payment_123 = FAILED

Provider:
payment_123 = SUCCESS

This is a serious problem.

Reconciliation compares our transaction records with external provider records.

Reconciliation Flow

A provider may give us a settlement or transaction report.

Provider Report
       |
       v
Reconciliation Service
       |
       +---- Compare ----+
       |                 |
       v                 v
Our Records         Provider Records
       |
       v
Find Mismatches

Examples:

Missing payment
Incorrect amount
Different status
Duplicate transaction
Missing refund
Settlement mismatch

These differences can be automatically repaired where safe or sent for investigation.

Why Reconciliation Is Essential

Distributed systems fail in unusual ways.

Networks time out.

Webhooks get delayed.

Workers crash.

External providers experience outages.

Reconciliation acts as a final safety net.

For payment systems:

Real-time processing
+
Asynchronous recovery
+
Reconciliation

is much safer than depending on one API response.

Step 30: Exactly-Once Payment Processing

Interviewers often ask:

How do you guarantee exactly-once payment?

True exactly-once execution across multiple independent distributed systems is extremely difficult.

Instead, we design for effectively-once business behavior.

We combine:

Idempotency keys
Unique constraints
Provider transaction references
Durable state
Duplicate detection
Safe retries
Reconciliation

The network may deliver the same request more than once.

Our business operation should still behave as though it happened only once.

Step 31: At-Least-Once Delivery

For asynchronous events, at-least-once delivery is common.

Example:

Payment Event
    |
    v
Queue
    |
    v
Consumer

If acknowledgement fails, the queue may redeliver the event.

Therefore, consumers should be idempotent.

For example:

event_id = evt_100

Before processing:

Have I processed evt_100?

Yes → Ignore safely
No  → Process

Step 32: Transactional Outbox Pattern

Consider this problem.

The Payment Service does:

1. Update payment = SUCCESS
2. Publish payment.success event

What if the database update succeeds but the service crashes before publishing the event?

Now:

Payment = SUCCESS

but downstream services never learn about it.

Outbox Solution

Store both the business update and event record in the same database transaction.

BEGIN TRANSACTION

UPDATE payment
SET status = SUCCESS

INSERT INTO outbox
(payment.success)

COMMIT

A background publisher reads the outbox and sends events to the message broker.

Database
   |
   v
Outbox Publisher
   |
   v
Message Queue

If publishing fails, it retries.

This prevents the classic dual-write problem.

Step 33: Why a Message Queue Helps

Not every operation should happen synchronously.

After a payment succeeds, we may need to:

Send merchant webhook
Send receipt
Update analytics
Update reporting
Trigger settlement processing
Run reconciliation jobs

Making the user wait for all these tasks would be unnecessary.

Instead:

Payment Success
      |
      v
Message Broker
      |
  +---+----+---------+
  |        |         |
  v        v         v
Webhook  Receipt  Analytics

The core payment flow remains small and reliable.

Step 34: What Should Be Synchronous?

Operations required to determine whether we can safely accept or process the payment may need to remain synchronous.

For example:

Request validation
Authentication
Idempotency check
Critical payment state persistence
Required provider interaction

Tasks such as analytics can normally be asynchronous.

The key principle is:

Keep the critical payment path as small as possible.

Step 35: Risk and Fraud Detection

Payment systems attract fraud.

A Risk Service may evaluate signals such as:

Transaction amount
Payment history
Velocity
Device information
IP signals
Merchant risk
Geographic patterns
Previous fraud signals

It may return:

ALLOW
BLOCK
REVIEW

Some risk checks must happen before processing.

Other deeper analysis can happen asynchronously.

Step 36: Rate Limiting

Attackers may send huge numbers of payment requests.

Rate limits can be applied by:

Merchant
User
IP address
Payment method
API key

This protects infrastructure and can also help reduce abuse.

But rate limits should be carefully designed so legitimate high-volume merchants are not accidentally blocked.

Step 37: Security

Security is critical in Payment Gateway architecture.

Important controls include:

TLS
Authentication
Authorization
Encryption at rest
Secret management
Tokenization
Audit logging
Access control
Network isolation
Fraud detection

Payment details should never be casually logged.

Step 38: Tokenization

Suppose a customer provides card information.

Instead of storing the raw card number everywhere, the sensitive value can be exchanged for a token.

Conceptually:

Card Details
    |
    v
Secure Tokenization System
    |
    v
tok_abc123

Other internal services use:

tok_abc123

instead of the actual card number.

This reduces the number of systems exposed to sensitive payment data.

Step 39: PCI DSS

Systems that handle cardholder data may fall under PCI DSS requirements.

From an architecture perspective, a useful goal is to minimize the number of components that directly handle sensitive card data.

For example:

Client
  |
  v
Secure Payment Component
  |
  v
Token
  |
  v
Merchant Backend

Reducing the sensitive-data footprint can reduce security risk and compliance complexity.

Step 40: Do Not Log Sensitive Data

Application logs should never casually contain:

Full card number
CVV
Authentication secrets
Private API keys

Logging systems often have broad internal access and long retention.

Sensitive values should be removed or masked before logging.

Step 41: Database Choice

Payment data is highly structured and often needs strong transactional guarantees.

A relational database is therefore a natural starting point.

For example, we may require atomic operations involving:

Payment record
Idempotency record
Ledger entries
Outbox event

Transactions and uniqueness constraints are extremely valuable here.

At very large scale, the database can still be partitioned.

But do not sacrifice correctness simply because “NoSQL scales.”

Step 42: Database Partitioning

Suppose one database can no longer handle all merchants.

We may partition data using:

merchant_id

For example:

Shard 1 → Merchants A–F
Shard 2 → Merchants G–M
Shard 3 → Merchants N–Z

Hash-based partitioning may distribute traffic more evenly.

The correct partition key depends on access patterns.

Hot Merchants

One giant merchant may generate a large percentage of traffic.

If all of that merchant's transactions live on one shard:

merchant_id → shard

that shard may become hot.

At extreme scale, additional partition dimensions may be needed.

For example:

hash(merchant_id + payment_id)

But this can make merchant-wide queries more expensive.

Again, partitioning creates trade-offs.

Step 43: Read Replicas

Payment status pages and reporting may generate many reads.

We can use read replicas for workloads that can tolerate replication delay.

But be careful.

Immediately after creating a payment:

Write Primary
     |
     v
Read Replica

the replica may not have the new data yet.

For strongly consistent payment-state checks, read from an authoritative source.

Step 44: Caching

Caching can help with relatively stable information such as:

Merchant configuration
Payment routing configuration
Currency metadata
Feature flags

But caching critical payment status requires care.

Showing stale financial state can create confusion.

Use caching where it improves performance without compromising correctness.

Step 45: Payment Gateway Scalability

The stateless parts of the system can scale horizontally.

             Load Balancer
                  |
       +----------+----------+
       |          |          |
       v          v          v
Payment API  Payment API  Payment API
   Server       Server       Server

More servers can be added during traffic spikes.

State should live in durable shared systems rather than only in application memory.

Step 46: Handling Traffic Spikes

Imagine a major sale begins at 8 PM.

Payment traffic suddenly increases by 10x.

The architecture should support:

Horizontal scaling
Queue buffering
Database capacity planning
Provider rate limiting
Backpressure
Timeouts
Circuit breakers

Queues can absorb asynchronous work.

But core payment authorization cannot simply be delayed forever.

Capacity planning remains important.

Step 47: Backpressure

Suppose Provider A supports:

5,000 requests/second

but our gateway receives:

20,000 requests/second

Sending everything immediately could overwhelm the provider.

The system may need:

Rate control
Alternative routing
Controlled queues
Load shedding

depending on payment semantics.

Backpressure prevents one overloaded dependency from bringing down the entire system.

Step 48: Circuit Breaker

Suppose Provider A starts failing almost every request.

Continuing to send all traffic to it wastes resources and increases latency.

A circuit breaker can temporarily stop calls.

Provider A
Failure Rate High
      |
      v
Circuit OPEN
      |
      v
Stop New Calls Temporarily

Traffic may be routed to another provider when safe.

After some time, test requests can determine whether Provider A has recovered.

Step 49: Can We Retry Through Another Provider?

This requires extreme care.

Suppose Provider A times out.

We do not know whether the payment succeeded.

Immediately sending the same charge to Provider B could charge the customer twice.

So:

Provider Timeout
≠
Safe to Retry Elsewhere

First determine the state of the original transaction whenever possible.

Provider failover is much safer when we know the first attempt definitely did not create a financial transaction.

Step 50: Refund Flow

Suppose Alice paid ₹5,000 but returns the product.

The merchant requests a refund.

Merchant
   |
   v
Refund Service
   |
   v
Validate Refund
   |
   v
Payment Provider
   |
   v
Refund Processing

The refund may also be asynchronous.

Possible states:

CREATED
PROCESSING
SUCCESS
FAILED
PENDING

Just like payments, refunds need idempotency.

Step 51: Partial Refund

Suppose:

Payment = ₹5,000

The merchant wants to refund only:

₹2,000

The system should allow it.

But later, another request for:

₹4,000

should fail because total refunds would exceed the original refundable amount.

This check must be concurrency-safe.

Two refund requests arriving simultaneously should not both pass based on an outdated refundable balance.

Step 52: Settlement

Payment success does not always mean the merchant instantly receives money in its bank account.

There may be a settlement process.

Conceptually:

Customer Payment
      |
      v
Payment Processed
      |
      v
Merchant Balance
      |
      v
Settlement Batch
      |
      v
Merchant Bank Account

Settlement may happen later according to business rules.

A mature payment architecture therefore separates:

Payment Processing

from:

Settlement

Step 53: Payment Gateway Observability

A Payment Gateway needs excellent monitoring.

Important metrics include:

Payment success rate
Payment failure rate
Pending-payment count
Provider latency
Provider error rate
Webhook failures
Refund success rate
Reconciliation mismatches
Database latency
Queue backlog

Metrics should also be broken down by:

Provider
Payment method
Merchant
Country
Currency
Error code

This helps engineers quickly identify whether a problem is global or isolated.

Step 54: Distributed Tracing

One payment may travel through:

API Gateway
→ Payment Service
→ Risk Service
→ Router
→ Provider Adapter
→ Provider
→ Event Queue
→ Webhook Service

A correlation or trace ID helps follow the transaction through the system.

For example:

trace_id = tr_90012

Every relevant log can include that ID.

This makes production debugging much easier.

Step 55: Payment Gateway Failure Scenarios

A Payment Gateway must handle failures without creating incorrect or duplicate transactions.

• API Server Crash: Another stateless server handles requests.

• Database Failure: Do not confirm payment without durable persistence.

• Provider Timeout: Mark as PENDING and verify the final status later.

• Webhook Failure: Retry asynchronously.

• Queue Consumer Crash: Redeliver and process idempotently.

• Duplicate Request: Use idempotency to prevent duplicate payments.

• Reconciliation Mismatch: Repair safely or flag for investigation.

Step 56: Complete End-to-End Payment Flow

Let us put everything together.

Alice purchases a phone for ₹30,000.

1. Merchant Creates an Order

order_100
amount = ₹30,000

2. Merchant Creates a Payment

The backend sends:

POST /payments

with an idempotency key.

3. Gateway Checks Idempotency

No existing request is found.

4. Payment Record Is Created

payment_900
status = CREATED

5. Risk Checks Run

The transaction is allowed.

6. Payment Router Selects a Provider

Provider B

7. Payment Becomes PROCESSING

The gateway stores the new state.

8. Provider Processes the Payment

The external payment system authorizes the transaction.

9. Gateway Receives Success

payment_900
status = SUCCESS

10. Ledger Entries Are Created

Financial movement is recorded.

11. Payment Event Is Published

payment.success

12. Webhook Worker Notifies the Merchant

The merchant receives:

payment_900 = SUCCESS

13. Merchant Marks the Order Paid

order_100 = PAID

14. Reconciliation Later Confirms the Transaction

Provider records and our records match.

That gives us a complete payment lifecycle.

Payment Gateway Architecture Trade-Offs

Payment systems contain many trade-offs, but financial correctness should guide the decisions.

• Synchronous vs Asynchronous Processing: Keep the critical payment path synchronous and move tasks such as webhooks, analytics, reporting, and some reconciliation work to asynchronous processing.

• Availability vs Consistency: If the system cannot safely determine whether another charge is valid, correctness should take priority over blindly accepting the request.

• SQL vs NoSQL: Relational databases are a strong default for transactional payment data; other databases may support specialized workloads.

• Immediate Retry vs Safe Retry: Known failures may be retried when safe, but unknown outcomes should be resolved before another financial operation is attempted.

Payment Gateway HLD vs LLD

HLD focuses on overall payment architecture and scalability, while LLD focuses on component-level implementation.

A common processor abstraction can be:

PaymentProcessor
       |
       +---- CardProcessor
       |
       +---- BankTransferProcessor
       |
       +---- WalletProcessor

Provider adapters may look like:

PaymentProvider
       |
       +---- ProviderAAdapter
       |
       +---- ProviderBAdapter
       |
       +---- ProviderCAdapter

The goal is to keep business logic separate from provider-specific integration code.

Most Asked Payment Gateway System Design Interview Questions

These questions focus on the most important payment architecture decisions, failure scenarios, and financial correctness concerns.

1. How Would You Design a Payment Gateway?

Start with a Payment Service that creates and tracks payments.

Use idempotency keys to prevent duplicate operations, durable storage for payment state, provider adapters for external integrations, and a ledger for financial accounting.

Add asynchronous webhooks, refunds, reconciliation, security, monitoring, and horizontal scaling.

2. How Do You Prevent Double Payments?

Use an idempotency key for each logical payment operation.

Enforce uniqueness atomically in durable storage.

Retries using the same key return the original payment rather than creating another transaction.

3. What Is Idempotency in Payments?

Idempotency means performing the same logical request multiple times produces the same business result as performing it once.

It is essential because clients and servers retry requests when networks fail.

4. What Happens If the Payment Succeeds but the Response Is Lost?

Do not assume failure.

The payment should move to a pending or unknown state.

Use the provider's transaction reference, status API, webhook, or reconciliation process to determine the final result.

5. Why Is a Timeout Not the Same as Payment Failure?

A timeout only means we did not receive the response in time.

The provider may have already processed the transaction successfully.

Blindly retrying can create duplicate charges.

6. How Do You Achieve Exactly-Once Payment Processing?

True distributed exactly-once execution is difficult.

Instead, create effectively-once behavior using idempotency keys, unique constraints, transaction references, idempotent consumers, durable state, safe retries, and reconciliation.

7. Why Do We Need a Ledger?

A ledger gives us an auditable history of financial movements.

Instead of only storing the current balance, it records the transactions that created that balance.

This helps with auditing, reconciliation, refunds, settlement, and debugging.

8. What Is a Double-Entry Ledger?

Every financial transaction creates balanced debit and credit entries.

The key invariant is:

Total Debits = Total Credits

This helps maintain accounting correctness.

9. How Do Payment Webhooks Work?

When a payment state changes, the Payment Gateway asynchronously sends an event to the merchant's configured endpoint.

Webhook delivery should support authentication, retries, unique event IDs, and duplicate-safe processing.

10. Why Can Webhooks Be Delivered More Than Once?

Network failures make it difficult to know whether the merchant received and processed an event.

Therefore, webhook systems commonly retry until they receive a successful acknowledgement.

This creates at-least-once delivery, so consumers must handle duplicates.

11. How Do You Secure Webhooks?

Sign webhook payloads using a merchant-specific secret.

The merchant verifies the signature before trusting the event.

Timestamps and replay protection can provide additional security.

12. What Is Reconciliation?

Reconciliation compares internal payment records with records from payment providers or financial institutions.

It detects mismatches caused by network failures, missing webhooks, software bugs, delayed events, or external-system inconsistencies.

13. How Do You Handle Payment Provider Failure?

Use timeouts, circuit breakers, monitoring, and provider-health information.

Alternative routing can be used when it is safe.

Never blindly retry an uncertain financial transaction through another provider because the first attempt may already have succeeded.

14. Why Use a Transactional Outbox?

It solves the dual-write problem.

The Payment Service stores both the payment update and the event-to-publish inside one database transaction.

A separate worker reliably publishes the event afterward.

15. How Do You Handle Duplicate Queue Events?

Give each event a unique identifier.

Consumers maintain idempotent processing so receiving the same event multiple times does not create duplicate business effects.

16. Which Database Is Best for a Payment Gateway?

A relational database is often a strong starting point because payment systems benefit from transactions, constraints, structured data, and strong consistency.

At very large scale, partitioning and specialized storage may be added based on workload.

17. How Do You Handle Refunds?

Model a refund as a separate financial operation with its own ID and lifecycle.

Support idempotency, partial refunds, provider references, asynchronous states, and ledger entries.

Ensure concurrent refunds cannot exceed the refundable amount.

18. How Do You Scale a Payment Gateway?

Keep API and worker services horizontally scalable.

Partition data when necessary, use queues for asynchronous work, replicate appropriate read workloads, cache safe configuration data, and scale provider integrations independently.

The database and external provider limits usually require particular attention.

19. How Do You Handle Payment Traffic Spikes?

Use horizontal scaling, capacity planning, queue buffering for asynchronous tasks, backpressure, rate limiting, and provider-aware routing.

Monitor downstream capacity so an overloaded provider does not create cascading failures.

20. What Is the Difference Between Payment and Settlement?

Payment processing determines whether the customer's transaction succeeded.

Settlement is the later movement of funds to the merchant according to the platform's financial process.

They should be modeled separately.

21. How Would You Handle Multiple Payment Providers?

Use a Payment Router and provider-specific adapters.

The router can select providers based on payment method, geography, availability, cost, success rates, and merchant configuration.

22. How Do You Store Money in a Database?

Avoid binary floating-point types for monetary calculations.

A common approach is storing amounts as integers in the currency's smallest supported unit together with the currency code.

For example:

amount = 200000
currency = INR

represents ₹2,000 when using paise.

23. How Do You Make Payment Processing Fault Tolerant?

Combine:

Durable database writes
Idempotency
Safe retries
Queues
Transactional outbox
Provider status checks
Webhooks
Reconciliation
Replication
Monitoring

No single mechanism is enough.

24. How Would You Handle a Payment Stuck in PENDING?

A background worker periodically checks unresolved transactions.

It can query the provider using the original provider transaction ID.

Webhooks may also resolve the state.

If normal recovery does not work, reconciliation provides another path for detecting the correct outcome.

25. What Are the Most Important Payment Gateway Design Principles?

A strong answer is:

Never assume timeout means failure.

Never blindly retry an unknown payment.

Make payment creation idempotent.

Persist critical state durably.

Keep an auditable financial ledger.

Make asynchronous consumers idempotent.

Reconcile with external systems.

Keep sensitive payment data isolated.

Design every important flow for failure.

These principles are more important than memorizing a particular technology stack.

Common Mistakes in Payment Gateway System Design

Payment systems should not be treated like normal CRUD applications because they involve external systems, financial state, retries, and uncertain outcomes.

A dangerous pattern is:

Provider Timeout
      |
      v
Mark FAILED
      |
      v
Retry Payment

The original payment may already have succeeded, so this can create duplicate charges.

Other common mistakes include:

• No idempotency

• Using floating-point numbers for money

• No PENDING state for uncertain outcomes

• Blindly retrying timed-out payments

• No reconciliation

• No immutable financial ledger

• Blind provider failover

• Non-idempotent webhook consumers

• No transactional outbox

• Storing sensitive payment data unnecessarily

• Ignoring partial refunds and concurrent refund requests

• Treating payment processing and settlement as the same operation

A strong Payment Gateway assumes failures will happen and makes every critical financial operation safe, traceable, and recoverable.