Sustained load against the M-Pesa sandbox put the service into a spiral:
throttling opened the circuit breaker, the breaker recorded untried payments
as Failed, and an uncapped reconciliation sweep held the operator's entire
call budget while real payments queued behind it.
Resilience observability
- ResilienceEventLogging registers RegistryEventConsumers for Retry,
CircuitBreaker and RateLimiter. Resilience4j publishes events but logs
nothing, so retries, state transitions and refused permits were invisible.
Subscribing at registry level catches instances as they are created lazily.
- ProviderBackoff extracts the ProviderBusyException backoff that all three
clients duplicated, and logs each wait.
- Everything routes through one logger name, com.test.payment.resilience.
- org.hibernate.SQL logging off by default.
Breaking the loop
- ProviderBusyException is ignored by the circuit breaker. Being throttled is
not the provider being unhealthy; counting 429s as failures turned a rate
limit into a self-inflicted outage.
- New Rejected status with resolvedBy = NOT_SENT for calls an open breaker or
a refused permit turned away. These never reached the operator, so recording
them as Failed produced rows that read as failed payments when nothing was
sent — and with no provider reference, nothing could ever resolve them.
- Reconciliation sweeps at most payments.reconciliation.max-per-run (10)
initiations per tick, oldest first, and logs what it deferred. Uncapped, a
backlog of 60 queued 60 provider calls per 60-second tick.
Rate limits calibrated to the provider
- mpesaLimiter is 1 permit per 2s, not 5 per second. Safaricom's Apigee spike
arrest allows 30/minute with a burst of 3, so the old setting was 10x more
permissive than the provider and never protected anything. Spike arrest
enforces spacing, so this is 1-per-2s rather than 30-per-60s.
- Client poll rates move to payments.status-poll-interval and are served by
GET /api/payments/config, so clients stop guessing at a budget only the
server can see. providerCallsPerMinute is derived from the limiter config.
Verified against a mock provider: the capped sweep reconciles exactly N oldest
and reports the rest; 429s no longer open the breaker; breaker-rejected calls
land as Rejected/NOT_SENT while genuine provider errors stay Failed/ERROR.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- StatusCatalog gains an Unresolved state and a PRECEDENCE order
(Pending -> Unresolved -> Failed -> Success -> Paid); canTransition only
allows moves up it, so a callback and a status query racing each other
cannot make the status flap. A refused transition still stores the row.
- PaymentInitiation carries a @Version optimistic lock; callback and query
paths go through blockingWithRetry, which replays the losing unit of work
once instead of dropping it.
- Reconciliation runs two passes per tick, stale first: anything Pending past
payments.reconciliation.stale-age (3h) is marked Unresolved with
resolvedBy = EXPIRY, then anything past pending-age (now 3m) is re-queried.
- Explicit lowercase @Table names on the entities that were relying on
derived naming.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
