Updates

CLAUDE.md

@@ -22,12 +22,14 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 ```
 
 Requires JDK 21 to run Gradle (Gradle 8.14 cannot run on JDK 25; set `JAVA_HOME` accordingly).
-The app starts on the default port 8080. Swagger UI is available at `http://localhost:8080/swagger-ui.html`.
+The app starts on port 8091 (`server.port`). Swagger UI is at `http://localhost:8091/swagger-ui.html`.
 Redis is optional at runtime (`docker run -d --name mpesa-redis -p 6379:6379 redis:7-alpine`) — the app degrades gracefully without it.
 
 ## Architecture Overview
 
-Spring Boot 4 / Java 21 reactive (WebFlux) multi-provider mobile-money service (M-Pesa STK Push, Airtel Money, MTN MoMo), using R2DBC for async DB access, Redis (Lettuce) for token caching, and Resilience4j for fault tolerance.
+Spring Boot 4 / Java 21 reactive (WebFlux) multi-provider mobile-money service (M-Pesa STK Push, Airtel Money, MTN MoMo), using **JPA/Hibernate over PostgreSQL** for persistence, Redis (Lettuce) for token caching, and Resilience4j for fault tolerance.
+
+**Blocking persistence under a reactive web layer.** HTTP, WebClient and Resilience4j stay reactive; JPA blocks. The bridge is a facade/store split — `PaymentLifecycleService` and `PaymentLimitService` keep their `Mono`/`Flux` API and hand each unit of work to `PaymentLifecycleStore` / `PaymentLimitStore` via `Mono.fromCallable(...).subscribeOn(Schedulers.boundedElastic())`. The stores are separate beans on purpose: `@Transactional` goes through a Spring AOP proxy that self-invocation would bypass (same trap as the Resilience4j annotations). **Never call a repository from the event loop, and never hold a transaction across a provider HTTP call** — `checkStatus` deliberately enters the store, exits, queries the provider, then re-enters.
 
 **Layering (same shape for every provider):**
 `<Provider>Controller` → `<Provider>Service` (implements `PaymentProviderService`; parses provider payloads) → `PaymentLifecycleService` (ALL persistence: initiation/response/callback/transaction, dedup, status transitions) and `<Provider>Client` (resilience-wrapped HTTP) → provider API.
@@ -39,6 +41,8 @@ Spring Boot 4 / Java 21 reactive (WebFlux) multi-provider mobile-money service (
 - `GET /api/payments/transactions?provider=` — consolidated transactions across providers.
 - `GET /api/payments/limits?provider=` / `PUT /api/payments/limits` — read and upsert the configurable payment ceilings (body: `provider`, `period`, `scope`, `maxAmount`, `currency`, `active`).
 
+**Provider identity is market-qualified.** `Operator` (MPESA/AIRTEL/MTN) is the integration — one client, one token, one set of Resilience4j instances. `PaymentProviderType` is `Operator` × `Country`: MPESA_KE, AIRTEL_KE, AIRTEL_UG, MTN_UG, … That enum name is what lands in every `provider` column and every provider-scoped lookup (limits, reconciliation dispatch, callback matching). Which market each operator runs in comes from `<operator>.country` in yml, resolved once by `ProviderMarkets`; an unsupported pairing fails at startup rather than mid-payment. `provider` columns are typed `@Enumerated(EnumType.STRING)`, never raw Strings — likewise `ProviderToken.provider`, which is an `Operator` because credentials are per operator, not per market.
+
 **Provider specifics:**
 - M-Pesa: `providerReference` = CheckoutRequestID, `secondaryReference` = MerchantRequestID; STK query "still processing" (errorCode 500.001.1001) maps to `ProviderProcessingException` → stays PENDING. Sandbox creds in yml are live.
 - Airtel: reference is a generated `ATL<uuid>` transaction id; status codes TIP (pending) / TS (success) / TF (failed); receipt = `airtel_money_id`. Credentials are placeholders (`airtel.*`).
@@ -46,23 +50,35 @@ Spring Boot 4 / Java 21 reactive (WebFlux) multi-provider mobile-money service (
 
 **Key design points:**
 
-- All I/O is non-blocking (`Mono`/`Flux` throughout). Never block a reactive pipeline with `.block()` (startup schema init is the one deliberate exception).
+- The web layer and all provider I/O are non-blocking (`Mono`/`Flux`). Never block the event loop and never call `.block()`; database work reaches JPA only through the store beans on `Schedulers.boundedElastic()`.
 - All outbound provider calls live in `client/*Client` classes so the Resilience4j annotations (`@CircuitBreaker`, `@RateLimiter`, `@Retry`) go through Spring AOP — they are silently skipped on self-invocation. Each provider has its own instances (`mpesa*`, `airtel*`, `mtn*`) configured **only** in `application.yml` (YAML anchors share the retry/circuit-breaker settings).
 - Shared exception model in `exceptions/`: `ProviderBusyException` (429/busy → reactive backoff retry, 503), `ProviderTransientException` (Resilience4j retry, 502), `ProviderPermanentException` (never retried, CB-ignored, 400), `ProviderProcessingException` (still processing → stays PENDING). HTTP-to-exception mapping is shared in `client/ProviderHttpErrors` (401 evicts the token so the retry refetches).
 - `TokenCacheService` is the tiered token cache for all providers: Redis (`<provider>:access_token`) → `PROVIDER_TOKENS` table → live OAuth fetch (persisted to both tiers). Redis being down never fails a request (800ms timeouts, falls through). `<Provider>TokenService` classes only supply the fetch call.
 - Boot 4 defaults to Jackson 3 (`tools.jackson.databind.ObjectMapper`) — inject that type, not `com.fasterxml`. The `com.fasterxml.jackson.annotation.*` annotations still work.
 - Resilience4j 2.4.0 with `resilience4j-spring-boot4`; annotations require `aspectjweaver` (Boot 4 removed `spring-boot-starter-aop`). Boot 4 also split `WebClient` auto-config into `spring-boot-starter-webclient`.
-- Package convention: `models` holds only database entities (`@Table`) plus `DatabaseSchema`; everything crossing an API boundary lives in `dto`.
+- Package convention: `models` holds only database entities (`@Entity`) and the enums they persist (`Operator`, `Country`, `PaymentProviderType`, `LimitPeriod`, `LimitScope`); `models/audit` holds the provider-call audit entities; everything crossing an API boundary lives in `dto`.
 
-**Persistence (in-memory H2 via R2DBC):**
-- The schema lives in code: `models/DatabaseSchema.STATEMENTS`, executed at startup by `configurations/DatabaseSchemaInitializer` (there is no schema.sql; `spring.sql.init` is not used).
-- `PAYMENT_INITIATIONS` — one row per payment attempt, any provider (status: PENDING → SUCCESS/FAILED).
-- `PAYMENT_RESPONSES` — the provider's answer, `initiation_id UNIQUE` (1:1) and `provider_reference UNIQUE` (lookup key for callbacks/status).
-- `PAYMENT_CALLBACKS` — the provider result callback, `initiation_id UNIQUE`, duplicates ignored, raw payload stored as JSON.
-- `TRANSACTIONS` — consolidated record upserted by `PaymentLifecycleService.recordTransaction` whenever an initiation reaches a terminal state, from whichever path resolved it (`resolvedBy`: CALLBACK, QUERY, REJECTION, ERROR, RECONCILIATION). `initiation_id UNIQUE`.
-- `PROVIDER_TOKENS` — OAuth tokens per provider with expiry.
-- `PROVIDER_LIMITS` — configurable payment ceilings, `UNIQUE (provider, period, scope)`. Periods come from the `LimitPeriod` enum (PER_TRANSACTION, DAILY, MONTHLY — adding a constant is all a new period needs); `scope` from the `LimitScope` enum (PER_PAYER buckets by paying MSISDN, MERCHANT sums every payer on the provider; defaults to PER_PAYER and is ignored by PER_TRANSACTION). Defaults are seeded by `DatabaseSchema.SEED_STATEMENTS` only when the triple is absent, so runtime edits survive on a persistent DB. `PaymentLimitService.enforce` runs in every `<Provider>Service.initiatePayment` **before** the initiation is persisted (breaches leave no DB row) and raises `PaymentLimitExceededException` → 422 `LIMIT_EXCEEDED`. Cumulative periods sum non-FAILED initiations inside the window (per MSISDN for PER_PAYER, provider-wide for MERCHANT), so PENDING pushes count against the cap. An unrecognised `scope` on a row falls back to PER_PAYER (the tighter interpretation).
-- `@Table` names must be UPPERCASE — H2 stores unquoted DDL identifiers uppercase and Spring Data quotes entity names verbatim.
+**Persistence (PostgreSQL via JPA/Hibernate):**
+- Connection settings live under `spring.datasource.*` (`jdbc:postgresql://localhost:5432/payments`, user `myapp`), driver `org.postgresql:postgresql`. `spring.jpa.open-in-view` is off — WebFlux has no OSIV filter, so lazy access outside a transaction should fail loudly.
+- The database must exist and be reachable before startup, so `./gradlew build` (which runs the `contextLoads` test) needs a live Postgres, or use `-x test`. On PostgreSQL 15+ the app's role also needs `GRANT USAGE, CREATE ON SCHEMA public`, or startup dies with SQLSTATE 42501.
+- Hibernate owns the DDL (`spring.jpa.hibernate.ddl-auto: update`). There is no schema.sql and no hand-rolled initializer; `configurations/DataSeeder` seeds only the reference data Hibernate cannot derive — the `STATUSES` rows and the default provider ceilings — each guarded by an existence check so runtime edits survive a restart.
+- `STATUSES` — the payment lifecycle states as rows, not an enum: Paid, Success, Pending, Failed. `PaymentInitiation` and `Transaction` reference it `@ManyToOne` (`status_id`). There is deliberately **no enum mirroring these values**: `StatusCatalog` is the single place that names the four the code branches on, and every caller asks it for a row (`statuses.pending()`, `.paid()`, `.success()`, `.failed()`) so the value served is whatever the table currently holds. `StatusCatalog.defaults()` is also what `DataSeeder` seeds from, so the names live in exactly one file. Rows are memoised, so a transition costs no query — call `invalidate()` after inserting. **Paid vs Success**: a successful callback carrying a receipt number resolves to Paid, one without to Success.
+- `PAYMENT_INITIATIONS` — one row per payment attempt, any provider.
+- `<operator>_PAYMENT_RESPONSES` — the provider's answer, one table per operator (`mpesa_payment_responses`, `airtel_…`, `mtn_…`), `initiation_id UNIQUE` (1:1) and `provider_reference UNIQUE` (lookup key for callbacks/status).
+- `<operator>_PAYMENT_CALLBACKS` — the result callback that resolved the payment, one table per operator, `initiation_id UNIQUE`, duplicates ignored, raw payload stored as JSON. (The duplicate-inclusive record is the separate `<operator>_callback_responses` audit table.)
+- These six entities are standalone — no shared supertype, matching the audit tables. `PaymentLifecycleStore` therefore dispatches on `Operator` and hands the rest of the lifecycle a provider-neutral `PaymentLifecycleService.StoredResponse` view record (not an entity, no table). **Keep the provider equality check when resolving a reference**: one operator's table holds every one of its markets, so a MPESA_TZ reference must not resolve against a MPESA_KE payment.
+- `TRANSACTIONS` — the spine that ties a payment together. Created by `saveInitiation` **at request time**, not at resolution, so a payment is never missing from this table; `recordTransaction` then updates it as the payment progresses (`resolvedBy`: CALLBACK, QUERY, REJECTION, ERROR, RECONCILIATION — left null while still open). References its `PaymentInitiation` `@ManyToOne` (`initiation_id`, UNIQUE — one transaction per attempt), its `Status`, and the provider's response and callback.
+- A transaction points at **the request** and **the callback**, not the response: the response already hangs off the request (`MpesaResponse.request`), so the request is the single anchor for everything the operator sent back. Because those tables are per operator with no shared supertype, each link is three nullable FKs — `mpesa_request_id` / `airtel_request_id` / `mtn_request_id`, and the same for callbacks — of which exactly one per trio is set. `attachRequest`/`attachCallback` in the store pick the right one; `hasRequest()` / `hasCallback()` hide the null-checking.
+- `attachRequest` links the **earliest** audit request for the initiation (later rows are status queries) and is **best effort**: audit rows are written asynchronously, so if one has not landed yet the link is picked up on the next update.
+**Naming convention — PascalCase entity fields (`CreatedByPartner` style).** Every `@Entity` field is PascalCase. Lombok still generates ordinary `getX()`/`setX()` and builders take the PascalCase name (`.Provider(...)`). Three consequences, all of which have already bitten:
+- **Spring Data derived queries break silently.** The persistent property is `Initiation`, not `initiation`, so `findByInitiationId` fails at runtime with `IllegalStateException: Binding property is null`. **Every repository therefore uses explicit `@Query` JPQL with PascalCase paths** (`WHERE t.Initiation.Id = :id`). Never add a derived query method. `findFirstBy…` has no JPQL equivalent either — those became `ORDER BY … ` queries returning a `List` whose head the caller takes.
+- **Associations are `@ManyToOne(targetEntity = X.class, fetch = FetchType.LAZY)` with `@JoinColumn(name = "<FieldName>")`** — the column is named after the field, so `transactions` has `initiation`, `status`, `mpesa_request`, `mpesa_callback` rather than `*_id`. No `foreignKey = @ForeignKey(name = ...)`; constraint names are left to Hibernate. Because everything is LAZY and `open-in-view` is off, **any query whose result is serialised by the web layer must `LEFT JOIN FETCH` its associations** — see `TransactionRepository.GRAPH`, which the listing queries share.
+- **Entity-returning endpoints now emit PascalCase JSON keys** (`{"Id":1,"Provider":"MPESA_KE"}`) — `/api/payments/transactions` and `/api/payments/limits`. The DTO-returning endpoints (`/pay`, `/status`, `/callback`) are unaffected and stay camelCase, because DTOs are not part of this convention.
+- `<operator>_requests.initiation_id` is a **String**, not the numeric id, so it can carry a NanoID; it is a loose reference with no FK.
+- `PROVIDER_TOKENS` — OAuth tokens per **operator** (not per market) with expiry.
+- `PROVIDER_LIMITS` — configurable payment ceilings, `UNIQUE (provider, period, scope)`. Periods come from the `LimitPeriod` enum (PER_TRANSACTION, DAILY, MONTHLY — adding a constant is all a new period needs); `scope` from the `LimitScope` enum (PER_PAYER buckets by paying MSISDN, MERCHANT sums every payer on the provider; defaults to PER_PAYER and is ignored by PER_TRANSACTION). Defaults are seeded by `DataSeeder` only when the triple is absent (and only for the markets `<operator>.country` actually configures), so runtime edits survive a restart. `PaymentLimitService.enforce` runs in every `<Provider>Service.initiatePayment` **before** the initiation is persisted (breaches leave no DB row) and raises `PaymentLimitExceededException` → 422 `LIMIT_EXCEEDED`. Cumulative periods sum non-Failed initiations inside the window (per MSISDN for PER_PAYER, provider-wide for MERCHANT), so PENDING pushes count against the cap. An unrecognised `scope` on a row falls back to PER_PAYER (the tighter interpretation).
+- **Provider-call audit trail** (`models/audit`, `repository/audit`, `ProviderCallAudit`): every outbound call and inbound callback is recorded per operator in its own independent tables — `mpesa_requests` / `mpesa_responses` / `mpesa_callback_responses`, and the `airtel_*` and `mtn_*` equivalents. Responses and callbacks reference their request `@ManyToOne`. The nine entities are **fully standalone** — no shared supertype, no `@MappedSuperclass`, no discriminator, no join. `ProviderCallAudit` therefore dispatches on `Operator` with an explicit branch per provider rather than polymorphically, which is verbose on purpose: a new operator will not compile until all three of its tables are wired up. Writes run on the `auditExecutor` pool (bounded queue, drop-on-overflow, daemon threads) so a payment is never delayed or failed by its audit trail; ordering within a call is kept by chaining onto the request's `CompletableFuture` rather than blocking on it. Credentials matching `ProviderCallAudit.SECRETS` (Password, passkey, api-key, secret, authorization, access_token) are masked before anything is stored.
+- `@Table` names must be lowercase — Postgres folds unquoted DDL identifiers to lowercase and Spring Data quotes an explicit entity name verbatim. Column names carry no `@Column` annotation, so they are derived and adapt to the dialect's casing on their own; keep it that way.
 - `PaymentReconciliationJob` reconciles PENDING initiations of **all** providers older than `payments.reconciliation.pending-age` (default 5m) by dispatching to the right `PaymentProviderService`; interval `payments.reconciliation.fixed-delay` (default 60s).
 
 **Configuration (`application.yml`):**