feat(backend, mock-ase): encrypted data exchange

[sanducb]

Changes proposed in this pull request

This PR adds support for sending encrypted additional data over ILP on which the receiving ASE can action upon i.e. approve/reject a specific payment (e.g. if AML/KYC checks were not successful).

Context

Closes Rafiki Encrypted Data Exchange project

Checklist

• Related issues linked using fixes #number
• Tests added/updated
• Make sure that all checks pass
• Bruno collection updated (if necessary)
• Documentation issue created with user-docs label (if necessary)
• OpenAPI specs updated (if necessary)

[netlify]

✅ Deploy Preview for brilliant-pasca-3e80ec canceled.

Name	Link
🔨 Latest commit	1fb8054
🔍 Latest deploy log	https://app.netlify.com/projects/brilliant-pasca-3e80ec/deploys/69e24b861790e6000911ed93

[github-actions]

🚀 Performance Test Results

Test Configuration:

• VUs: 4
• Duration: 1m0s

Test Metrics:

• Requests/s: 67.80
• Iterations/s: 22.62
• Failed Requests: 0.00% (0 of 4077)

📜 Logs

> performance@1.0.0 run-tests:testenv /home/runner/work/rafiki/rafiki/test/performance
> ./scripts/run-tests.sh -e test "-k" "-q" "--vus" "4" "--duration" "1m"

Cloud Nine GraphQL API is up: http://localhost:3101/graphql
Cloud Nine Wallet Address is up: http://localhost:3100/
Happy Life Bank Address is up: http://localhost:4100/
cloud-nine-wallet-test-backend already set
cloud-nine-wallet-test-auth already set
happy-life-bank-test-backend already set
happy-life-bank-test-auth already set
     data_received..................: 1.5 MB 25 kB/s
     data_sent......................: 3.1 MB 52 kB/s
     http_req_blocked...............: avg=7.23µs   min=2.37µs   med=5.22µs   max=1.1ms    p(90)=6.65µs   p(95)=7.3µs   
     http_req_connecting............: avg=502ns    min=0s       med=0s       max=1.05ms   p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=58.38ms  min=6.65ms   med=48.09ms  max=383.07ms p(90)=98.8ms   p(95)=107.21ms
       { expected_response:true }...: avg=58.38ms  min=6.65ms   med=48.09ms  max=383.07ms p(90)=98.8ms   p(95)=107.21ms
     http_req_failed................: 0.00%  ✓ 0         ✗ 4077
     http_req_receiving.............: avg=87.84µs  min=28.04µs  med=75.6µs   max=2.68ms   p(90)=120.42µs p(95)=160.77µs
     http_req_sending...............: avg=37.06µs  min=9.79µs   med=27.22µs  max=1.72ms   p(90)=42.39µs  p(95)=59.43µs 
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=58.25ms  min=6.5ms    med=47.96ms  max=382.85ms p(90)=98.64ms  p(95)=107.13ms
     http_reqs......................: 4077   67.798784/s
     iteration_duration.............: avg=176.75ms min=120.91ms med=171.91ms max=787.89ms p(90)=208.21ms p(95)=229.28ms
     iterations.....................: 1360   22.616224/s
     vus............................: 4      min=4       max=4 
     vus_max........................: 4      min=4       max=4 

[njlie]

LGTM just needs the merge conflicts fixed, of course

[BlairCurrey]

I think the implementation differs from the spec here. we have partialPayment here but it looks like the implementation is doing partialPaymentId. I think it was originally correct when we implemented here but then something happened: #3810

As far as I know this isnt breaking anything (no spec validation against webhooks I think?) but it should match one way or another.

[BlairCurrey]

This value will be publicly visible in the metadata field if this incoming payment is requested through Open Payments

I dont think we make good on this promise. rejectPartialIncomingPayment doesnt pass reason through. Should we be putting it on the metadata?

I don't have any context on this, but for whatever its worth, im not sure I like putting this on metadata. Historically it's been a client-controlled field and now we're injecting our own data there. We'd need to be careful in merging the data and it may not be very intuitive for the ASE. Im not sure if the new encrypted data has changed the role of this metadata field or something.

[sanducb]

That is a good point, I can't quite figure if the description of this field is stale and we forgot to update/remove it. CC @njlie

[BlairCurrey]

I think this might be too loose. I see in #3686 that it says if value is missing by the deadline → proceed (next()). But if success is undefined (which I believe happens if it did not find anything by deadline), this will not call next.

So I think maybe this should be if (decision?.success !== false) { to ensure its a success. This is just from me tracing though - would be good to have a test for this.

[BlairCurrey]

a test for this case - currently failing

test.each([
    ['no success field (timeout)', { message: 'No response from ASE' }],
    [
      'explicit success: undefined',
      { success: undefined, message: 'No response from ASE' }
    ]
  ])(
    'allows payment when ASE does not respond (%s)',
    async (_, decisionResult) => {
      const { services } = makeServices()
      const ctx = makeContext(undefined, services)
      mockIncomingMoneyReply(ctx)
      jest
        .spyOn(services.incomingPayments, 'processPartialPayment')
        .mockResolvedValue(decisionResult)

      const next = jest.fn()
      await expect(middleware(ctx, next)).resolves.toBeUndefined()

      expect(next).toHaveBeenCalledTimes(1)
      expect(ctx.response.reply).toBeUndefined()
    }
  )

[BlairCurrey]

Does the reject reason need to get back to the sending ASE? Doesnt look like it does, currently. This is what I see when I setup the mock ase to reject instead of confirm:

cloud-nine-backend-1              | {"level":40,"time":1776363393572,"pid":31,"hostname":"cloud-nine-wallet-backend","service":"OutgoingPaymentService","payment":"db31cbdd-aa28-4c54-a037-908c3d2b41a2","from_state":"SENDING","state":"SENDING","error":"Received error during ILP pay","stateAttempts":1,"errorDescription":"ApplicationError","msg":"payment lifecycle failed"}
cloud-nine-mock-ase-1             | received webhook:  {"id":"10038171-7744-492b-90e4-ca09a0a187f4","type":"outgoing_payment.failed","data":{"id":"db31cbdd-aa28-4c54-a037-908c3d2b41a2","walletAddressId":"a74e201e-a81a-4736-a410-b38c0c3bf8b0","client":"https://happy-life-bank-backend/accounts/pfry","state":"FAILED","receiver":"https://happy-life-bank-backend/cf5fd7d3-1eb1-4041-8e43-ba45747e9e5d/incoming-payments/d5b7a04d-0023-42ee-8d73-1e3b043f0df4","debitAmount":{"value":"205","assetCode":"USD","assetScale":2},"receiveAmount":{"value":"100","assetCode":"USD","assetScale":2},"sentAmount":{"value":"0","assetCode":"USD","assetScale":2},"stateAttempts":0,"createdAt":"2026-04-16T18:16:33.045Z","balance":"205","metadata":{"description":"Free Money!"},"error":"Received error during ILP pay","grantId":"b8cd2c65-661f-477c-8616-118304da6141"}}

I set the reason as "KYC checked failed" on the reject request. I was expecting to see this reason on the webhook to the sending ASE. But I just see a generic "Received error during ILP pay". Im not sure if the reject reason is exposed by the pay library currently.

I'm not specifically sure of the requirements but I assumed we needed this. It was briefly noted here although this issue is old and was closed. #3689

[sanducb]

I added the rejection reason on the webhook on the error field of the outgoing_payment.failed webhook in this commit as you suggested. If I am not mistaken #3689 was cancelled because of us not needing a new webhook, but IMO it makes sense to still put the rejection reason on outgoing_payment.failed (CC @mkurapov).

[BlairCurrey]

looks good to me

cloud-nine-backend-1              | {"level":50,"time":1776710737471,"pid":31,"hostname":"cloud-nine-wallet-backend","service":"IlpPaymentService","err":{"type":"PaymentMethodHandlerError","message":"Received error during ILP pay","stack":"PaymentMethodHandlerError: Received error during ILP pay\n    at pay (/home/rafiki/packages/backend/src/payment-method/ilp/service.ts:372:17)\n    at Object.handleSending (/home/rafiki/packages/backend/src/open_payments/payment/outgoing/lifecycle.ts:117:23)\n    at handlePaymentLifecycle (/home/rafiki/packages/backend/src/open_payments/payment/outgoing/worker.ts:106:5)\n    at /home/rafiki/packages/backend/src/open_payments/payment/outgoing/worker.ts:32:9","name":"PaymentMethodHandlerError","description":"KYC verification failed: identity could not be confirmed.","retryable":false},"destination":"test.happy-life-bank.mp0E2F-vlTkUM0iCE4KZMMdV7DLouTgqs0dhjqYX-9BqhmPsNK_AfsCW4j1nDtytOkujCm3B_2cbX8eSL1G0u96Sv_seOc0","errorDescription":"KYC verification failed: identity could not be confirmed.","msg":"Received error during ILP pay"}
cloud-nine-backend-1              | {"level":40,"time":1776710737485,"pid":31,"hostname":"cloud-nine-wallet-backend","service":"OutgoingPaymentService","payment":"034979cb-9eaa-4f83-950a-8430ea331eb0","from_state":"SENDING","state":"SENDING","error":"KYC verification failed: identity could not be confirmed.","stateAttempts":1,"errorDescription":"KYC verification failed: identity could not be confirmed.","msg":"payment lifecycle failed"}
cloud-nine-mock-ase-1             | received webhook:  {"id":"24e806f5-d330-419c-b3b1-4d1079b6c369","type":"outgoing_payment.failed","data":{"id":"034979cb-9eaa-4f83-950a-8430ea331eb0","walletAddressId":"8a34ee74-78b1-4d7d-8d60-90015cc0dd0e","client":"https://happy-life-bank-backend/accounts/pfry","state":"FAILED","receiver":"https://happy-life-bank-backend/cf5fd7d3-1eb1-4041-8e43-ba45747e9e5d/incoming-payments/f207ab03-615e-4914-9478-8c4f0b996d5c","debitAmount":{"value":"205","assetCode":"USD","assetScale":2},"receiveAmount":{"value":"100","assetCode":"USD","assetScale":2},"sentAmount":{"value":"0","assetCode":"USD","assetScale":2},"stateAttempts":0,"createdAt":"2026-04-20T18:45:37.043Z","balance":"205","metadata":{"description":"Free Money!"},"error":"KYC verification failed: identity could not be confirmed.","grantId":"bf958282-c390-4c56-895b-be8b79e58fd1"}}

[sanducb]

Seems like backend build is failing because of some failing Tigerbeetle tests, but all the other (build) jobs are successful. Currently looking into this issue, might be related to the new lockfile or overrides that I added on this branch. Nothing related to the data exchange logic, though.

[BlairCurrey]

I looked at the failing backend tests and why testcontainer couldnt find docker. No resolution yet. Here is what I found:

Last working commit: 683096a.

After that the lockfile is broken - once its fixed and the tests can be run we see the error. See this on install: Broken lockfile: no entry for '/regenerator-runtime/0.14.0' in pnpm-lock.yaml

I pinned testcontainers to 10.16 (previous version before lockfile fix upraded it) locally and dont see the errors. Not sure about the root cause though. I thought maybe it was how an internal test container dependency (docker-modem: apocas/docker-modem#156, apocas/docker-modem#157) resolves the docker socket when not provided one but i couldnt confirm. I tried to set the DOCKER_HOST env var explicitly. Saw the same error.

The failure we are seeing, for reference:

FAIL packages/backend/src/accounting/tigerbeetle/service.test.ts
  ● Test suite failed to run

    Could not find a working container runtime strategy

      27 |     ])
      28 |     .withAddedCapabilities('IPC_LOCK')
    > 29 |     .withCommand([
         |                          ^
      30 |       'format',
      31 |       '--cluster=' + tigerBeetleClusterId,
      32 |       '--replica=0',

      at getContainerRuntimeClient (node_modules/.pnpm/testcontainers@10.28.0/node_modules/testcontainers/src/container-runtime/clients/client.ts:63:9)
      at GenericContainer.start (node_modules/.pnpm/testcontainers@10.28.0/node_modules/testcontainers/src/generic-container/generic-container.ts:86:20)
      at startTigerBeetleContainer (packages/backend/src/tests/tigerbeetle.ts:29:26)
      at TigerBeetleEnvironment.setup (packages/backend/jest.tigerbeetle-environment.ts:22:29)

[BlairCurrey]

I looked at the failing backend tests and why testcontainer couldnt find docker. No resolution yet. Here is what I found:

Last working commit: 683096a.

After that the lockfile is broken - once its fixed and the tests can be run we see the error. See this on install: Broken lockfile: no entry for '/regenerator-runtime/0.14.0' in pnpm-lock.yaml

I pinned testcontainers to 10.16 (previous version before lockfile fix upraded it) locally and dont see the errors. Not sure about the root cause though. I thought maybe it was how an internal test container dependency (docker-modem: apocas/docker-modem#156, apocas/docker-modem#157) resolves the docker socket when not provided one but i couldnt confirm. I tried to set the DOCKER_HOST env var explicitly. Saw the same error.

The failure we are seeing, for reference:

FAIL packages/backend/src/accounting/tigerbeetle/service.test.ts
  ● Test suite failed to run

    Could not find a working container runtime strategy

      27 |     ])
      28 |     .withAddedCapabilities('IPC_LOCK')
    > 29 |     .withCommand([
         |                          ^
      30 |       'format',
      31 |       '--cluster=' + tigerBeetleClusterId,
      32 |       '--replica=0',

      at getContainerRuntimeClient (node_modules/.pnpm/testcontainers@10.28.0/node_modules/testcontainers/src/container-runtime/clients/client.ts:63:9)
      at GenericContainer.start (node_modules/.pnpm/testcontainers@10.28.0/node_modules/testcontainers/src/generic-container/generic-container.ts:86:20)
      at startTigerBeetleContainer (packages/backend/src/tests/tigerbeetle.ts:29:26)
      at TigerBeetleEnvironment.setup (packages/backend/jest.tigerbeetle-environment.ts:22:29)

As mentioned in Slack I think it would be best to revert the testcontainer upgrade and handle that in our normal renovate package updating process. The update was just incidental to fixing the lockfile. Should make an issue for it though.