Insights Contact

Product and Program Management

Release Management in Payments: From Plan to Go-Live Without Surprises

By SP Sankar

⏱ 2-minute read · open “Read more” for the details

In most software, a bad release means a bug. In payments, a bad release means money in the wrong place, customers who can’t pay their bills, and regulators asking questions. That is why release management matters more in payments than almost anywhere else.

Releasing a payment change is like landing a plane: there is a checklist, a clear go or no-go decision, and a way to back out safely if something looks wrong. This post explains each step.

Release management is simply the way a team gets a change from “done” to “live” safely: planned, tested, approved, deployed, checked and learned from. Every time, in the same order.

The config file that stopped every release

Some of the most expensive mistakes in payments aren’t in the payment logic at all. One of mine was a few lines in a config file.

  • Passwords for a keystore and a database were hard-coded in plain text.
  • Secrets belong in a vault, never in config files.
Read more: The config file that stopped every release

The passwords for a keystore, where the bank’s certificates and keys live, and for a database had been hard-coded in plain text in the configuration files. They worked, nobody complained, and they traveled quietly from release to release. Then the compliance team noticed.

That triggered an audit, and everything stopped. Not just the change that had the passwords. Every release was frozen until the audit was done. A few lines of config had put the whole delivery pipeline on hold.

The fix itself wasn’t complicated: the passwords were taken out of the config and encrypted as per the bank’s security standard. The cost was everything else, all the releases that sat waiting while it was put right.

What I took away: no passwords in code or config, ever, and scan for them automatically on every build so a person doesn’t have to spot them. And remember that a security finding doesn’t stop one release. It stops all of them.

What happens when it goes wrong

In April 2018, the UK bank TSB moved its customers to a new platform in one big switch. It failed straight away.

  • A large share of its 5.2 million customers were hit.
  • The lesson: big changes need a clear, boring, repeatable release process.
Read more: What happens when it goes wrong

In April 2018, the UK bank TSB moved its customers to a new IT platform in one big switch. The platform failed straight away. A large share of its 5.2 million customers were hit, and it took until December 2018 to get back to normal. TSB paid £32.7 million to put things right for customers, and UK regulators later fined it £48.65 million. The regulators’ finding was simple: the bank did not plan and control the change properly. (The FCA is the UK’s financial regulator; the PRA is its bank safety regulator.)

The lesson isn’t “never change anything”. It’s that big changes need a clear, boring, repeatable release process.

The six stages of a payments release

A release moves through six stages, like landing a plane.

  • Plan, then build and test.
  • Approve it at a go / no-go meeting.
  • Deploy in small steps, check with real money, then learn.
  • A tested rollback is the safety net.
The payments release lifecycleEvery release goes through the same six stages, every time1Planscope, date,owners2Build & testcode, regression,UAT3Approvego / no-gocheck4Deploysmall, safesteps5Verifypenny test,alerts6Learnreview,improveSomething wrong? Roll back, fix, test againRule of thumb: no release starts stage 4 without a tested way back.
Six stages, in the same order, for every release
Read more: The six stages of a payments release

Read the boxes from left to right. A release moves from planning to learning in six steps. The dashed line underneath is the safety net: if something goes wrong while deploying or checking, the team rolls back, fixes it and tests again before trying once more.

StageWhat happensThe question to answer
1. PlanAgree scope, date, owners and the release calendarWhat is going out, when, and who owns it?
2. Build & testBuild, then regression tests (checking old features still work), end-to-end tests and user acceptance tests (business users try it before go-live)Does it work, and did we break anything else?
3. ApproveChange approval and a go / no-go meetingAre we truly ready, and can we undo it?
4. DeployRelease in small steps, often behind feature flags (on/off switches in the software)Can we switch it off without a new release?
5. VerifyPenny tests with real small payments, dashboards and alertsIs real money moving correctly?
6. LearnShort review of what went well and what didn’tWhat will we do differently next time?

What is a penny test?

Right after a release, the team sends a tiny real payment.

  • Usually between one cent and one dollar.
  • It proves live certificates, routing and firewall rules work.
  • Test environments can’t prove that.
Read more: What is a penny test?

A penny test is a tiny real payment, usually between one cent and one dollar, sent through the live system right after a release. It proves real money moves correctly from end to end, through real bank links and real accounts, before customer payments start.

Test environments can’t prove everything. Live certificates, network settings, bank routing and firewall rules only exist in production. A penny test is the cheapest way to find a problem there, while the stakes are one cent instead of millions.

How a penny test works:

  1. Use your own accounts. Send from and to company-owned test accounts, never a customer’s.
  2. Test every changed path. If the release touched wires, ACH, RTP, FedNow or cards, send one small payment through each.
  3. Check every step. The payment is accepted, debited and credited; statuses and messages are correct; fees are right; the ledger and reconciliation match; notifications go out.
  4. Return the money and keep the evidence. Reverse or return the payment, and save screenshots and references for audit.
  5. Then open the gates. Only after a clean penny test do you switch the change on for customers, or widen the feature flag.

Tip: decide before release night who sends the penny test, who checks it, and which result means “go” and which means “roll back”. At 2 a.m. is not the time to debate it.

Verifying each payment type: online and batch

Each payment type moves money differently, so each needs its own checks.

  • Online checks: in the first minutes after go-live.
  • Batch checks: after the next processing cycle.
  • ACH, wires and instant payments each have their own checks.
Read more: Verifying each payment type: online and batch

Every payment type moves money differently, so each needs its own checks. And verification happens in two layers:

  • Online checks happen in the first minutes after go-live: penny tests and live dashboards. They answer: does it work right now?
  • Batch checks happen after the next processing cycle or at end of day: reconciliation, returns and exceptions. They answer: did everything settle and balance?

A release isn’t finished until both layers pass.

Payment typeHow it movesOnline check (minutes after release)Batch check (next cycle / end of day)Watch out for
ACHIn batches: payments are grouped into files and sent in processing windowsThe file builds and passes format checks; batch counts and totals balance; the bank or operator accepts the fileSettlement is confirmed; returns and change notices are processed; entry counts and amounts reconcileProblems can show up a day later, through returns. Don’t release just before a file cut-off.
Wire (Fedwire, SWIFT, the secure network banks use to send payments abroad)One at a time, in real time, during operating hours; final once sentA penny wire is sent and confirmed with its reference; ISO 20022 fields are correct (like structured addresses); sanctions screening and limits workEnd-of-day reconciliation against the bank’s statement; no wires stuck in the repair queueWires can’t be pulled back. Use only your own accounts, and finish well before the daily cut-off.
RTP / FedNowInstantly, 24/7, one at a time; final once acceptedA penny payment is accepted within seconds; the status message coming back (called pacs.002) is correct; the receiver’s account is credited at once; time-outs and rejects are handledThe total owed between banks (the settlement position) matches; no payments stuck as pending; limits and balances are correctThere’s no quiet night. Release at the lowest volume, behind a feature flag, with someone watching live.

The live dashboard to watch during online verification, for every payment type:

  • Acceptance rate: should match normal levels within minutes.
  • Rejects by reason code: any new or growing reason code is a red flag.
  • Speed: real-time payments should still complete in seconds.
  • Queues: nothing piling up in repair, pending or exception queues.

The batch rule: the release is only closed when reconciliation shows zero unexplained breaks (records that don’t match). Every payment sent must match what settled, to the cent.

Who does what

Everyone has a clear role.

  • Release manager: owns the calendar and runs go / no-go.
  • Product owner: confirms the scope and that the business is ready.
  • Engineering and QA: build, test and prove the rollback.
  • Operations, risk and compliance: watch, approve and record.
Read more: Who does what
  • Release manager: owns the calendar, runs go / no-go, makes the final call.
  • Product owner: confirms the scope and the business is ready.
  • Engineering and QA: build, test, and prove the rollback works.
  • Operations and support: watch the dashboards and handle customer questions.
  • Risk and compliance: confirm controls, approvals and audit evidence.

The go / no-go checklist

Before any payments release, every answer must be “yes”.

  • All critical tests passed.
  • The rollback plan has been tested, not just written.
  • The window avoids cut-offs, and monitoring is ready.
  • A penny test plan and sign-offs are in place.
Read more: The go / no-go checklist

Before any payments release, every answer should be “yes”:

  • All critical tests passed, including regression and end-to-end payment flows.
  • Network or clearing house certification is done, where needed.
  • The release window avoids cut-off times, settlement windows, month-end and payroll days.
  • Monitoring and alerts are ready, and someone is watching them.
  • Support, operations and key clients know what is changing.
  • A penny test plan is ready to prove real money moves correctly.
  • Change approval and sign-offs are recorded for audit.

If any answer is “no”, the decision is “no-go”. Moving a date is cheap. Fixing a failed payment release is not.

The release documentation checklist

In payments, if it isn’t written down, it didn’t happen.

  • Release plan and change request.
  • Test evidence and UAT sign-off.
  • Security, compliance and certification evidence.
Read more: The release documentation checklist

In payments, if it isn’t written down, it didn’t happen. Auditors, regulators and your own team will ask for proof. These are the documents every release should have, matched to the six stages above:

#DocumentWhat it coversOwnerStage
1Release planScope, version, date, release window, ownersRelease managerPlan
2Change requestWhat is changing, why, risk level and impactRelease managerPlan
3Test evidenceTest plan, regression and end-to-end results, defects found and fixedQA leadBuild & test
4UAT sign-offBusiness confirms it works the way they needProduct ownerBuild & test
5Security and compliance checksSecurity scans, open-source checks, control evidence, access approvalsSecurity / riskBuild & test
6Certification evidenceNetwork or clearing house tests, message format changes (for example ISO 20022), where neededEngineeringBuild & test
7Deployment runbookStep-by-step plan with timings and who does each stepEngineering / opsApprove
8Rollback planHow to undo the change, when to trigger it, and proof it was testedEngineeringApprove
9Go / no-go recordThe decision, who attended, and every sign-offRelease managerApprove
10Communication planWho hears what and when: support, operations, clients and partnersProduct ownerApprove
11Release notesWhat changed, in plain English, plus any known issuesProduct ownerDeploy
12Penny test resultsWhich payments were sent, what was checked, and the outcomePayment opsVerify
13Support handoverDashboards, alerts, on-call contacts and FAQs for the support teamOperationsVerify
14Post-release reviewWhat went well, any incidents, and actions with ownersRelease managerLearn

Two simple habits make this easy:

  • One folder per release. Keep every document in one place and link it from the change ticket.
  • The one-year test. Could someone who wasn’t there rebuild the full story of this release a year from now, using only these documents? If not, something is missing.

Payments rules that make releases safer

A few simple rules keep releases calm.

  • Small beats big.
  • Ship code switched off, behind feature flags.
  • Respect the payments calendar.
  • Agree in advance who calls a rollback.
Read more: Payments rules that make releases safer
  • Small beats big. Release in small pieces instead of one “big bang”. If something fails, less breaks.
  • Feature flags. Ship the code switched off, then turn it on for a few customers first.
  • Respect the payments calendar. Avoid holidays, month-end, payroll days and network maintenance windows.
  • Stay backward compatible. When message formats change, like ISO 20022 fields, old and new versions must work side by side for a while.
  • Bad news travels fast. Agree in advance who calls a rollback (going back to the previous version if something breaks) and how quickly. More on this in In Payments, Bad News Must Travel Faster Than Money.

Measure it: four numbers

Four numbers show how well you release.

  • How often you release.
  • How long finished code takes to go live.
  • How many releases fail.
  • How fast you recover.
Read more: Measure it: four numbers

Google’s DORA (DevOps Research and Assessment) team has studied software delivery for over ten years. It tracks four numbers. Here is what the best teams in the 2024 report achieve:

NumberWhat it meansBest teams
Deployment frequencyHow often you releaseOn demand, many times a day
Lead timeTime from finished code to liveLess than a day
Change failure rateShare of releases that need a fix or rollback5% or lower
Recovery timeTime to recover from a failed releaseLess than an hour

The surprising part: the fastest teams are also the safest. Compared with the lowest group, the best teams fail 8 times less often and recover from failed releases about 2,293 times faster. Speed and safety come from the same thing: small, well-tested, well-practiced releases.

Payments teams don’t need to release many times a day. But they should know their four numbers and improve them every quarter.

My take

Good release management in payments isn’t heroics on release night. It’s a calm, repeatable process: plan, test, approve, deploy small, verify with real money, and learn. Want to see how this runs week by week? Read my release manager’s playbook for shipping every week.

Working on a payments project?

I help banks and fintechs deliver real-time payments, ISO 20022, and AI-driven AML and compliance change, from strategy to go-live.

Connect on LinkedInSee how I can help

Leave a Reply

Your email address will not be published. Required fields are marked *