
⏱ 2-minute read · open “Read more” for the details
In most software, a bad release means a bug. In payments, a bad release means money in the wrong place, customers who can’t pay their bills, and regulators asking questions. That is why release management matters more in payments than almost anywhere else.
Releasing a payment change is like landing a plane: there is a checklist, a clear go or no-go decision, and a way to back out safely if something looks wrong. This post explains each step.
Release management is simply the way a team gets a change from “done” to “live” safely: planned, tested, approved, deployed, checked and learned from. Every time, in the same order.
The config file that stopped every release
Some of the most expensive mistakes in payments aren’t in the payment logic at all. One of mine was a few lines in a config file.
- Passwords for a keystore and a database were hard-coded in plain text.
- Secrets belong in a vault, never in config files.
Read more: The config file that stopped every release
The passwords for a keystore, where the bank’s certificates and keys live, and for a database had been hard-coded in plain text in the configuration files. They worked, nobody complained, and they traveled quietly from release to release. Then the compliance team noticed.
That triggered an audit, and everything stopped. Not just the change that had the passwords. Every release was frozen until the audit was done. A few lines of config had put the whole delivery pipeline on hold.
The fix itself wasn’t complicated: the passwords were taken out of the config and encrypted as per the bank’s security standard. The cost was everything else, all the releases that sat waiting while it was put right.
What I took away: no passwords in code or config, ever, and scan for them automatically on every build so a person doesn’t have to spot them. And remember that a security finding doesn’t stop one release. It stops all of them.
What happens when it goes wrong
In April 2018, the UK bank TSB moved its customers to a new platform in one big switch. It failed straight away.
- A large share of its 5.2 million customers were hit.
- The lesson: big changes need a clear, boring, repeatable release process.
Read more: What happens when it goes wrong
In April 2018, the UK bank TSB moved its customers to a new IT platform in one big switch. The platform failed straight away. A large share of its 5.2 million customers were hit, and it took until December 2018 to get back to normal. TSB paid £32.7 million to put things right for customers, and UK regulators later fined it £48.65 million. The regulators’ finding was simple: the bank did not plan and control the change properly. (The FCA is the UK’s financial regulator; the PRA is its bank safety regulator.)
The lesson isn’t “never change anything”. It’s that big changes need a clear, boring, repeatable release process.
The six stages of a payments release
A release moves through six stages, like landing a plane.
- Plan, then build and test.
- Approve it at a go / no-go meeting.
- Deploy in small steps, check with real money, then learn.
- A tested rollback is the safety net.
Read more: The six stages of a payments release
Read the boxes from left to right. A release moves from planning to learning in six steps. The dashed line underneath is the safety net: if something goes wrong while deploying or checking, the team rolls back, fixes it and tests again before trying once more.
| Stage | What happens | The question to answer |
|---|---|---|
| 1. Plan | Agree scope, date, owners and the release calendar | What is going out, when, and who owns it? |
| 2. Build & test | Build, then regression tests (checking old features still work), end-to-end tests and user acceptance tests (business users try it before go-live) | Does it work, and did we break anything else? |
| 3. Approve | Change approval and a go / no-go meeting | Are we truly ready, and can we undo it? |
| 4. Deploy | Release in small steps, often behind feature flags (on/off switches in the software) | Can we switch it off without a new release? |
| 5. Verify | Penny tests with real small payments, dashboards and alerts | Is real money moving correctly? |
| 6. Learn | Short review of what went well and what didn’t | What will we do differently next time? |
What is a penny test?
Right after a release, the team sends a tiny real payment.
- Usually between one cent and one dollar.
- It proves live certificates, routing and firewall rules work.
- Test environments can’t prove that.
Read more: What is a penny test?
A penny test is a tiny real payment, usually between one cent and one dollar, sent through the live system right after a release. It proves real money moves correctly from end to end, through real bank links and real accounts, before customer payments start.
Test environments can’t prove everything. Live certificates, network settings, bank routing and firewall rules only exist in production. A penny test is the cheapest way to find a problem there, while the stakes are one cent instead of millions.
How a penny test works:
- Use your own accounts. Send from and to company-owned test accounts, never a customer’s.
- Test every changed path. If the release touched wires, ACH, RTP, FedNow or cards, send one small payment through each.
- Check every step. The payment is accepted, debited and credited; statuses and messages are correct; fees are right; the ledger and reconciliation match; notifications go out.
- Return the money and keep the evidence. Reverse or return the payment, and save screenshots and references for audit.
- Then open the gates. Only after a clean penny test do you switch the change on for customers, or widen the feature flag.
Tip: decide before release night who sends the penny test, who checks it, and which result means “go” and which means “roll back”. At 2 a.m. is not the time to debate it.
Verifying each payment type: online and batch
Each payment type moves money differently, so each needs its own checks.
- Online checks: in the first minutes after go-live.
- Batch checks: after the next processing cycle.
- ACH, wires and instant payments each have their own checks.
Read more: Verifying each payment type: online and batch
Every payment type moves money differently, so each needs its own checks. And verification happens in two layers:
- Online checks happen in the first minutes after go-live: penny tests and live dashboards. They answer: does it work right now?
- Batch checks happen after the next processing cycle or at end of day: reconciliation, returns and exceptions. They answer: did everything settle and balance?
A release isn’t finished until both layers pass.
| Payment type | How it moves | Online check (minutes after release) | Batch check (next cycle / end of day) | Watch out for |
|---|---|---|---|---|
| ACH | In batches: payments are grouped into files and sent in processing windows | The file builds and passes format checks; batch counts and totals balance; the bank or operator accepts the file | Settlement is confirmed; returns and change notices are processed; entry counts and amounts reconcile | Problems can show up a day later, through returns. Don’t release just before a file cut-off. |
| Wire (Fedwire, SWIFT, the secure network banks use to send payments abroad) | One at a time, in real time, during operating hours; final once sent | A penny wire is sent and confirmed with its reference; ISO 20022 fields are correct (like structured addresses); sanctions screening and limits work | End-of-day reconciliation against the bank’s statement; no wires stuck in the repair queue | Wires can’t be pulled back. Use only your own accounts, and finish well before the daily cut-off. |
| RTP / FedNow | Instantly, 24/7, one at a time; final once accepted | A penny payment is accepted within seconds; the status message coming back (called pacs.002) is correct; the receiver’s account is credited at once; time-outs and rejects are handled | The total owed between banks (the settlement position) matches; no payments stuck as pending; limits and balances are correct | There’s no quiet night. Release at the lowest volume, behind a feature flag, with someone watching live. |
The live dashboard to watch during online verification, for every payment type:
- Acceptance rate: should match normal levels within minutes.
- Rejects by reason code: any new or growing reason code is a red flag.
- Speed: real-time payments should still complete in seconds.
- Queues: nothing piling up in repair, pending or exception queues.
The batch rule: the release is only closed when reconciliation shows zero unexplained breaks (records that don’t match). Every payment sent must match what settled, to the cent.
Who does what
Everyone has a clear role.
- Release manager: owns the calendar and runs go / no-go.
- Product owner: confirms the scope and that the business is ready.
- Engineering and QA: build, test and prove the rollback.
- Operations, risk and compliance: watch, approve and record.
Read more: Who does what
- Release manager: owns the calendar, runs go / no-go, makes the final call.
- Product owner: confirms the scope and the business is ready.
- Engineering and QA: build, test, and prove the rollback works.
- Operations and support: watch the dashboards and handle customer questions.
- Risk and compliance: confirm controls, approvals and audit evidence.
The go / no-go checklist
Before any payments release, every answer must be “yes”.
- All critical tests passed.
- The rollback plan has been tested, not just written.
- The window avoids cut-offs, and monitoring is ready.
- A penny test plan and sign-offs are in place.
Read more: The go / no-go checklist
Before any payments release, every answer should be “yes”:
- All critical tests passed, including regression and end-to-end payment flows.
- Network or clearing house certification is done, where needed.
- The release window avoids cut-off times, settlement windows, month-end and payroll days.
- Monitoring and alerts are ready, and someone is watching them.
- Support, operations and key clients know what is changing.
- A penny test plan is ready to prove real money moves correctly.
- Change approval and sign-offs are recorded for audit.
If any answer is “no”, the decision is “no-go”. Moving a date is cheap. Fixing a failed payment release is not.
The release documentation checklist
In payments, if it isn’t written down, it didn’t happen.
- Release plan and change request.
- Test evidence and UAT sign-off.
- Security, compliance and certification evidence.
Read more: The release documentation checklist
In payments, if it isn’t written down, it didn’t happen. Auditors, regulators and your own team will ask for proof. These are the documents every release should have, matched to the six stages above:
| # | Document | What it covers | Owner | Stage |
|---|---|---|---|---|
| 1 | Release plan | Scope, version, date, release window, owners | Release manager | Plan |
| 2 | Change request | What is changing, why, risk level and impact | Release manager | Plan |
| 3 | Test evidence | Test plan, regression and end-to-end results, defects found and fixed | QA lead | Build & test |
| 4 | UAT sign-off | Business confirms it works the way they need | Product owner | Build & test |
| 5 | Security and compliance checks | Security scans, open-source checks, control evidence, access approvals | Security / risk | Build & test |
| 6 | Certification evidence | Network or clearing house tests, message format changes (for example ISO 20022), where needed | Engineering | Build & test |
| 7 | Deployment runbook | Step-by-step plan with timings and who does each step | Engineering / ops | Approve |
| 8 | Rollback plan | How to undo the change, when to trigger it, and proof it was tested | Engineering | Approve |
| 9 | Go / no-go record | The decision, who attended, and every sign-off | Release manager | Approve |
| 10 | Communication plan | Who hears what and when: support, operations, clients and partners | Product owner | Approve |
| 11 | Release notes | What changed, in plain English, plus any known issues | Product owner | Deploy |
| 12 | Penny test results | Which payments were sent, what was checked, and the outcome | Payment ops | Verify |
| 13 | Support handover | Dashboards, alerts, on-call contacts and FAQs for the support team | Operations | Verify |
| 14 | Post-release review | What went well, any incidents, and actions with owners | Release manager | Learn |
Two simple habits make this easy:
- One folder per release. Keep every document in one place and link it from the change ticket.
- The one-year test. Could someone who wasn’t there rebuild the full story of this release a year from now, using only these documents? If not, something is missing.
Payments rules that make releases safer
A few simple rules keep releases calm.
- Small beats big.
- Ship code switched off, behind feature flags.
- Respect the payments calendar.
- Agree in advance who calls a rollback.
Read more: Payments rules that make releases safer
- Small beats big. Release in small pieces instead of one “big bang”. If something fails, less breaks.
- Feature flags. Ship the code switched off, then turn it on for a few customers first.
- Respect the payments calendar. Avoid holidays, month-end, payroll days and network maintenance windows.
- Stay backward compatible. When message formats change, like ISO 20022 fields, old and new versions must work side by side for a while.
- Bad news travels fast. Agree in advance who calls a rollback (going back to the previous version if something breaks) and how quickly. More on this in In Payments, Bad News Must Travel Faster Than Money.
Measure it: four numbers
Four numbers show how well you release.
- How often you release.
- How long finished code takes to go live.
- How many releases fail.
- How fast you recover.
Read more: Measure it: four numbers
Google’s DORA (DevOps Research and Assessment) team has studied software delivery for over ten years. It tracks four numbers. Here is what the best teams in the 2024 report achieve:
| Number | What it means | Best teams |
|---|---|---|
| Deployment frequency | How often you release | On demand, many times a day |
| Lead time | Time from finished code to live | Less than a day |
| Change failure rate | Share of releases that need a fix or rollback | 5% or lower |
| Recovery time | Time to recover from a failed release | Less than an hour |
The surprising part: the fastest teams are also the safest. Compared with the lowest group, the best teams fail 8 times less often and recover from failed releases about 2,293 times faster. Speed and safety come from the same thing: small, well-tested, well-practiced releases.
Payments teams don’t need to release many times a day. But they should know their four numbers and improve them every quarter.
My take
Good release management in payments isn’t heroics on release night. It’s a calm, repeatable process: plan, test, approve, deploy small, verify with real money, and learn. Want to see how this runs week by week? Read my release manager’s playbook for shipping every week.
Working on a payments project?
I help banks and fintechs deliver real-time payments, ISO 20022, and AI-driven AML and compliance change, from strategy to go-live.
Connect on LinkedInSee how I can help