Insights Contact

Case Studies

Case Study: Payment Outages and Release Safety

What 17 major payment outages from 2014 to 2026 have in common, and a 90-day release-safety plan a bank can run to cut their impact.

← All case studies

Case study · Operational resilience

Most payment outages start with a change we made ourselves

I studied 17 major public payment outages from 2014 to 2026. Seven came from a software or system change and four from a vendor or telecom provider. This case study turns that pattern into a release-safety plan a bank can run in 90 days.

17major payment outages studied
7caused by a software or system change
4caused by vendors or telecom providers
12years of incidents, 2014 to 2026

The situation

Payments now run around the clock, and customers notice a failure within minutes. When a large bank goes down, it can look like an industry outage, because other banks cannot complete payments with it.

  • United States, Nov 16, 2026. JPMorgan Chase said a software bug in on-premise hardware systems slowed transactions. Customers at other banks saw transfer and Zelle delays (Associated Press).
  • Global, Jul 19, 2024. A faulty CrowdStrike security update crashed Windows systems at banks, payment firms and merchants at the same time.
  • India, Apr 12, 2025. NPCI linked a UPI outage to a surge in API calls from banks.
  • United Kingdom, Apr 22, 2018. TSB moved customers to a new platform in one go and many could not use their accounts. Regulators later fined the bank £48.65 million.

What the 17 incidents tell us

7 of 17

Change risk

Upgrades, migrations, configuration and automation errors. The failure often shows up hours after release, at peak load.

4 of 17

Third parties

One vendor or telecom update stops many firms at once, and you cannot fix it yourself.

6 of 17

Load and recovery

Hardware, network, capacity, cyber and undisclosed causes. Backups that did not take over cleanly, and retries that turned a slowdown into an outage.

What caused the 17 outages

Count of major public payment outages by stated cause, 2014 to 2026

Software or system change7 · 41%
Third-party vendor or telecom4 · 24%
Hardware failure2 · 12%
Network configuration1 · 6%
Cyberattack1 · 6%
Traffic surge and retries1 · 6%
Not disclosed1 · 6%

Outages by year

Major public payment outages in the study, by year (years with none are left out)

1
2
1
1
3
2
5
2
20142018202120222023202420252026

The approach: a 90-day release-safety plan

The release-safety loop

Every payment change goes round this loop; no step is skipped for a “small” release

1AssessImpact on critical journeys and vendors
2Prove the way backRollback rehearsed, feature flag ready
3Release in wavesSmall group first, go/no-go at each wave
4Watch customersSuccess rate, response time, queues
5Continue or roll backDecide in minutes, then review and learn

↺ Lessons from step 5 feed the next assessment

Weeks 1 to 2

Know the journeys customers feel first

List the critical journeys: login, balances, transfers, card authorisation, instant payments and wires. Set a target for each (success rate, response time, longest acceptable outage) and map every system and vendor behind it.

Weeks 3 to 6

Make every change reversible

An impact assessment for each release. A rollback rehearsed before go-live, not written after. No big-bang cutovers: move customers in waves with a go/no-go at each wave. Feature flags to switch new code off without a redeploy.

Weeks 3 to 8

Test what breaks in production

Automated regression on the critical journeys for every release. Performance tests at twice the expected peak, including payday and holiday dates. Failover tests that prove the backup really takes over. Retry tests so apps back off instead of flooding the system. Penny tests in production right after release.

Weeks 5 to 8

Treat vendors as part of your system

Rank critical vendors. Roll vendor updates out in stages, a pilot group first. Contract for fast incident notice and root-cause reports. Practise running without each critical vendor.

Weeks 7 to 10

Detect early and talk fast

Monitor what customers feel (payment success rate, response time, queue depth) and peer status pages, not just servers. Send the first customer message within 30 minutes, with scripts ready for branches, call centres and social media.

Weeks 10 to 13

Learn and prove it

A blameless review within 10 working days, fixes tracked to closure, and the evidence regulators ask for kept in one place.

Controls by failure type

Failure typeReal exampleControl that reduces impactEvidence to keep
Software or system changeChase, Nov 2026Staged rollout, rehearsed rollback, feature flagsRelease record, rollback test result
Platform migrationTSB, Apr 2018Migrate in waves with go/no-go gates and a tested way backWave plan, gate sign-offs
Vendor updateCrowdStrike, Jul 2024Staged vendor updates, a tested way to run without the vendorVendor register, drill results
Traffic surge and retriesUPI, Apr 2025Load tests at twice peak, back-off rules, rate limitsLoad test report, rate-limit settings
Hardware failureVisa Europe, Jun 2018Regular failover drills under real loadFailover test report
Network configurationSquare, Sep 2023Peer-reviewed configuration changes and automated checksChange approvals, test logs

Numbers to watch

Payment success ratePer critical journey, every minute
Time to detectMinutes from first failure to alert
Time to restoreMinutes from alert to normal service
Change failure rateShare of releases that cause an incident
Tested rollbacksTarget: 100% of payment releases
Vendor failover testedTarget: every critical vendor, every year

If I had one week: pick the three journeys customers feel first, check that each has a tested rollback and a working failover, and alert on payment success rate rather than server health. That alone shortens most of the outages in this study.

Sources

Get the case studies as a PDF, free

Take this case study and the other two with you, to share with your team.

  • All three case studies with charts, plans, risks and controls (14 pages)
  • Plus the Cloud vs On-Prem study of 29 US banks (10 slides)

Enter your email and confirm it. The download links arrive straight away.

I use your email to send the case studies and, now and then, payments insights. Nothing is sold or shared. Every email has a one-click unsubscribe link, and you can ask me to delete your details at any time at sp@paymentconsultant.top. Didn’t receive it? Check your spam folder, or email me and I’ll send it personally.

Want this plan for your bank or fintech?

I help banks and fintechs launch and change payment products safely: discovery, delivery plans, testing, release and go-live.

Book a 15-minute callOpen the incident tracker

Industry case study built from public sources and my payments delivery experience. It does not describe any client engagement. Figures are as published by the sources listed; general guidance, not legal advice.