Case study · Operational resilience
Most payment outages start with a change we made ourselves
I studied 17 major public payment outages from 2014 to 2026. Seven came from a software or system change and four from a vendor or telecom provider. This case study turns that pattern into a release-safety plan a bank can run in 90 days.
The situation
Payments now run around the clock, and customers notice a failure within minutes. When a large bank goes down, it can look like an industry outage, because other banks cannot complete payments with it.
- United States, Nov 16, 2026. JPMorgan Chase said a software bug in on-premise hardware systems slowed transactions. Customers at other banks saw transfer and Zelle delays (Associated Press).
- Global, Jul 19, 2024. A faulty CrowdStrike security update crashed Windows systems at banks, payment firms and merchants at the same time.
- India, Apr 12, 2025. NPCI linked a UPI outage to a surge in API calls from banks.
- United Kingdom, Apr 22, 2018. TSB moved customers to a new platform in one go and many could not use their accounts. Regulators later fined the bank £48.65 million.
What the 17 incidents tell us
Change risk
Upgrades, migrations, configuration and automation errors. The failure often shows up hours after release, at peak load.
Third parties
One vendor or telecom update stops many firms at once, and you cannot fix it yourself.
Load and recovery
Hardware, network, capacity, cyber and undisclosed causes. Backups that did not take over cleanly, and retries that turned a slowdown into an outage.
What caused the 17 outages
Count of major public payment outages by stated cause, 2014 to 2026
Outages by year
Major public payment outages in the study, by year (years with none are left out)
The approach: a 90-day release-safety plan
The release-safety loop
Every payment change goes round this loop; no step is skipped for a “small” release
↺ Lessons from step 5 feed the next assessment
Weeks 1 to 2
Know the journeys customers feel first
List the critical journeys: login, balances, transfers, card authorisation, instant payments and wires. Set a target for each (success rate, response time, longest acceptable outage) and map every system and vendor behind it.
Weeks 3 to 6
Make every change reversible
An impact assessment for each release. A rollback rehearsed before go-live, not written after. No big-bang cutovers: move customers in waves with a go/no-go at each wave. Feature flags to switch new code off without a redeploy.
Weeks 3 to 8
Test what breaks in production
Automated regression on the critical journeys for every release. Performance tests at twice the expected peak, including payday and holiday dates. Failover tests that prove the backup really takes over. Retry tests so apps back off instead of flooding the system. Penny tests in production right after release.
Weeks 5 to 8
Treat vendors as part of your system
Rank critical vendors. Roll vendor updates out in stages, a pilot group first. Contract for fast incident notice and root-cause reports. Practise running without each critical vendor.
Weeks 7 to 10
Detect early and talk fast
Monitor what customers feel (payment success rate, response time, queue depth) and peer status pages, not just servers. Send the first customer message within 30 minutes, with scripts ready for branches, call centres and social media.
Weeks 10 to 13
Learn and prove it
A blameless review within 10 working days, fixes tracked to closure, and the evidence regulators ask for kept in one place.
Controls by failure type
| Failure type | Real example | Control that reduces impact | Evidence to keep |
|---|---|---|---|
| Software or system change | Chase, Nov 2026 | Staged rollout, rehearsed rollback, feature flags | Release record, rollback test result |
| Platform migration | TSB, Apr 2018 | Migrate in waves with go/no-go gates and a tested way back | Wave plan, gate sign-offs |
| Vendor update | CrowdStrike, Jul 2024 | Staged vendor updates, a tested way to run without the vendor | Vendor register, drill results |
| Traffic surge and retries | UPI, Apr 2025 | Load tests at twice peak, back-off rules, rate limits | Load test report, rate-limit settings |
| Hardware failure | Visa Europe, Jun 2018 | Regular failover drills under real load | Failover test report |
| Network configuration | Square, Sep 2023 | Peer-reviewed configuration changes and automated checks | Change approvals, test logs |
Numbers to watch
If I had one week: pick the three journeys customers feel first, check that each has a tested rollback and a working failover, and alert on payment success rate rather than server health. That alone shortens most of the outages in this study.
Sources
- Global Payments Incident Tracker: all 17 incidents with causes and sources (snapshot as of Nov 2026)
- Associated Press: Chase outage, Nov 16, 2026
- NYSSCPA: CrowdStrike outage and financial firms
- Outlook Money: NPCI on the UPI outage
- Financial Conduct Authority: TSB fined
- Bank of England: Visa Europe outage
- Square: incident summary
Get the case studies as a PDF, free
Take this case study and the other two with you, to share with your team.
- All three case studies with charts, plans, risks and controls (14 pages)
- Plus the Cloud vs On-Prem study of 29 US banks (10 slides)
Enter your email and confirm it. The download links arrive straight away.
I use your email to send the case studies and, now and then, payments insights. Nothing is sold or shared. Every email has a one-click unsubscribe link, and you can ask me to delete your details at any time at sp@paymentconsultant.top. Didn’t receive it? Check your spam folder, or email me and I’ll send it personally.
Want this plan for your bank or fintech?
I help banks and fintechs launch and change payment products safely: discovery, delivery plans, testing, release and go-live.
Book a 15-minute callOpen the incident trackerIndustry case study built from public sources and my payments delivery experience. It does not describe any client engagement. Figures are as published by the sources listed; general guidance, not legal advice.