Insights Contact

Data, AI and Testing

How to Retest an AI-Assisted Fix

By SP Sankar

⏱ 3-minute read · open “Read more” for the details

AI assistants such as GitHub Copilot can now draft a bug fix in minutes. That is real progress. But the time you save on the fix moves the risk to the retest. An AI fix often looks right, passes the obvious check, and still breaks something nearby. In payments, “nearby” can mean a customer paid twice.

When AI helps fix a bug, test the fix more carefully, not less. Prove the bug is gone, prove the new test really catches it, check nothing nearby broke, and attach the proof before you close the bug.

This post goes deeper into one part of Sell me this defect.

Why an AI fix needs a careful retest

Maya’s developer used an AI assistant to fix a payment bug in minutes. The time saved on the fix moves the risk to the retest.

  • AI doesn’t know your payment rules, cut-off times or downstream systems.
  • It can fix the symptom and miss the cause, or change more than it should.
  • It can sound confident and still be wrong.
Read more: Why an AI fix needs a careful retest

An AI tool does not know your payment rules, your cut-off times or your downstream systems. It only sees the code and the text you give it. So its fixes fail in a few typical ways:

What can go wrongWhat to check
It fixes the symptom, not the cause. The error message is gone, but the real problem is still there.Ask the developer for the root cause (the real reason behind the bug) in one sentence. If nobody can say it, the fix is not ready.
It changes more than it should. The AI “tidies up” other code while it is there.Read the list of changed files. Anything outside the bug’s area needs its own test.
Its test checks the wrong thing. The new test passes, but it would also have passed before the fix.Run the new test on the old code. It must fail there.
It misses business rules. Cut-off times, holidays, limits, rounding, currencies.Test the payment edge cases below, not only the happy path.
It sounds confident. The explanation reads well even when it is wrong.Trust the test results, not the explanation.

The retest flow

Here is Maya’s retest flow on one page.

  • If any check fails, the bug goes back to the developer with proof.
  • It is never closed “with a note”.
1Check what changedRead the fix and the files it touched2Show the bug on the old versionSame steps and the new test both fail3Show it is fixed on the new versionSame steps and the new test both pass4Check nothing else brokeNearby features and payment edge casesEverything OK?YesClose the bug and attach the proof ✓NoSend it backto the developerFix againA payment wentout twice!Sent back:fix againEverythingOK ✓Bug closed,proof attached ✓

Step by step

Maya retests in four steps.

  • Check what changed in the code.
  • Show the bug on the old version.
  • Show it is fixed on the new version, with the same steps and data.
  • Check that nothing else broke.
Read more: Step by step
  1. Check what changed. Before you test anything, read the fix: which files, screens and rules it touched, and why. Ask whether AI helped write it. That changes how closely you look, not whether you test.
  2. Show the bug on the old version. Use the build from before the fix. Repeat the original steps with the same test data, and the bug must still happen. Then run the new automated test on it, and it must fail too. A test that passes on the old version can’t prove anything was fixed. This is the most useful check for AI-written tests.
  3. Show it is fixed on the new version. Same steps, same data, same channel (mobile, web or API): the bug is gone, and the new test passes. Then try one or two small variations, such as a different amount, account type or browser.
  4. Check nothing else broke. Run the existing tests for every area the fix touched, not only the feature in the bug title. Testers call this regression testing: re-checking things that worked before. Then test the payment edge cases the fix touches, and, where the fix touches them, security, access, performance and compliance (both below).

If everything is OK, close the bug and attach the proof: test results, screenshots, logs and the version number. If anything fails, send it back to the developer with that proof.

Payment edge cases to test

Most payment bugs that reach customers come from the unhappy paths.

  • Leading zeros in account and routing numbers.
  • Duplicate requests, timeouts and retries.
  • Cut-off times, holidays and rounding.
Read more: Payment edge cases to test

Most payment bugs that reach customers come from the unhappy paths (also called edge cases): the cases where something goes wrong. Pick the ones the fix touches:

CaseWhat to tryWhat must be true
Leading zerosUse account and routing numbers that start with 0, such as 0012345678The zeros are kept everywhere: screen, API, payment file and statement.
Duplicate requestSend the same payment twice with the same idempotency key (a unique id that tells the system “this is the same request”)Only one payment is made. The second call returns the first result.
Timeout and retryMake the downstream system slow, then let the app retryNo double payment. The status is clear to the customer.
Cut-off timeSend just before and just after the daily cut-off (the latest time a payment goes out the same day)The value date (the day money moves) is correct on both sides.
Holidays and weekendsSchedule a payment for a bank holidayIt moves to the next business day, as the rules say.
Rounding and currencyUse amounts like 0.005, very large amounts and a second currencyTotals match to the cent. No amount is lost or added.
LimitsGo just under, at and just over the daily limitThe limit is applied exactly at the boundary.
Return or reversalReturn the payment after it is sentBalances, status and statements all agree.
Status with other systemsCheck the same payment in the upstream, downstream and ledger systemsEvery system shows the same status and amount.
Sensitive dataLook at the logs and error messages the fix addedNo full account or card numbers appear.

Example: the leading zero

This is a common one. A payment fails with a “number format” error. The AI suggests changing the account number field from text to a number, and the error goes away. But a number cannot start with 0, so account 0012345678 quietly becomes 12345678. The payment is rejected by the bank, or worse, goes to a different account. Account numbers, routing numbers and sort codes look like numbers but must always be kept as text. Always include at least one test account that starts with 0.

Security, access, performance and compliance

A fix can pass every functional test and still cause harm somewhere else.

  • Security: no new weakness.
  • Access: the right people can still approve, and only them.
  • Performance and compliance: no slowdown at month-end, and no rule broken.
Read more: Security, access, performance and compliance

A fix can pass every functional test and still open a security hole, let the wrong person approve a payment, slow down month-end, or break a compliance rule. AI tools help in each area, but a person still needs to check each one:

AreaHow AI helpsWhat a person still checks
SecurityScans the change for common weaknesses, such as injection (bad input that runs as code), passwords or keys left in the code, and sensitive data written to logs. Suggests a fix and tests with bad input.A developer reviews every finding. For high-risk fixes, a security tester retests. No full card or account numbers appear in logs or error messages.
Access (who can do what)Lists the roles the change touches, such as maker, checker, viewer and admin, and writes one test per role, including the “must be denied” cases.Maker-checker still works: one person creates a payment and a different person approves it. Nobody can approve their own payment. Limits by role still apply.
PerformanceCompares response times before and after the fix, spots slow database queries the fix added, and drafts a short load test (a test with many users at once).The load test runs on a production-like environment at peak volume, such as payroll day or month-end. Response times stay within the agreed limits.
ComplianceLists the rules the change may affect, such as sanctions screening (checking names against blocked lists), transaction limits, the audit trail and data privacy. Drafts the evidence pack.Screening and limits cannot be skipped. The audit trail records who did what and when. The compliance or risk team signs off when a rule is touched.

A simple rule: if the fix touches login, roles, limits, screening, logs or anything at high volume, add that area to step 4, even if the bug looked small.

Check the AI-written test

AI writes tests fast, and sometimes they pass for the wrong reason.

  • Red flag: it only checks the status code.
  • Red flag: it fakes the very thing it is testing.
  • Red flag: there is no negative case, or it retries until it passes.
Read more: Check the AI-written test

AI is good at writing tests quickly. It is also good at writing tests that pass for the wrong reason. Watch for these red flags:

  • 🚩 It only checks the status code, such as “201 Created”, and never looks at the amount, the status or the account.
  • 🚩 It mocks the thing being tested. A mock is a fake version of a system. If the payment service itself is faked, the test checks the fake.
  • 🚩 The expected values were copied from today’s output. If the code is wrong, the test now says “wrong is right”.
  • 🚩 There is no negative case. Nothing checks what happens with a bad amount, a closed account or a duplicate.
  • 🚩 It retries or skips until it passes. A flaky test (one that passes and fails at random) hides real bugs.

Here is a real-world style example in Rest-Assured (a Java library for API tests). The bug: a retried request paid the customer twice. The first test is the kind AI often writes. The second one actually proves the fix.

// Weak: passes before and after the fix
given().spec(spec).body(payment)
    .post("/payments")
    .then().statusCode(201);

// Strong: proves a retry does not pay twice
String key = UUID.randomUUID().toString();
String firstId = given().spec(spec).header("Idempotency-Key", key).body(payment)
    .post("/payments")
    .then().statusCode(201)
    .extract().path("paymentId");

given().spec(spec).header("Idempotency-Key", key).body(payment)
    .post("/payments")
    .then().statusCode(200)
    .body("paymentId", equalTo(firstId));

// and only one debit on the account
given().spec(spec).get("/accounts/{id}/debits?key=" + key, accountId)
    .then().body("size()", equalTo(1));

Run the strong test on the old build first: it must fail. More on building these tests in API automation using Rest-Assured.

How much to retest

Match the retest to how serious the bug is.

  • Critical bugs that touch money: every step, with senior sign-off.
  • Low, cosmetic bugs: repeat the original steps.
Read more: How much to retest

Not every fix needs the full list. Match the depth of the retest to the severity of the bug and to how much the fix touched money:

SeverityRetestWho signs off
S1, Critical (money, data or security at risk)All four steps, every payment edge case the fix touches, and all the tests for the payment flowTest lead and the product owner
S2, HighAll four steps, and the tests for the changed areaTest lead
S3, MediumSteps 2 and 3, and the automated testsThe tester
S4, Low (cosmetic)Repeat the original steps and check the screen on the main channelsThe tester

One rule overrides the table: if the fix touches how money is calculated or moved, treat it as S1, even if the bug itself was small.

Close with evidence

Maya closes the bug only with proof.

  • The build number and environment.
  • The new test failing on the old code.
  • The regression results, and a note that AI helped and who reviewed it.
Read more: Close with evidence

A bug closed without evidence comes back later as an argument. Attach these before you close:

  • The build number that was tested, and the environment.
  • The fail-first result: the new test failing on the old build and passing on the new one.
  • The regression run result, with a link to the report.
  • Screenshots or logs for the original steps, with sensitive data masked.
  • A note that the fix was AI-assisted, and who reviewed it. Auditors increasingly ask this.

A short closing note is enough:

Retested on build 4.18.2 (SIT). New test fails on 4.18.1 and passes on 4.18.2. Original steps pass on web and mobile. Regression for payments and limits: 214 of 214 passed. Duplicate, timeout and cut-off cases checked: one payment only. Fix was AI-assisted; code reviewed by the payments team lead. Closing.

Guardrails for payments

Some rules never bend.

  • Never paste real customer data into an AI tool.
  • AI fixes get the same code review as any other fix.
  • The reviewer isn’t the person who asked the AI.
  • AI never signs off.
Read more: Guardrails for payments
  • Never paste real customer data, account numbers or production logs into a public AI tool. Use your company’s approved tool, with masked data.
  • Every AI-suggested fix gets the same code review as any other change. In payments, that review is often a required control (a check the bank must be able to prove it did).
  • The person who reviews the fix is not the person who asked the AI for it. Two sets of eyes, as with any change to money movement.
  • Keep the AI out of the sign-off. AI can draft the closing note; a person reads the evidence and closes the bug.

Is AI helping? Measure it

Faster fixes only count if they stay fixed.

  • Reopen rate: how many fixed bugs come back.
  • Defect age: how long bugs stay open.
  • Leakage: how many bugs slip to later test levels.
Read more: Is AI helping? Measure it

Faster fixes only count if they stay fixed. Compare AI-assisted fixes with the others, using the same numbers from the weekly report:

MeasureWhat it tells you
Reopen rate (bugs reopened ÷ bugs fixed)If AI fixes are reopened more often, the retest is catching problems the review missed. Look harder at review, not only at testing.
Defect age (days a bug stays open)Should drop. If it does not, the time is going into rework.
Leakage (bugs found after a test level ÷ all bugs)If more bugs reach UAT or production after AI fixes, the retest is too light.
Fail-first rate (new tests that fail on the old build)Should be close to 100%. Lower means AI-written tests are not proving the fix.

My take

AI changes who writes the first draft of a fix. It does not change who is accountable when a customer is paid twice. The retest is where a test lead earns that trust: prove the bug is gone, prove the test can catch it, and leave evidence that anyone can check later.

Working on a payments project?

I help banks and fintechs deliver real-time payments, ISO 20022, and AI-driven AML and compliance change, from strategy to go-live.

Connect on LinkedInSee how I can help