Visa benchmark finds AI shopping agents fail almost half of test runs
For the head of card disputes at a regional bank
Visa said in a press release on Nov. 5 that it launched Visa Agentic Commerce Benchmark, a test in which 22 AI models from 12 labs failed 42.8% of runs across 15 shopping challenges.12
Visa said shopping is hard for AI because consumers often do not know all their requirements at the outset, so models must identify needs through conversation and decline to buy if nothing fits.12 To pass a run, a model must uncover every task-defined customer requirement and make a correct purchase or no-purchase decision, according to Visa's release.1
The best models passed 82.2% of attempts and the worst passed 28.9%, with an overall pass rate of 57.2%, Visa said.12
About 54% of failed runs still ended in the AI making a purchase, either one that did not meet the customer's identified requirements or one made without uncovering all requirements, according to Visa.12
Every model got three attempts at each scenario, and all three attempts passed 34.2% of the time across all models and scenarios.12
The five models that uncovered the most hidden buying factors averaged a 70.2% pass rate, against 44.4% for the five that uncovered the least; the five with the highest average cost per run passed 67.1% of runs, against 50.2% for the five least expensive.12
The test stops before checkout and payment, so it does not show how failed purchases would play out in disputes or liability. Banks and fintechs building or accepting agent-initiated payments should expect agents to make wrong purchases that the customer did authorize, and should plan for that in product design and dispute handling. This is a Visa-run lab test, and Visa has shared it with model providers for feedback.
Our take. AI agents in Visa's test failed 42.8% of runs, and 54% of those failures still bought something. Your cardholder authorized the agent, not the TV. Decide now which side of that line your chargeback rules put the loss.
Action. Card disputes: Map agent-initiated purchases that miss the customer's stated needs to a reason code, and set who bears the loss: issuer, merchant or cardholder.
From our records
- Visafintech · company Card network, founded 1958, United States · bank partners Barclays US, DBS, BBVA
- Commerce Bancsharesbank · latest Nolan & Associates acquisition completed (Oct 2026)