Visa benchmark finds AI shopping models fail 42.8% of test runs

Reliability remains uneven

Visa launched its Agentic Commerce Benchmark on 5 November after testing 22 AI models from 12 labs across 15 realistic shopping tasks. Models had to identify hidden customer requirements through conversation and either select a suitable product or decline to buy. The overall pass rate was 57.2%, meaning 42.8% of runs failed. More importantly for merchants, Visa said 54% of failed runs still ended in a purchase, either because the choice did not meet known requirements or the model had not gathered enough information. Performance was also inconsistent: all three attempts at the same scenario passed only 34.2% of the time. Visa plans to broaden the benchmark to checkout, payment and order placement. The findings measure selected test scenarios rather than every model or live merchant implementation, but they highlight a concrete risk in delegating product selection too early.