What should a bank look for in AI ai infrastructure software?
AI infrastructure is the compute and software layer under a bank's models: GPUs and other AI accelerators, the servers and racks that hold them, and the software that trains, fine-tunes and serves models. Banks buy it to run open-weight and in-house models in their own data centres or a private cloud, alongside or instead of paying per token for a hosted model. The questions that separate suppliers are cost per token at real utilisation, data residency, model and software support, lead times, and dependence on one supplier.
Observations from how these products are sold and regulated, not advice.
01
On-premises, private cloud or rented inference
Running models in the bank's own data centre keeps prompts and customer data inside its perimeter and under its residency rules, but the bank carries capacity, power, cooling and upgrades. Hosted per-token APIs move that burden to the provider. Several banks on this site now run both, sending sensitive or high-volume workloads to their own hardware.
02
Cost per token at real utilisation, not price per chip
Hardware becomes a unit cost only once utilisation, power, software licences and staff are counted. Bank-reported savings from in-house inference are usually measured on selected workloads against a per-token API price; the utilisation behind a cost-per-interaction figure decides whether it holds at scale.
03
Software stack and model support
Drivers, libraries and serving software decide which open-weight models run well and how quickly a new model can be deployed. A stack tied to one accelerator family makes a later move expensive, so the supported models, frameworks and production support terms matter as much as raw speed.
04
Supply, lead times and concentration
Accelerator supply is concentrated among a few chip designers and manufacturers, lead times have been long, and export controls can change what may be shipped where. For hardware a bank will run for several years, support continuity and the supplier's financial position are part of the purchase.
Which rules apply to a bank using AI ai infrastructure software?
| Rule | Why it applies | Documents |
|---|---|---|
| Third-party and concentration risk | Hardware, software subscriptions and support from an AI infrastructure supplier are third-party relationships under the 2023 interagency guidance; the Basel Committee's 2025 principles add nth-party supply chains and concentration risk. | OCC Bulletin 2023-17 SR 23-4 FDIC FIL-29-2023 BCBS Third-Party Risk Principles (Dec 2025) |
| Operational resilience | On-premises AI systems that support critical operations fall under business-continuity, ICT-resilience and incident expectations; in the EU, DORA treats AI systems as ICT assets and their suppliers as ICT third-party service providers. | BCBS Principles for Operational Resilience (2021) DORA (Regulation (EU) 2022/2554) |
| Hosted and private-cloud GPUs | Where accelerators are rented or hosted by a cloud provider, the ECB's cloud outsourcing guide sets expectations on data location, availability, exit strategies and audit of the provider. | ECB Guide on outsourcing cloud services |
| Frontier AI and critical technology providers | The FSB's August 2026 letter asks firms to ensure resilience among critical third-party technology providers as frontier models raise cyber risk; NIST's Generative AI Profile covers the value-chain and security risks of the models a bank serves on its own hardware. | FSB Chair's letter to G20 (Aug 2026) NIST AI 600-1 (Generative AI Profile) |
Documents link to the Regulation tracker. By use case: Generative & agentic AI · Data & privacy · Third-party & vendor AI.
How do the AI ai infrastructure vendors compare?
| Vendor | What it does | Deployment | Pricing model | Published price | US bank customers* | Founded · ownership |
|---|---|---|---|---|---|---|
| SambaNova | AI-inference chip maker selling RDU systems for on-premises use (SambaRack, SambaStack) and a per-token open-model API (SambaCloud). | On-premises systems (SambaRack, SambaStack), dedicated hosting in the cloud, or API (cloud) through SambaCloud | Per-token API pricing for SambaCloud; on-premises systems (SambaRack, SambaStack) through sales-led contracts | $0.22 input / $0.59 output per 1M tokens (SambaCloud — gpt-oss-120b) | 1 | 2017 · Private |
| NVIDIA | GPU and AI-systems maker: DGX systems plus the NVIDIA AI Enterprise stack (NeMo, NIM, Triton) banks run on-premises or in the cloud. | On-premises GPU servers and DGX systems, cloud GPU instances, and NVIDIA AI Enterprise software licensed per GPU or by the hour on cloud marketplaces | GPUs and DGX systems through NVIDIA and its partners on quotes; NVIDIA AI Enterprise software per GPU per year, as a perpetual licence, or per GPU-hour on cloud marketplaces | $4,500 per GPU (NVIDIA AI Enterprise — Self-managed subscription, 1 year (includes Business Standard support)) | 6 | 1993 · NASDAQ: NVDA |
* Banks among the 120 largest US banks on this site publicly named as customers, each sourced on the vendor's page. Prices are shown only where the vendor publishes them for the AI product itself; a price for another product (a credit score, a notary plan, contact-centre seats) is listed on the vendor's page, not here. More vendors in this category are being researched.
Private · founded 2017
SambaNova
Designs its own reconfigurable dataflow unit (RDU) AI chips and sells them as on-premises inference racks, dedicated hosted systems and a per-token cloud API, pitching banks, governments and cloud providers on running large open models on infrastructure they control.
Banks among the 120: JPMorgan
NASDAQ: NVDA · founded 1993
NVIDIA
Designs GPUs and AI systems and sells them, with a licensed software stack for training, customising and serving models, to cloud providers, AI labs and enterprises; banks run the stack in their own data centres or rent it in the cloud.
Banks among the 120: BNY, Amex, Capital One, PNC, Santander, Deutsche Bank
Why do banks buy their own GPUs instead of only using cloud AI?
The banks on this site give three reasons: keeping sensitive data and prompts inside their own perimeter, lowering the cost per token for high-volume workloads compared with per-token API prices, and reducing dependence on one model or cloud provider. Most still use hosted models as well and add their own capacity for specific workloads.
How is AI infrastructure priced?
Accelerators and systems are sold by the vendor and its partners on quotes that are rarely published; serving and development software is often licensed per GPU per year, and hosted inference per token or per GPU-hour. The comparison table shows a published price only where the vendor's own site lists one.
Which AI ai infrastructure vendors are compared here?
SambaNova (AI-inference chip maker selling RDU systems for on-premises use (SambaRack, SambaStack) and a per-token open-model API (SambaCloud)); NVIDIA (GPU and AI-systems maker: DGX systems plus the NVIDIA AI Enterprise stack (NeMo, NIM, Triton) banks run on-premises or in the cloud). Each has its own page with products, pricing, ownership and the banks among the 120 largest US banks that are named as customers. More vendors in this category are being added as their pages are researched and sourced.
When a bank picks an AI ai infrastructure vendor, you'll read it here first.
when a bank picks an AI vendor, or a vendor ships for banks, it is in the next morning's brief · six sourced stories · 7 am PT · free
plus every regulation, bank and agent page update, the morning after · leave any morning