Nvidia Nemotron 3.5 vs Claude API: Self-Host or Not?

By Lazar Milicevic · Published October 20, 2026 · 10 min read

You run a small business or a small team, and you handle client documents, invoices, contracts, or support tickets you'd rather not send to a third-party API. Nvidia just released open models built for exactly that worry, and now you're asking whether to self-host an agent or keep paying per token. This post gives you the decision framework, the cost math you can run yourself, and a working setup to test it.

What Nvidia actually released (and what is still a claim)

On October 15, 2026, Nvidia expanded its Nemotron family with three models, according to Nvidia's announcement:

They join the existing family: Lightning (August) for high-throughput work, Ultra for the hardest reasoning, and Bolt for the edge.

A few details matter for a small shop:

• Super VL is multimodal. It reads images, webpages and PDFs, and Nvidia targets OCR, tables, charts and structured data in visual documents. That makes it a direct fit for document pipelines.
• It's a mixture-of-experts model. 120B total parameters, 12B active per token, per Unite.AI's coverage. You need memory for the full weights, but compute per token behaves more like a 12B model.
• It ships open. Nvidia says open weights, open datasets and training recipes, with customization through NeMo and Python pipelines.
• Where you can get it. Super VL is on Hugging Face, OpenRouter, ModelScope and as an NVIDIA NIM. Bolt 74B-A9B is on Hugging Face with support for llama.cpp, Ollama, LM Studio, Unsloth and vLLM.
• Who's using it. TechCrunch reports Nvidia named Skydance, EY, Ramp, CrowdStrike, Perplexity and Salesforce as users of Super VL.

Now the part you should hold loosely. Almost every performance number comes from Nvidia:

• Super VL matching its largest reasoning model on multimodal agentic tasks at roughly one-fifth the parameter count.
• Up to 5x higher throughput than comparable models on the Artificial Analysis Intelligence Index leaderboard.
• Up to 8x more users served at 85% lower latency on NVIDIA Dynamo.
• 63.44% on Hermes Bench and 87.05% on LangChain behavioral evaluations, versus 56.33% and 84.3% for Nemotron 3 Ultra.
• Bolt 74B-A9B ranking #1 among open models in its class on PinchBench and #2 on BrowseComp.

Independent measurement is thin so far. Artificial Analysis has published its own measurements, including output speed on a pre-release endpoint and an Intelligence Index score, which partially backs the speed claim. Treat the other numbers above, such as the Hermes Bench and PinchBench results or the Dynamo throughput and latency figures, as vendor claims until someone outside Nvidia reproduces them. The same applies to the TechCrunch report that Nvidia says a post-trained Super VL finishes certain tasks faster than competing Anthropic and OpenAI models. Notice the qualifier: post-trained and task-specific. That's a tuned specialist against general-purpose models, which is a different comparison than "drop-in replacement."

The real question: what are you protecting?

Nobody should self-host a 120B model because it's trendy. The decision comes down to three things: data sensitivity, volume, and who maintains it.

Nvidia's pitch is that organizations can run Super VL wherever security and data-residency rules require, with its example being a customized insurance or financial-services agent where the model and data stay on premises. That's a legitimate need. Here is how it maps to small-business situations:

Self-hosting earns serious consideration when:

• A contract or regulation says client data can't leave your environment, and a vendor's data-handling terms don't satisfy it.
• You process high, steady volume of a narrow task (the same document type, the same extraction schema) where a tuned smaller model can do the job.
• You need the agent to work offline or on a laptop in the field.
• You want to fine-tune on your own labeled data.

An API is usually the better default when:

• Volume is low or spiky. Idle GPUs cost money; idle API access costs nothing.
• The task needs broad judgment: ambiguous emails, messy multi-step reasoning, anything where quality beats cost.
• Nobody on your team wants to be on call for an inference server.
• You're still figuring out what the workflow should even be.

A middle path exists and is often the right answer: route by sensitivity. Send the regulated documents to a local model and everything else to a hosted API. More on that below.

One caution on privacy. "Open weights" doesn't automatically mean "private." Privacy comes from where inference runs and what leaves that box. A NIM container on your own server is private. Super VL called through OpenRouter is still a third-party hop, which is fine for some data and a non-starter for others. Read where each path sends your bytes before you assume.

Cost math you can run yourself

I won't quote prices here. For NIM licensing in production, ask Nvidia or your reseller for a current quote, and check the model card on Hugging Face for the license terms and any commercial-use conditions before you build anything on this.

What I can give you is the structure of the comparison. Fill in current numbers from the vendors' pricing pages:

# Break-even sketch: self-hosted vs. API.
# Every number below is a PLACEHOLDER. Replace with current quotes.

# --- API side ---
requests_per_month = 20_000          # your real volume
avg_input_tokens   = 6_000           # a typical multi-page doc
avg_output_tokens  = 800
api_in_per_mtok    = 0.0             # fill from the pricing page
api_out_per_mtok   = 0.0             # fill from the pricing page

api_monthly = requests_per_month * (
    avg_input_tokens  / 1_000_000 * api_in_per_mtok +
    avg_output_tokens / 1_000_000 * api_out_per_mtok
)

# --- Self-hosted side ---
hardware_cost      = 0.0             # GPU server or workstation, amortized
amortize_months    = 36
power_and_hosting  = 0.0             # monthly
license_cost       = 0.0             # monthly; check the model card + Nvidia
eng_hours_per_mo   = 6               # patching, monitoring, upgrades
eng_hourly_rate    = 0.0             # your time has a price

self_monthly = (
    hardware_cost / amortize_months
    + power_and_hosting
    + license_cost
    + eng_hours_per_mo * eng_hourly_rate
)

print(f"API:  ${api_monthly:,.2f}/mo")
print(f"Self: ${self_monthly:,.2f}/mo")
print("Self-host wins" if self_monthly < api_monthly else "API wins")

Two things people forget in this calculation:

1. Engineering time is a real line item. For a solopreneur, six hours a month of server babysitting is six hours not spent on clients. Put a number on it.
2. Utilization decides everything. A GPU running at 10% load looks very different from one at 70%. If your pipeline processes a batch overnight and sits idle otherwise, the API's pay-per-use model usually wins unless your sensitivity requirement forces the issue.

Run the sketch with three scenarios: current volume, 3x volume, and one-third volume. If the answer flips between scenarios, your decision is really about growth assumptions, and you should say so out loud before buying hardware.

Can it run on hardware you already own?

This is where I'd slow down. Nvidia says Bolt is built for its RTX Spark laptops (which it said arrive October 16), and also runs on GeForce RTX PCs, DGX Spark, DGX Station and RTX PRO workstations. Bolt 74B-A9B uses native 2-bit quantization to fit in 24GB and run alongside other applications. TechCrunch notes the RTX Spark machines are coming through partners including Microsoft and Dell.

What I could not find: independent measurements of real-world speed, memory pressure, or total cost on a typical small-business PC. Nvidia's description is promising. A claim that it runs "alongside other applications" in 24GB deserves a test on your own workload before you tell a client it works. Don't assume your current laptop qualifies; check the model card's stated requirements.

The honest test is cheap. Pull the Bolt 74B-A9B weights from Hugging Face and try them in a runtime you already know:

# Option A: Ollama (check the model card for the exact published tag)
ollama run <nemotron-bolt-tag-from-model-card>

# Option B: llama.cpp server with a GGUF from the model card
./llama-server -m ./nemotron-bolt-74b-a9b.gguf -c 8192 --port 8080

# Option C: vLLM for a proper OpenAI-compatible endpoint
vllm serve <hugging-face-repo-id> --max-model-len 8192

I've deliberately left the tags and repo IDs as placeholders. Look them up on the Hugging Face model page rather than trusting a guess from me. Once any of these is serving, you can point existing code at it because they expose OpenAI-compatible endpoints:

from openai import OpenAI

# Local server; no data leaves this machine
local = OpenAI(base_url="localhost api_key="not-needed")

resp = local.chat.completions.create(
    model="nemotron-bolt",   # whatever name your server registers
    messages=[
        {"role": "system", "content": "Extract invoice fields as JSON."},
        {"role": "user", "content": invoice_text},
    ],
    temperature=0,
)
print(resp.choices[0].message.content)

Run your own 30 to 50 real documents through it and score the output against what a human would extract. That's worth more than any leaderboard.

Where document pipelines fit

Super VL's visual-document focus is the most relevant piece for small businesses. Invoices, receipts, scanned contracts, intake forms, bank statements: the daily grind is turning PDFs and images into structured data. Nvidia targets OCR, tables and charts specifically.

A pipeline for this has a predictable shape regardless of which model sits in the middle:

pipeline: invoice_intake
steps:
  - name: ingest
    source: shared_inbox        # e.g., a monitored mailbox or folder
  - name: classify
    model: small_local          # cheap: is this an invoice, receipt, or other?
  - name: extract
    router:
      sensitive: local_vlm      # self-hosted multimodal model
      routine: hosted_api       # API for non-sensitive docs
    output_schema: invoice_v1   # strict JSON schema
  - name: validate
    rules:
      - totals_match_line_items
      - vendor_in_known_list_or_flag
  - name: human_review
    when: confidence_below_threshold_or_validation_failed
  - name: write
    target: accounting_system

Notice what does the heavy lifting: the validation and human-review steps, not the model. A wrong extraction that gets written to your books silently costs far more than an API bill. Whichever model you choose, build the checks (do the line items sum to the total? is the vendor one you've seen?) and keep a person in the loop for low-confidence results. A model swap is a configuration change in a pipeline built this way; a pipeline with no validation can't be rescued by any model.

A routing pattern that avoids the either/or

You don't have to pick one side. The setup that fits most small teams is a router that decides per request:

def choose_backend(doc):
    if doc.contains_regulated_data or doc.client_requires_onprem:
        return "local"          # self-hosted open model
    if doc.complexity == "high" or doc.needs_broad_reasoning:
        return "hosted_api"     # stronger general model
    return "local" if local_has_capacity() else "hosted_api"

The sensitivity flag should come from something deterministic (client contract tags, document source, a classifier you've tested), not from the model deciding for itself. And log which backend handled each request, so when a client asks "did my data leave your environment?" you can answer with a record instead of a guess.

This pattern also protects you from the churn in this market. Models and rankings reshuffle quickly; Nvidia's own release is another entry in a crowded field. If your pipeline treats the model as a swappable component behind a stable interface, a new release is an afternoon of re-testing rather than a rebuild.

What to verify before you commit

Before you move any real workload, close these gaps yourself, because I couldn't close them from public sources:

• License terms. Read the model card for the license name and commercial-use conditions. Don't rely on secondhand summaries.
• Pricing and enterprise licensing. If you plan to use NIM in production, request a current quote from Nvidia or your reseller.
• Context window. Third-party trackers such as Artificial Analysis list a figure, but that is a single outside source. Confirm it on the model card, and test with your longest real document.
• Independent benchmarks. Wait for, or run, tests on your own tasks. Vendor-versus-competitor speed claims apply to tuned setups and aren't a prediction of what you'll see out of the box.
• Bolt E4B availability. It's scheduled for October 23, 2026 and isn't out as of this writing. Its edge use cases, per SiliconANGLE, include wearables, point-of-service devices and car dashboards, which matters more to product builders than to most SMB back offices.

A sensible evaluation takes a week: pick one real workflow, collect 50 representative documents, run them through the local model and your current API option, score both against human-verified answers, then fill in the cost sketch with real quotes. If the local model clears your accuracy bar and your privacy requirement is real, you have a case. If it doesn't clear the bar, you've spent a week instead of a quarter of hardware budget.

Where BizFlowAI fits

BizFlowAI is about practical AI automation for solopreneurs and small teams. Whether the model in your pipeline is self-hosted, a hosted API, or a routed mix of the two, the useful first step is running the cost and privacy math on your actual numbers rather than a vendor's.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Request a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.