From Ollama to vLLM Source

Markdown source • October 21, 2026

1---2title: "From Ollama to vLLM"3date: "2026-10-21"4published: true5tags: ["ai", "llm", "local-ai", "ollama", "vllm", "nvidia", "dgx-spark", "deepseek", "networking"]6author: "Gavin Jackson"7excerpt: "From 24GB Blackwell workstation cards to one 128GB DGX Spark, then two Sparks and a QSFP cable: running DeepSeek-V4-Flash-Vision-Exp locally, understanding vLLM, and planning the next step with a 200Gb/s switch."8---910# From Ollama to vLLM1112<p align="center"><img src="/assets/ai/vllm-hero-v1-two-sparks-one-cable.jpg" alt="Two engraved desktop computing units connected by a single glowing golden cable carrying sparks of data" width="800"></p>1314My local AI experiments started with the RTX PRO 4000 Blackwell cards. I wanted something useful on a developer workstation: help with code, explanations of unfamiliar systems, and an assistant that could work inside a network where sending everything to a cloud API was not an option.1516In [my earlier Blackwell article](/post/running-edge-llms-rtx-pro-4000-blackwell), the recurring constraint was **24GB of VRAM**. Ollama made running the models straightforward. The interesting work was choosing a model and quantisation that fitted, leaving room for context, and connecting it to tools such as Hermes and Continue.1718That setup taught me a pattern: get the model working directly, check the quality, then add the agent or IDE integration. When something broke, work back through those layers.1920I took essentially the same approach to a **DGX Spark with 128GB of unified memory**, still using Ollama. Suddenly I could explore models roughly five times the size of my earlier workstation baseline. Then I connected **two Sparks with a single QSFP cable**, moved into vLLM, and started running **DeepSeek-V4-Flash-Vision-Exp**, a roughly 300-billion-parameter model.2122And it is actually starting to look good.2324That is my assessment of the responses so far. I still want repeatable measurements of speed, coding quality and tool use. But the progression from fitting a useful assistant onto a workstation card to serving something this large across two small computers has been quite a journey.2526## One Spark: the same pattern, much more room2728The first Spark was a fairly comfortable transition. Pick a model, choose a sensible quant, load it into Ollama, and point the clients at it.2930The big change was the memory budget. The [Spark hardware guide](https://docs.nvidia.com/dgx/dgx-spark/hardware.html) lists **128GB of LPDDR5x unified system memory**, shared coherently by the CPU and GPU. Compared with the Blackwell card's 24GB, that is about **5.3 times the physical capacity**.3132It gave me room to move from the smaller models I had been testing into the 70-120B range I discussed in [my earlier Spark post](/post/heretic-models-dgx-spark). The exact increase in model size depends on quantisation and context; the fivefold comparison describes the scale of the opportunity, rather than a speed improvement.3334The operating system uses that memory too. So do the inference engine, temporary buffers and conversation state. A 128GB Spark does not give a model 128GB exclusively for its weights.3536Still, there was a pleasant period where I could reuse what I already knew and just try much larger models.3738Then, naturally, I wanted another Spark.3940## Two Sparks: follow the bouncing ball4142My starting points were NVIDIA's [Connect Multiple DGX Sparks playbook](https://github.com/NVIDIA/dgx-spark-playbooks/tree/main/nvidia/playbook-connect-multiple-sparks) and the [intermediate vLLM cluster guide](https://build.nvidia.com/spark/vllm-cluster/intermediate).4344I followed the bouncing ball. Connect the machines, establish the cluster network, prepare the inference environment, and work towards a model endpoint.4546The physical setup is surprisingly modest: two Sparks and one direct QSFP cable. NVIDIA documents this as a supported two-device layout, with a switch becoming an option as more devices join.4748In my network checks, the dedicated link came up at **200,000Mb/s**, with the two ends reachable at `10.0.0.1` and `10.0.0.2`. That established basic connectivity. It did not establish that inference was using the best transport or that the model was correctly distributed.4950That distinction becomes much more relevant once the network is part of generating every answer.5152The other conceptual shift was moving beyond the single-machine Ollama setup I had been using. For this experiment, I needed a serving engine that could explicitly distribute one model across two machines. That brought me into **vLLM**.5354## What is vLLM actually doing?5556[vLLM](https://github.com/vllm-project/vllm) is an open-source inference and serving engine. It loads a trained model, executes it on supported hardware, manages requests and memory, and provides an API that applications can use.5758It does not add knowledge to the model. If the responses improve because I can now run a more capable model, that capability comes from the model. vLLM makes the deployment practical.5960There are several different pieces to that practicality.6162### Splitting the model across the GPUs6364With **tensor parallelism**, supported weight tensors are divided across GPUs. Both Sparks participate in the calculation, exchanging intermediate results as inference progresses. The model's entire weight set does not have to fit on either Spark individually.6566With **pipeline parallelism**, groups of layers run in different stages. Work passes between those stages. This changes the communication pattern, but introduces pipeline scheduling and utilisation trade-offs.6768These are established model-parallel techniques, described in [NVIDIA's parallelism overview](https://docs.nvidia.com/nemo/megatron-bridge/latest/parallelisms.html) and supported by [vLLM's distributed serving implementation](https://docs.vllm.ai/en/stable/serving/parallelism_scaling/). A recipe chooses a layout that its model implementation supports; adding another computer does not make every possible split valid.6970For a two-Spark tensor-parallel deployment, each GPU holds a share of the model, along with whatever state and buffers that implementation needs. Some data can be replicated, and the split need not consume precisely equal memory.7172The useful consequence is that a model too large for one Spark can become possible across the pair.7374### Coordinating the workers7576There is a separate question: how do the processes find each other and start working together?7778Some vLLM deployments use **Ray** to manage distributed workers. Current versions also support native multi-node multiprocessing. NVIDIA's current GLM recipe uses the native route, so a separate Ray cluster is not a universal requirement.7980The relevant native-launch settings appear in [vLLM's command reference](https://docs.vllm.ai/en/latest/cli/serve/):8182| Setting | Role |83| --- | --- |84| `--tensor-parallel-size 2` | Divide supported tensors across two GPU ranks. |85| `--nnodes 2` | Participate in a two-node deployment. |86| `--node-rank 0` / `1` | Give each node a distinct identity. |87| `--master-addr` / `--master-port` | Tell the processes where to rendezvous. |88| `--headless` | Run the worker without its own public API server. |8990These are pieces of a complete recipe. The checkpoint, container, kernels and network settings still need to match.9192Worker coordination also has a different job from moving inference data. GPU collectives commonly use **NCCL**, while particular Spark recipes add specialised RoCE communication paths. Ray being healthy, or both processes starting successfully, does not prove that those transfers are efficient.9394### Using conversation memory efficiently9596Generating a response involves more than reading the weights. The engine also retains attention state for tokens already processed: the **KV cache**.9798The [original vLLM PagedAttention explanation](https://vllm.ai/blog/2023-06-20-vllm) describes managing that cache in blocks, allowing logical token sequences to use non-contiguous physical memory. This reduces the waste associated with reserving large contiguous areas for conversations whose eventual lengths are unknown.99100The broad benefit is better use of the available memory, especially when serving several requests. Modern models have different attention and recurrent-state designs, so the actual cache layout and kernels depend on the architecture.101102For my purposes, the practical lesson survives all those details: **context has a memory cost even after the model has loaded**.103104### Scheduling work and reusing it105106vLLM also supports continuous batching: active requests can share execution batches as requests arrive and finish. Chunked prefill divides long prompt processing into smaller pieces, helping the scheduler balance that work with ongoing generation. These are among the [engine's documented serving features](https://docs.vllm.ai/en/v0.20.2/).107108[Prefix caching](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/) can reuse computed state when requests begin with the same token prefix. An agent repeatedly sending the same instructions or repository background is an obvious candidate. It mainly saves prompt-processing work; it does not eliminate the cost of generating the answer.109110Some recipes also use [speculative decoding](https://docs.vllm.ai/en/latest/features/speculative_decoding/), proposing several tokens and checking them with the target model. A proposal mechanism can use a separate draft model or the model's own multi-token prediction components. Whether it helps depends on acceptance rates and execution overhead.111112This helps explain why a working launch command can be much longer than `ollama run`. There are more decisions exposed to the person operating the service.113114## How does a 300B model fit?115116The model I am running is [DeepSeek-V4-Flash-Vision-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp). Its Hugging Face page lists approximately **305B parameters**. This is the experimental multimodal release, which adds visual modules and continued training to the Flash architecture.117118Start with the weight arithmetic. Using **305 billion parameters** as a rough sizing figure, the idealised payload is:119120| Weight precision | Bytes per parameter | Weight payload |121| --- | ---: | ---: |122| BF16 / FP16 | 2 | 610GB |123| 8-bit | 1 | 305GB |124| 4-bit | 0.5 | 152.5GB |125126These are decimal GB calculations, before quantisation scales, higher-precision tensors, caches, workspaces and system overhead. Real checkpoints mix precisions and have their own storage layouts.127128Two Sparks provide **256GB of installed memory in aggregate**. A suitable four-bit checkpoint therefore has a plausible path to fitting. Its actual shards and runtime allocations must each fit within an individual Spark's budget.129130The network does not create a coherent 256GB memory pool spanning the two computers. The serving engine distributes the model and exchanges the data needed to execute it.131132Mixture-of-Experts models add another distinction: **total parameters versus parameters active for each token**. Routing selects a subset of experts, reducing computation relative to activating the entire model. For resident inference, those other experts still need storage.133134The Vision-Exp release makes this especially interesting for the work I do. Alongside text and code, visual understanding opens up experiments with screenshots, diagrams and documentation. Those are capabilities I want to evaluate; my encouraging first impressions do not establish how well it handles each of them. Image processing also adds to the runtime memory budget.135136The combination is what makes this interesting: a large pool of model weights, sparse computation, quantisation, and an engine that can divide the work between machines.137138## The cable is part of the computer now139140A 200Gb/s link has a theoretical line rate of **25GB/s**, before protocol overhead. NVIDIA lists the Spark's local memory bandwidth at **273GB/s**. Those figures describe different paths, but they illustrate why moving data between boxes has to be treated carefully.141142Every exchange has bandwidth and latency costs. For repeated small collectives, latency can matter as much as the headline link rate.143144The checks I want around a deployment now go beyond a successful ping:1451461. Verify the negotiated link speed and interface error counters.1472. Identify the ConnectX interfaces and RoCE devices that the recipe expects.1483. Confirm the inference transport is using the intended high-speed path.1494. Run communication tests before using a large model to diagnose the network.150151NVIDIA's [NCCL playbook](https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/nccl/README.md) provides communication tests for Spark topologies. The [vLLM transport guidance](https://docs.vllm.ai/en/stable/serving/parallelism_scaling/#optimizing-network-communication-for-tensor-parallelism) also explains how NCCL logs can reveal a TCP socket fallback. Recipe-specific RoCE kernels need their own checks too.152153One physical QSFP connection can expose multiple logical interfaces. That makes guessing from a device name an unreliable way to decide which interface should be used or disabled. I want the documented mapping in front of me before changing it.154155## What I want to run next156157My next substantial test is [NVIDIA's GLM-5.3-Flash recipe for two 128GB Sparks](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-spark-2gpu/index.html?model=z-ai%2Fglm-5.3-flash&os=linux&backend=vllm&memory=128&precision=fp4-local&memoryConfig=128).158159There are good alternatives worth comparing alongside it. My shortlist is based on documented deployment paths and the jobs I want these machines to do, rather than an assertion that one model wins every benchmark.160161| Candidate | Why it is on my list | Deployment evidence |162| --- | --- | --- |163| **GLM-5.3-Flash FP4-Local** | My first choice for the next large coding and agent test. The [upstream model](https://huggingface.co/zai-org/GLM-5.3-Flash) has 320B total parameters and 18B active. | NVIDIA provides the exact two-128GB-Spark recipe linked above. |164| **Qwen3.8-Flash-Next FP4-Local** | A coding, tool-use and multimodal comparison; also useful for testing whether a smaller deployment is sufficient. | The [NVIDIA checkpoint card](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local) documents Spark use, and NVIDIA's cluster guide links a two-Spark recipe. |165| **Nemotron 3 Super 120B-A12B NVFP4** | A more manageable agent and reasoning baseline, with 120B total parameters and 12B active. | The [official model instructions](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4/blob/main/README.md) include a single-Spark vLLM configuration. Two machines could run independent instances. |166| **DeepSeek-V4-Flash-Vision-Exp** | My existing roughly 305B multimodal point of comparison: promising responses, with proper task evaluation still to do. | Running across my two Sparks. Reproducing that deployment also requires the particular quantised weights and runtime settings. |167168For Qwen in particular, the model card lists a 125B main model plus additional n-gram embeddings and MTP parameters, and a checkpoint around 96GiB on disk. A headline parameter count alone is a poor guide to the deployed memory footprint.169170My order would be GLM first, then Qwen and Nemotron on the same tasks. If a model running on one Spark produces comparable results, that leaves the second machine available for another user or workload.171172### Why the precise GLM recipe matters173174The download I want is **`nvidia/GLM-5.3-Flash-FP4-Local`**. NVIDIA's [checkpoint card](https://huggingface.co/nvidia/GLM-5.3-Flash-FP4-Local) describes quantisation-aware distillation and mixed tensor precisions. It also explicitly requires a matching community vLLM container while support for its format is being integrated upstream.175176That is a stronger requirement than having four-bit weights and a recent generic container. The loader and kernels must understand that particular checkpoint.177178The selected recipe currently pins **`eugr/spark-vllm-b12x:nightly-20261015`**, uses tensor parallelism across two nodes, FP8 KV cache, and native MTP speculative decoding. Its configured request limit is **500,000 tokens** with up to four sequences; those settings are a reference configuration, not evidence that every workload at those limits will be comfortable.179180NVIDIA reports **33.94 tokens/sec decode throughput** for its SPEED-Bench qualitative coding workload, measured on **16 October 2026** on two 128GB Sparks. That result belongs to the [published recipe and workload](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-spark-2gpu/index.html?model=z-ai%2Fglm-5.3-flash&os=linux&backend=vllm&memory=128&precision=fp4-local&memoryConfig=128); it is a useful reference for my next experiment.181182I would start with a shorter context and a single request, then increase the workload while watching memory and latency.183184## Getting back to a normal chat interface185186After all that machinery, the application-facing part becomes familiar again. vLLM provides an [OpenAI-compatible serving API](https://docs.vllm.ai/en/latest/serving/online_serving/), including model discovery and chat completions.187188For example, from the head machine, once the selected recipe is running:189190```bash191curl http://localhost:8000/v1/models192193curl http://localhost:8000/v1/chat/completions \194  -H 'Content-Type: application/json' \195  -d '{196    "model": "nvidia/GLM-5.3-Flash-FP4-Local",197    "messages": [198      {"role": "user", "content": "Explain tensor parallelism using two DGX Sparks."}199    ],200    "max_tokens": 512,201    "stream": true202  }'203```204205The model name must match what `/v1/models` reports. A recipe can publish an alias instead of its Hugging Face repository name.206207A chat UI or agent that supports a custom OpenAI-compatible endpoint can then use the head node's `/v1` base URL. For coding agents, the model-specific chat template, reasoning parser and tool-call parser are part of making that integration work properly.208209I want to repeat the same testing pattern that worked with Ollama: verify the direct API first, then connect the clients.210211> ## Spinning Up vLLM for Browser Users212>213> Everything so far ends at an OpenAI-compatible API on the head node. Turning that into something an end user can reach from a web browser takes two pieces: the server has to listen on an address the LAN can reach, and a browser chat interface has to point at that address.214>215> On a single machine, the serving side is one command:216>217> ```bash218> vllm serve nvidia/GLM-5.3-Flash-FP4-Local \219>   --host 0.0.0.0 \220>   --port 8000221> ```222>223> In the two-Spark recipes, the serving step is part of the recipe itself, and only the head node publishes the API. The second node runs `--headless`, so there is nothing to connect to on it. Users always point at the head node's regular LAN address, not the `10.0.0.x` interconnect, which is reserved for inference traffic between the Sparks.224>225> For the browser side, [Open WebUI](https://github.com/open-webui/open-webui) is the quickest self-hosted chat interface I have found. It talks to OpenAI-compatible endpoints and runs as a single container:226>227> ```bash228> docker run -d --name open-webui --network host \229>   -e OPENAI_API_BASE_URL=http://<head-node>:8000/v1 \230>   -e OPENAI_API_KEY=unused \231>   -v open-webui:/app/backend/data \232>   ghcr.io/open-webui/open-webui:main233> ```234>235> Browse to `http://<open-webui-host>:8080` and register; the first account created becomes the administrator. The model picker is populated from `/v1/models`, so the name shown matches whatever the recipe publishes, which may be an alias rather than the Hugging Face repository name.236>237> | Setting | Value |238> | --- | --- |239> | vLLM API base URL | `http://<head-node>:8000/v1` |240> | API key | vLLM accepts any key unless `--api-key` is set |241> | Open WebUI default port | `8080` |242>243> A few operational notes:244>245> - vLLM itself has no user accounts. Setting `--api-key` gives one shared secret, not per-user identity; per-user logins and separate chat histories come from Open WebUI. For anything beyond a home LAN, I would put the endpoint behind a reverse proxy that handles authentication properly.246> - If the served model needs a specific reasoning parser or tool-call parser, those options belong on the serving command. Without them, streamed thinking blocks and tool calls will not render the way the UI expects.247> - [LibreChat](https://github.com/danny-avila/LibreChat) and [LobeChat](https://github.com/lobehub/lobe-chat) follow the same pattern: set the base URL and key, pick the model, chat.248249## Once the 200Gb/s switch arrives250251A switch gives us a more convenient way to connect additional Sparks. The useful specification is **200Gb/s Ethernet per Spark-facing connection**, with compatible cables or optics. QSFP describes a connector family; the connector shape alone does not establish protocol or link speed.252253NVIDIA has a [multiple-Sparks-through-a-switch playbook](https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/multi-sparks-through-switch/README.md). Moving to that topology changes addressing, interface selection and communication paths, so the direct-cable setup needs to be revalidated.254255The memory arithmetic becomes appealing:256257| 128GB Sparks | Installed memory in aggregate | What I would explore |258| --- | ---: | --- |259| Two | 256GB | The current 300B-class experiment and the GLM comparison. |260| Three | 384GB | A supported three-way model split, or a two-node model plus a separate service. |261| Four | 512GB | Larger quantised models, two independent two-node deployments, or several single-node services. |262263Three-way tensor parallelism is not guaranteed to be supported by a given architecture. Four nodes are not guaranteed to be twice as fast as two. Those are measurements to make after establishing a supported configuration.264265As a capacity estimate, **671B parameters at an idealised four bits need about 335.5GB for weights alone**. Four Sparks make that class of experiment worth investigating, subject to a suitable checkpoint, supported partitioning and memory headroom. A 1.6T model would need around 800GB even at that idealised precision, beyond four Sparks' installed capacity.266267These calculations identify experiments to investigate. They do not establish a working launch recipe.268269### More machines give me several useful choices270271One option is to spread a larger model across all the nodes. Another is to run replicas of a model that already fits, increasing the number of requests we can serve independently. A third is to give different services their own machines: coding, chat, embeddings or reranking.272273For MoE models, [expert parallelism](https://docs.vllm.ai/en/v0.15.1/serving/expert_parallel_deployment/) offers another layout by placing experts on different GPUs. Its routing and communication requirements make it another configuration to benchmark, rather than an automatic improvement.274275My first switch-connected milestone would be to reproduce the current two-node behaviour through the switch. Then add a node, validate communications, and compare supported layouts. Under load, I would also check the switch and NIC congestion counters using [NVIDIA's NCCL networking guidance](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/networking_troubleshooting.html), including the relevant RoCE configuration for the actual switch.276277For the models themselves, I want to record:278279- **Time to first token**, including longer prompts.280- **Generation speed for one user**, and total throughput with several users.281- **Memory use and stable context limits** on every node.282- **Task success** on a fixed set of coding, Ansible, Linux troubleshooting and tool-use exercises.283- **Exact model revision, quantisation, container and launch settings**, so the comparison can be repeated.284285That last part is becoming essential. A model name is no longer enough to describe the system that produced an answer.286287## A small cluster with a useful job288289The Blackwell workstations established that local AI could be useful inside an ordinary developer workflow. One Spark let me take the same Ollama patterns into much larger models. Two Sparks have pushed me into distributed inference and made DeepSeek-V4-Flash-Vision-Exp a practical experiment.290291There is more to operate now: workers, network transport, memory budgets, quantised checkpoints and kernels that need to agree with one another. Understanding those pieces is becoming as interesting as trying the next model.292293GLM-5.3-Flash is next on my list. After that, the switch should give us room to find out whether the best use of more Sparks is a larger model, more simultaneous users, or a few specialised services.294295The encouraging part is that the responses are already making me want to use the system. Now I want the measurements to catch up with that impression.296297---298299## Sources and further reading300301Technical recipes and model details checked on **21 October 2026**. Links in the relevant sections identify the source of specifications and reported results.302303- [My earlier RTX PRO 4000 Blackwell article](/post/running-edge-llms-rtx-pro-4000-blackwell)304- [My earlier DGX Spark article](/post/heretic-models-dgx-spark)305- [NVIDIA: connect multiple DGX Sparks](https://github.com/NVIDIA/dgx-spark-playbooks/tree/main/nvidia/playbook-connect-multiple-sparks)306- [NVIDIA: intermediate vLLM cluster guide](https://build.nvidia.com/spark/vllm-cluster/intermediate)307- [DeepSeek-V4-Flash-Vision-Exp model card](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp)308- [NVIDIA: GLM-5.3-Flash on two 128GB Sparks](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-spark-2gpu/index.html?model=z-ai%2Fglm-5.3-flash&os=linux&backend=vllm&memory=128&precision=fp4-local&memoryConfig=128)309- [vLLM: parallelism and scaling](https://docs.vllm.ai/en/latest/serving/parallelism_scaling/)310- [NVIDIA: Spark NCCL tests](https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/nccl/README.md)311- [NVIDIA: multiple Sparks through a switch](https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/multi-sparks-through-switch/README.md)312

---
title: "From Ollama to vLLM"
date: "2026-10-21"
published: true
tags: ["ai", "llm", "local-ai", "ollama", "vllm", "nvidia", "dgx-spark", "deepseek", "networking"]
author: "Gavin Jackson"
excerpt: "From 24GB Blackwell workstation cards to one 128GB DGX Spark, then two Sparks and a QSFP cable: running DeepSeek-V4-Flash-Vision-Exp locally, understanding vLLM, and planning the next step with a 200Gb/s switch."
---

# From Ollama to vLLM

<p align="center"><img src="/assets/ai/vllm-hero-v1-two-sparks-one-cable.jpg" alt="Two engraved desktop computing units connected by a single glowing golden cable carrying sparks of data" width="800"></p>

My local AI experiments started with the RTX PRO 4000 Blackwell cards. I wanted something useful on a developer workstation: help with code, explanations of unfamiliar systems, and an assistant that could work inside a network where sending everything to a cloud API was not an option.

In [my earlier Blackwell article](/post/running-edge-llms-rtx-pro-4000-blackwell), the recurring constraint was **24GB of VRAM**. Ollama made running the models straightforward. The interesting work was choosing a model and quantisation that fitted, leaving room for context, and connecting it to tools such as Hermes and Continue.

That setup taught me a pattern: get the model working directly, check the quality, then add the agent or IDE integration. When something broke, work back through those layers.

I took essentially the same approach to a **DGX Spark with 128GB of unified memory**, still using Ollama. Suddenly I could explore models roughly five times the size of my earlier workstation baseline. Then I connected **two Sparks with a single QSFP cable**, moved into vLLM, and started running **DeepSeek-V4-Flash-Vision-Exp**, a roughly 300-billion-parameter model.

And it is actually starting to look good.

That is my assessment of the responses so far. I still want repeatable measurements of speed, coding quality and tool use. But the progression from fitting a useful assistant onto a workstation card to serving something this large across two small computers has been quite a journey.

## One Spark: the same pattern, much more room

The first Spark was a fairly comfortable transition. Pick a model, choose a sensible quant, load it into Ollama, and point the clients at it.

The big change was the memory budget. The [Spark hardware guide](https://docs.nvidia.com/dgx/dgx-spark/hardware.html) lists **128GB of LPDDR5x unified system memory**, shared coherently by the CPU and GPU. Compared with the Blackwell card's 24GB, that is about **5.3 times the physical capacity**.

It gave me room to move from the smaller models I had been testing into the 70-120B range I discussed in [my earlier Spark post](/post/heretic-models-dgx-spark). The exact increase in model size depends on quantisation and context; the fivefold comparison describes the scale of the opportunity, rather than a speed improvement.

The operating system uses that memory too. So do the inference engine, temporary buffers and conversation state. A 128GB Spark does not give a model 128GB exclusively for its weights.

Still, there was a pleasant period where I could reuse what I already knew and just try much larger models.

Then, naturally, I wanted another Spark.

## Two Sparks: follow the bouncing ball

My starting points were NVIDIA's [Connect Multiple DGX Sparks playbook](https://github.com/NVIDIA/dgx-spark-playbooks/tree/main/nvidia/playbook-connect-multiple-sparks) and the [intermediate vLLM cluster guide](https://build.nvidia.com/spark/vllm-cluster/intermediate).

I followed the bouncing ball. Connect the machines, establish the cluster network, prepare the inference environment, and work towards a model endpoint.

The physical setup is surprisingly modest: two Sparks and one direct QSFP cable. NVIDIA documents this as a supported two-device layout, with a switch becoming an option as more devices join.

In my network checks, the dedicated link came up at **200,000Mb/s**, with the two ends reachable at `10.0.0.1` and `10.0.0.2`. That established basic connectivity. It did not establish that inference was using the best transport or that the model was correctly distributed.

That distinction becomes much more relevant once the network is part of generating every answer.

The other conceptual shift was moving beyond the single-machine Ollama setup I had been using. For this experiment, I needed a serving engine that could explicitly distribute one model across two machines. That brought me into **vLLM**.

## What is vLLM actually doing?

[vLLM](https://github.com/vllm-project/vllm) is an open-source inference and serving engine. It loads a trained model, executes it on supported hardware, manages requests and memory, and provides an API that applications can use.

It does not add knowledge to the model. If the responses improve because I can now run a more capable model, that capability comes from the model. vLLM makes the deployment practical.

There are several different pieces to that practicality.

### Splitting the model across the GPUs

With **tensor parallelism**, supported weight tensors are divided across GPUs. Both Sparks participate in the calculation, exchanging intermediate results as inference progresses. The model's entire weight set does not have to fit on either Spark individually.

With **pipeline parallelism**, groups of layers run in different stages. Work passes between those stages. This changes the communication pattern, but introduces pipeline scheduling and utilisation trade-offs.

These are established model-parallel techniques, described in [NVIDIA's parallelism overview](https://docs.nvidia.com/nemo/megatron-bridge/latest/parallelisms.html) and supported by [vLLM's distributed serving implementation](https://docs.vllm.ai/en/stable/serving/parallelism_scaling/). A recipe chooses a layout that its model implementation supports; adding another computer does not make every possible split valid.

For a two-Spark tensor-parallel deployment, each GPU holds a share of the model, along with whatever state and buffers that implementation needs. Some data can be replicated, and the split need not consume precisely equal memory.

The useful consequence is that a model too large for one Spark can become possible across the pair.

### Coordinating the workers

There is a separate question: how do the processes find each other and start working together?

Some vLLM deployments use **Ray** to manage distributed workers. Current versions also support native multi-node multiprocessing. NVIDIA's current GLM recipe uses the native route, so a separate Ray cluster is not a universal requirement.

The relevant native-launch settings appear in [vLLM's command reference](https://docs.vllm.ai/en/latest/cli/serve/):

| Setting | Role |
| --- | --- |
| `--tensor-parallel-size 2` | Divide supported tensors across two GPU ranks. |
| `--nnodes 2` | Participate in a two-node deployment. |
| `--node-rank 0` / `1` | Give each node a distinct identity. |
| `--master-addr` / `--master-port` | Tell the processes where to rendezvous. |
| `--headless` | Run the worker without its own public API server. |

These are pieces of a complete recipe. The checkpoint, container, kernels and network settings still need to match.

Worker coordination also has a different job from moving inference data. GPU collectives commonly use **NCCL**, while particular Spark recipes add specialised RoCE communication paths. Ray being healthy, or both processes starting successfully, does not prove that those transfers are efficient.

### Using conversation memory efficiently

Generating a response involves more than reading the weights. The engine also retains attention state for tokens already processed: the **KV cache**.

The [original vLLM PagedAttention explanation](https://vllm.ai/blog/2023-06-20-vllm) describes managing that cache in blocks, allowing logical token sequences to use non-contiguous physical memory. This reduces the waste associated with reserving large contiguous areas for conversations whose eventual lengths are unknown.

The broad benefit is better use of the available memory, especially when serving several requests. Modern models have different attention and recurrent-state designs, so the actual cache layout and kernels depend on the architecture.

For my purposes, the practical lesson survives all those details: **context has a memory cost even after the model has loaded**.

### Scheduling work and reusing it

vLLM also supports continuous batching: active requests can share execution batches as requests arrive and finish. Chunked prefill divides long prompt processing into smaller pieces, helping the scheduler balance that work with ongoing generation. These are among the [engine's documented serving features](https://docs.vllm.ai/en/v0.20.2/).

[Prefix caching](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/) can reuse computed state when requests begin with the same token prefix. An agent repeatedly sending the same instructions or repository background is an obvious candidate. It mainly saves prompt-processing work; it does not eliminate the cost of generating the answer.

Some recipes also use [speculative decoding](https://docs.vllm.ai/en/latest/features/speculative_decoding/), proposing several tokens and checking them with the target model. A proposal mechanism can use a separate draft model or the model's own multi-token prediction components. Whether it helps depends on acceptance rates and execution overhead.

This helps explain why a working launch command can be much longer than `ollama run`. There are more decisions exposed to the person operating the service.

## How does a 300B model fit?

The model I am running is [DeepSeek-V4-Flash-Vision-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp). Its Hugging Face page lists approximately **305B parameters**. This is the experimental multimodal release, which adds visual modules and continued training to the Flash architecture.

Start with the weight arithmetic. Using **305 billion parameters** as a rough sizing figure, the idealised payload is:

| Weight precision | Bytes per parameter | Weight payload |
| --- | ---: | ---: |
| BF16 / FP16 | 2 | 610GB |
| 8-bit | 1 | 305GB |
| 4-bit | 0.5 | 152.5GB |

These are decimal GB calculations, before quantisation scales, higher-precision tensors, caches, workspaces and system overhead. Real checkpoints mix precisions and have their own storage layouts.

Two Sparks provide **256GB of installed memory in aggregate**. A suitable four-bit checkpoint therefore has a plausible path to fitting. Its actual shards and runtime allocations must each fit within an individual Spark's budget.

The network does not create a coherent 256GB memory pool spanning the two computers. The serving engine distributes the model and exchanges the data needed to execute it.

Mixture-of-Experts models add another distinction: **total parameters versus parameters active for each token**. Routing selects a subset of experts, reducing computation relative to activating the entire model. For resident inference, those other experts still need storage.

The Vision-Exp release makes this especially interesting for the work I do. Alongside text and code, visual understanding opens up experiments with screenshots, diagrams and documentation. Those are capabilities I want to evaluate; my encouraging first impressions do not establish how well it handles each of them. Image processing also adds to the runtime memory budget.

The combination is what makes this interesting: a large pool of model weights, sparse computation, quantisation, and an engine that can divide the work between machines.

## The cable is part of the computer now

A 200Gb/s link has a theoretical line rate of **25GB/s**, before protocol overhead. NVIDIA lists the Spark's local memory bandwidth at **273GB/s**. Those figures describe different paths, but they illustrate why moving data between boxes has to be treated carefully.

Every exchange has bandwidth and latency costs. For repeated small collectives, latency can matter as much as the headline link rate.

The checks I want around a deployment now go beyond a successful ping:

1. Verify the negotiated link speed and interface error counters.
2. Identify the ConnectX interfaces and RoCE devices that the recipe expects.
3. Confirm the inference transport is using the intended high-speed path.
4. Run communication tests before using a large model to diagnose the network.

NVIDIA's [NCCL playbook](https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/nccl/README.md) provides communication tests for Spark topologies. The [vLLM transport guidance](https://docs.vllm.ai/en/stable/serving/parallelism_scaling/#optimizing-network-communication-for-tensor-parallelism) also explains how NCCL logs can reveal a TCP socket fallback. Recipe-specific RoCE kernels need their own checks too.

One physical QSFP connection can expose multiple logical interfaces. That makes guessing from a device name an unreliable way to decide which interface should be used or disabled. I want the documented mapping in front of me before changing it.

## What I want to run next

My next substantial test is [NVIDIA's GLM-5.3-Flash recipe for two 128GB Sparks](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-spark-2gpu/index.html?model=z-ai%2Fglm-5.3-flash&os=linux&backend=vllm&memory=128&precision=fp4-local&memoryConfig=128).

There are good alternatives worth comparing alongside it. My shortlist is based on documented deployment paths and the jobs I want these machines to do, rather than an assertion that one model wins every benchmark.

| Candidate | Why it is on my list | Deployment evidence |
| --- | --- | --- |
| **GLM-5.3-Flash FP4-Local** | My first choice for the next large coding and agent test. The [upstream model](https://huggingface.co/zai-org/GLM-5.3-Flash) has 320B total parameters and 18B active. | NVIDIA provides the exact two-128GB-Spark recipe linked above. |
| **Qwen3.8-Flash-Next FP4-Local** | A coding, tool-use and multimodal comparison; also useful for testing whether a smaller deployment is sufficient. | The [NVIDIA checkpoint card](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local) documents Spark use, and NVIDIA's cluster guide links a two-Spark recipe. |
| **Nemotron 3 Super 120B-A12B NVFP4** | A more manageable agent and reasoning baseline, with 120B total parameters and 12B active. | The [official model instructions](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4/blob/main/README.md) include a single-Spark vLLM configuration. Two machines could run independent instances. |
| **DeepSeek-V4-Flash-Vision-Exp** | My existing roughly 305B multimodal point of comparison: promising responses, with proper task evaluation still to do. | Running across my two Sparks. Reproducing that deployment also requires the particular quantised weights and runtime settings. |

For Qwen in particular, the model card lists a 125B main model plus additional n-gram embeddings and MTP parameters, and a checkpoint around 96GiB on disk. A headline parameter count alone is a poor guide to the deployed memory footprint.

My order would be GLM first, then Qwen and Nemotron on the same tasks. If a model running on one Spark produces comparable results, that leaves the second machine available for another user or workload.

### Why the precise GLM recipe matters

The download I want is **`nvidia/GLM-5.3-Flash-FP4-Local`**. NVIDIA's [checkpoint card](https://huggingface.co/nvidia/GLM-5.3-Flash-FP4-Local) describes quantisation-aware distillation and mixed tensor precisions. It also explicitly requires a matching community vLLM container while support for its format is being integrated upstream.

That is a stronger requirement than having four-bit weights and a recent generic container. The loader and kernels must understand that particular checkpoint.

The selected recipe currently pins **`eugr/spark-vllm-b12x:nightly-20261015`**, uses tensor parallelism across two nodes, FP8 KV cache, and native MTP speculative decoding. Its configured request limit is **500,000 tokens** with up to four sequences; those settings are a reference configuration, not evidence that every workload at those limits will be comfortable.

NVIDIA reports **33.94 tokens/sec decode throughput** for its SPEED-Bench qualitative coding workload, measured on **16 October 2026** on two 128GB Sparks. That result belongs to the [published recipe and workload](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-spark-2gpu/index.html?model=z-ai%2Fglm-5.3-flash&os=linux&backend=vllm&memory=128&precision=fp4-local&memoryConfig=128); it is a useful reference for my next experiment.

I would start with a shorter context and a single request, then increase the workload while watching memory and latency.

## Getting back to a normal chat interface

After all that machinery, the application-facing part becomes familiar again. vLLM provides an [OpenAI-compatible serving API](https://docs.vllm.ai/en/latest/serving/online_serving/), including model discovery and chat completions.

For example, from the head machine, once the selected recipe is running:

```bash
curl http://localhost:8000/v1/models

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "nvidia/GLM-5.3-Flash-FP4-Local",
    "messages": [
      {"role": "user", "content": "Explain tensor parallelism using two DGX Sparks."}
    ],
    "max_tokens": 512,
    "stream": true
  }'
```

The model name must match what `/v1/models` reports. A recipe can publish an alias instead of its Hugging Face repository name.

A chat UI or agent that supports a custom OpenAI-compatible endpoint can then use the head node's `/v1` base URL. For coding agents, the model-specific chat template, reasoning parser and tool-call parser are part of making that integration work properly.

I want to repeat the same testing pattern that worked with Ollama: verify the direct API first, then connect the clients.

> ## Spinning Up vLLM for Browser Users
>
> Everything so far ends at an OpenAI-compatible API on the head node. Turning that into something an end user can reach from a web browser takes two pieces: the server has to listen on an address the LAN can reach, and a browser chat interface has to point at that address.
>
> On a single machine, the serving side is one command:
>
> ```bash
> vllm serve nvidia/GLM-5.3-Flash-FP4-Local \
>   --host 0.0.0.0 \
>   --port 8000
> ```
>
> In the two-Spark recipes, the serving step is part of the recipe itself, and only the head node publishes the API. The second node runs `--headless`, so there is nothing to connect to on it. Users always point at the head node's regular LAN address, not the `10.0.0.x` interconnect, which is reserved for inference traffic between the Sparks.
>
> For the browser side, [Open WebUI](https://github.com/open-webui/open-webui) is the quickest self-hosted chat interface I have found. It talks to OpenAI-compatible endpoints and runs as a single container:
>
> ```bash
> docker run -d --name open-webui --network host \
>   -e OPENAI_API_BASE_URL=http://<head-node>:8000/v1 \
>   -e OPENAI_API_KEY=unused \
>   -v open-webui:/app/backend/data \
>   ghcr.io/open-webui/open-webui:main
> ```
>
> Browse to `http://<open-webui-host>:8080` and register; the first account created becomes the administrator. The model picker is populated from `/v1/models`, so the name shown matches whatever the recipe publishes, which may be an alias rather than the Hugging Face repository name.
>
> | Setting | Value |
> | --- | --- |
> | vLLM API base URL | `http://<head-node>:8000/v1` |
> | API key | vLLM accepts any key unless `--api-key` is set |
> | Open WebUI default port | `8080` |
>
> A few operational notes:
>
> - vLLM itself has no user accounts. Setting `--api-key` gives one shared secret, not per-user identity; per-user logins and separate chat histories come from Open WebUI. For anything beyond a home LAN, I would put the endpoint behind a reverse proxy that handles authentication properly.
> - If the served model needs a specific reasoning parser or tool-call parser, those options belong on the serving command. Without them, streamed thinking blocks and tool calls will not render the way the UI expects.
> - [LibreChat](https://github.com/danny-avila/LibreChat) and [LobeChat](https://github.com/lobehub/lobe-chat) follow the same pattern: set the base URL and key, pick the model, chat.

## Once the 200Gb/s switch arrives

A switch gives us a more convenient way to connect additional Sparks. The useful specification is **200Gb/s Ethernet per Spark-facing connection**, with compatible cables or optics. QSFP describes a connector family; the connector shape alone does not establish protocol or link speed.

NVIDIA has a [multiple-Sparks-through-a-switch playbook](https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/multi-sparks-through-switch/README.md). Moving to that topology changes addressing, interface selection and communication paths, so the direct-cable setup needs to be revalidated.

The memory arithmetic becomes appealing:

| 128GB Sparks | Installed memory in aggregate | What I would explore |
| --- | ---: | --- |
| Two | 256GB | The current 300B-class experiment and the GLM comparison. |
| Three | 384GB | A supported three-way model split, or a two-node model plus a separate service. |
| Four | 512GB | Larger quantised models, two independent two-node deployments, or several single-node services. |

Three-way tensor parallelism is not guaranteed to be supported by a given architecture. Four nodes are not guaranteed to be twice as fast as two. Those are measurements to make after establishing a supported configuration.

As a capacity estimate, **671B parameters at an idealised four bits need about 335.5GB for weights alone**. Four Sparks make that class of experiment worth investigating, subject to a suitable checkpoint, supported partitioning and memory headroom. A 1.6T model would need around 800GB even at that idealised precision, beyond four Sparks' installed capacity.

These calculations identify experiments to investigate. They do not establish a working launch recipe.

### More machines give me several useful choices

One option is to spread a larger model across all the nodes. Another is to run replicas of a model that already fits, increasing the number of requests we can serve independently. A third is to give different services their own machines: coding, chat, embeddings or reranking.

For MoE models, [expert parallelism](https://docs.vllm.ai/en/v0.15.1/serving/expert_parallel_deployment/) offers another layout by placing experts on different GPUs. Its routing and communication requirements make it another configuration to benchmark, rather than an automatic improvement.

My first switch-connected milestone would be to reproduce the current two-node behaviour through the switch. Then add a node, validate communications, and compare supported layouts. Under load, I would also check the switch and NIC congestion counters using [NVIDIA's NCCL networking guidance](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/networking_troubleshooting.html), including the relevant RoCE configuration for the actual switch.

For the models themselves, I want to record:

- **Time to first token**, including longer prompts.
- **Generation speed for one user**, and total throughput with several users.
- **Memory use and stable context limits** on every node.
- **Task success** on a fixed set of coding, Ansible, Linux troubleshooting and tool-use exercises.
- **Exact model revision, quantisation, container and launch settings**, so the comparison can be repeated.

That last part is becoming essential. A model name is no longer enough to describe the system that produced an answer.

## A small cluster with a useful job

The Blackwell workstations established that local AI could be useful inside an ordinary developer workflow. One Spark let me take the same Ollama patterns into much larger models. Two Sparks have pushed me into distributed inference and made DeepSeek-V4-Flash-Vision-Exp a practical experiment.

There is more to operate now: workers, network transport, memory budgets, quantised checkpoints and kernels that need to agree with one another. Understanding those pieces is becoming as interesting as trying the next model.

GLM-5.3-Flash is next on my list. After that, the switch should give us room to find out whether the best use of more Sparks is a larger model, more simultaneous users, or a few specialised services.

The encouraging part is that the responses are already making me want to use the system. Now I want the measurements to catch up with that impression.

---

## Sources and further reading

Technical recipes and model details checked on **21 October 2026**. Links in the relevant sections identify the source of specifications and reported results.

- [My earlier RTX PRO 4000 Blackwell article](/post/running-edge-llms-rtx-pro-4000-blackwell)
- [My earlier DGX Spark article](/post/heretic-models-dgx-spark)
- [NVIDIA: connect multiple DGX Sparks](https://github.com/NVIDIA/dgx-spark-playbooks/tree/main/nvidia/playbook-connect-multiple-sparks)
- [NVIDIA: intermediate vLLM cluster guide](https://build.nvidia.com/spark/vllm-cluster/intermediate)
- [DeepSeek-V4-Flash-Vision-Exp model card](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp)
- [NVIDIA: GLM-5.3-Flash on two 128GB Sparks](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-spark-2gpu/index.html?model=z-ai%2Fglm-5.3-flash&os=linux&backend=vllm&memory=128&precision=fp4-local&memoryConfig=128)
- [vLLM: parallelism and scaling](https://docs.vllm.ai/en/latest/serving/parallelism_scaling/)
- [NVIDIA: Spark NCCL tests](https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/nccl/README.md)
- [NVIDIA: multiple Sparks through a switch](https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/multi-sparks-through-switch/README.md)