8 DGX Sparks: LLM Serving Capacity

Research question.

How much concurrent work can a local inference fleet sustain at an interactive generation rate? I evaluated eight DGX Spark systems with English prose and Python workloads, using a median per-stream rate of 20 generated tokens per second as a capacity reference. The aim was to establish practical concurrency limits and identify the bottlenecks behind them.

Experimental setup.

Eight NVIDIA DGX Spark systems, each with a GB10 processor and 128 GB of unified memory, host four deployments. The four-node DeepSeek deployment uses a direct-cabled ring and a modified NCCL build. All three model configurations use speculative decoding.

Measurement protocol.

Synchronized waves of 1–16 streams use distinct prompts from sets of 16 Python tasks and 16 English writing topics. Requested output lengths are 1,000 tokens for code at temperature 0, and 300–600 tokens for prose at temperature 0.7. Reasoning is disabled. Each concurrency level has a warm-up wave and one to five repeated trials.

Each stream’s decode rate is (output tokens − 1) divided by the interval between its first and last token. The figures take the median across streams within each trial, then the median across trials. Queueing and prefill are assessed separately through time to first token. Aggregate throughput divides the sum of (output tokens − 1) across streams by the shared interval from the earliest first token to the latest last token. It therefore differs from concurrency multiplied by the median stream rate.

Concurrency depends on the workload.

Code generation sustains higher concurrency at the selected median-rate threshold in every tested deployment. Summing the deployment capacities gives approximately 11 English-text streams or 28 code streams under their respective test conditions. These are alternative workloads, not a simultaneous mixed-load result. Higher speculative draft acceptance on code is consistent with the advantage, although temperature and output length also differ.

¹ DeepSeek’s two-stream text estimate depends on the shorter-output observation. The 600-token measurements alone place the threshold between 1 and 4 streams. At four code streams, GLM’s median is 21.1 tok/s and its minimum is 19.4 tok/s.

Prefill can dominate responsiveness.

A 32k-token uncached prompt on Qwen reduced concurrent decode rates to roughly 4 tok/s for about 16 seconds. New short requests waited 13–15 seconds for their first token. On DeepSeek, reusing a cached 40k-token document reduced time to first token from about 17 seconds to 0.4 seconds.

These observations make prompt reuse and prefill isolation relevant to capacity planning. A decode-only concurrency estimate does not capture the delays introduced by long incoming documents.

Diagnosing an interconnect bottleneck.

The four-node ring initially sustained only 12.5–13 Gb/s per link. RDMA loopback reproduced the slowdown without a cable, directing the investigation toward the host-side NIC path. System logs associated the affected hosts with NIC hot-plug reinitialization. Rebooting with cables connected restored link performance; the initialization path remained the working root-cause explanation.

After link recovery, DeepSeek’s aggregate code throughput at 16 streams improved by 27%, and prefill throughput roughly doubled.

Implications and limitations.

The results favor workload-aware routing and separate budgets for decode concurrency and long-context prefill. Configured request slots alone are insufficient for interactive capacity planning. Model selection still requires an independent evaluation of output quality.

The 20 tok/s reference applies to median decode rate and does not guarantee a minimum speed for every request. Prompts are synthetic, repetition counts vary, and the figures report point estimates without confidence intervals. English-text results from one Qwen replica are extrapolated to the second; full agent workflows and output quality were not evaluated.

GLM code measurements were collected on 25 September and its text measurements on 26 September with the same configuration but a different server session. Output length also matters: in a separate GLM comparison at eight streams, extending responses from 400 to 1,000 tokens reduced the rate from 19.8 to 17.3 tok/s.

DeepSeek’s four- and sixteen-stream code records flag output-length validation failures; individual completion lengths were not retained in those exports. Long-context tests include prompts up to approximately 86k tokens and 40–56k-token cache checks, but do not establish performance at the full configured context limits.