Choosing a GPU for LLMs: RTX 4090, A100, H100 or GX10
Compare memory requirements, compatibility and the cost of a test to choose hardware for your LLM inference workload.
Define the model and workload
To choose a GPU, specify the exact model revision, weight format, context length and expected concurrency. Two deployments of the same LLM can have very different requirements: a short conversation with one user and parallel processing of long documents consume different amounts of memory.
Decide whether your priority is time to first token, generation speed, throughput or the experiment budget. Compare configurations using the same prompts and limits. Otherwise, differences in the workload can obscure differences in the hardware.
RTX 4090: check the 24 GB limit
The RTX 4090 has 24 GB of VRAM. Consider it when the model weights, KV cache and engine buffers fit within that capacity. Loading the model successfully is only the first check: the target context and concurrency must fit during generation as well.
Quantization can reduce weight memory, but your runtime must support the chosen format. Check answer quality after changing precision. Offloading work to the CPU also changes performance, so record it as part of the benchmark configuration.
A100 and H100: compare the exact configuration
The family name does not describe the entire server. The catalog includes different A100 and H100 configurations. Compare memory per GPU, accelerator count and PCIe or SXM form factor, then check host RAM, storage and CPU resources.
For multiple GPUs, verify the interconnect and support for distributed execution. Adding up the memory of several cards does not establish that a single process can use it as one contiguous pool. The application needs to partition the model or workload, and communication introduces overhead.
GX10: unified memory and ARM64
The NVIDIA GB10-based ASUS Ascent GX10 shares 128 GB of memory between CPU and GPU. The operating system and background processes use part of that capacity. Compare the memory actually available to your workload, as well as the headline specification.
GX10 uses an ARM64 CPU. Check container architecture, dependencies and CUDA extensions before moving your project. A GX10 pair consists of two separate 128 GB nodes, and using them together requires application support. Having enough memory alone does not predict generation speed.
Compare the cost of completing the test
A lower hourly rate can come with a longer run. Measure preparation time, execution time, peak memory and successful outputs for the same test. Then multiply the billable duration by the agreed rate.
All rentGPU configurations are listed on request. Hardware, dates, software environment and access terms need confirmation before rental. The current website form is a demo: it shows what a request includes, but does not submit data or reserve a server.
Sources and documentation
Choose a GPU for your workload
Compare specifications and rates in the catalog. Prepare your model, environment and preferred dates to discuss the configuration and access terms.
Keep exploring
GPU rental cost: how to estimate your budget
GPUs for LoRA and QLoRA: preparing to fine-tune an LLM