Using GPUs

Configure GPU hardware in cerebrium.toml, compare available GPU models by VRAM and plan tier, and size GPUs for your model’s memory needs.

GPUs accelerate computational workloads through parallel processing. Originally designed for graphics rendering, modern GPUs are essential for AI models, large-scale data processing, and other compute-intensive applications.

Cerebrium provides GPU access through configuration in the cerebrium.toml file, without requiring infrastructure management.

Specifying GPUs

Configure GPUs in the [cerebrium.hardware] section of cerebrium.toml, specifying the type (compute parameter) and quantity (gpu_count). Additional deployment and scaling considerations are covered in the sections below.

Available GPUs

The platform offers GPUs ranging from cost-effective development options to high-end enterprise hardware.

GPU Model	Identifier	VRAM (GB)	Max GPUs	Plan required
NVIDIA B300	BLACKWELL_B300	262	8	Enterprise
NVIDIA B200	BLACKWELL_B200	180	8	Enterprise
NVIDIA H200	HOPPER_H200	141	8	Enterprise
NVIDIA H100	HOPPER_H100	80	8	Enterprise
RTX PRO 6000	BLACKWELL_RTX6000	96	8	Standard
NVIDIA A100	AMPERE_A100_80GB	80	8	Standard
NVIDIA A100	AMPERE_A100_40GB	40	8	Standard
NVIDIA L40s	ADA_L40	48	8	Hobby+
NVIDIA L4	ADA_L4	24	8	Hobby+
NVIDIA A10	AMPERE_A10	24	8	Hobby+
NVIDIA T4	TURING_T4	16	8	Hobby+
AWS Trainium	TRN1	32	8	Hobby+

The identifier is used in the cerebrium.toml file. It consists of the GPU model generation and model name to avoid ambiguity.

GPU selection is also possible using the --compute and --gpu-count flags during application initialization.

Choosing a GPU for Your Model

Pick a GPU based on the total VRAM your workload needs at runtime, not just the on-disk size of the model files. Total VRAM has to fit the model weights plus activations, the KV cache (for LLMs), and any intermediate buffers your framework allocates.

As a rule of thumb, size VRAM to roughly 1.2×–2× the loaded weight size, depending on batch size, context length, and precision. For example:

Loaded weight size	Typical VRAM to target	Example single-GPU options
Under 8 GB	16–24 GB	TURING_T4, AMPERE_A10, ADA_L4
8–20 GB	24–48 GB	AMPERE_A10, ADA_L4, ADA_L40
20–40 GB	48–80 GB	ADA_L40, AMPERE_A100_80GB
40–70 GB	80–96 GB	AMPERE_A100_80GB, BLACKWELL_RTX6000
70 GB and above	96 GB or multi-GPU	BLACKWELL_RTX6000, HOPPER_H100 (multi-GPU), HOPPER_H200, BLACKWELL_B200

If a model does not fit on a single GPU, set gpu_count to shard weights across multiple GPUs of the same type. Multi-GPU inference requires framework support (for example, vLLM tensor parallelism or device_map="auto" in Transformers).

Quantized formats such as GGUF Q4/Q5, FP8, and INT8 reduce weight size significantly. A Q4 quantized 7B model, for example, fits comfortably on a 24 GB GPU. Consult the model card for its runtime VRAM requirements at the precision you plan to use — Cerebrium does not maintain per-model requirements.

Match RAM to VRAM as a starting point so your framework can stage weights from CPU to GPU without swapping. See CPU and memory for details.

Multi-GPU Configuration

Multiple GPUs are configured in the cerebrium.toml file:

[cerebrium.hardware]
compute = "AMPERE_A100_80GB"
gpu_count = 4        # Number of GPUs needed
cpu = 8
memory = 128.0

GPU availability varies by region and provider. Narrowing the provider and region constraints increases the likelihood of request queuing. For guaranteed burst capacity, contact the enterprise plan team.