SiliconMark™ Benchmarks

Available Benchmarks

SiliconMark supports various benchmarks for GPU performance testing.

QuickMark - Comprehensive single-node GPU compute and memory performance test
Cluster Network - Multi-node network connectivity and bandwidth testing
Inference Benchmark - Multi-engine LLM inference performance (NVIDIA and AMD, using vLLM)
Llama 3 Inference - Single-node LLM inference performance using NVIDIA NIM
Llama 3 Fine-Tuning - Single-node LLM fine-tuning performance using NVIDIA NeMo

Each benchmark section includes configuration options, execution actions, result structures, and field metadata for interpreting the performance metrics.

QuickMark Benchmark

Overview

Field	Value
Benchmark ID	`quick_mark`
Type	Single-node
Min Nodes	1
Description	Comprehensive GPU compute, memory, and interconnect performance test

Configuration

No configuration required — uses defaults.

Result Structure

Results include one entry per GPU in test_results and a combined aggregate_results. For single-GPU systems, aggregate_results mirrors the single GPU result.

{
  "test_results": [
    {
      "gpu_id": "GPU-000...",
      "gpu_model": "NVIDIA H100 80GB HBM3",
      "fp32_tflops": 367.5,
      "fp32_cuda_core_tflops": 53.6,
      "fp16_tflops": 684.6,
      "bf16_tflops": 729.6,
      "fp8_tflops": 1456.2,
      "mixed_precision_tflops": 648.9,
      "memory_bandwidth_gbs": 3025.0,
      "l2_bandwidth_gbs": 415.5,
      "host_to_device_bandwidth_gbs": 27.7,
      "device_to_host_bandwidth_gbs": 28.5,
      "kernel_launch_overhead_us": 7.6,
      "power_consumption_watts": 654.3,
      "temperature_centigrade": 59,
      "fp32_tflops_per_peak_watt": 0.564,
      "fp16_tflops_per_peak_watt": 1.082,
      "fp8_tflops_per_peak_watt": 2.163,
      "energy_consumption_wh": 45.2,
      "total_vram_mib": 81559.0,
      "gpu_clocks": {
        "compute": { "base_mhz": 1110, "max_mhz": 1980 },
        "memory":  { "base_mhz": 2619, "max_mhz": 2619 }
      },
      "sample_frequency_s": 5.0,
      "measurements_temp": [58, 59, 60, 59],
      "measurements_power_draw": [640.1, 654.3, 651.0, 648.2],
      "core_utilization_percent": [98.0, 99.0, 98.5, 99.0],
      "memory_utilization_percent": [95.0, 96.0, 95.5, 96.0],
      "core_clock_mhz": [1965, 1980, 1975, 1980],
      "memory_clock_mhz": [2619, 2619, 2619, 2619]
    }
  ],
  "aggregate_results": {
    "gpu_id": "aggregate",
    "fp32_tflops": 2863.0,
    "fp32_cuda_core_tflops": 424.2,
    "fp16_tflops": 5492.8,
    "bf16_tflops": 5755.8,
    "mixed_precision_tflops": 4222.2,
    "memory_bandwidth_gbs": 22717.3,
    "allreduce_bandwidth_gbs": 275.2,
    "broadcast_bandwidth_gbs": 392.0,
    "host_to_device_bandwidth_gbs": 110.6,
    "device_to_host_bandwidth_gbs": 112.9,
    "power_consumption_watts": 5078.5,
    "temperature_centigrade": 68,
    "fp32_tflops_per_peak_watt": 0.564,
    "fp16_tflops_per_peak_watt": 1.082,
    "gpu_bandwidth_matrix": {
      "gpu0_to_gpu1": {
        "connection_type": "nvlink_v4_18x",
        "simplex_gbs": 388.1,
        "duplex_gbs": 389.2
      },
      "gpu0_to_gpu2": {
        "connection_type": "nvlink_v4_18x",
        "simplex_gbs": 389.2,
        "duplex_gbs": 389.9
      }
    }
  },
  "timestamp": "YYYY-MM-DDT20:03:06Z"
}

Field Metadata

Field	Display Name	Unit	Notes
`fp32_tflops`	FP32 Performance	TFLOPS	Tensor Core (TF32)
`fp32_cuda_core_tflops`	FP32 CUDA Core Performance	TFLOPS	CUDA cores only
`fp16_tflops`	FP16 Performance	TFLOPS
`bf16_tflops`	BF16 Performance	TFLOPS
`fp8_tflops`	FP8 Performance	TFLOPS	Where supported
`mixed_precision_tflops`	Mixed Precision Performance	TFLOPS	FP16 compute, FP32 accumulate
`memory_bandwidth_gbs`	Memory Bandwidth	GB/s	HBM bandwidth
`l2_bandwidth_gbs`	L2 Cache Bandwidth	GB/s
`host_to_device_bandwidth_gbs`	Host to Device Bandwidth	GB/s	PCIe
`device_to_host_bandwidth_gbs`	Device to Host Bandwidth	GB/s	PCIe
`kernel_launch_overhead_us`	Kernel Launch Overhead	μs
`allreduce_bandwidth_gbs`	AllReduce Bandwidth	GB/s	Multi-GPU only
`broadcast_bandwidth_gbs`	Broadcast Bandwidth	GB/s	Multi-GPU only
`gpu_bandwidth_matrix`	GPU Bandwidth Matrix	GB/s	Per-pair simplex/duplex with connection type
`fp32_tflops_per_peak_watt`	FP32 TFLOPS per Peak Watt	TFLOPS/W
`fp16_tflops_per_peak_watt`	FP16 TFLOPS per Peak Watt	TFLOPS/W
`fp8_tflops_per_peak_watt`	FP8 TFLOPS per Peak Watt	TFLOPS/W	Where supported
`energy_consumption_wh`	Energy Consumption	Wh	Total for benchmark run
`power_consumption_watts`	Power Consumption	W	Peak power draw
`temperature_centigrade`	GPU Temperature	°C	Peak temperature
`total_vram_mib`	Total VRAM	MiB
`gpu_clocks`	GPU Clock Speeds	MHz	Compute and memory base/max/application/boost
`sample_frequency_s`	Monitoring Sample Frequency	s
`measurements_temp`	Temperature Timeseries	°C	One sample per interval
`measurements_power_draw`	Power Draw Timeseries	W	One sample per interval
`core_utilization_percent`	Core Utilization Timeseries	%	One sample per interval
`memory_utilization_percent`	Memory Utilization Timeseries	%	One sample per interval
`core_clock_mhz`	Core Clock Timeseries	MHz	One sample per interval
`memory_clock_mhz`	Memory Clock Timeseries	MHz	One sample per interval

Cluster Network Benchmark

Overview

Field	Value
Benchmark ID	`cluster_network`
Type	Multi-node
Min Nodes	2
Description	Tests network throughput and latency between all cluster nodes

Result Structure

One measurement per directed node pair.

{
  "network_results": [
    {
      "host_ip": "192.168.1.10",
      "dest_ip": "192.168.1.11",
      "throughput_mbps": 45200.0,
      "throughput_gbps": 45.2,
      "latency_ms": 0.75
    }
  ],
  "measurement_count": 12
}

Field Metadata

Field	Display Name	Unit
`throughput_gbps`	Throughput	Gbps
`throughput_mbps`	Throughput	Mbps
`latency_ms`	Latency (RTT)	ms
`measurement_count`	Total Links Tested

Inference Benchmark — vLLM

This benchmark measures LLM inference serving performance using vLLM. It supports both NVIDIA (CUDA) and AMD (ROCm) GPUs and runs without requiring an NGC API key.

Overview

Field	Value
Benchmark ID	`inference_benchmark`
Type	Single-node
Min Nodes	1
GPU Support	NVIDIA and AMD
Description	Multi-engine LLM inference performance benchmark using vLLM

Configuration

Field	Type	Required	Description	Default
`inference_engine`	string	No	Inference engine	`"vllm"`
`model`	string	No	Model name/path	`"openai/gpt-oss-120b"`
`tp`	int	No	Tensor parallel size	`1`
`concurrency`	int	Yes	Concurrent requests	—
`isl`	int	Yes	Input sequence length (1–131072)	—
`osl`	int	Yes	Output sequence length (1–131072)	—
`random_range_ratio`	float	No	Prompt length variation	`0.0`
`num_prompts`	int	No	Number of prompts (0 = concurrency × 10)	`0`

Result Structure

{
  "duration": 120.5,
  "total_input_tokens": 512000,
  "total_output_tokens": 128000,
  "request_throughput": 42.3,
  "output_token_throughput": 5890.4,
  "total_token_throughput": 8750.2,
  "mean_ttft_ms": 145.2,
  "median_ttft_ms": 138.7,
  "p99_ttft_ms": 312.4,
  "mean_tpot_ms": 18.4,
  "median_tpot_ms": 17.9,
  "p99_tpot_ms": 28.6,
  "mean_itl_ms": 18.4,
  "median_itl_ms": 17.9,
  "p99_itl_ms": 28.6,
  "mean_e2el_ms": 2340.5,
  "median_e2el_ms": 2180.3,
  "p99_e2el_ms": 4120.8
}

Field Metadata

Field	Display Name	Unit
`request_throughput`	Request Throughput	req/s
`output_token_throughput`	Output Token Throughput	tok/s
`total_token_throughput`	Total Token Throughput	tok/s
`total_input_tokens`	Total Input Tokens	tokens
`total_output_tokens`	Total Output Tokens	tokens
`mean_ttft_ms`	TTFT (Mean)	ms
`median_ttft_ms`	TTFT (Median)	ms
`p99_ttft_ms`	TTFT (P99)	ms
`mean_tpot_ms`	TPOT (Mean)	ms
`median_tpot_ms`	TPOT (Median)	ms
`p99_tpot_ms`	TPOT (P99)	ms
`mean_itl_ms`	Inter-Token Latency (Mean)	ms
`median_itl_ms`	Inter-Token Latency (Median)	ms
`p99_itl_ms`	Inter-Token Latency (P99)	ms
`mean_e2el_ms`	End-to-End Latency (Mean)	ms
`median_e2el_ms`	End-to-End Latency (Median)	ms
`p99_e2el_ms`	End-to-End Latency (P99)	ms
`duration`	Benchmark Duration	s

Llama 3 Inference — NIM

This benchmark measures LLM inference serving performance using NVIDIA NIM containers, driven by GenAI-Perf.

Overview

Field	Value
Benchmark ID	`llama3_inf_single`
Type	Single-node
Min Nodes	1
Description	Llama 3 inference performance benchmark using NVIDIA NIM

Requirements

NGC API Key: Required (set as NGC_API_KEY environment variable)
Podman: Required to run NIM and GenAI-Perf containers

Configuration

Field	Type	Required	Description
`concurrency`	int	Yes	Concurrent requests (1–10000)
`isl`	int	Yes	Input sequence length (1–131072)
`osl`	int	Yes	Output sequence length (1–131072)

Multiple configurations can be submitted in a single job run.

Result Structure

Each configuration produces a BenchmarkMetrics object. Each metric field contains a full statistical distribution.

{
  "request_throughput":     { "unit": "req/s", "avg": 42.3, "p50": 41.8, "p90": 45.1, "p99": 47.2, "min": 38.0, "max": 48.5 },
  "request_latency":        { "unit": "ms",    "avg": 2340.5, "p50": 2180.3, "p90": 3800.1, "p99": 4120.8 },
  "time_to_first_token":    { "unit": "ms",    "avg": 145.2, "p50": 138.7, "p90": 290.4, "p99": 312.4 },
  "time_to_second_token":   { "unit": "ms",    "avg": 163.6, "p50": 156.8 },
  "inter_token_latency":    { "unit": "ms",    "avg": 18.4,  "p50": 17.9,  "p90": 25.1,  "p99": 28.6 },
  "output_token_throughput":           { "unit": "tok/s", "avg": 5890.4 },
  "output_token_throughput_per_request": { "unit": "tok/s", "avg": 139.1 },
  "output_sequence_length": { "unit": "tokens", "avg": 512.0 },
  "input_sequence_length":  { "unit": "tokens", "avg": 256.0 }
}

Each metric object may include: avg, p25, p50, p75, p90, p95, p99, min, max, std.

Field Metadata

Field	Display Name	Unit
`request_throughput`	Request Throughput	req/s
`request_latency`	Request Latency	ms
`time_to_first_token`	Time to First Token (TTFT)	ms
`time_to_second_token`	Time to Second Token	ms
`inter_token_latency`	Inter-Token Latency (ITL)	ms
`output_token_throughput`	Output Token Throughput	tok/s
`output_token_throughput_per_request`	Output Token Throughput per Request	tok/s
`output_sequence_length`	Output Sequence Length	tokens
`input_sequence_length`	Input Sequence Length	tokens

Llama 3 Fine-Tuning — NeMo

This benchmark measures LLM fine-tuning performance using NVIDIA’s NeMo framework with automatic memory-aware parallelism configuration.

Overview

Field	Value
Benchmark ID	`llama3_ft_single`
Type	Single-node
Min Nodes	1
Description	Llama 3 fine-tuning performance benchmark using NVIDIA NeMo

Configuration

Field	Type	Required	Description	Default	Constraints
`model_size`	string	No	Model size	`"8b"`	`"8b"`, `"70b"`, `"405b"`
`dtype`	string	No	Data type	`"fp8"`	`"fp8"`, `"bf16"`
`fine_tune_type`	string	No	Fine-tuning method	`"lora"`	`"lora"`, `"full"`
`max_steps`	int	No	Training steps	`50`

Fixed Parameters

Sequence Length: 4096 tokens
Micro Batch Size: 1 (optimized for packed sequences)
Training Data: Synthetic (SquadDataModule)

Requirements

Software Requirements

NeMo Container: nvcr.io/nvidia/nemo:25.11.01 — downloaded automatically if not present
HuggingFace Token: Required (set as HF_TOKEN environment variable). Get your token from https://huggingface.co/settings/tokens
Docker: Required to run the NeMo container
Disk Space:
- 8B model: ~75GB (55GB base + 20GB model)
- 70B model: ~205GB (55GB base + 150GB model)
- 405B model: ~905GB (55GB base + 850GB model)

export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxx"  # Required
export STAGE_PATH="$HOME/workspace/benchmark_stage"  # Optional, defaults to ~/workspace/benchmark_stage

Hardware Requirements

The benchmark automatically calculates memory requirements based on model configuration:

Model	FP8 Memory	BF16 Memory	LoRA reduction (~30%)
8B	20GB total	35GB total	~14GB / ~25GB
70B	85GB total	160GB total	~60GB / ~112GB
405B	450GB total	850GB total	~315GB / N/A

Minimum GPU requirements:

8B LoRA: 1× GPU with ≥16GB VRAM
8B full: 1× GPU with ≥24GB VRAM
70B: ≥2 GPUs
405B: ≥8 GPUs (FP8 + LoRA only)

Parallelism Strategy

The benchmark automatically calculates optimal parallelism using a memory-aware strategy:

Tensor Parallelism (TP) = smallest power of 2 such that total_memory / TP ≤ gpu_memory
Data Parallelism (DP) = total_gpus / TP
Global Batch Size (GBS) = min(DP × 2, model_cap) — caps: 8B→64, 70B→32, 405B→16

Example Configurations

GPUs	Model	dtype	Fine-tune	TP	DP	GBS
8× 80GB	8B	fp8	lora	1	8	16
8× 80GB	70B	fp8	lora	1	8	16
8× 80GB	405B	fp8	lora	8	1	2

Result Structure

{
  "tokens_per_step": 65536,
  "tokens_per_second": 48617.2,
  "train_step_time_mean": 1.348,
  "train_step_time_std": 0.003,
  "step_time_cv_percent": 0.223,
  "time_to_1t_tokens_days": 238.1,
  "peak_memory_gb": 68.4,
  "memory_efficiency_percent": 85.5
}

Metrics Calculation

Tokens per Step = global_batch_size × sequence_length
Tokens per Second = tokens_per_step ÷ train_step_time_mean
Time to 1T Tokens = 10¹² ÷ (tokens_per_second × 86400) days
Step Time CV = (train_step_time_std ÷ train_step_time_mean) × 100

Field Metadata

Field	Display Name	Unit
`tokens_per_step`	Tokens per Step	tokens
`tokens_per_second`	Tokens per Second	tok/s
`train_step_time_mean`	Training Step Time (Mean)	s
`train_step_time_std`	Training Step Time (Std Dev)	s
`step_time_cv_percent`	Step Time Coefficient of Variation	%
`time_to_1t_tokens_days`	Time to 1T Tokens	days
`peak_memory_gb`	Peak GPU Memory Usage	GB
`memory_efficiency_percent`	GPU Memory Efficiency	%

​Available Benchmarks

​QuickMark Benchmark

​Overview

​Configuration

​Result Structure

​Field Metadata

​Cluster Network Benchmark

​Overview

​Result Structure

​Field Metadata

​Inference Benchmark — vLLM

​Overview

​Configuration

​Result Structure

​Field Metadata

​Llama 3 Inference — NIM

​Overview

​Requirements

​Configuration

​Result Structure

​Field Metadata

​Llama 3 Fine-Tuning — NeMo

​Overview

​Configuration

​Fixed Parameters

​Requirements

​Software Requirements

​Hardware Requirements

​Parallelism Strategy

​Example Configurations

​Result Structure

​Metrics Calculation

​Field Metadata

Available Benchmarks

QuickMark Benchmark

Overview

Configuration

Result Structure

Field Metadata

Cluster Network Benchmark

Overview

Result Structure

Field Metadata

Inference Benchmark — vLLM

Overview

Configuration

Result Structure

Field Metadata

Llama 3 Inference — NIM

Overview

Requirements

Configuration

Result Structure

Field Metadata

Llama 3 Fine-Tuning — NeMo

Overview

Configuration

Fixed Parameters

Requirements

Software Requirements

Hardware Requirements

Parallelism Strategy

Example Configurations

Result Structure

Metrics Calculation

Field Metadata