Welcome to TensorRT-LLM’s Documentation!

Getting Started

• Overview
  • About TensorRT-LLM
  • What Can You Do With TensorRT-LLM?
• Quick Start Guide
  • Prerequisites
  • LLM API
  • Compile the Model into a TensorRT Engine
  • Run the Model
  • Deploy with Triton Inference Server
  • Next Steps
  • Related Information
• Key Features
• Release Notes
  • TensorRT-LLM Release 0.16.0
  • TensorRT-LLM Release 0.15.0
  • TensorRT-LLM Release 0.14.0
  • TensorRT-LLM Release 0.13.0
  • TensorRT-LLM Release 0.12.0
  • TensorRT-LLM Release 0.11.0
  • TensorRT-LLM Release 0.10.0
  • TensorRT-LLM Release 0.9.0
  • TensorRT-LLM Release 0.8.0
  • TensorRT-LLM Release 0.7.1

Installation

• Installing on Linux
• Building from Source Code on Linux
  • Prerequisites
  • Building a TensorRT-LLM Docker Image
  • Building the Python Bindings for the C++ Runtime
  • Linking with the TensorRT-LLM C++ Runtime
• Installing on Windows
• Building from Source Code on Windows
  • Prerequisites
  • Building a TensorRT-LLM Docker Image
  • Building TensorRT-LLM on Bare Metal
  • Linking with the TensorRT-LLM C++ Runtime
• Installing on Grace Hopper

Architecture

• TensorRT-LLM Architecture
  • Model Weights
• Model Definition
• Compilation
  • TensorRT Compiler
  • Model Engine
  • Weight Bindings
  • Pattern-Matching and Fusion
  • Plugins
• Runtime
• Multi-GPU and Multi-Node Support
  • Examples
• TensorRT-LLM Checkpoint
  • Overview
  • Prepare the TensorRT-LLM Checkpoint
  • Build Checkpoint into TensorRT Engine
  • Make Evaluation
• TensorRT-LLM Build Workflow
  • Overview
  • Conversion APIs
  • Quantization APIs
  • Build APIs
  • CLI Tools
• Adding a Model
  • Step 1. Write Modeling Part
  • Step 2. Implement Weight Conversion
  • Step 3. Register New Model
  • Step 4. Verify New Model
  • Reference

Advanced

• Multi-Head, Multi-Query, and Group-Query Attention
  • Important Note
  • Padded and Packed Tensors
  • Context and Generation Phases
  • In-flight Batching
  • Chunked Context
  • KV Cache
  • INT8/FP8 KV Caches
  • Sliding Window Attention, Cyclic (Rolling Buffer) KV Cache
  • StreamingLLM
  • Beam-Search
  • Input QKV tensor
• C++ GPT Runtime
  • Overview
  • The Session
  • In-flight Batching Support
  • Know Issues and Future Changes
• Executor API
  • API
  • C++ Executor API Example
  • Python Bindings for the Executor API
  • In-flight Batching with the Triton Inference Server
• Graph Rewriting Module
  • When to Use Graph Rewriting?
  • Graph Rewriting APIs
  • Classical Workflow
• Inference Request
• Responses
• Run gpt-2b + LoRA using GptManager / cpp runtime
  • LoRA tensor format details
• Expert Parallelism in TensorRT-LLM
  • Mixture of Experts (MoE)
  • Tensor Parallel vs Expert Parallel
  • How to Enable
• KV cache reuse
  • How to enable kv cache reuse
  • Performance expectations
  • Situations that can prevent kv cache reuse
  • Offloading to host memory
• Speculative Sampling
  • About Speculative Sampling
  • Performance Improvements
  • Draft-Target-Model
  • Prompt-Lookup-Decoding
  • Medusa
  • ReDrafter
  • EAGLE
  • Lookahead Decoding
• Disaggregated-Service (experimental)
  • Usage
  • Benchmarks
  • Troubleshooting and FAQ

Performance

• Overview
  • Known Issues
  • Throughput Measurements
  • Reproducing Benchmarked Results
  • Online Serving Measurements
• Benchmarking
  • Supported Networks for Benchmarking
  • Support Quantization Modes
  • Inflight Benchmarking with a Dataset
  • Preparing a Dataset
  • Building a Benchmark Engine
  • Max Throughput Benchmark
  • Low Latency Benchmark
  • Summary
• Best Practices
  • How To Measure Performance?
  • Build Options to Optimize the Performance of TensorRT-LLM Models
  • Runtime Options to Optimize the Performance of TensorRT-LLM Models
• Performance Analysis
  • Feature Descriptions
  • Usage
  • Coordinating with NVIDIA Nsight Systems Launch
  • Examples
  • Profiling a single IFB iteration executing on a single rank of a multi-GPU model

Reference

• Troubleshooting
  • Installation Errors
  • Debug on Unit Tests
  • Debug on E2E Models
  • Debug Execution Errors
  • Tips
• Support Matrix
  • Models
  • Hardware
  • Software
• Numerical Precision
  • FP32, FP16 and BF16
  • Quantization and Dequantization (Q/DQ)
  • INT8 SmoothQuant (W8A8)
  • INT4 and INT8 Weight-Only (W4A16 and W8A16)
  • GPTQ and AWQ (W4A16)
  • FP8 (Hopper)
  • Support matrix
  • Technical Detail: The QuantMode Flags
• Memory Usage of TensorRT-LLM
  • Understand inference time GPU memory usage
  • Memory pool
  • Known Issues
  • FAQ

Indices and tables

• Index

• Module Index

• Search Page