Quantizing Models with GPTQ for INT4: VRAM and Latency Analysis

Introduction

In our last lesson, we successfully quantized a model to INT8 using bitsandbytes, achieving a nearly 2x reduction in the VRAM needed for model weights. We saw that this came with a three-way trade-off between memory, latency, and model quality.

Today, we're pushing the boundary of memory optimization even further. Our goal is to quantize a model to INT4 using an algorithm like GPTQ and measure the resulting VRAM savings and latency change. This is a more aggressive compression, promising a theoretical 4x reduction in weight memory compared to fp16. However, squeezing model weights into just 16 possible values (4 bits) without catastrophic quality loss requires a more sophisticated approach than we used for INT8. We'll explore the GPTQ algorithm, understand why it's so effective, and then apply it in a hands-on benchmark.

1. The Challenge of 4-Bit Quantization

Moving from 8 bits (256 values) to 4 bits (16 values) dramatically increases the potential for quantization error. The outlier problem we discussed in the last lesson becomes even more severe. If we used a simple scaling method like absmax, a single large weight value would force the vast majority of other weights to be quantized to just one or two of the 16 available integer values. This would effectively erase most of the learned information in the weight matrix.

To solve this, researchers developed advanced Post-Training Quantization (PTQ) algorithms. One of the most influential is GPTQ (Generative Pre-trained Transformer Quantization).

The core insight of GPTQ is to quantize the model's weights layer by layer (or even block by block within a layer) and, crucially, to iteratively update the remaining unquantized weights to compensate for the error introduced by the weights it just quantized. This makes the process error-aware, leading to significantly higher accuracy for the final 4-bit model.

2. The GPTQ Algorithm at a High Level

Let's unpack how GPTQ achieves this accurate quantization.

As the article describes, GPTQ improves upon prior work by making the quantization process computationally feasible for billion-parameter models. The key is that instead of finding the perfect order to quantize weights (which is computationally expensive), it processes them in a fixed order and uses clever numerical techniques to update the remaining weights efficiently.

The process can be visualized as follows:

A final important piece is dynamic dequantization. During inference, the weights are stored in VRAM as INT4. When a layer is executed, a specialized CUDA kernel fetches the INT4 weights, dequantizes them back to fp16 on-the-fly, and then performs the matrix multiplication. Because moving data from VRAM to the GPU's compute units is often a bottleneck, fetching smaller INT4 data results in a significant speedup, even with the added dequantization step.

The article "GPTQ: Accurate Post-Training Quantization..." describes this process well. You can review the section "Dynamic Dequantization in Inference" (resource LINK, section 3) for more detail.

3. Hands-On: Benchmarking GPTQ INT4 vs. FP16

Now, let's apply GPTQ to the same TinyLlama model from our previous lesson and measure the impact. We will use the auto-gptq library, which is integrated into Hugging Face transformers.

Setup:
This practical requires the auto-gptq library.

# Ensure you have the basics from the last lesson
pip install torch transformers accelerate bitsandbytes



# Install the GPTQ library
pip install auto-gptq

Benchmark Script:
The script below is an extension of our previous benchmark. It first measures the FP16 baseline, then performs GPTQ quantization, and finally benchmarks the new INT4 model.

A key part of the GPTQ algorithm is the use of a calibration dataset. The algorithm uses this small sample of text to observe the activation patterns and make more informed decisions about how to quantize weights to minimize error on realistic inputs. This is specified in the GPTQConfig.

import torch
import time
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig
import numpy as np

def get_vram_usage_gb():
    """Returns the maximum allocated VRAM in gigabytes."""
    if not torch.cuda.is_available():
        return 0
    return torch.cuda.max_memory_allocated() / (1024 ** 3)

def benchmark_latency(model, tokenizer, prompt, num_tokens=100, num_runs=10):
    """Measures average latency and throughput."""
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    



    # Warm-up runs
    for _ in range(2):
        _ = model.generate(**inputs, max_new_tokens=5)
    
    timings = []
    for _ in range(num_runs):
        torch.cuda.synchronize()
        t0 = time.perf_counter()
        _ = model.generate(**inputs, max_new_tokens=num_tokens, pad_token_id=tokenizer.eos_token_id)
        torch.cuda.synchronize()
        t1 = time.perf_counter()
        timings.append(t1 - t0)
        
    avg_latency = np.mean(timings)
    tokens_per_second = (num_tokens - 1) / avg_latency
    return avg_latency, tokens_per_second

def run_benchmark(model_name: str, quantization_config=None, is_quantized=False):
    """Loads a model, performs quantization if needed, and benchmarks it."""
    config_name = "INT4 (GPTQ)" if is_quantized else "FP16"
    print(f"\n--- Benchmarking: {config_name} ---")

    if torch.cuda.is_available():
        torch.cuda.empty_cache()
        torch.cuda.reset_max_memory_allocated()




    # --- Load Model and Tokenizer ---
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    if tokenizer.pad_token is None:
        tokenizer.pad_token = tokenizer.eos_token




    # --- Perform Quantization (if applicable) ---
    print("Loading model...")
    quantization_start_time = time.perf_counter()
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype=torch.float16,
        quantization_config=quantization_config,
        device_map="auto"
    )
    quantization_end_time = time.perf_counter()
    
    if is_quantized:
        print(f"Quantization process took: {quantization_end_time - quantization_start_time:.2f}s")

    vram_after_load = get_vram_usage_gb()
    print(f"VRAM after load: {vram_after_load:.2f} GB")




    # --- Benchmark Performance ---
    prompt = "An AI Systems Engineer needs to consider several factors when deploying a model, such as"
    avg_latency, tokens_per_second = benchmark_latency(model, tokenizer, prompt)
    
    print(f"Average latency: {avg_latency*1000:.2f} ms")
    print(f"Throughput: {tokens_per_second:.2f} tokens/sec")




    # Clean up to free VRAM for the next run
    del model
    if torch.cuda.is_available():
        torch.cuda.empty_cache()
        
    return {
        "VRAM (GB)": vram_after_load,
        "Tokens/sec": tokens_per_second
    }

if __name__ == "__main__":
    if not torch.cuda.is_available():
        print("CUDA not available. This benchmark requires a GPU.")
        exit()

    MODEL_ID = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
    



    # 1. Benchmark FP16 (baseline)
    fp16_results = run_benchmark(MODEL_ID, is_quantized=False)




    # 2. Define GPTQ configuration and benchmark INT4
    # The dataset is used for calibration during quantization.
    gptq_config = GPTQConfig(
        bits=4,
        dataset="c4", # A standard dataset for calibration. 'wikitext2', 'ptb' are also options.
        tokenizer=MODEL_ID,
        desc_act=False # Workaround for a potential bug with some model architectures
    )
    int4_results = run_benchmark(MODEL_ID, quantization_config=gptq_config, is_quantized=True)




    # 3. Print comparison
    print("\n--- Comparison Summary (FP16 vs. INT4-GPTQ) ---")
    print(f"{'Metric':<15} | {'FP16':<10} | {'INT4':<10} | {'Change':<10}")
    print("-" * 55)

    vram_fp16 = fp16_results['VRAM (GB)']
    vram_int4 = int4_results['VRAM (GB)']
    vram_change = ((vram_int4 - vram_fp16) / vram_fp16) * 100
    print(f"{'VRAM (GB)':<15} | {vram_fp16:<10.2f} | {vram_int4:<10.2f} | {vram_change:<9.2f}%")

    speed_fp16 = fp16_results['Tokens/sec']
    speed_int4 = int4_results['Tokens/sec']
    speed_change = ((speed_int4 - speed_fp16) / speed_fp16) * 100
    print(f"{'Tokens/sec':<15} | {speed_fp16:<10.2f} | {speed_int4:<10.2f} | {speed_change:<9.2f}%")

4. Analyzing the Trade-offs

After running the script, you will have concrete data on the impact of INT4 GPTQ quantization. Here's what to look for:

1. Quantization Time: Notice that unlike loading a pre-quantized model, the from_pretrained call for the GPTQ model takes a significant amount of time. This is the one-time cost of running the GPTQ algorithm on the FP16 weights. For TinyLlama, this might be a few minutes; for a 70B model, it could take hours.
2. VRAM Savings: This is the main prize. The VRAM usage for the INT4 model should be dramatically lower. For the ~1.1B parameters of TinyLlama, the weights go from 1.1B * 2 bytes = 2.2 GB in FP16 to 1.1B * 0.5 bytes = 0.55 GB in INT4. Your measurement of total VRAM will reflect this dramatic drop, confirming a memory footprint close to 25% of the original.
3. Latency Change (Throughput): You should see a noticeable increase in tokens/sec. This demonstrates the benefit of reduced memory bandwidth pressure and the efficiency of the custom GPTQ kernels. Unlike the bitsandbytes INT8 approach which can sometimes be slower for small models, GPTQ's optimized inference path generally yields a speedup.
4. Quality Degradation: While our script doesn't measure perplexity, it's the hidden part of the trade-off. GPTQ minimizes this degradation, but it's not zero.

To better frame these trade-offs, let's consider the broader context of model size versus precision.

This conclusion is critical for an AI Systems Engineer. Given a fixed hardware budget (e.g., a single GPU with 24GB of VRAM), 4-bit quantization allows you to run a much larger, more capable model (e.g., a 30B parameter model) than you could at FP16 (a ~10B model). The superior intelligence of the larger model almost always outweighs the minor quality loss from quantization.

Conclusion

In this lesson, we've taken a deep dive into 4-bit quantization with GPTQ, a cornerstone technique for efficient LLM serving. You've moved beyond simple quantization to a sophisticated, error-aware algorithm and have empirically verified its benefits.

Key Takeaways:

• INT4 offers ~4x weight memory reduction: This aggressive quantization is key to fitting larger, more powerful models onto existing hardware.
• GPTQ preserves accuracy: By intelligently compensating for quantization errors during its process, GPTQ maintains high model quality even at 4-bit precision.
• Performance is a balance: GPTQ provides substantial VRAM savings and often improves inference speed, at the cost of a small, one-time quantization compute cost and a slight, often negligible, drop in perplexity.
• System Design Principle: For a given hardware constraint, deploying a larger model at 4-bit precision is generally a better strategy than deploying a smaller model at full precision.

Preview of the Next Lesson:

GPTQ is a post-training quantization method that cleverly minimizes the error of quantizing an already-trained model. But what if we could design the quantization process with more awareness of the model's internal workings? In the next lesson, we will explore Activation-aware Weight Quantization (AWQ). This method analyzes the model's activations to identify which weights are most important to the model's performance and protects them during quantization, offering another powerful approach to the quality-vs-compression trade-off.