Speed is the variable nobody talks about enough when evaluating LLM APIs. OpenAI and Anthropic produce excellent models, but their inference throughput is throttled by GPU memory bandwidth in ways that make streaming feel sluggish for real-time applications. Groq takes a radically different approach with its Language Processing Unit (LPU) hardware, and the results are legitimately startling — 800 to 1,500 tokens per second on production models, compared to 40–80 tokens per second on typical GPU-based endpoints.

This guide explains why Groq is fast, how to get started with its API, which models are available, and what the rate limits look like in practice.

The LPU Architecture Explained

Traditional LLM inference runs on GPUs, which are designed for parallel matrix math but communicate through memory hierarchies that create bottlenecks during the sequential token-generation phase of autoregressive decoding. Each new token requires loading model weights from memory, and the memory bandwidth ceiling — not raw compute — is usually what limits throughput.

Groq’s LPU (Language Processing Unit) is purpose-built for this sequential workload. It uses a deterministic single-core architecture with enormous on-chip SRAM, eliminating the DRAM bottleneck almost entirely. The chip does not parallelize across a batch of requests the way a GPU does — it runs each request independently but completes each token generation step in a predictable, extremely fast cycle.

The practical outcome: Groq serves Llama 3 70B at speeds that make real-time voice applications, interactive coding assistants, and sub-second RAG retrieval genuinely feasible.

Supported Models

Groq’s model lineup focuses on top-tier open-weight models:

Model	Parameters	Context	Speed (tokens/sec)
`llama-3.3-70b-versatile`	70B	128K	~900
`llama-3.1-8b-instant`	8B	128K	~1,800
`llama3-groq-70b-8192-tool-use-preview`	70B	8K	~700
`mixtral-8x7b-32768`	8x7B MoE	32K	~1,100
`gemma2-9b-it`	9B	8K	~1,600
`whisper-large-v3`	—	Audio	Real-time STT

The llama-3.1-8b-instant model at over 1,800 tokens per second is the fastest option and is genuinely useful for low-latency applications where an 8B model’s capability is sufficient. The 70B versatile variant hits the better capability ceiling while still running at roughly 10x the speed of a typical GPU-hosted endpoint.

API Key Setup

Visit console.groq.com and sign up with a Google or GitHub account.
Navigate to API Keys → Create API Key.
Copy the key — it starts with gsk_.

export GROQ_API_KEY="gsk_your-key-here"

Install the Groq Python SDK:

pip install groq

Making Your First Request

from groq import Groq

client = Groq(api_key="gsk_your-key-here")

chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "Explain the transformer attention mechanism in plain English.",
        }
    ],
    model="llama-3.3-70b-versatile",
)

print(chat_completion.choices[0].message.content)

The API is OpenAI-compatible, so you can also use the openai Python library:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.groq.com/openai/v1",
    api_key="gsk_your-key-here",
)

response = client.chat.completions.create(
    model="mixtral-8x7b-32768",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Streaming for Real-Time Applications

Streaming is where Groq’s speed advantage is most visible. With a GPU-hosted model, you might wait 2–3 seconds for the first token. With Groq, the first token often arrives in under 200 milliseconds.

stream = client.chat.completions.create(
    messages=[{"role": "user", "content": "Write a Python quicksort implementation."}],
    model="llama-3.1-8b-instant",
    stream=True,
)

for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)

The time-to-first-token (TTFT) on llama-3.1-8b-instant is consistently under 100 ms, making it viable for voice synthesis pipelines where latency directly affects perceived naturalness.

Speed Benchmarks vs OpenAI

These are approximate figures based on publicly available benchmarks and community testing. Actual performance varies with load:

Provider	Model	Avg tokens/sec	TTFT
Groq	Llama 3.1 8B	~1,800	<100 ms
Groq	Llama 3.3 70B	~900	~200 ms
OpenAI	GPT-4o	~80	~500 ms
OpenAI	GPT-4o mini	~200	~300 ms
Anthropic	Claude 3.5 Haiku	~150	~400 ms

For applications where GPT-4o quality is not required, Groq’s Llama 70B delivers competitive results at 10x the output speed and at significantly lower cost.

Rate Limits

The free tier rate limits are generous for development but will constrain production use:

Model	Requests/min	Tokens/min	Tokens/day
llama-3.3-70b-versatile	30	6,000	100,000
llama-3.1-8b-instant	30	20,000	500,000
mixtral-8x7b-32768	30	5,000	500,000

Paid plans (GroqCloud Developer and above) raise these limits substantially and are priced per million tokens — competitive with direct OpenAI pricing while delivering far better latency.

Rate limit headers are returned with every response:

x-ratelimit-limit-requests: 30
x-ratelimit-remaining-requests: 28
x-ratelimit-reset-requests: 2s

Handle 429 responses with exponential backoff:

import time
from groq import RateLimitError

def call_with_retry(client, **kwargs):
    for attempt in range(5):
        try:
            return client.chat.completions.create(**kwargs)
        except RateLimitError:
            time.sleep(2 ** attempt)
    raise RuntimeError("Max retries exceeded")

Tool Use and Function Calling

Groq supports OpenAI-compatible function calling on the llama3-groq-70b-8192-tool-use-preview model:

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
                },
                "required": ["location"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="llama3-groq-70b-8192-tool-use-preview",
    messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
    tools=tools,
    tool_choice="auto",
)

Best Use Cases for Groq

Groq’s speed makes it the obvious choice for real-time voice assistants, interactive coding tools, high-throughput document processing, and any application where the gap between human reading speed and model output speed creates a frustrating experience. For batch processing of large documents where latency is irrelevant, a standard GPU endpoint may be more cost-effective. But for anything interactive, Groq’s LPU changes the user experience in a way that is hard to go back from once you have tried it.