Apple Silicon has quietly become one of the best platforms for running local LLMs, and the secret weapon is unified memory. An M3 Max with 96 GB of RAM can run a 70B parameter model that would require a multi-GPU workstation on any other consumer hardware. MLX is Apple’s open-source machine learning framework designed specifically for Apple Silicon, and mlx-lm is the package that puts LLM inference on top of it.

This guide covers installation, running Llama 3 and Mistral locally, performance expectations across the M-chip lineup, how MLX compares to Ollama on Mac, and quantization options for fitting larger models into available memory.

Why MLX Instead of Ollama?

Ollama is the easiest path to local LLMs on any platform, including Mac. It uses llama.cpp under the hood, which runs on Apple Silicon via Metal GPU acceleration. MLX is different — it is a framework built specifically by Apple’s machine learning research team to exploit Apple Silicon’s architecture, particularly the unified memory bus that allows the CPU, GPU, and Neural Engine to share the same memory pool without data copies.

In practice, MLX squeezes more tokens per second out of Apple Silicon than llama.cpp does, especially on the higher-end chips with more GPU cores and memory bandwidth. The tradeoff is a steeper setup process and a more technical user experience.

Installation

You need a Mac with Apple Silicon (M1 or newer) and Python 3.10+. The setup is straightforward:

# Install MLX and the LLM package
pip install mlx-lm

# Verify installation
python -c "import mlx; print(mlx.__version__)"

That is the entire setup. The mlx-lm package pulls in MLX as a dependency. No additional GPU drivers, CUDA, or special runtime configuration needed.

Downloading and Running Models

MLX models are served from Hugging Face under the mlx-community organization. These are pre-converted model weights in MLX’s native format, often available in multiple quantization levels.

Running Llama 3

# Run Llama 3.2 3B — works on any Apple Silicon Mac
mlx_lm.generate \
  --model mlx-community/Llama-3.2-3B-Instruct-4bit \
  --prompt "Explain how the transformer architecture works" \
  --max-tokens 500

# Run Llama 3.3 70B — requires 40+ GB unified memory
mlx_lm.generate \
  --model mlx-community/Llama-3.3-70B-Instruct-4bit \
  --prompt "Write a Python implementation of the A* pathfinding algorithm" \
  --max-tokens 1000

Running Mistral

mlx_lm.generate \
  --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
  --prompt "What are the key differences between RAG and fine-tuning?" \
  --max-tokens 400

Using the Python API

For integration into your own scripts:

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Llama-3.2-3B-Instruct-4bit")

prompt = "Write a function to parse ISO 8601 dates in Python"

if hasattr(tokenizer, "apply_chat_template") and tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )

response = generate(model, tokenizer, prompt=prompt, max_tokens=500, verbose=True)
print(response)

The verbose=True flag prints token generation speed during inference.

Running a Local Chat Server

mlx-lm includes an OpenAI-compatible server:

mlx_lm.server \
  --model mlx-community/Llama-3.2-3B-Instruct-4bit \
  --port 8080

This exposes a /v1/chat/completions endpoint that any OpenAI-compatible client can use — Continue.dev, Open WebUI, LangChain, etc.

Performance on M-Series Chips

Real-world token generation speeds for 4-bit quantized models:

Chip	Memory	Llama 3.2 3B	Mistral 7B	Llama 3.3 70B
M1 (8 GB)	8 GB	~40 tok/s	~18 tok/s	Not feasible
M2 (16 GB)	16 GB	~65 tok/s	~30 tok/s	Not feasible
M3 Pro (18 GB)	18 GB	~80 tok/s	~38 tok/s	Not feasible
M3 Max (48 GB)	48 GB	~120 tok/s	~70 tok/s	~12 tok/s
M4 Pro (24 GB)	24 GB	~110 tok/s	~55 tok/s	Not feasible
M4 Max (128 GB)	128 GB	~140 tok/s	~85 tok/s	~18 tok/s

Memory is the primary constraint, not compute. With a 4-bit quantized Llama 3.3 70B requiring roughly 40 GB, you need at least an M3 Max or M4 Max with 48+ GB to run it at all. On chips with sufficient memory, the generation speed is impressive — 12–18 tokens per second on a 70B model on a laptop is genuinely remarkable.

Quantization Options

Quantization reduces model size by representing weights in fewer bits. More quantization = smaller file and faster inference, but at some cost to accuracy.

Common formats on the mlx-community Hugging Face org:

Suffix	Bits per weight	Size (7B model)	Quality
`-4bit`	4	~4 GB	Good (most popular)
`-8bit`	8	~7 GB	Near-lossless
`-3bit`	3	~3 GB	Moderate degradation
`-fp16`	16	~14 GB	Full quality

4-bit is the practical choice for most users. The quality difference versus 8-bit or fp16 is minor for conversational tasks and code generation. You notice the gap more with complex mathematical reasoning or very long context.

You can also quantize models yourself using mlx-lm:

mlx_lm.convert \
  --hf-path meta-llama/Llama-3.2-3B-Instruct \
  --mlx-path ./llama-3.2-3b-4bit \
  -q \
  --q-bits 4

This converts a Hugging Face model to MLX format with 4-bit quantization. You need the original model files, which for gated models like Llama requires a Hugging Face account with access approval.

MLX vs Ollama on Mac: Practical Comparison

Aspect	MLX (mlx-lm)	Ollama
Setup difficulty	Moderate (pip)	Easy (installer)
Speed (Apple Silicon)	10–30% faster	Baseline
Model availability	mlx-community HF models	Ollama model library
OpenAI-compatible server	Yes (mlx_lm.server)	Yes (default)
UI	None (CLI/API only)	None (CLI/API only)
Custom model support	Yes (convert any HF model)	Yes (Modelfile)
Memory efficiency	Excellent	Good
Cross-platform	Mac only	Mac, Linux, Windows

The speed advantage of MLX over Ollama on Apple Silicon is real — typically 15–25% more tokens per second on the same model and chip. Whether that matters depends on your use case. For interactive chat, both feel fast enough on a 7B model. For batch processing or generation of long documents, the MLX speed advantage compounds.

Use Ollama if you want the simplest setup, the broadest model selection from a single tool, and a workflow that also works when you are on a Linux server. Use MLX if you are committed to Mac, want maximum performance from your hardware, and are comfortable with a command-line-first workflow. Both are excellent — the choice is mostly about ergonomics.