Uploading confidential documents to ChatGPT or Claude raises real questions about data privacy, especially when those documents contain client data, internal financials, or proprietary research. PrivateGPT is an open-source application that lets you chat with your documents entirely on your own hardware — no internet connection required once set up, no data sent to external servers, and no per-query API costs.

This guide covers installation, the document formats it handles, how to ingest PDFs and Word documents, how to choose an embedding model, and how PrivateGPT stacks up against its main alternative, AnythingLLM.

How PrivateGPT Works

PrivateGPT implements a retrieval-augmented generation (RAG) pipeline entirely locally:

You ingest documents into a local vector store (ChromaDB by default).
Your question is converted to a vector embedding using a local embedding model.
The embedding is used to retrieve the most semantically relevant document chunks.
Those chunks are injected as context into a prompt sent to a local LLM.
The LLM answers based on that context.

The critical difference from cloud-based RAG tools: both the embedding step and the generation step happen on your machine. Nothing leaves.

Installation

PrivateGPT requires Python 3.11 and Poetry for dependency management.

# Clone the repository
git clone https://github.com/zylon-ai/private-gpt
cd private-gpt

# Install Poetry if you don't have it
pip install poetry

# Install dependencies (CPU-only profile)
poetry install --extras "ui llms-ollama embeddings-ollama vector-stores-qdrant"

# Or for a GPU-accelerated setup with llama.cpp
poetry install --extras "ui llms-llama-cpp embeddings-huggingface vector-stores-chroma"

PrivateGPT has a modular extras system — you compose your installation by selecting the LLM backend, embedding provider, and vector store you want.

Quick Start with Ollama

The easiest setup uses Ollama for both inference and embeddings. Install Ollama first, then pull the required models:

ollama pull llama3:8b           # LLM for generation
ollama pull nomic-embed-text    # Embedding model

Configure PrivateGPT by editing settings-ollama.yaml:

llm:
  mode: ollama
  max_new_tokens: 512
  context_window: 3900

ollama:
  llm_model: llama3:8b
  embedding_model: nomic-embed-text
  api_base: http://localhost:11434
  keep_alive: 5m
  tfs_z: 1.0
  num_predict: 512

Launch:

PGPT_PROFILES=ollama make run

The web UI opens at http://localhost:8001.

Supported Document Formats

PrivateGPT handles a broad range of file types out of the box:

Format	Extension	Notes
PDF	`.pdf`	Text extraction; scanned PDFs need OCR pre-processing
Word	`.docx`, `.doc`	Full paragraph and table extraction
Text	`.txt`, `.md`	Direct ingestion
CSV	`.csv`	Each row treated as a document chunk
HTML	`.html`	Tag stripping applied
EPUB	`.epub`	E-book chapter chunking
PowerPoint	`.pptx`	Slide text extracted

Scanned PDFs (image-based, not text-based) require OCR pre-processing before ingestion. Run them through tesseract or Adobe Acrobat first to produce text-extractable PDFs.

Ingesting PDFs and Word Documents

Via the Web UI

Open http://localhost:8001.
Click Upload File(s) in the left panel.
Select one or more files — PDFs and .docx files are accepted directly.
PrivateGPT chunks the documents, generates embeddings, and stores them in the vector database.
A success indicator appears when ingestion completes.

Via the API

PrivateGPT exposes a REST API for programmatic ingestion:

# Ingest a single file
curl -X POST http://localhost:8001/v1/ingest/file \
  -H "Content-Type: multipart/form-data" \
  -F "file=@/path/to/your/document.pdf"

# List ingested documents
curl http://localhost:8001/v1/ingest/list

# Delete a document by ID
curl -X DELETE http://localhost:8001/v1/ingest/YOUR_DOC_ID

Bulk Ingestion from a Folder

Place files in the source_documents/ directory and run:

poetry run python scripts/ingest_folder.py

This recursively processes all supported files in the folder — useful for initial bulk ingestion of a document archive.

Choosing an Embedding Model

The embedding model determines how well semantic search works. A poor embedding model means the retrieval step surfaces the wrong chunks, and the LLM cannot answer accurately regardless of its own quality.

Model	Size	Speed	Quality
`nomic-embed-text` (via Ollama)	274 MB	Fast	Good
`all-MiniLM-L6-v2` (HuggingFace)	22 MB	Very fast	Moderate
`BAAI/bge-large-en-v1.5` (HuggingFace)	1.3 GB	Moderate	Excellent
`text-embedding-3-small` (OpenAI API)	Cloud	Fast	Excellent

For a fully local setup, nomic-embed-text via Ollama is the practical choice — it is easy to set up and delivers strong retrieval quality. If you can tolerate a larger download and want the best local retrieval accuracy, BAAI/bge-large-en-v1.5 is worth the extra storage.

To use a HuggingFace embedding model, switch the profile:

embedding:
  mode: huggingface
  
huggingface:
  embedding_hf_model_name: BAAI/bge-large-en-v1.5

Querying Your Documents

Once documents are ingested, select Query Documents mode in the web UI (as opposed to LLM Chat mode, which ignores your documents and just chats with the model).

The query UI shows retrieved source chunks below the answer, with citations indicating which document and page the information came from. This transparency is one of PrivateGPT’s strengths — you can verify that the answer is grounded in your documents rather than model hallucination.

PrivateGPT vs AnythingLLM

Both tools solve the same core problem, but with different priorities:

Feature	PrivateGPT	AnythingLLM
Setup difficulty	Moderate (Poetry, profiles)	Easy (Docker or installer)
UI quality	Functional, minimal	Polished, feature-rich
Multi-workspace	No	Yes
Web scraping ingestion	No	Yes
Agent tools	Basic	Advanced (web search, etc.)
API	Full REST API	Full REST API
Performance tuning	Extensive config	Moderate config
Cloud LLM support	Yes (OpenAI, etc.)	Yes
Fully local capable	Yes	Yes

Choose PrivateGPT if you want a hackable, API-first tool with deep configuration options and are comfortable with a code-level setup. Choose AnythingLLM if you want a polished UI, multi-workspace document organization, and a Docker-based setup that non-developers can manage.

For teams with sensitive documents and a technical operator available to maintain it, PrivateGPT’s transparency and configurability make it the more trustworthy choice for serious deployments.