KoboldCpp: Run Local LLMs for Creative Writing and RP

KoboldCpp is a single-file application that runs GGUF-format language models locally, designed specifically for creative writing and interactive fiction. It features an integrated story editor (KoboldAI Lite), AI Horde support, and robust sampling controls that authors use to tune exactly how creative or constrained the model writes.

What Makes KoboldCpp Different

Most local LLM tools (Ollama, LM Studio, Jan) focus on chat assistants. KoboldCpp is purpose-built for:

• Long-form creative writing: maintains story context over long passages
• Interactive fiction: the AI plays a character or narrator while you direct the story
• Character roleplay: consistent persona maintenance
• Sampling parameters: extensive controls over randomness and repetition

Installation

KoboldCpp ships as a single executable — no Python environment needed.

Windows (easiest):

1. Download koboldcpp.exe from the latest release at github.com/LostRuins/koboldcpp/releases
2. Double-click to run — the launcher opens in your browser at http://localhost:5001

Linux:

# Download binary
wget https://github.com/LostRuins/koboldcpp/releases/latest/download/koboldcpp-linux-x64

chmod +x koboldcpp-linux-x64

# Run with a model
./koboldcpp-linux-x64 --model /path/to/model.gguf --gpulayers 35

macOS (Apple Silicon):

# Download Mac binary
# Run with Metal GPU support
./koboldcpp --model model.gguf --usemetal

Downloading Models

KoboldCpp works with any GGUF model. Download from Hugging Face:

Best models for creative writing:

• Llama-3.1-8B-Instruct Q6_K: excellent instruction following, uncensored fine-tunes available
• Mistral-7B-v0.3-GGUF: strong creative output, fast
• Phi-4-14B Q4: Microsoft’s capable model, good narrative sense
• Llama-3.3-70B Q3 (24GB VRAM): near-GPT-4 quality locally

Popular uncensored creative writing models from Hugging Face:

• Dolphin series (by Eric Hartford): Dolphin-Llama3-8B, Dolphin-Mistral-7B
• Noromaid-series: roleplay and creative focused
• Midnight-Miqu: story and creative writing specific

Launch Configuration

When you run KoboldCpp, a launcher dialog appears:

• Model File: path to your GGUF model
• GPU Layers: number of layers to offload to GPU (higher = faster); set to 99 to maximize GPU use
• Context Size: how much text the model “remembers” — 4096 (default), 8192, 32768 depending on model support and VRAM
• BLAS Batch Size: 512 default — increase for GPU throughput

Or launch from command line:

./koboldcpp \
  --model dolphin-llama3-8b.Q6_K.gguf \
  --gpulayers 35 \
  --contextsize 8192 \
  --port 5001

The KoboldAI Lite Interface

Open http://localhost:5001 in your browser.

Story Mode: the classic interactive fiction interface. You write or the AI writes, alternating. Use AI to let the model continue, Submit to add your own text.

Instruct Mode: for instruct-tuned models — send prompts and get responses like a chat interface.

Chat Mode: conversational interface with persistent memory.

Sampling Parameters (The Secret Sauce)

KoboldCpp exposes the full sampling stack — this is where creative writing users spend time tuning:

Temperature: higher = more creative but less coherent. 0.7–1.0 for fiction, 0.3–0.6 for factual.

Top-P (nucleus sampling): only sample from top P probability mass. 0.9 is a common starting point.

Top-K: limit to top K tokens at each step. 40–100 typical.

Repetition Penalty: prevents the model from repeating phrases. 1.1–1.2 recommended for fiction. Too high causes incoherent output.

Min-P: newer sampling method that works well with higher temperatures — filters low-probability tokens while allowing expressive variation.

Recommended creative writing presets:

Temperature: 1.0
Top-P: 0.9
Top-K: 100
Repetition Penalty: 1.1
Min-P: 0.05

Story Memory and Author’s Notes

Memory: persistent context always prepended to the prompt. Use for world description, character sheets, story rules. Example:

Setting: Post-apocalyptic Seattle, 2050. The city flooded in 2031.
Protagonist: Maya Chen, 28, former marine turned scavenger.
Tone: Gritty, realistic, sparse dialogue.

Author’s Note: appears closer to the current text (more influential). Use for immediate story direction:

[Current scene: tense confrontation. Maya is outnumbered. She needs to talk her way out.]

World Info: A database of keywords that inject relevant information when mentioned. Define “Cascade faction” → inject lore about them whenever the word appears.

API for Integration

KoboldCpp provides an API compatible with KoboldAI and partially with OpenAI format:

import requests

response = requests.post("http://localhost:5001/api/v1/generate", json={
    "prompt": "In the ruins of Seattle, Maya found",
    "max_length": 200,
    "temperature": 0.9,
    "top_p": 0.9,
    "rep_pen": 1.1
})

print(response.json()["results"][0]["text"])

KoboldCpp is the right tool if you want a local LLM for anything storytelling-related. Its sampling controls, story memory, and Author’s Notes system are purpose-built for writers in ways that general chat interfaces aren’t.