Seamlessly run large GGUF models on low-RAM machines, even without a discrete GPU.
A primary appeal of tinyinference is making capable LLMs seamlessly runnable on low-spec hardware using CPU inference and file-backed model weights via mmap. For frontier open-weight models (which are typically extremely large), it will not be fast, often painfully slow. The point is that it runs at all. However, for balanced models, it can be fast enough even if your machine doesn't have enough RAM to hold all of the model's weights in RAM simultaneously.
All you need to get going is a single binary. Download one portable build and replace it as updates come. You still need llama.cpp.
Inside the app
Two surfaces: the native control panel, and chat in the browser.
What you can do
- Start / stop / restart
- Manage the state of the llama.cpp server
- Chat
- Custom tinyinference chat interface
- Manage models
- Swap between or run multiple models at once
- Configure
- Configure performance presets and more
- Logs
- Monitor tinyinference and llama.cpp activity
- Stats
- See LLM inference statistics
- Copy /v1 URL
- Copy the OpenAI-compatible /v1 API endpoint
- Share API
- LLLM network sharing. Tailscale preferred; HTTPS + API key
Requirements
- llama-server from llama.cpp (on PATH, or set its path in Configure)
- A GGUF model available locally or on Hugging Face
- A modern web browser (for the chat page)
- A system webview for the control-panel window (WebView2 / WKWebView / WebKitGTK), or run with --no-window
Run
Download a release binary and run it directly.
# Windows .\tinyinference.exe # macOS / Linux ./tinyinference # Windowless ./tinyinference --no-window