Overview & Backend Selection
InterGenOS runs local AI inference on the hardware in front of it. The local assistant, InterGen, detects the host’s RAM and GPU at startup and selects an appropriately sized model, then serves it through a llama-server subprocess from llama.cpp over a local HTTP API. The compute backend that subprocess uses to talk to your GPU is the subject of this page.
The guiding principle is the same one that runs through the rest of the system: a machine you understand, can modify, and can trust. The default backend is chosen to work on the widest range of hardware without a vendor-specific driver stack. Vendor-tuned backends are available, but they are opt-in.
What ships today: Vulkan GPU inference, with a CPU fallback
The shipping llama.cpp build is compiled with a Vulkan compute backend and a portable CPU fallback (an AVX2-class instruction-set floor, with non-native codegen so the binary runs on every supported target rather than only the build host). On a machine with a supported GPU and a working Vulkan driver, InterGen’s inference is GPU-accelerated through Vulkan; on a machine without one, it runs on the CPU. Selection is automatic — there is no vendor driver stack to install for the default path.
Vulkan is the default because it is vendor-neutral: one API that runs across AMD, Intel, and NVIDIA GPUs through each vendor’s installable client driver, rather than a vendor-specific SDK baked in at build time. For AMD and Intel the open Mesa drivers provide the Vulkan driver in the default image; on NVIDIA the Vulkan driver arrives with the proprietary-driver opt-in. This keeps the default install small, inspectable, and portable while still using the GPU where one is present.
Two different “Vulkans.” The Vulkan graphics stack —
radv(AMD) andanv(Intel), used for the desktop and rendering — and the Vulkan compute backend that accelerates LLM inference are related but distinct. Both ship and work on InterGenOS today. See Vulkan for the details.
Opt-in vendor backends
Beyond the vendor-neutral GPU path, two vendor-tuned backends are available as opt-in choices for users who want maximum throughput on specific hardware. Both are in-tree today; neither is on the installed image:
- ROCm for AMD GPUs. See ROCm (AMD).
- CUDA for NVIDIA GPUs. See CUDA (NVIDIA).
Each typically delivers higher throughput than the vendor-neutral path on its hardware, at the cost of pulling in a vendor-specific runtime that enlarges the trusted surface — so neither is enabled by default. Unlike the vendor-neutral Vulkan backend, which ships enabled, ROCm ships from source as an opt-in, mirror-only compute tier; CUDA is provided via the NVIDIA driver runtime (the CUDA Toolkit is not packaged). The per-backend pages track each one’s status.
How a backend is selected
Backend selection follows a local-first, hardware-detected philosophy. The shipping llama.cpp is built with a Vulkan GPU backend and a CPU fallback: on a machine with a supported GPU and a working Vulkan driver, inference is GPU-accelerated through the vendor-neutral Vulkan path; on a machine without one, it runs on the CPU (a portable AVX2 instruction-set floor). This happens automatically, with no configuration and no vendor lock-in.
The vendor-specific ROCm backend ships as an explicit opt-in (mirror-only), used only where enabled and where AMD hardware is present; the CUDA path is the NVIDIA driver runtime, likewise opt-in. The out-of-the-box path stays vendor-neutral.
Relationship to model tiers
Backend selection is independent of, but interacts with, the model tier InterGen picks for your hardware. The compute backend determines how a model is accelerated (Vulkan on a supported GPU, CPU otherwise); the tier determines which model runs. Both keys off the same property — the presence and size of dedicated VRAM — so on a machine with a real card the model is accelerated by that hardware and sized for it. System RAM is not part of the tier decision.
For the tier rules and thresholds, see the GPU section overview. For per-GPU notes, see Per-GPU guidance.
What this is not
This page covers the local inference backend only. It is unrelated to optional cloud features:
- InterGen Sentinel, the pluggable security scanner, runs entirely on-device by default (a fast local-rules pass plus an optional local Qwen classifier). Cloud scanning, backed by one of six opt-in providers, is configured by you and off by default.
- Phone-A-Friend (Frontier/Cloud Escalation) is an optional, consent-first path that hands a request to a frontier model in the cloud. It is off by default; no provider is configured out of the box.
Neither feature depends on the GPU backend, and neither is enabled out of the box. The default install runs and reasons entirely on your own hardware.