GPU Compute & AI Workloads
This section covers how InterGenOS uses the GPU: the graphics backends it can target, the compute stacks used for machine-learning work, and how the built-in AI assistant detects and scales to the hardware it finds on your machine.
InterGenOS is built from source and aligned on a single principle. That posture shapes the GPU stack the same way it shapes the rest of the system: you get a machine you understand, can modify, and can trust, with no opaque vendor blobs running outside your knowledge or consent.
These pages describe InterGenOS as it ships. Where a backend, command, or package is not yet documented in detail, the page says so plainly rather than guessing.
What ships today
The desktop is GNOME 49 on Wayland. GPU acceleration for the desktop session runs through that stack.
For compute and AI work, the system pairs a vendor-neutral default with opt-in vendor stacks. The individual pages in this section document each backend; the high-level shape is:
- A vendor-neutral default backend that works across GPUs without committing you to a single vendor’s toolchain. See Vulkan (default, vendor-neutral).
- AMD compute via a from-source ROCm path. See ROCm for AMD (automated from-source).
- NVIDIA compute via an opt-in CUDA path. See CUDA for NVIDIA (opt-in mirror).
The ROCm compute-tier package set is in-tree at ROCm 7.2.4 and is entirely mirror-only: none of it ships on the installed image, and you install it when you want it. CUDA remains driver-runtime-only — the CUDA Toolkit is not packaged.
For the per-vendor specifics — which driver lands, how it is selected, and the exact commands — read:
- Overview & Backend Selection
- PyTorch / TensorFlow / JAX
- Language-Runtime Foundations
- Per-GPU Driver Notes
How the GPU feeds the AI assistant
The clearest consumer of GPU compute in InterGenOS is InterGen, the local AI assistant. InterGen is offline-first and runs inference on local hardware by default, so your data stays on your machine. It carries zero telemetry.
InterGen is hardware-tiered. It probes the host’s GPU and its dedicated VRAM, selects a model for the detected tier, and serves it locally over an HTTP API backed by llama.cpp. The 2B floor (InternVL3.5-2B) runs on all hardware as the verified-everywhere default; the 9B tier (Qwen3.5-9B) ships for capable hardware. RAM is detected and reported, but it does not decide the tier — see the table below.
Tier assignment is decided by the GPU and its dedicated VRAM. System RAM is never an input. That is deliberate: a machine without a discrete GPU serves the small model for latency regardless of how much RAM it has, because the 9B on CPU takes roughly 50 seconds a query and is unusable — so there is no decision RAM could inform. A GPU whose capability cannot be determined always fails down to the smaller model, never up.
A card counts as discrete when it exposes at least 3 GB of dedicated VRAM. That threshold separates a real card from an integrated GPU carving a buffer out of system memory; vendor identity alone is not sufficient, since an integrated part and a discrete card can report the same vendor.
The detector reports the tier level, the detected RAM and GPU, and the recommended model and quantisation for that tier. You can see the result for your own machine with intergen tier. The rules are evaluated top-down:
| Tier | Hardware rule | Model |
|---|---|---|
| Tier 3 (shipping; no native lane) | discrete GPU, VRAM ≥ ~22 GB | Qwen3.5-35B-A3B MoE (Q4_K_M, ~21 GB) |
| Tier 2 (shipping; the native lane) | discrete GPU, VRAM ≥ ~7 GB | Qwen3.5-9B (Q4_K_M, ~5.5 GB) |
| Tier 1 (universal floor) | no discrete GPU, or a discrete GPU with less VRAM than Tier 2 needs, or VRAM that cannot be read | InternVL3.5-2B (Q4_K_M, ~1.2 GB) |
So an 8 GB card qualifies for Tier 2 and a 6 GB card does not; a 24 GB card qualifies for Tier 3 and a 20 GB card deliberately does not. The thresholds are sized for the model plus its vision projector and the buffers allocated at load, not for the model file alone.
All three models are pinned. Each model file, and each paired vision projector, carries a SHA-256 pin in the signed models manifest, so each can be downloaded and load-verified — a Tier-3-capable machine runs the 35B. What does not ship at Tier 3 is the native tool-calling lane: the 9B carries the only shipped native lane, so a machine serving the 35B decides actions through the code-owned path, the same posture as the Tier 1 floor. A recommendation is only ever capped when the tier’s model has no shipped pin, and such a cap is logged rather than silent.
InterGen’s bundled llama.cpp is built with a Vulkan GPU backend and a CPU fallback (a portable AVX2 instruction-set floor, so it also runs on a machine with no usable GPU). In R001 and R001.1, published 2026-08-16 and 2026-08-20, the detected tier decided the serving path, not just the model: on Tier 1 inference ran on the CPU only, whether or not the machine had a GPU. On Tier 2 and Tier 3 every layer was offloaded to the GPU through the Vulkan loader and the GPU’s installed driver. If tier detection failed, the machine offloaded, matching the assume-Tier-2 posture used elsewhere for absent detection. An explicit llama_server.gpu_layers integer in your configuration is honoured verbatim on every tier and remains the last word, so you can pin CPU-only serving with 0 or force offload with a layer count.
That tier rule for the serving path has since been replaced in the source tree: how much of the model goes on the card is decided by whether the model fits the card’s measured video memory, and the tier decides only which model is served. The difference shows on a discrete card between the discreteness floor and the Tier 2 threshold — roughly 3 to 7 GB. Under the tier rule such a card is detected and reported but not used for inference; under the fit rule it serves the 2B floor from the card. An explicit llama_server.gpu_layers integer stays the last word either way. Check your release’s notes for which rule your machine runs.
A small embedding model (nomic-ai/nomic-embed-text-v1.5) ships alongside the assistant to power its semantic-matching layer.
The full assistant architecture — the priority router, the AUTO/CONFIRM/BLOCKED safety classifier, D-Bus and MCP integration, and memory — is documented in The AI Assistant.
Security scanning and cloud escalation
Two related features sit alongside the local assistant. Both are local-by-default and consent-first.
InterGen Sentinel is a pluggable security scanner that inspects content crossing the device boundary: data coming in from external and MCP tools, and content about to be sent off-device. Its default configuration runs entirely on-device — a fast local-rules pass plus an optional deeper pass backed by a small local Qwen classifier. For deeper analysis you may opt in to a cloud scanner backed by one of six named providers — Claude (Anthropic), Gemini (Google), Copilot (Microsoft), ChatGPT (OpenAI), Grok (xAI), or DeepSeek — or by any custom OpenAI-compatible endpoint you supply. None is configured by default.
Phone-A-Friend (Frontier/Cloud Escalation) is an optional path for handing a request to a more capable frontier model when the local assistant cannot satisfy it. It is off by default, asks before reaching out, stores API keys in the system keyring, and scans every outbound payload through Sentinel’s egress policy first. Cloud assistance is available when you ask for it, never imposed.
The rest of the platform
InterGenOS ships its own package manager (pkm), its own installer (Forge), a signed Secure Boot chain, dm-verity integrity, and UKI signing. The system is assembled across seven package tiers, built in this order — toolchain, core, base, desktop, extra, compute, ai — through a 21-phase build pipeline. That is over 1,100 package definitions, of which roughly three quarters ship on the installed image and the rest are mirror-only; these counts drift as the system grows, so treat any figure as a snapshot rather than a fixed total.
For installation and the GPU-relevant steps of setup, see the Forge installer guide.
Where to go next
- New to the GPU stack? Start with Overview & Backend Selection.
- Running ML frameworks? See PyTorch / TensorFlow / JAX.
- Specific card behaving oddly? Check Per-GPU Driver Notes.