GPU Compute & AI Workloads
This section covers how InterGenOS uses the GPU: the graphics backends it can target, the compute stacks used for machine-learning work, and how the built-in AI assistant detects and scales to the hardware it finds on your machine.
InterGenOS is built from source and aligned on a single principle. That posture shapes the GPU stack the same way it shapes the rest of the system: you get a machine you understand, can modify, and can trust, with no opaque vendor blobs running outside your knowledge or consent.
InterGenOS is at version 1.0-dev (build id v1.0-dev1). The pages below describe the shipping system. Where a backend, command, or package is not yet documented in detail, the page says so plainly rather than guessing.
What ships today
The desktop is GNOME 49 on Wayland. GPU acceleration for the desktop session runs through that stack.
For compute and AI work, the system pairs a vendor-neutral default with opt-in vendor stacks. The individual pages in this section document each backend; the high-level shape is:
- A vendor-neutral default backend that works across GPUs without committing you to a single vendor’s toolchain. See Vulkan (default, vendor-neutral).
- AMD compute via a from-source ROCm path. See ROCm for AMD (automated from-source).
- NVIDIA compute via an opt-in CUDA path. See CUDA for NVIDIA (opt-in mirror).
The ROCm compute-tier package set is in-tree at ROCm 7.2.4 and is entirely mirror-only: none of it ships on the installed image, and you install it when you want it. CUDA remains driver-runtime-only — the CUDA Toolkit is not packaged.
For the per-vendor specifics — which driver lands, how it is selected, and the exact commands — read:
- Overview & Backend Selection
- PyTorch / TensorFlow / JAX
- Language-Runtime Foundations
- Per-GPU Driver Notes
How the GPU feeds the AI assistant
The clearest consumer of GPU compute in InterGenOS is InterGen, the local AI assistant. InterGen is offline-first and runs inference on local hardware by default, so your data stays on your machine. It carries zero telemetry.
InterGen is hardware-tiered. It probes the host’s GPU and its dedicated VRAM, selects a model for the detected tier, and serves it locally over an HTTP API backed by llama.cpp. The 2B floor (InternVL3.5-2B) runs on all hardware as the verified-everywhere default; the 9B tier (Qwen3.5-9B) ships for capable hardware. RAM is detected and reported, but it does not decide the tier — see the table below.
Tier assignment is decided by the GPU and its dedicated VRAM. System RAM is never an input. That is deliberate: a machine without a discrete GPU serves the small model for latency regardless of how much RAM it has, because the 9B on CPU takes roughly 50 seconds a query and is unusable — so there is no decision RAM could inform. A GPU whose capability cannot be determined always fails down to the smaller model, never up.
A card counts as discrete when it exposes at least 3 GB of dedicated VRAM. That threshold separates a real card from an integrated GPU carving a buffer out of system memory; vendor identity alone is not sufficient, since an integrated part and a discrete card can report the same vendor.
The detector reports the tier level, the detected RAM and GPU, and the recommended model and quantisation for that tier. You can see the result for your own machine with intergen tier. The rules are evaluated top-down:
| Tier | Hardware rule | Model |
|---|---|---|
| Tier 3 | discrete GPU, VRAM ≥ ~22 GB | Qwen3.5-35B-A3B MoE (Q4_K_M, ~21 GB) |
| Tier 2 (shipping) | discrete GPU, VRAM ≥ ~7 GB | Qwen3.5-9B (Q4_K_M, ~5.5 GB) |
| Tier 1 (universal floor) | no discrete GPU, or a discrete GPU with less VRAM than Tier 2 needs, or VRAM that cannot be read | InternVL3.5-2B (Q4_K_M, ~1.2 GB) |
So an 8 GB card qualifies for Tier 2 and a 6 GB card does not; a 24 GB card qualifies for Tier 3 and a 20 GB card deliberately does not. The thresholds are sized for the model plus its vision projector and the buffers allocated at load, not for the model file alone.
The Tier 3 model is catalogued but ships without a pin, and an unpinned model can be neither downloaded nor load-verified. Rather than dead-ending such an install, the assistant caps the recommendation to the highest tier that does have a shipped pin — so a Tier-3-capable machine runs the 9B today, and does so by an explicit, logged decision rather than a silent substitution.
InterGen’s bundled llama.cpp is built with a Vulkan GPU backend and a CPU fallback (a portable AVX2 instruction-set floor, so it also runs on a machine with no usable GPU). On a machine with a supported GPU, inference is GPU-accelerated through the Vulkan loader and the GPU’s installed driver; on a machine without one, it runs on the CPU. Every machine resolves to the largest pinned tier at or below its detected level, so a Tier-3-capable machine runs the 9B until the 35B ships with a pin.
A small embedding model (nomic-ai/nomic-embed-text-v1.5) ships alongside the assistant to power its semantic-matching layer.
The full assistant architecture — the priority router, the AUTO/CONFIRM/BLOCKED safety classifier, D-Bus and MCP integration, and memory — is documented in The AI Assistant.
Security scanning and cloud escalation
Two related features sit alongside the local assistant. Both are local-by-default and consent-first.
InterGen Sentinel is a pluggable security scanner that inspects content crossing the device boundary: data coming in from external and MCP tools, and content about to be sent off-device. Its default configuration runs entirely on-device — a fast local-rules pass plus an optional deeper pass backed by a small local Qwen classifier. For deeper analysis you may opt in to a cloud scanner backed by one of six named providers — Claude (Anthropic), Gemini (Google), Copilot (Microsoft), ChatGPT (OpenAI), Grok (xAI), or DeepSeek — or by any custom OpenAI-compatible endpoint you supply. None is configured by default.
Phone-A-Friend (Frontier/Cloud Escalation) is an optional path for handing a request to a more capable frontier model when the local assistant cannot satisfy it. It is off by default, asks before reaching out, stores API keys in the system keyring, and scans every outbound payload through Sentinel’s egress policy first. Cloud assistance is available when you ask for it, never imposed.
The rest of the platform
InterGenOS ships its own package manager (pkm), its own installer (Forge), a signed Secure Boot chain, dm-verity integrity, and UKI signing. The system is assembled across seven package tiers, built in this order — toolchain, core, base, desktop, extra, compute, ai — through a 21-phase build pipeline. That is over 1,100 package definitions, of which roughly three quarters ship on the installed image and the rest are mirror-only; these counts drift as the system grows, so treat any figure as a snapshot rather than a fixed total.
For installation and the GPU-relevant steps of setup, see the Forge installer guide.
Where to go next
- New to the GPU stack? Start with Overview & Backend Selection.
- Running ML frameworks? See PyTorch / TensorFlow / JAX.
- Specific card behaving oddly? Check Per-GPU Driver Notes.