Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Language-Runtime Foundations

InterGenOS ships a built-from-source toolchain and a set of language runtimes that the AI subsystem and other workloads run on. This page explains what those foundations are, where they come from, and how the local AI assistant uses them. The goal is the same one that runs through the whole system: a machine you understand, can modify, and can trust.

This is version 1.0-dev (build id v1.0-dev1). Package counts below are derived from the live package tree and drift over time; treat them as a snapshot, not a fixed contract.

Where runtimes come from

Everything is built from source through a staged pipeline rather than assembled from prebuilt binaries. The build runs in 21 phases:

validate → verify-sources → setup → toolchain → chroot-prep → chroot-tools → core → config → core-extra → base → kernel → desktop → extra → compute → ai → bootloader → image → manifest → squashfs → ukis-verity → iso

A publish phase is optional and runs after the others.

The runtime foundations are laid early. The toolchain phase builds the compiler and core build tooling that every later phase depends on; the core and base phases bring in the system libraries and language runtimes that user-facing software links against; and the ai phase builds the assistant on top of those runtimes.

Packages are organized into seven tiers. The live tree holds over 1,100 package definitions — desktop is the largest at 460+, core follows at 310+, and toolchain is the smallest at under three dozen. These numbers move as packages are added and consolidated. Derive the current counts from the package tree rather than quoting any figure as permanent.

Not every definition ships on the installed image: roughly three quarters do, and the rest are mirror-only — present in the repository, installed only when you ask for them. Several language toolchains are in that mirror-only set rather than on the image, so a default installation does not carry them.

The runtime the AI subsystem uses

The local assistant, InterGen, is a tiered, hardware-detected, offline-first local assistant with zero telemetry. It runs entirely on the local machine by default, with no cloud dependency, and selects a model sized to the hardware it detects.

InterGen has two distinct runtime layers, and separating them helps because they place different demands on the system:

  1. The model-serving layer. InterGen serves its selected model through llama-server, the HTTP server from llama.cpp, managed as a subprocess (intergen/llama_manager.py). This is the native, compiled inference runtime that does the heavy numeric work. It is built with a Vulkan GPU compute backend and a CPU fallback — on a supported GPU with a working Vulkan driver it accelerates through Vulkan, otherwise it runs on the CPU (see Overview & Backend Selection). It exposes the model over a local HTTP API.

  2. The orchestration layer. The router, safety classifier, hardware probe, model manager, memory store, D-Bus service, and MCP client are organized as modules (intergen/router.py, intergen/safety.py, intergen/hardware.py, intergen/model_manager.py, intergen/memory.py, intergen/state_cache.py, intergen/dbus_daemon.py, intergen/mcp_client.py). These coordinate the assistant: deciding how to answer a request, classifying its safety, and managing state.

These orchestration modules run on the shipping Python runtime, which is Python 3.14 (3.14.3 in the current build).

Rust is installed, but not on your PATH

Rust 1.95.0 is a core-tier package and is present on an installed system, but its binaries live under /opt/rustc-1.95.0/bin (reached through the /opt/rustc symlink) rather than in /usr/bin. On the current release, nothing on a freshly installed system adds that directory to PATH — which cargo comes back empty on a machine where Rust is installed, and cargo and rustc appear missing. A fix has landed (the profile script that adds /opt/rustc/bin to PATH now installs correctly on the target) and takes effect from the next release image; until your system carries it, add the directory yourself:

export PATH=/opt/rustc/bin:$PATH      # this session
echo 'export PATH=/opt/rustc/bin:$PATH' >> ~/.bashrc   # persist for your user

This is recorded here as observed behaviour rather than intent. The recipe does author a /etc/profile.d/rustc.sh that sets the PATH, and the installer does run each package’s post-install step on the installed target — but rust’s step aborted at an early line (a path valid only in the build environment, under set -e), so the profile script was never written on machines installed to date. The cause is fixed in the current source tree; systems installed from images built after the fix get PATH set as intended, and the page will say so plainly once an installed system demonstrates it.

The split matters for runtime planning. The orchestration layer is light and predictable. The model-serving layer is where available GPU memory decides which model can run at all; on a machine falling back to the CPU, throughput on the larger tiers is CPU-bound, which is exactly why such a machine is assigned the small model instead.

Hardware tiers and what they need

InterGen probes the GPU and its dedicated VRAM at startup (intergen/hardware.py) and assigns a tier; the model catalog and selection live in intergen/model_manager.py. System RAM is detected and reported but is never an input to the tier decision — a machine with no discrete GPU serves the small model regardless of how much RAM it has, because the 9B on CPU runs at roughly 50 seconds a query and is unusable. Capability that cannot be determined always fails down. A card counts as discrete only if it exposes at least 3 GB of dedicated VRAM, which is what separates a real card from an integrated GPU borrowing system memory.

The tiers are:

  • Tier 1 (2B, the universal floor) — InternVL3.5-2B Q4_K_M, about 1.2 GB. Selected when there is no discrete GPU, when the card has less VRAM than Tier 2 requires, or when VRAM cannot be read. It handles semantic matching, system queries, keyword extraction, and everyday explanation, and its tool dispatch is code-owned. It is not aimed at large-scale code generation.
  • Tier 2 (9B, shipping) — Qwen3.5-9B Q4_K_M, about 5.5 GB, selected for a discrete GPU with at least ~7 GB of VRAM, sized so the model, its vision projector and the load-time buffers all fit. An 8 GB card qualifies; a 6 GB card does not. On such a machine InterGen runs the 9B natively and unlocks native tool-calling, gated by the safety classifier and Sentinel.
  • Tier 3 (35B) — Qwen3.5-35B-A3B Q4_K_M, a Mixture-of-Experts model, about 21 GB, selected for a discrete GPU with at least ~22 GB of VRAM so the model stays resident. A 24 GB card qualifies; a 20 GB card deliberately does not. The model is catalogued but ships without a pin, and an unpinned model can be neither downloaded nor load-verified — so rather than dead-ending the install, the recommendation is capped to the highest pinned tier and a Tier-3 machine runs the 9B today. The cap is logged, not silent.

A small embedding model (nomic-ai/nomic-embed-text-v1.5, Q8_0, about 139 MB) ships alongside every tier to power the router’s semantic-matching layer.

GPU presence and VRAM size are the signals InterGen uses to select a tier. Inference is served by a bundled llama.cpp built with a Vulkan GPU backend and a CPU fallback: on a machine with a supported GPU the model is GPU-accelerated through the Vulkan loader and the installed driver, and on a machine without one it runs on the CPU (a portable AVX2 instruction-set floor). See the Overview & Backend Selection page.

How the runtime resolves a request

InterGen does not hand every prompt to the model. A priority-ordered router (intergen/router.py) tries to satisfy each request with the cheapest, most predictable method first and only escalates the runtime cost when it has to:

  • Priority 0 (Decomposition) — splits compound requests into sub-tasks and routes each in turn.
  • Priority 1 (Keyword/Regex Match) — fast pattern matches for common system commands, dispatched to a built-in tool without invoking the model.
  • Priority 2 (Semantic Embedding Match) — lightweight embedding search against a precomputed capability catalog; a high-confidence match dispatches to a built-in tool.
  • Priority 3 (LLM Tool Calling) — a gated path in which the query plus a tool schema go to the model, which selects a tool and arguments for the router to execute. On the 2B floor this path is disabled: tool dispatch is code-owned — the deterministic Priority 1/2 matchers select the tool and extract the arguments, and the model never decides a tool call. Native tool-calling unlocks on the shipped 9B tier (intergen/dispatch_policy.py), where a capable machine lets the model decide tool calls under the same safety-classifier and Sentinel gates; the 2B floor stays code-owned.
  • Priority 4 (LLM Free Response) — the fallback: the model answers conversationally from its own knowledge and the conversation context.

The practical effect is that most interactions never touch the model-serving runtime at all, which keeps the system responsive even on modest hardware.

Safety classification at the runtime boundary

Every action the router proposes passes through the classifier in intergen/safety.py, which sorts it into one of three tiers:

  • AUTO — read-only or harmless operations (for example ls, grep, systemctl status). Run immediately.
  • CONFIRM — state-changing operations (for example systemctl restart, pkm install, or editing a config file). The assistant pauses and asks for approval first.
  • BLOCKED — destructive or security-bypassing operations (for example rm -rf /). Refused outright, with an explanation.

Nothing that changes the system runs without explicit approval, and the most dangerous commands cannot run at all.

These three classes are the assistant-facing view of a larger trust model. For how AUTO/CONFIRM/BLOCKED relate to the system’s gating outcomes and trust zones, see Registry & Trust Boundary.

Crossing the network boundary

Two features can take a request off-device, and both run only when you ask:

  • InterGen Sentinel is a pluggable security scanner that inspects content crossing the ingress and egress boundaries. Its default configuration runs two local stages: a fast local-rules pass and an optional deep pass backed by a small local Qwen classifier. For deeper analysis you may opt in to one of six named cloud providers — Claude (Anthropic), Gemini (Google), Copilot (Microsoft), ChatGPT (OpenAI), Grok (xAI), or DeepSeek — or to any custom OpenAI-compatible endpoint you supply. None is configured by default, so a default install scans entirely on-device.
  • Phone-A-Friend (Frontier/Cloud Escalation) is an optional, consent-first path that hands a request to a more capable frontier model in the cloud when the local assistant cannot satisfy it. It is off by default; the same provider set can be configured. An API key is written to the system keyring and never into the configuration file, which records only the keyring identifier. Every outbound payload is scanned by Sentinel’s egress policy before it leaves the machine.

The rest of the platform

Beyond the AI subsystem, the same source-built foundations support the shipping system: pkm (the package manager), Forge (the installer), a signed Secure Boot chain, dm-verity integrity, and UKI signing. The shipping desktop is GNOME 49 on Wayland.

See also