Skip to content

machine: a Spec says the files QEMU opens and what it uses beyond its guest - #434

Merged
aledbf merged 2 commits into
mainfrom
machine-host-files-overhead
Nov 21, 2026
Merged

aledbf merged 2 commits into
mainfrom
machine-host-files-overhead

Conversation

@aledbf

@aledbf aledbf commented Nov 21, 2026

Copy link
Copy Markdown
Member

Why

spin-stack/machine owns a VM's lifecycle end to end, and spin keeps host policy. spin's runner still hard-codes two facts that depend on the machine and its release, not on spin's fleet. It links a release's whole firmware directory into each QEMU's chroot (internal/runner/qemu/root.go). It also bounds each QEMU's cgroup at the guest's ceiling plus 768 MiB, and at its vCPU ceiling plus 256 tasks (internal/runner/qemu/bounds.go). This is the last lifecycle piece before a release. spin keeps building the root and applying the cgroup, and it asks the machine what goes in them.

What

  • Spec.Files() lists the files QEMU opens by name: its binary, the kernel, the initrd, and from Firmware the BIOS, linuxboot_dma.bin and kvmvapic.bin. Start hands QEMU everything else as a descriptor. A root holding these and /dev/null starts the machine. efi-virtio.rom is never opened, since every NIC has romfile=. SeaBIOS is opened only when BIOS names it.
  • Spec.Overhead() (Overhead{Memory, Tasks}, error) is what QEMU's cgroup holds beyond the guest's RAM at its ceiling, counting nothing the host can reclaim. Memory is 64 MiB of QEMU's own, the kernel's and the initrd's sizes (QEMU keeps a copy of each), 1 MiB a vCPU, 64 KiB a thread, and 1/256 of the ceiling for page tables (the host's and KVM's, at 4 KiB pages). Tasks are counted up to their most: one a vCPU slot, QEMU's four, a vhost worker for the vsock and each NIC, a checkpoint's 8 channels and their driver, QEMU's thread pool (64, and a thread it cannot start aborts QEMU) and io_uring's workers (the ring's 128 entries).
  • Disk.Overhead() is each qcow2 image's L2 cache (its whole L2 table, up to 32 MiB) plus four clusters of refcounts, QEMU's defaults in read_cache_sizes. It reads each header through the image's File, so a hand-built Disk works too. A 64 GiB image of 64 KiB clusters costs 8.25 MiB.
  • task boot:overhead (TestAMachineRunsInARootOfItsFilesWithinItsOverhead, root through sudo) holds both.

Measured

Laptop, real KVM, v20261121.01, 2026-11-21, spin's shape: 1 of 16 vCPUs, 128 of 8192 MiB, 16 hotplug targets, a root and a home, a vsock and a NIC on a TAP. It ran as nobody in a root holding only Files and /dev/null, in a cgroup of its own.

Point Beyond the guest Tasks
booted, sized to 2048 MiB 31 MiB 9
ceiling: 16 vCPUs, 8192 MiB filled, 4 disks attached (one 64 GiB), every disk read, written and trimmed 73 MiB (27 page tables, 16 bzImage, 8 L2 cache) 27
checkpointed at the ceiling 79 MiB 34 (pids.peak)
Overhead + Disk.Overhead 153 MiB 223

The threads by name were 16 CPU n/KVM, call_rcu, IO mon_iothread, kvm-nx-lpage-recovery, 2 vhost, up to 5 iou-wrk, 1 pool worker, and through a checkpoint mig/src/main and 8 mig/src/send_. The lane counts QEMU's own memory from smaps, outside the guest's RAM mappings, plus the kernel's memory charged to the cgroup, less reclaimable slab. The cgroup's anon overstated it by 160 MiB once. A huge page the guest gave back in part stays charged until the kernel splits it under pressure.

Not measured: a restore, and a QEMU that has run for days.

Found on the way

Verification

  • go tool task lint test mutate: coverage 80.6% (floor 80), mutate 40 refused, 0 survived.
  • task boot:overhead passed (laptop, real KVM).
  • task boot:record passed.

🤖 Generated with Claude Code

aledbf and others added 2 commits November 21, 2026 18:50
… guest

spin built each QEMU's root from a release's whole firmware directory and bounded its cgroup at
the guest's ceiling plus 768 MiB and 256 tasks. Both depend on the machine and its release.

Spec.Files is what QEMU opens by name: its binary, the kernel, the initrd, and the BIOS,
linuxboot_dma.bin and kvmvapic.bin. Start hands it everything else as a descriptor. Spec.Overhead
is the memory and the tasks QEMU's cgroup holds beyond the guest's RAM at its ceiling, and
Disk.Overhead each qcow2 image's caches at QEMU's defaults.

task boot:overhead starts spin's shape as nobody in a root of Files and /dev/null alone, takes it
to its ceiling while the guest fills its memory and works every disk, checkpoints it, and holds
QEMU to both numbers. Measured on the laptop: 73 MiB and 27 tasks at the ceiling, 79 MiB and 34
tasks through the checkpoint, against 153 MiB and 223 tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aledbf
aledbf merged commit 25fde61 into main Nov 21, 2026
6 checks passed
@aledbf
aledbf deleted the machine-host-files-overhead branch November 21, 2026 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant