Initial support for VMs

[jonathan-conder]

Description

Introduces VM workshops: after running snap set workshop workshop.experimental-vms=1, add confinement: virtual-machine to the workshop definition file. Only limited functionality is supported (e.g. no SDKs).

Major chages

• Starting workshops (including during launch and refresh, but not at system boot) now relies on workshop-waitready.service. This is installed in workshops launched or refreshed with Workshop 0.9.5 or later. Older workshops are still supported, but will be dropped in future. After we drop support, lxc start can be run manually as a workaround.
• Adds support for freezing VM filesystems. LXD doesn't natively support this for VM root filesystems, but support will be added in future, possibly based on this implementation. This is required to safely snapshot a workshop VM (but snapshots aren't used anywhere yet).
• Workshop can now launch, start, stop, refresh, restore and remove stock Ubuntu VMs.

Freeze limitations

I think the fsfreeze implementation is slightly better than QEMU's guest agent, but has some of the same limitations: no ZFS support and potential timeout issues in obscure cases like mounting a loop file stored on a network drive. The latter is probably not worth worrying about; the former is something we could potentially support but there's no reason to right now. LXD VMs typically use ext4 for the / partition even when the block device is backed by ZFS.

Minor changes

• Adds a --vm flag to workshop init.
• Adds confinement in workshop info output.
• Fixes an issue where LXD client websocket goroutines can race with error handling when writing to a stderr buffer.
• Splits tests/main/launch and tests/integration/workshop-snapshots into multiple spread variants. This is a step towards enabling them for VMs, but we need to clean up snapshots more aggressively or the GitHub runners will run out of space.

Self-review quick check

• Make decisions that cost a lot to reverse explicit in the PR description.
• Avoid nested conditions.
• Delete dead code and redundant comments.
• Normalise symmetries by sticking to doing identical things identically.

// one way to handle errors
if err := f(); err != nil {
   ...
}

// one way to handle multiple returns
val, err := f()
if err != nil {
   ...
}
...

• Check that coupled code elements, files, and directories are adjacent. For example, test data is stored as close as possible to a test.
• Put variable declaration and initialisation together.
• Divide large expressions into digestable and self-explanatory ones. Use multiple variables if required.
• Put a blank line between two logically different chunks of code.
• Follow the style guide for new error messages.

Docs

Procedure:

• I have checked and added or updated relevant documentation.
• I have checked and added or updated relevant release notes.
• I have included the technical author in the review.

Content:

• Headings and titles accurately describe the content.
• New and updated pages include correct metadata.
• Documentation tests are added or updated where applicable (for tutorial/ and how-to/ sections).
• Documentation follows the style guide.
• If needed, docs/.coverage.yaml updated, coverage tags added (.. artefact).

---

Or:

• I confirm the PR has no implications for documentation.

[jonathan-conder]

@dmitry-lyfar updated

New commits:

• 0df6c68 Fix potential data race in projectFsRoot
• 71998b5 Add temporary support for starting pre-0.9.4 workshops
• b54a7a3 Add support for freezing filesystems before taking snapshots
• 3e3dfa3 Stop VMs gracefully during refresh

Major changes:

• Added fsfreeze to cloud-init and updated snapshot tests (f5fc4dc)
• Gated VMs behind snap set workshop.experimental-vms=1 (948add5)
• Added VMs and 26.04 to tests/main/launch (948add5)

Minor changes:

• Resolved a merge conflict in lxd_backend.go (1d70691 vs fcf0d26)
• Resolve a rebase conflict in lxd_backend.go (187fa1c vs a673c8c)
• Updated snapshot tests again to account for GRUB config (a673c8c)
• Fix snapshot tests after adding more whitespace to raw.idmap (46664be)
• Two new manifestSuite tests: TestRefreshRequiresContainer and TestLaunchRejectsVMsWithSDKs
• Added test for workshop init --vm (c21dbbc)

[Copilot]

Pull request overview

Adds experimental LXD virtual-machine workshops while preserving container behavior.

Changes:

• Adds confinement-aware definitions, APIs, CLI output, image handling, and VM launch support.
• Reworks readiness detection and introduces filesystem freezing and ID mapping.
• Refactors unit and integration coverage for multiple bases and confinement types.

Reviewed changes

Copilot reviewed 60 out of 61 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/main/start/task.yaml	Updates readiness tests.
tests/main/launch/workshop.yaml.in	Parameterizes base and confinement.
tests/main/launch/task.yaml	Splits launch variants and validates ID maps.
tests/main/launch/.workshop/ws-24.yaml	Removes duplicated fixture.
tests/main/launch/.workshop/ws-22.yaml	Removes duplicated fixture.
tests/main/launch-multiple/task.yaml	Adds DNS validation and cleanup.
tests/lib/utils.sh	Enables experimental VMs in tests.
tests/integration/workshop-snapshots/task.yaml	Splits snapshot test variants.
snap/local/commands/run_daemon	Exports the VM feature setting.
internal/workshop/workshop_file.go	Adds confinement parsing.
internal/workshop/workshop_file_test.go	Tests confinement parsing.
internal/workshop/lxd/tests/integration/workshop_test.go	Updates LXD integration coverage.
internal/workshop/lxd/tests/integration/workshop_exec_test.go	Reuses shared test devices.
internal/workshop/lxd/tests/integration/snapshot-format.yaml	Updates snapshot expectations.
internal/workshop/lxd/tests/integration/snapshot_test.go	Adds VM snapshot comparisons.
internal/workshop/lxd/tests/integration/project_test.go	Enables multicall test binaries.
internal/workshop/lxd/tests/helper/helper.go	Adds shared VM test helpers.
internal/workshop/lxd/start_command.sh	Removes legacy startup script.
internal/workshop/lxd/lxd_base_manager.go	Selects images by confinement.
internal/workshop/lxd/lxd_backend.go	Implements VM launch and readiness behavior.
internal/workshop/lxd/lxd_backend_test.go	Tests container and VM configuration.
internal/workshop/lxd/lxd_backend_snapshots.go	Adds VM filesystem freezing.
internal/workshop/lxd/lxd_backend_project.go	Includes VMs in project operations.
internal/workshop/fakebackend/backend.go	Propagates confinement through tests.
internal/workshop/backend.go	Extends backend contracts for confinement.
internal/waitready/waitready.go	Makes timeout test-configurable.
internal/overlord/workshopstate/request.go	Adjusts refresh stops and execution lookup.
internal/overlord/workshopstate/request_test.go	Updates snapshot fixtures.
internal/overlord/workshopstate/manifest.go	Gates VM use and rejects VM SDKs.
internal/overlord/workshopstate/manifest_test.go	Tests VM manifest restrictions.
internal/overlord/workshopstate/manager.go	Warns about legacy workshop formats.
internal/overlord/workshopstate/handlers_test.go	Updates confinement fixtures.
internal/overlord/sdkstate/handlers_test.go	Updates confinement fixtures.
internal/overlord/ifacestate/ifacemgr_test.go	Updates confinement fixtures.
internal/overlord/hookstate/handlers_test.go	Updates confinement fixtures.
internal/overlord/healthstate/healthstate_test.go	Updates confinement fixtures.
internal/interfaces/lxd_device/tests/integration/backend_test.go	Adapts device integration setup.
internal/idmap/structs.go	Defines ID-map structures.
internal/idmap/idmapset_linux.go	Implements Linux ID-map operations.
internal/fsfreeze/sys/zsysnum_linux.go	Defines generated ioctl constants.
internal/fsfreeze/sys/sysnum_linux.go	Provides ioctl generation input.
internal/fsfreeze/sys/syscall_linux.go	Wraps freeze/thaw ioctls.
internal/fsfreeze/sys/generate.sh	Generates ioctl constants.
internal/fsfreeze/fsfreeze.go	Implements guest filesystem freezing.
internal/fsfreeze/fsfreeze_test.go	Tests filesystem freezing.
internal/fsfreeze/export_test.go	Exposes freeze test hooks.
internal/dirs/dirs.go	Adds the guest fsfreeze path.
internal/daemon/snapshot-ingredients.yaml	Includes confinement in snapshots.
internal/daemon/api_workshops.go	Exposes confinement through the API.
internal/daemon/api_workshops_test.go	Updates API expectations.
internal/daemon/api_sdks_test.go	Updates snapshot fixtures.
internal/daemon/api_exec_test.go	Updates snapshot fixtures.
internal/daemon/api_connections.go	Locks state during workshop lookup.
internal/daemon/api_connections_test.go	Updates snapshot fixtures.
docs/reference/definition-files/schema.json	Adds the confinement schema field.
cmd/workshopctl/main.go	Adds the fsfreeze multicall mode.
cmd/workshop/init.go	Adds workshop init --vm.
cmd/workshop/init_test.go	Tests VM initialization.
cmd/workshop/info.go	Displays confinement.
cmd/workshop/info_test.go	Updates info output expectations.
client/workshop.go	Exposes confinement to clients.

Files not reviewed (1)

• internal/fsfreeze/sys/zsysnum_linux.go: Generated file

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jonathan-conder]

@tomponline we took this code from LXD, this comment looks relevant for you as well

[jonathan-conder]

Summary of changes since my last comment:

• Only check for old workshops when using the LXD backend.
• Fixed shellcheck issue in the fsfreeze package.
• Add agent.nic_config=true to LXD VM config; seems to be necessary to override eth0 config in systemd-networkd, but I'm not sure it's reliable yet.
• Switched back to force-stopping containers in the snapshot tests. We need to stop VMs gracefully but there's an issue where gracefully stopping containers times out.
• Use ID-mapped mounts for container root filesystems in the snapshot tests. This allows another container to modify the files, so we can share test code with the VM case.
• Ignore /var/snap/lxd/common when diffing snapshot filesystems.
• Check /project idmap in tests/main/launch.
• Limit workshop.experimental-vms gate to workshop launch.
• Add confinement field to workshop info.
• Split tests/main/launch and tests/integration/workshop-snapshots. VM tests are disabled until I have time to experiment with the limits of our GitHub runners.

[dmitry-lyfar]

I need to finish tests review, these are the points, I've come across so far.

[dmitry-lyfar]

Can I ask you to summarise the idmap logic and problems we address with calculating idmaps here? Links to the existing issues work too.

[jonathan-conder]

Problem it addresses

Containers and VMs have inconsistent idmaps by default. The ideal solution is to fix this problem in LXD: canonical/lxd#18875. But for now, this workaround does the same thing with some minor caveats.

I don't know of any real world cases that require consistency, but suppose you want to limit your agent to a certain subdirectory of /project/. If you try that with a VM before the idmap commit, you'll see something like this:

$ mkdir slop
$ chmod 777 slop
$ workshop exec -- sudo useradd -m agent
$ workshop exec -- sudo -u agent touch /project/slop/example.go
touch: cannot touch '/project/slop/example.go': Invalid argument

Whereas it works in containers.

I added a test, which should pass for both containers and VMs, here.

Summary of the logic

For containers, the outcome is the same as before (just different whitespace). The implementation differs in that we express the idmap for the workshop user and group as a []idmap.IdmapEntry and render it using formatIdMap.

For VMs, the outcome is an extended raw.idmap, but volatile.idmap.next should be the same as containers.

The logic to generate raw.idmap for VMs is modelled on how LXD computes the container idmap here, but:

• we skip parsing raw.idmap and just start with the resulting list of IdmapEntrys.
• LXD merges this into a copy of state.OS.IdmapSet.Idmap, we use idmap.KernelDefaultMap() instead.

When using the LXD snap, the results are identical:

• state.OS.IdmapSet.Idmap is constructed using idmap.GetIdmapSet().
• idmap.GetIdmapSet() is just some logging and error handling on top of idmap.DefaultIdmapSet("", ""); we copy the error handling.
• idmap.DefaultIdmapSet("", "") returns idmap.kernelDefaultMap() unless it finds /etc/subuid or /etc/subgid; these don't exist in the confined LXD snap.

If we wanted to support non-snap LXD installations, the most reliable way would be to call idmap.DefaultIdmapSet("/proc/<LXD PID>/root", "root"), but this would make it harder to run workshopd as an unprivileged user. Instead we could try to detect whether LXD is a snap or not; I opted not to bother implementing this because it's not a case we support anyway.

Another difference is that LXD also tries to extend raw.idmap for VMs here, but only does it when the existing idmap doesn't handle UID 0 or GID 0. I think this is possible, but it would require running LXD in a user namespace with lots of available UIDs and few available GIDs, or vice versa; I don't think it's likely to occur in practice.