lainos-kernel-backup

Version: 1.0.0
Author: Grayson Giles
License: GPL3
Target: LainOS Layer 02 (baremetal, systemd-free, OpenRC, BTRFS)

---

Why This Exists

The Real Risk: Experimental Init-Replacement

LainOS Layer 02 is not a standard Arch system. It replaces systemd with OpenRC and a custom Protocol 7 compatibility layer — a set of early-boot services (lainos-init, lainos-dbus-bridge, cgroup-delegate, lainos-wifi-dns, etc.) that depend on specific kernel behaviors.

When Arch upstream upgrades the kernel, they test against systemd. They do not test against:

• Custom cgroup delegation outside systemd's unified hierarchy
• Netlink socket behavior relied on by lainos-dbus-bridge
• Specific sysfs device enumeration paths used by lainos-init
• Module loading sequences in an initramfs built without systemd-udev

A kernel upgrade that works perfectly on standard Arch can silently break LainOS Layer 02 because the init-replacement stack exercises different kernel surface area.

What "Drastic Upstream Changes" Look Like

Here are concrete examples of kernel changes that have broken or could break LainOS Layer 02:

Upstream Change	How It Breaks LainOS	Recovery
cgroups v2 hierarchy refactor (e.g., new cgroup.subtree_control behavior, cpuset delegation changes)	cgroup-delegate service fails; containers and sandboxed services lose resource limits or fail to start entirely	Boot backup kernel → patch cgroup-delegate to match new hierarchy → test in VM → upgrade
sysfs device class restructuring (e.g., network devices moved in sysfs, new dev links)	lainos-init or lainos-dbus-bridge cannot enumerate WiFi/eth interfaces; boot hangs waiting for devices that exist but are at unexpected paths	Boot backup kernel → update device discovery logic → test → upgrade
Module compression format change (e.g., Arch switches from xz to zstd-compressed modules)	Initramfs kmod cannot load critical modules (disk controller, filesystem, wireless); boot drops to emergency shell with no storage or network	Boot backup kernel → rebuild initramfs with updated kmod/mkinitcpio → upgrade
Kernel boot protocol changes (e.g., init= parameter handling refactored, new rd.* requirements)	Libreboot chainloads the kernel, but the kernel misinterprets LainOS's init handoff and drops to a broken prompt instead of starting OpenRC	Boot backup kernel → update kernel command line or init wrapper → upgrade
BPF verifier hardening	Stricter BPF verifier rejects existing LainOS BPF programs used for network sandboxing or syscall filtering; AppArmor-like behaviors fail silently	Boot backup kernel → rewrite BPF programs for new verifier → test → upgrade
DRM/Framebuffer changes (e.g., simpledrm replaces efifb, console handoff changes)	tuigreet shows black screen or garbled output; user cannot log in despite system being otherwise functional	Boot backup kernel → add nomodeset or update console service → upgrade
Netlink socket behavior changes (e.g., RTM_NEWLINK message format changes, wireless extensions deprecation)	iwd or lainos-wifi-dns fails to detect interface state changes; WiFi appears "up" but never associates	Boot backup kernel → update netlink parsing in affected services → upgrade

Why BTRFS Snapshots Are Not Enough

LainOS Layer 02 uses BTRFS by default. You might think: "Just snapshot / and roll back." This is insufficient for kernel recovery because:

1. The kernel image is what GRUB loads. If /boot/vmlinuz-linux is overwritten with a broken kernel, a BTRFS snapshot of the root filesystem does not help you boot — GRUB still points to the broken file on disk. You need the previous kernel binary to exist somewhere GRUB can load it.

2. The module tree is version-specific. /usr/lib/modules/$(uname -r)/ is deleted on upgrade. Even if you BTRFS-snapshot /, booting an old kernel with a missing module tree causes modprobe failures for disk controllers, WiFi, graphics, etc.

3. Snapshots are all-or-nothing. Rolling back a BTRFS snapshot reverts your entire root filesystem — all package updates, user data, logs, and configuration changes since the snapshot. This is a sledgehammer. Kernel backup is a scalpel: you boot the old kernel while keeping your current root filesystem intact.

4. Boot partition may be outside BTRFS. On many baremetal installs (including Libreboot T480 setups), /boot is on a separate ESP (FAT32) or a small ext4 partition. BTRFS snapshots do not cover these.

Why Pre-Transaction Backup

The backup runs as a PreTransaction pacman hook. This means it fires before pacman modifies any file. If the hook fails (disk full, permissions issue), the entire upgrade aborts. Your system is not left in a half-upgraded state with a missing kernel.

A PostTransaction hook would be too late — the old kernel would already be gone.

Why Save Headers

Kernel headers (/usr/lib/modules/<kver>/build/) are required for:

• Building out-of-tree kernel modules (wireless drivers, VirtualBox, etc.)
• dkms auto-rebuilds when modules update
• Compiling software that depends on specific kernel interfaces

If you restore a kernel but not its headers, dkms will fail to rebuild modules against the restored kernel, leaving you with a bootable but partially broken system.

Why No Automatic Pruning

Old backups are not automatically deleted. On a privacy-focused system, automatic deletion of recovery artifacts is a risk. You decide when to remove old backups based on your own disk constraints and risk tolerance.

---

What This Is

A lightweight, deterministic kernel backup and restore system. It snapshots the currently running kernel, initramfs, modules, and headers before pacman touches them.

If an upgrade breaks your system, you restore the backup in one command and reboot into a known-good kernel — without rolling back your entire BTRFS root filesystem.

---

What Gets Backed Up

For every backup, the following are copied to /var/lib/lainos/kernel-backups/<kver>_<timestamp>/:

File / Directory	Description
vmlinuz-linux-<kver>	The kernel image from /boot/
initramfs-linux-<kver>.img	The initramfs image
initramfs-linux-fallback-<kver>.img	The fallback initramfs (if present)
modules-<kver>/	The full module tree from /usr/lib/modules/<kver>/
headers-<kver>/	The kernel headers (follows the build symlink)
.backup-info	Metadata: kernel version, date, hostname

---

Installation

Build the package:

cd lainos-kernel-backup/
makepkg -f

Install on the target system:

doas pacman -U lainos-kernel-backup-1.0.0-1-any.pkg.tar.zst

No further configuration is required. The pacman hook activates immediately.

---

Automatic Backup (Default Behavior)

Every time pacman -Syu upgrades any of the following packages, the hook fires before the transaction begins:

• linux
• linux-lts
• linux-hardened
• linux-zen

You will see this during the upgrade:

(1/1) Backing up current kernel before upgrade...
==> Backing up kernel 6.12.8-arch1-1...
    Destination: /var/lib/lainos/kernel-backups/6.12.8-arch1-1_20260728_143022
==> Kernel backup complete: /var/lib/lainos/kernel-backups/6.12.8-arch1-1_20260728_143022
    To restore: doas lainos-kernel-restore 6.12.8-arch1-1_20260728_143022

Then pacman proceeds with the upgrade normally.

---

Manual Backup

Before a risky manual kernel install or a custom build:

doas lainos-kernel-backup

---

Listing Backups

lainos-kernel-list

Output:

Available kernel backups:
────────────────────────────────────────────────────────────
  6.12.8-arch1-1_20260728_143022    6.12.8-arch1-1    20260728_143022
  6.12.7-arch1-1_20260721_091512    6.12.7-arch1-1    20260721_091512
────────────────────────────────────────────────────────────
Restore: doas lainos-kernel-restore <backup-name>

---

Restoring a Kernel

If the new kernel panics, fails to boot, or breaks hardware support:

# Boot from a LainOS live USB or drop to an emergency shell
# Mount your root partition and chroot if needed
# Then:

doas lainos-kernel-restore 6.12.8-arch1-1_20260728_143022

# This copies the backup back to /boot/ and /usr/lib/modules/
# Then regenerate GRUB:
doas grub-mkconfig -o /boot/grub/grub.cfg

doas reboot

At the GRUB menu, select:

LainOS Kernel Backup: 6.12.8-arch1-1

Boot. Your system is back to the known-good kernel. Your BTRFS root filesystem is untouched — all your data, packages, and configs from after the upgrade are still there.

---

Recovery Workflow After an Incompatible Kernel Release

Scenario: Upstream kernel 6.13 breaks cgroup-delegate

1. Pacman upgrades linux → hook auto-backs up 6.12.8 before upgrade
2. Reboot → kernel 6.13 panics or services fail to start
3. Power cycle → GRUB menu appears
4. Select "LainOS Kernel Backup: 6.12.8-arch1-1" → system boots normally
5. Debug from a working system:

   # Check what broke
   dmesg | grep -i cgroup
   journalctl -u cgroup-delegate
   
   # Test the new kernel in a VM before deploying
   qemu-system-x86_64 -kernel /boot/vmlinuz-linux -append "init=/sbin/openrc-init"
   
   # Patch the service
   vim /etc/init.d/cgroup-delegate
   
   # When fixed, remove the backup entry or keep it until next cycle
   rm -rf /var/lib/lainos/kernel-backups/6.12.8-arch1-1_20260728_143022
   doas grub-mkconfig -o /boot/grub/grub.cfg
   

Scenario: Upstream switches to zstd-compressed modules

1. Pacman upgrades linux → hook auto-backs up working kernel
2. Reboot → initramfs cannot load modules; emergency shell
3. Boot backup kernel from GRUB → system works
4. Fix from working system:

   # Update mkinitcpio or kmod to support zstd
   doas pacman -S kmod mkinitcpio
   # Rebuild initramfs with new tools
   doas mkinitcpio -P
   # Test reboot
   

---

How Restore Works

1. Reads .backup-info to verify the backup is valid
2. Copies vmlinuz-linux-<kver> back to /boot/
3. Copies initramfs-linux-<kver>.img back to /boot/ (and fallback if present)
4. Replaces /usr/lib/modules/<kver>/ with the backed-up module tree
5. Replaces /usr/lib/modules/<kver>/build/ (headers) with the backed-up headers
6. Runs grub-mkconfig to add the backup entry to the GRUB menu

The restore is atomic per-file. If a step fails, the script exits with an error and does not leave a half-restored state.

---

Trust Model

Threat	Mitigation
Hook fires after pacman deletes files	PreTransaction hook — fires before any package file is modified
Hook is silently bypassed	Only pacman -Sdd bypasses hooks; normal -Syu always triggers them
Backup is incomplete or corrupted	Verifies kernel image, modules, headers, and initramfs exist before copying; aborts if anything is missing
Headers are not saved	Follows the /usr/lib/modules/<kver>/build symlink and copies the actual header tree
Backup metadata is tampered with	Restore refuses to proceed if .backup-info is missing or malformed
Restore overwrites the wrong kernel	Reads KVER from metadata; refuses to restore if metadata is missing
BTRFS snapshot rollback destroys user data	Kernel backup is surgical — only the kernel is restored, root filesystem stays current

This package does not:

• Prevent kernel upgrades
• Modify the pacman package database
• Hold packages
• Install a second bootloader
• Replace BTRFS snapshots (it complements them)

---

Storage Requirements

Each backup is approximately:

• Kernel image: ~10 MB
• Initramfs: ~30–50 MB
• Modules: ~100–200 MB
• Headers: ~100–150 MB

Total per backup: ~250–400 MB

Backups are stored in /var/lib/lainos/kernel-backups/. Monitor disk usage and remove old backups manually:

# Remove a specific backup
rm -rf /var/lib/lainos/kernel-backups/6.12.7-arch1-1_20260721_091512

# Or remove all but the last N
# (No automatic pruning is implemented — this is intentional.)

---

Troubleshooting

"No kernel backups found"

The hook only fires on kernel package upgrades. If you have not upgraded since installing this package, no backups exist yet. Run doas lainos-kernel-backup manually.

"Backup metadata missing — refusing to restore"

The .backup-info file was deleted or corrupted. The backup directory may have been tampered with. Do not restore it.

GRUB does not show the backup entry

lainos-kernel-restore runs grub-mkconfig automatically. If the entry is still missing, check:

# Is the kernel image in /boot/?
ls /boot/vmlinuz-linux-*

# Is the GRUB script executable?
ls -l /etc/grub.d/41_lainos_kernel_backups

# Regenerate manually
doas grub-mkconfig -o /boot/grub/grub.cfg

"Headers symlink broken"

linux-headers is not installed, or the symlink /usr/lib/modules/$(uname -r)/build is missing. Install linux-headers before running a manual backup.

"Can I just use a BTRFS snapshot instead?"

No. A BTRFS snapshot rolls back your entire root filesystem — all packages, data, and configs since the snapshot. Kernel backup lets you boot a known-good kernel while keeping your current root filesystem intact. This is critical when you need to debug an incompatible kernel from a working system without losing post-upgrade changes.

---

Changelog

Version	Date	Changes
1.0.0	2026-07-28	Initial release — automatic pre-transaction backup, restore, GRUB integration

---

End of document.