--- name: nixos-usb-recovery-10b852bf6cba description: "Recover or rebuild an installed NixOS system from a NixOS USB/live ISO. Use for boot failures, broken generations, bootloader repairs, or rescue rebuilds." --- # NixOS USB Recovery Use this skill when repairing an installed NixOS system from a NixOS USB/live environment. The goal is to recover the existing installation in place: mount the target system, inspect generations/configuration, rebuild or repair, then reboot safely. ## When to use - The machine is booted into a NixOS installer/live USB for rescue work. - A NixOS generation, boot entry, activation script, or package change broke boot. - You need to build a new generation for the installed system without reinstalling. - You need to repair bootloader entries, profile symlinks, or `/boot` contents. ## Safety principles - Do not reinstall unless the user explicitly asks. - Do not assume device names, pool names, hostnames, or users. - Prefer inspection before mutation: identify disks, filesystems, mountpoints, and boot entries first. - Keep a note of every mount and symlink you change so you can undo or explain it. - Use pinned flake refs or the user's requested source when rebuilding. - Treat secrets and SSH keys carefully; do not print private key contents. ## Quick recovery flow 1. **Confirm live environment and network access** ```bash hostname ip addr lsblk -f findmnt ``` If working remotely, confirm SSH stays available before starting a long rebuild. 2. **Identify the installed system** Inspect disks and filesystems without assuming names: ```bash lsblk -f sudo blkid sudo zpool import 2>/dev/null || true sudo btrfs filesystem show 2>/dev/null || true ``` Look for the installed root filesystem, Nix store filesystem, home datasets, and EFI System Partition. 3. **Import or unlock storage if needed** ZFS example, replacing placeholders with discovered names: ```bash sudo zpool import -N -R /mnt sudo zfs list ``` LUKS example: ```bash sudo cryptsetup open /dev/disk/by-uuid/ ``` 4. **Mount the installed system under `/mnt`** Use discovered devices/datasets. Examples: ```bash sudo mount /dev/disk/by-uuid/ /mnt sudo mkdir -p /mnt/nix /mnt/boot /mnt/home sudo mount /dev/disk/by-uuid/ /mnt/nix # if separate sudo mount /dev/disk/by-uuid/ /mnt/boot # or /mnt/boot/efi ``` ZFS example: ```bash sudo zfs mount / sudo mkdir -p /mnt/nix /mnt/boot sudo zfs mount / sudo mount /dev/disk/by-uuid/ /mnt/boot ``` Verify before continuing: ```bash findmnt -R /mnt test -e /mnt/etc/NIXOS || echo "Warning: /mnt does not look like NixOS root" test -d /mnt/nix/store || echo "Warning: Nix store is not mounted" ``` 5. **Inspect generations and boot entries** ```bash ls -l /mnt/nix/var/nix/profiles/system* find /mnt/boot -maxdepth 3 -type f \( -name '*.conf' -o -name 'loader.conf' \) -print cat /mnt/boot/loader/loader.conf 2>/dev/null || true ls /mnt/boot/loader/entries 2>/dev/null || true ``` If the task is a rollback, update the boot default and system profile only after confirming the target generation exists. 6. **Prepare for `nixos-enter` or rebuild** Ensure DNS works inside the installed root if the rebuild fetches inputs: ```bash sudo rm -f /mnt/etc/resolv.conf sudo cp -L /etc/resolv.conf /mnt/etc/resolv.conf ``` Enter the installed environment when activation scripts or installed paths matter: ```bash sudo nixos-enter --root /mnt ``` 7. **Rebuild or repair** For a new generation from a flake: ```bash sudo nixos-enter --root /mnt -c \ 'nixos-rebuild boot --refresh --flake #' ``` If private flake inputs are fetched over SSH, pass an installed deploy key or agent deliberately: ```bash sudo nixos-enter --root /mnt -c \ 'env GIT_SSH_COMMAND="ssh -i /home//.ssh/ -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new" \ nixos-rebuild boot --refresh --flake #' ``` Use `boot` for rescue work when you want the new generation selected on next boot without switching live services in the USB environment. Use `switch` only when you intentionally want activation inside the entered system. 8. **Verify the installed result** ```bash readlink -f /mnt/nix/var/nix/profiles/system cat /mnt/boot/loader/loader.conf 2>/dev/null || true ls -lt /mnt/boot/loader/entries 2>/dev/null | head sudo nixos-enter --root /mnt -c 'nixos-version; systemctl --version | head -1' ``` If the repair was for a specific command or package, test it from `nixos-enter`: ```bash sudo nixos-enter --root /mnt -c ' --help | head' ``` 9. **Unmount and reboot safely** ```bash sync sudo umount -R /mnt || true sudo zpool export 2>/dev/null || true sudo reboot ``` If a pool export says it is busy, do not panic. Run `findmnt -R /mnt`, close shells using `/mnt`, run `sync`, and document the busy export if rebooting anyway. ## Common repair patterns ### Build a fixed generation instead of rolling back Use this when the user has already fixed the flake/config and wants that fix deployed: ```bash sudo nixos-enter --root /mnt -c \ 'nixos-rebuild boot --refresh --flake #' ``` Then verify the new generation number, boot default, and the fixed command/package. ### Select a known-good generation Use only when rollback is requested or rebuilding is not possible: ```bash ls -l /mnt/nix/var/nix/profiles/system-*-link sudo ln -sfn /nix/var/nix/profiles/system--link /mnt/nix/var/nix/profiles/system sudo sed -i 's/^default .*/default nixos-generation-.conf/' /mnt/boot/loader/loader.conf ``` Verify the entry exists before changing the default: ```bash test -f /mnt/boot/loader/entries/nixos-generation-.conf ``` ### Repair systemd-boot entries If the installed profile is correct but boot entries are missing or stale, rebuild boot files from the installed environment: ```bash sudo nixos-enter --root /mnt -c 'nixos-rebuild boot --install-bootloader --flake #' ``` ## Post-boot checks After removing the USB and booting from disk, SSH into the installed system and check: ```bash readlink /nix/var/nix/profiles/system systemctl is-system-running || true systemctl --failed --no-pager systemctl is-active sshd tailscaled 2>/dev/null || true ``` If a service initially failed during boot but later recovered, use `systemctl reset-failed` only after confirming the service is now healthy. ## Final report checklist Summarize: - How the installed system was identified and mounted. - What generation or flake ref was deployed. - Any bootloader/profile changes made. - Verification performed before reboot and after boot. - Any remaining failed units or follow-up work. --- Source attribution: edmundmiller Original project: https://github.com/edmundmiller/dotfiles/blob/a60ca3c122ffc91fdb3d63db156b1e30851c9872/.agents/skills/nixos-usb-recovery/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice The MIT License (MIT) Copyright (c) 2020 Edmund Miller Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Mega AI Hub resource metadata Creator / publisher: edmundmiller Original project: https://github.com/edmundmiller/dotfiles/blob/a60ca3c122ffc91fdb3d63db156b1e30851c9872/.agents/skills/nixos-usb-recovery/SKILL.md Attribution basis: Source repository record Fields of work: Development & Engineering, Product Management Roles: Backend Developer, DevOps / SRE, Technical Product Manager Specialties: Backend, DevOps Tools: Nixos Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: 14233d371cc780a3f1027f76f991b4b9c165748eb20c9a57ad13f49e592cc220