guix-north-america/README.org

#+TITLE: Guix North America
#+AUTHOR: Collin J. Doering

#+begin_abstract
This repository contains setup and management instructions for a Guix North American Build
Farm.
#+end_abstract

* Install Guix on debian to be used to bootstrap the Guix os installation

Optionally, the below steps can be completed within tmux or screen. Tmux was installed and
used in this case using the following.

#+begin_src shell
  sudo apt update
  sudo apt install tmux
  tmux
#+end_src

Following the [[https://guix.gnu.org/manual/en/html_node/Binary-Installation.html][Binary Installation]] section from the Guix manual to install guix.

#+begin_src shell
  sudo apt install -y guix
#+end_src

This installs the Debian's packaged version of Guix, which likely is older then what's
available upstream. As such, update our installation of Guix (following the [[https://guix.gnu.org/manual/en/html_node/Upgrading-Guix.html][Updating Guix]]
documentation specific to foreign distros').

#+begin_src shell
  sudo -i guix pull
  sudo systemctl restart guix-daemon.service
#+end_src

* Define Guix operating-system for the machine

See: [[file:balg02.scm][balg02.scm]]

** Bootloader configuration

For this installation, debian and its bootloader Grub will be left in place. Because we want
to retain Guix's interactions with Grub (eg. to allow for restoring from failed upgrades to
an earlier generation), we will have debian's Grub chainload Guix's Grub. To do so, we will
need to manually adjust Debians' Grub in order to add another menu entry, and set it as the
default menu item.

Below is a snippet from debian's ~/etc/default/grub~.

#+begin_src text
  GRUB_DEFAULT=0
  GRUB_TIMEOUT=5
  GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
  GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0,115200n8"
  GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0,115200n8"
  GRUB_TERMINAL="console serial"
  GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=1 --word=8 --parity=no --stop=1"
#+end_src

From this we extract the necessary guix bootloader configuration options (for serial).

- serial-unit :: 1
- serial-speed :: 115200
- terminal-inputs :: console serial
- terminal-outputs :: console serial

*** Manual modifications to Debian's Grub
:PROPERTIES:
:CUSTOM_ID: manual_modifications_to_debians_grub
:END:

Modify grub config on debian to add an additional (and default) option to chainload Guix
grub.

- Add a menuitem for Guix in ~/etc/grub.d/40_custom~, where ~<EFI-UUID>~ is replaced with the
  efi partition UUID.

  #+begin_src text
    menuentry "Gnu Guix" {
        insmod part_gpt
        insmod search_fs_uuid
        insmod chain
        search --fs-uuid --no-floppy --set=root <EFI-UUID>
        chainloader ($root)/EFI/Guix/grubx64.efi
    }
  #+end_src

- Modify ~/etc/default/grub~ setting ~GRUB_DEFAULT="Gnu Guix"~

- Run ~grub-mkconfig -o /boot/grub/grub.cfg~

** Network configuration

Using the a snippet taken from ~/etc/network/interfaces~ on the existing debian installation
(below), we can extract the necessary details to configure Guix's static-networking-service.

- Interface        :: eno8303
- Address          :: 216.37.76.55/24
- Gateway          :: 216.37.76.1
- DNS Name Servers :: 216.37.64.2 216.37.64.3
- DNS Search       :: genenetwork.org

#+begin_src text
  # The primary network interface
  allow-hotplug eno8303
  iface eno8303 inet static
          address 216.37.76.55/24
          gateway 216.37.76.1
          # dns-* options are implemented by the resolvconf package, if installed
          dns-nameservers 216.37.64.2 216.37.64.3
          dns-search genenetwork.org
#+end_src

** Disk Partitioning
:PROPERTIES:
:CUSTOM_ID: disk_partitioning
:END:

For this installation we are using ~/dev/sdb~ (a 1.5T ssd which is faster then the
alternative 3.6T ssd in the server).

First, we require a variety of tools to setup and partition the disk destined for Guix
installation. These could be installed on debian, however an alternative approach would be to
use Guix from debian as a package manager to temporarily provide the prerequisite tools. This
can be done using the shell spawned from the following command.

#+begin_src shell
  guix shell parted btrfs-progs dosfstools
#+end_src

*** Create disk partition table and layout

#+begin_src bash
  parted /dev/sda mklabel gpt
#+end_src

*** Create partitions

A simple™️ partition layout is used for this installation, consisting of an EFI ESP partition,
and the remaining disk partitions for use by btrfs, where btrfs subvolumes and a swapfile
will be used.

#+begin_src bash
  parted /dev/sda mkpart primary fat32 0% 512MiB
  parted /dev/sda mkpart primary 512MiB 100%
#+end_src

*** Create EFI partition

#+begin_src bash
  parted /dev/sda set 1 esp on
  mkfs.fat -F32 /dev/sda1
#+end_src

*** Create btrfs 'pool' (file-system) and subvolumes

**** Create btrfs file-system

#+begin_src bash
  mkfs.btrfs --label root /dev/sda2
#+end_src

**** Create btrfs subvolumes

First mount the btrfs top-level file-system.

#+begin_src bash
  mount /dev/sda2 /mnt
#+end_src

Then create the root subvolume, and a subvolume for swapfiles.

#+begin_src bash
  btrfs subvolume create /mnt/@
  btrfs subvolume create /mnt/@swap
#+end_src

Unmount the top-level btrfs file-system.

#+begin_src bash
  umount /mnt
#+end_src

Mount the root subvolume.

#+begin_src bash
  mount -o subvol=@,compress=zstd /dev/sda2 /mnt
#+end_src

Create nested subvolumes for ~/gnu/store~ and ~/home~.

#+begin_src bash
  mkdir -p /mnt/gnu

  btrfs subvolume create /mnt/gnu/store
  btrfs subvolume create /mnt/home
  btrfs subvolume create /mnt/var
#+end_src

*** Create swap

#+begin_src bash
  mkdir /mnt/swap
  mount -o subvol=@swap /dev/sda2 /mnt/swap
  chmod 600 /mnt/swap/swapfile
  touch /mnt/swap/swapfile
  chattr +C /mnt/swap/swapfile
  dd if=/dev/zero of=/mnt/swap/swapfile bs=1M count=32768

  mkswap /mnt/swap/swapfile
#+end_src

*** Prepare ~/mnt~ for Guix installation

Create ~/boot/efi~ directory for UEFI boot and mount the ESP partition there.

#+begin_src bash
  mkdir -p /mnt/boot/efi
  mount /dev/sda1 /mnt/boot/efi
#+end_src

Both root and swap are already mounted and ready due to earlier steps.

** Testing

To test the configuration in a vm before deployment, the following can be used.

#+begin_src shell
  $(guix time-machine -C channels.scm -- system vm -e '(@ (guix-na config balg02) %system)') -m 2G -smp 2 -nic user,model=virtio-net-pci
#+end_src

** Manual Testing of bootstrapping Guix from a Debian VM

To correctly test this deployment, a environment that mimics bal02g should be used. The
closest to this is a VM with debian installed, with an additional virtual disk to bootstrap
guix onto. This will enable validating bootloader changes required to chainboot Guix's Grub.

This testing could be automated, but was done manually as we do not expect to have to
bootstrap a system like this often.

*** Setup Debian VM

1. Using ~qemu~, ~libvirt~, ~virtualbox~, etc.. create a VM that boots using UEFI firmware.

   1. Create an additional virtual disk that will be used to bootstrap Guix onto from Debian.
      This disk should be ~>20GiB~.

   2. Ensure that there is a serial device attached to the VM.

2. Install Debian 12 on the VM created during step 1 (this can be a minimal server
   installation, no desktop, etc..).

   1. It's worth noting that for some reason debian didn't setup a efi boot
      entry for some reason. Not sure why. To create one I used:

      #+begin_src shell
        efibootmgr --create --disk /dev/vda -p 1 -L "Debian" -l "\EFI\debian\grub64.efi"
      #+end_src

      After which I would have adjusted the boot order with:

      #+begin_src shell
        efibootmgr -o X,Y,...
      #+end_src

      However, in my case it was not needed as the boot order had debian first.

3. Reboot VM; further configure Debian.

   1. Enable serial for debian grub

      Modify ~/etc/default/grub~, adjusting ~GRUB_TERMINAL~ and ~GRUB_CMDLINE_LINUX_DEFAULT~ as
      follows.

      #+begin_src text
        GRUB_TERMINAL="console serial"
        GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0,115200n8"
      #+end_src

   2. Enable getty over serial

      #+begin_src shell
        systemctl enable getty@ttyS0.service
        systemctl start getty@ttyS0.service
      #+end_src

*** Test Bootstrapping Gnu Guix from Debian

With the Debian VM setup, we can now apply the documented bootstrapping steps.

1. [[#disk_partitioning][Disk Partitioning]], but with disks adjusted to match the testing VM.
2. [[#bootstrap_guix][Bootstrap Guix]], ensure ~<EFI-UUID>~ matches the VM efi partition used for Guix.
3. [[#manual_modifications_to_debians_grub][Manual modifications to Debian's Grub]], again ensuring ~<EFI-UUID>~ matches the VM efi
   partition used for Guix.
4. Reboot

Following rebooting the VM, its expected that:

- Debian Grub boots first, has "Gnu Guix" as its default selected option, which boots Guixs'
  Grub.
- Serial access works for:
  - Debian and Guix Grub/s
  - Debian and Guix linux console

As this testing is occurring in a VM, its worth noting things that are NOT expected to to be
testable.

- The network interfaces are not going to match what is on balg02, so its expected that the
  networking service will not be able to start.

* Bootstrap Guix
:PROPERTIES:
:CUSTOM_ID: bootstrap_guix
:END:

Using Guix on debian, bootstrap the machine using the configuration in [[*Define Guix operating-system for the machine][Define Guix
operating-system for the machine]].

** Configure Guix Channels

First, fetch the most recent channel file from the target machine.

#+begin_src shell
  curl -O https://git.rekahsoft.ca/rekahsoft/guix-north-america/raw/branch/master/channels.scm
#+end_src

** Create and Bootstrap System

Create a ~bootstrap.scm~ file like below, but where ~<EFI-UUID>~ is replaced with the efi
partition UUID.

#+begin_src scheme
  ((@ (guix-na config balg02) balg02) "<EFI-UUID>")
#+end_src

Use ~guix system init ...~ to instantiate the system, but using guix time-machine to use
pinned dependencies.

#+begin_src shell
  guix time-machine -C channels.scm -- system init bootstrap.scm /mnt
#+end_src

** Post Boostrapping

After guix has been bootstrapped, its useful to do an initial ~guix pull~ using the same
channels that were used during bootstrapping.

#+begin_src shell
  guix pull -C /run/current-system/channels.scm
#+end_src

To ensure your shell refers to the correct guix after its been updated, run ~hash guix~.