Compare commits

...

4 Commits

4 changed files with 312 additions and 91 deletions

368
TODO.org
View File

@ -1035,9 +1035,67 @@ CLOSED: [2023-03-28 Tue 14:45] DEADLINE: <2023-06-15 Thu>
- State "CANCELED" from "TODO" [2023-03-28 Tue 14:45] \\
No longer relevant as I'm moving away from nextcloud.
See: https://ownyourbits.com/2017/04/18/different-ways-to-access-your-nextcloud-files/
*** CANCELED [#A] Update nextcloud :home_network:
CLOSED: [2023-04-01 Sat 22:25] DEADLINE: <2023-04-02 Sun>
:PROPERTIES:
:ARCHIVE_TIME: 2023-04-01 Sat 22:26
:END:
- State "CANCELED" from "PAUSE" [2023-04-01 Sat 22:25] \\
Have made the decision to not go back to nextcloud, however I will re-use the cloud0 vm (it
has been updated as part of another TODO)
- State "PAUSE" from "DOING" [2023-03-12 Sun 22:15] \\
Pausing this as the update is now borked and I'm using it as an opportunity to replace
nextcloud
- Note taken on [2023-03-10 Fri 09:41] \\
I attempted to update right to the latest version out of laziness. I should have done it
version by version, because the update paused for some time, and then failed saying multiple
major versions upgrades are not supported. Fair, but the update also wrote changes (not sure
exactly where, but the ~/var/www/html~ volume was adjusted, and perhaps other stuff too).
Anyways, the old nextcloud container no longer boots, and the system is officially bored.
Though this was unexpected, in a way it could be viewed as a blessing in disguise. Namely,
now I can replace nextcloud (which I have had more and more performance issues with as it is
more heavily used).
- State "DOING" from "NEXT" [2023-03-09 Thu 21:16]
- Note taken on [2023-01-05 Thu 12:36] \\
I may just replace nextcloud with [[id:a4009c1c-a32e-4b73-b0b4-af41d95747d0][radicle]] + some file storage web/mobile solution (eg. [[id:8f0253f9-968f-4332-996a-31f42f5c14d3][SyncThing]])
- Note taken on [2022-09-01 Thu 16:49] \\
Nextcloud reports:
#+begin_quote
Nextcloud 22.2.5
The version you are running is not maintained anymore. Please make sure to update to a supported version as soon as possible.
#+end_quote
Thus, its important to update ASAP
- State "NEXT" from "TODO" [2022-09-01 Thu 16:49]
*** DONE Resolve deprecation's in [[file:.guix/rekahsoft/guix-config/proxmox-vm-lvm-minimal.scm][.guix/rekahsoft/guix-config/proxmox-vm-lvm-minimal.scm]]
CLOSED: [2023-04-01 Sat 23:29] DEADLINE: <2023-04-04 Tue>
:PROPERTIES:
:ARCHIVE_TIME: 2023-04-02 Sun 08:14
:END:
- Note taken on [2023-04-01 Sat 23:29] \\
Done as part of another TODO
- State "DONE" from "TODO" [2023-04-01 Sat 23:29]
#+begin_src text
.guix/rekahsoft/guix-config/proxmox-vm-lvm-minimal.scm:82:11: warning: 'accountsservice-service' is deprecated, use 'accountsservice-service-type' instead
.guix/rekahsoft/guix-config/proxmox-vm-lvm-minimal.scm:84:11: warning: 'elogind-service' is deprecated, use 'elogind-service-type' instead
.guix/rekahsoft/guix-config/proxmox-vm-lvm-minimal.scm:85:11: warning: 'dbus-service' is deprecated, use 'dbus-root-service-type' instead
#+end_src
** Backlog :backlog:
DEADLINE: <1990-05-08 Tue>
*** TODO Setup data volumes for all vms, migrating persistent data to them
DEADLINE: <2023-07-01 Sat>
For the purposes of backups, it would be preferred that all important vm data is stored on a
separate volume that is attached to the vm. The reason for this, is because then external
backups of this data can be made (using proxmox backup server replication), without having to
backup the ~/gnu/store~ and other files that can simply be recreated by ~guix~.
*** TODO Improve metrics collection [0/4] :metrics:
**** TODO Make prometheus setup production ready and used everywhere [0%] :metrics:
**** TODO Setup [[https://github.com/msroest/sabnzbd_exporter][sabnzbd_exporter]] (Prometheus exporter for sabnzbd) :home_network:metrics:
- Note taken on [2021-07-29 Thu 11:49] \\
May also be able to use [[https://grafana.com/grafana/dashboards/9567][this grafana community dashboard for sabnzbd]]
@ -1081,12 +1139,33 @@ DEADLINE: <2023-04-30 Sun>
- [ ] [[id:831d3046-3412-4f9b-a66e-cf34e8808250][Sonarr]]
- [ ] [[id:789e5ec4-f800-478a-a1d8-c0fb58a5bd3a][Radarr]]
- [ ] [[id:817e3ed9-0857-40cb-8515-682d69ac943d][Lidarr]]
*** TODO Setup loki (and associated services) in a High-Availability configuration
- [ ] loki
- [ ] Storage considerations
I originally was thinking of just storing logs on disk, but after further consideration
and consultation of the [[https://grafana.com/docs/loki/latest/operations/storage/filesystem][loki documentation]], decided it would be best to use the s3
backend with my local ceph/rados powered s3. See the [[https://grafana.com/docs/loki/latest/operations/storage/filesystem][s3-exapnded-config.yaml]] they provide
in their documentation as a starting point.
- [ ] rsyslog (syslog collector)
- [ ] promtail
- [ ] orchestrating deployment
I expect this to be done with ~guix deploy~ (but would require support for running
docker-compose from guix).
*** TODO Setup guix on personal pinebook pro
*** TODO Setup guix on dell personal laptop
** Ideas :spike:
DEADLINE: <1990-05-08 Tue>
*** TODO Setup [[https://github.com/home-assistant/core][home-assistant]] accessible on ~<SOMETHING>.home.rekahsoft.ca~
DEADLINE: <2023-05-01 Mon>
*** TODO [#C] Investigate usefulness of [[https://github.com/photoprism/photoprism][photoprism]] to manage photos similar to google photos :home_network:
DEADLINE: <2023-03-26 Sun>
DEADLINE: <2023-04-08 Sat>
*** TODO Investigate and setup [[id:cca06776-de84-4146-ba61-abe8697b178d][Readarr]]
*** TODO Setup [[https://octoprint.org/][octoprint]] accessible on ~3d.home.rekahsoft.ca~ :home_network:
@ -1101,8 +1180,9 @@ This follows maintenance work completed as part of [[*Rotate ssh-host keys][Rota
intervention.
- [ ] Remove data (or use ~pg_upgrade~ or reset postgres if the data upgrade didn't work out)
** TODO Run/configure docker/docker-compose from within guix configurations
** NEXT Run/configure docker/docker-compose from within guix configurations
- State "NEXT" from "TODO" [2023-04-02 Sun 08:14]
Better docker support within guix configuration is required to make this issue go away. Once
completed, the ~unguix~ directory can be removed.
@ -1121,8 +1201,21 @@ Checked indicates VMs that use docker/docker-compose (and either depend on files
- [X] [[file:.guix/rekahsoft/guix-config/vms/searx0-home-rekahsoft-ca.scm]]
- [X] [[file:.guix/rekahsoft/guix-config/vms/vault0-home-rekahsoft-ca.scm]]
** TODO [#A] Setup btrfs backup system [0%]
DEADLINE: <2023-04-04 Tue>
*** TODO Setup local 'time-machine' like backups of personal data (using [[id:86cd693b-b56e-40a4-a56d-7b912c62e6f2][Btrbk]] + cron)
- [ ] Personal "Work" laptop
- [ ] Personal laptop
- [ ] pinebook pro
*** TODO Setup remote (in home-network) backup solution for btrbk
This requires a host with a btrfs volume available for backup storage.
** DOING Add missing vm's to Prometheus configuration [75%] :home_network:
DEADLINE: <2023-04-01 Sat>
DEADLINE: <2023-04-08 Sat>
- State "DOING" from "TODO" [2022-07-01 Fri 13:09]
- [X] vault0.home.rekahsoft.ca
@ -1134,17 +1227,8 @@ DEADLINE: <2023-04-01 Sat>
- [ ] dash0.home.rekahsoft.ca
- [X] pbs.home.rekahsoft.ca
** DOING [#A] Make prometheus/loki setup production ready and used everywhere :home_network:metrics:
DEADLINE: <2023-04-01 Sat>
- State "DOING" from "TODO" [2023-03-01 Wed 11:37]
- Note taken on [2021-07-29 Thu 11:51] \\
See:
- https://github.com/grafana/loki/blob/v1.5.0/docs/clients/promtail/scraping.md#syslog-receiver
- https://www.gnu.org/software/inetutils/manual/html_node/syslogd-invocation.html#syslogd-invocation
- https://github.com/prometheus/snmp_exporter
- https://www.robustperception.io/snmp-monitoring-with-prometheus
** NEXT [#A] Investigate replacing nextcloud with another dav + file storage solution
DEADLINE: <2023-04-01 Sat>
DEADLINE: <2023-04-08 Sat>
- State "NEXT" from "TODO" [2023-03-10 Fri 09:44]
I'm currently thinking [[id:a4009c1c-a32e-4b73-b0b4-af41d95747d0][radicle]] + [[id:8f0253f9-968f-4332-996a-31f42f5c14d3][SyncThing]], but more research is required.
@ -1153,52 +1237,10 @@ The primary goal of this change would to avoid the maintenance and challenges th
with running nextcloud. As well as some of the performance challenges I have seen when
handling many files (eg. media not loading or crashing the nextcloud android app).
** TODO Allow for using ~guix home~ for minimal server dotfiles (eg so emacs-tramp works)
DEADLINE: <2023-04-01 Sat>
** PAUSE [#A] Update nextcloud :home_network:
DEADLINE: <2023-04-02 Sun>
- State "PAUSE" from "DOING" [2023-03-12 Sun 22:15] \\
Pausing this as the update is now borked and I'm using it as an opportunity to replace
nextcloud
- Note taken on [2023-03-10 Fri 09:41] \\
I attempted to update right to the latest version out of laziness. I should have done it
version by version, because the update paused for some time, and then failed saying multiple
major versions upgrades are not supported. Fair, but the update also wrote changes (not sure
exactly where, but the ~/var/www/html~ volume was adjusted, and perhaps other stuff too).
Anyways, the old nextcloud container no longer boots, and the system is officially bored.
Though this was unexpected, in a way it could be viewed as a blessing in disguise. Namely,
now I can replace nextcloud (which I have had more and more performance issues with as it is
more heavily used).
- State "DOING" from "NEXT" [2023-03-09 Thu 21:16]
- Note taken on [2023-01-05 Thu 12:36] \\
I may just replace nextcloud with [[id:a4009c1c-a32e-4b73-b0b4-af41d95747d0][radicle]] + some file storage web/mobile solution (eg. [[id:8f0253f9-968f-4332-996a-31f42f5c14d3][SyncThing]])
- Note taken on [2022-09-01 Thu 16:49] \\
Nextcloud reports:
#+begin_quote
Nextcloud 22.2.5
The version you are running is not maintained anymore. Please make sure to update to a supported version as soon as possible.
#+end_quote
Thus, its important to update ASAP
- State "NEXT" from "TODO" [2022-09-01 Thu 16:49]
** TODO Resolve deprecation's in [[file:.guix/rekahsoft/guix-config/proxmox-vm-lvm-minimal.scm][.guix/rekahsoft/guix-config/proxmox-vm-lvm-minimal.scm]]
DEADLINE: <2023-04-04 Tue>
#+begin_src text
.guix/rekahsoft/guix-config/proxmox-vm-lvm-minimal.scm:82:11: warning: 'accountsservice-service' is deprecated, use 'accountsservice-service-type' instead
.guix/rekahsoft/guix-config/proxmox-vm-lvm-minimal.scm:84:11: warning: 'elogind-service' is deprecated, use 'elogind-service-type' instead
.guix/rekahsoft/guix-config/proxmox-vm-lvm-minimal.scm:85:11: warning: 'dbus-service' is deprecated, use 'dbus-root-service-type' instead
#+end_src
** TODO [#A] Setup btrfs backups (using [[id:86cd693b-b56e-40a4-a56d-7b912c62e6f2][Btrbk]] + cron) [0%]
DEADLINE: <2023-04-04 Tue>
- [ ] Personal "Work" laptop
- [ ] Personal laptop
** TODO Enable using ~guix home~ to setup minimal server dotfiles (eg so emacs-tramp works)
DEADLINE: <2023-04-08 Sat>
** DOING Perform manual update of all systems
DEADLINE: <2023-03-31 Fri>
DEADLINE: <2023-04-08 Sat>
- State "DOING" from "TODO" [2023-03-28 Tue 20:07]
- [X] Manual
@ -1233,9 +1275,26 @@ DEADLINE: <2023-03-31 Fri>
- [X] [[file:./.guix/rekahsoft/guix-config/vms/searx0-home-rekahsoft-ca.scm]]
- [X] [[file:./.guix/rekahsoft/guix-config/vms/vault0-home-rekahsoft-ca.scm]]
** TODO Look into issues with syslog forwarding not working until service is manually restarted
** DOING [#A] Make loki setup production ready and used everywhere [25%] :metrics:
DEADLINE: <2023-04-15 Sat>
- State "DOING" from "TODO" [2023-03-01 Wed 11:37]
- Note taken on [2021-07-29 Thu 11:51] \\
See:
- https://github.com/grafana/loki/blob/v1.5.0/docs/clients/promtail/scraping.md#syslog-receiver
- https://www.gnu.org/software/inetutils/manual/html_node/syslogd-invocation.html#syslogd-invocation
- https://github.com/prometheus/snmp_exporter
- https://www.robustperception.io/snmp-monitoring-with-prometheus
*** DONE Setup loki permanent log storage :logging:
CLOSED: [2023-04-02 Sun 08:57]
- State "DONE" from "NEXT" [2023-04-02 Sun 08:57]
- State "NEXT" from "TODO" [2023-04-02 Sun 08:50]
I was having issues getting this to work until I came across the following [[https://github.com/grafana/loki/pull/1834][loki PR]].
*** NEXT Look into issues with syslog forwarding not working until service is manually restarted :logging:
DEADLINE: <2023-04-04 Tue>
- State "NEXT" from "TODO" [2023-04-02 Sun 08:52]
I noticed this after upgrading all guix vms to enable syslog forwarding. It seems that I need
to manually restart the syslogd service after reboots. More investigations required.
@ -1252,9 +1311,10 @@ recent upgrades, or is an issue upon every boot. It appears to happen every boot
My first guess is that the syslogd service needs to depend on network (but imho that doesn't
make that much sense, as it should still eventually work once the network is online).
** TODO Setup docker syslog driver on vm's that use docker
*** NEXT Setup docker syslog driver on vm's that use docker :logging:
DEADLINE: <2023-04-06 Thu>
- State "NEXT" from "TODO" [2023-04-02 Sun 08:52]
See [[https://docs.docker.com/config/containers/logging/syslog/][docker syslog driver documentation]] for more details on how to configure it. Note however
that in guix, the guix docker-configuration (documentation in [[info:guix#Miscellaneous Services][Miscellaneous Services]] section)
does not allow this to be configured. However, as documented in [[info:guix#Service Reference][Service Reference]] section,
@ -1263,3 +1323,191 @@ configuration file in place.
Alternatively, the docker-configuration could be updated to support setting logging
configuration and other options.
** DOING [#A] Setup guix on lenovo t80s personal laptop
DEADLINE: <2023-03-19 Sun>
- State "DOING" from "TODO" [2022-03-14 Mon 11:50]
*** Notes
**** Partitioning
***** TODO Create disk partition table and layout
#+begin_src bash
# TODO: Create gpt labeled disk
#+end_src
***** Create EFI partition
#+begin_src bash
parted /dev/nvme0n1p1 set 1 esp on
mkfs.fat -F32 /dev/nvme0n1p1
#+end_src
***** Create LUKS container on remainder of disk
#+begin_src bash
cryptsetup luksFormat -l crypt /dev/nvme0n1p2
#+end_src
****** Unlock LUKS container after creation
#+begin_src bash
cryptsetup luksOpen /dev/nvme0n1p2
#+end_src
***** Create LVM2 container inside of LUKS container
****** Create Physical Volume (pv)
#+begin_src bash
pvcreate /dev/mapper/crypt
#+end_src
****** Create Volume Group (vg)
#+begin_src bash
vgcreate vg0 /dev/mapper/crypt
#+end_src
****** Create Logical Volume/s (vg)
#+begin_src bash
vgcreate -L 442G vg0 -n root
vgcreate -l +100%FREE vg0 -n swap
#+end_src
***** Create btrfs 'pool' (file-system) and subvolumes
****** Create btrfs file-system
#+begin_src bash
mkfs.btrfs -l root /dev/vg0/root
#+end_src
****** Create btrfs subvolumes
First mount the btrfs top-level file-system.
#+begin_src bash
mount /dev/vg0/root /mnt
#+end_src
Then create the root subvolume.
#+begin_src bash
btrfs subvolume create /mnt/@
#+end_src
Unmount the top-level btrfs file-system.
#+begin_src bash
umount /mnt
#+end_src
Mount the root subvolume.
#+begin_src bash
mount -o subvol=@,compress=zstd /dev/vg0/root /mnt
#+end_src
Create nested subvolumes for ~/gnu/store~ and ~/home~.
#+begin_src bash
mkdir -p /mnt/gnu /mnt/var/log
btrfs subvolume create /mnt/gnu/store
btrfs subvolume create /mnt/home
# TODO: Should have created these
#btrfs subvolume create /mnt/var/log
#+end_src
***** Create and activate swap
#+begin_src bash
mkswap -l swap /dev/vg0/swap
swapon /dev/vg0/swap
#+end_src
***** Prepare ~/mnt~ for Guix installation
Create ~/boot/efi~ directory for UEFI boot and mount the ESP partition there.
#+begin_src bash
mkdir -p /mnt/boot/efi
mount /dev/nvme0n1p1 /mnt/boot/efi
#+end_src
Both root and swap are already mounted and ready due to earlier steps.
**** Install Guix
Start ~cow-store~ to allow later steps to write store changes to ~/mnt/gnu/store~ as well as
to bootstrap the system.
#+begin_src bash
herd start cow-store /mnt
#+end_src
#+begin_src bash
guix system init /mnt/etc/config.scm /mnt
#+end_src
***** DONE Figure out how config file should be retrieved
CLOSED: [2022-04-20 Wed 11:47]
- State "DONE" from "TODO" [2022-04-20 Wed 11:47]
Figured out. This should be done through a channel, and is now implemented in the
[[https://git.home.rekahsoft.ca/rekahsoft-public/guix-machines][guix-machines]] repository.
***** DONE Determine what setup looks like if non-free firmware is required
CLOSED: [2022-04-20 Wed 11:48]
- State "DONE" from "TODO" [2022-04-20 Wed 11:48]
This is going to require using a pre-built guix image that includes the appropriate firmware
and blob loader (non-libre linux kernel).
***** TODO Streamline setup as my own disk image
**** Setup user-space
***** Setup flatpak
Flatpak comes installed on Guix without any system of user remotes. Additionally, a
~/var/lib/flatpak~ folder does not exist by default on arch. It could be created, with a
group controlling its permissions, but I instead prefer to keep this to be managed on a
per-user basis. As such, add flathub as a user remote:
#+begin_src sh :results output
flatpak --user remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
#+end_src
****** DONE Look into output
CLOSED: [2022-04-20 Wed 11:48]
- State "DONE" from "TODO" [2022-04-20 Wed 11:48]
#+begin_src text
Note that the directories
'/var/lib/flatpak/exports/share'
'/home/collin/.local/share/flatpak/exports/share'
are not in the search path set by the XDG_DATA_DIRS environment variable, so
applications installed by Flatpak may not appear on your desktop until the
session is restarted.
#+end_src
****** TODO Install flatpak applications
I don't currently know which applications I will be using from flatpak. It will be a fallback
if the package is not available in Non-Guix or Guix repositories, and cannot be easily
packaged.
I will need to develop a nice way to automatically make sure flatpak applications are
installed. For the time being, I will just keep track of what has been installed here.
#+begin_src bash
flatpak install flathub us.zoom.Zoom
flatpak install flathub com.slack.Slack
#+end_src

View File

@ -34,6 +34,7 @@ services:
- "1514:1514"
volumes:
- /var/lib/loki/config:/mnt/config
- /var/lib/loki/data:/loki
networks:
net-bridge:

View File

@ -15,7 +15,7 @@ ingester:
chunk_retain_period: 30s
max_transfer_retries: 0
wal:
dir: /tmp/wal
dir: /loki/wal
schema_config:
configs:
@ -29,10 +29,10 @@ schema_config:
storage_config:
boltdb:
directory: /tmp/loki/index
directory: /loki/index
filesystem:
directory: /tmp/loki/chunks
directory: /loki/chunks
limits_config:
enforce_metric_name: false

View File

@ -1,28 +0,0 @@
#!/usr/bin/env bash
docker run -d \
--restart unless-stopped \
--name loki \
-v /var/lib/loki/config:/mnt/config \
-p 3100:3100 \
docker.nexus.home.rekahsoft.ca/grafana/loki:2.7.4 -config.file=/mnt/config/loki-config.yaml
docker run -d \
--restart unless-stopped \
--name rsyslog \
-v /var/lib/rsyslog/config:/config \
-v /var/lib/rsyslog/work:/work \
-v /var/lib/rsyslog/logs:/logs \
-p 514:514 \
-p 514:514/udp \
-e RSYSLOG_CONF=/config/rsyslog.conf \
docker.nexus.home.rekahsoft.ca/rsyslog/syslog_appliance_alpine:8.36.0-3.7
docker run -d \
--restart unless-stopped \
--name promtail \
-v /var/lib/loki/config:/mnt/config \
-p 1514:1514 \
-p 1514:1514/udp \
-p 9080:9080 \
docker.nexus.home.rekahsoft.ca/grafana/promtail:2.7.4 -config.file=/mnt/config/promtail-config.yaml