TODO.org: Update TODOs

* unguix/loki-home-rekahsoft-ca/loki.sh: Remove /var/log volume (related change in promtail-config.yaml). This matches the docker-compose.yaml file introduced in this commit.

* unguix/loki-home-rekahsoft-ca/rsyslog.conf: Correct omfwd action target. Now that docker-compose is being used, and both the promtail and rsyslog containers are in the same non-default bridge network, we can depend on container dns resolution to handle the fact that promtail's ip is dynamic.

* unguix/loki-home-rekahsoft-ca/promtail-config.yaml (scrape_configs): Remove system /var/log job (its not necessary now that syslog is setup, and was primarily there for testing purposes in the first place).

* unguix/loki-home-rekahsoft-ca/docker-compose.yaml: New file; depreciates loki.sh. This enables correcting an issue with rsyslog configuration referencing the correct docker instance of promtail.

TODO.org

@@ -1233,3 +1233,33 @@ DEADLINE: <2023-03-31 Fri>
   - [X] [[file:./.guix/rekahsoft/guix-config/vms/searx0-home-rekahsoft-ca.scm]]
   - [X] [[file:./.guix/rekahsoft/guix-config/vms/vault0-home-rekahsoft-ca.scm]]
 
+** TODO Look into issues with syslog forwarding not working until service is manually restarted
+DEADLINE: <2023-04-04 Tue>
+
+I noticed this after upgrading all guix vms to enable syslog forwarding. It seems that I need
+to manually restart the syslogd service after reboots. More investigations required.
+
+Confirmed on:
+- git0
+- vault0
+- grocy0
+- dash0
+- ci0
+
+Rebooting vault0 to see if the change was just a one-off after rebooting following my most
+recent upgrades, or is an issue upon every boot. It appears to happen every boot 😢.
+
+My first guess is that the syslogd service needs to depend on network (but imho that doesn't
+make that much sense, as it should still eventually work once the network is online).
+
+** TODO Setup docker syslog driver on vm's that use docker
+DEADLINE: <2023-04-06 Thu>
+
+See [[https://docs.docker.com/config/containers/logging/syslog/][docker syslog driver documentation]] for more details on how to configure it. Note however
+that in guix, the guix docker-configuration (documentation in [[info:guix#Miscellaneous Services][Miscellaneous Services]] section)
+does not allow this to be configured. However, as documented in [[info:guix#Service Reference][Service Reference]] section,
+the ~etc-service-type~ can be extended to place the appropriate ~/etc/docker/daemon.json~
+configuration file in place.
+
+Alternatively, the docker-configuration could be updated to support setting logging
+configuration and other options.

unguix/loki-home-rekahsoft-ca/docker-compose.yaml

@@ -0,0 +1,39 @@
+version: "3.9"
+
+services:
+  loki:
+    image: docker.nexus.home.rekahsoft.ca/grafana/loki:2.7.4
+    command: "-config.file=/mnt/config/loki-config.yaml"
+    networks:
+      - net-bridge
+    ports:
+      - "3100:3100"
+    volumes:
+      - /var/lib/loki/config:/mnt/config
+  rsyslog:
+    image: docker.nexus.home.rekahsoft.ca/rsyslog/syslog_appliance_alpine:8.36.0-3.7
+    networks:
+      - net-bridge
+    ports:
+      - "514:514"
+      - "514:514/udp"
+    environment:
+      RSYSLOG_CONF: /config/rsyslog.conf
+    volumes:
+      - /var/lib/rsyslog/config:/config
+      - /var/lib/rsyslog/work:/work
+      - /var/lib/rsyslog/logs:/logs
+  promtail:
+    image: docker.nexus.home.rekahsoft.ca/grafana/promtail:2.7.4
+    command: "-config.file=/mnt/config/promtail-config.yaml"
+    networks:
+      - net-bridge
+    ports:
+      - "1514:1514/udp"
+      - "9080:9080"
+      - "1514:1514"
+    volumes:
+      - /var/lib/loki/config:/mnt/config
+
+networks:
+  net-bridge:

unguix/loki-home-rekahsoft-ca/loki.sh

@@ -22,7 +22,6 @@ docker run -d \
   --restart unless-stopped \
   --name promtail \
   -v /var/lib/loki/config:/mnt/config \
-  -v /var/log:/var/log \
   -p 1514:1514 \
   -p 1514:1514/udp \
   -p 9080:9080 \

unguix/loki-home-rekahsoft-ca/promtail-config.yaml

@@ -9,13 +9,6 @@ clients:
   - url: http://loki:3100/loki/api/v1/push
 
 scrape_configs:
-- job_name: system
-  static_configs:
-  - targets:
-      - localhost
-    labels:
-      job: varlogs
-      __path__: /var/log/*log
 - job_name: syslog
   syslog:
     listen_address: 0.0.0.0:1514

unguix/loki-home-rekahsoft-ca/rsyslog.conf

@@ -25,7 +25,7 @@ ruleset(name="log_to_files") {
 
 ruleset(name="remote") {
   # TODO: the target is a docker bridge ip (on the default bridge, so an alias cannot be used); this should be fixed
-  action(type="omfwd" Target="172.17.0.2" Port="1514" Protocol="tcp" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted")
+  action(type="omfwd" Target="promtail" Port="1514" Protocol="tcp" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted")
 }
 
 #################### default ruleset begins ####################