Improve development environment setup and documentation

* src/site.hs (siteRules): Refactor site building Rules into a new variable allowing for
reuse. Remove duplicate rules ()
(myConfig): Set deployCommand to something temporary.
(_devWatch): Entrypoint mean for use in development; it rebuilds the site builder, then
rebuilds the site.

* site (Removed file): Removed as it is no longer necessary. If additional cli functionality
is required, this can be done directly in haskell.

* guix.scm: Symlink 'site' now that it is no longer provided as a script.
(setenv): setenv only applies to the first (non-cached) run of `guix shell ...`,
because of this, remove setting PS1.
(native-inputs): Include development environment dependencies

* channels.scm (channel): Update rekahsoft-guix channel (noop, but originally updated while
looking at using overmind).

* bootstrap.sh: A script that generates various development scripts from the projects
README.org file.

* README.org: Cleanup and minor restructuring.
(Repository Structure): New (incomplete) section
(Prerequisites): New section
(Quick Start): New section
(Hakyll Site Commands): New section, with subheadings for various hakyll sub-commands.
(Makefile): New section; generates a makefile useful for development

* .gitignore: Ignore files generated by ./bootstrap.sh

* .ghci: ghci configuration file for hakyll development.

* .envrc: direnv configuration that loads a development environment for those who have
allowed it
This commit is contained in:
Collin J. Doering 2023-05-30 22:05:30 -04:00
parent fa89f9f158
commit bad2d07ce9
Signed by: rekahsoft
GPG Key ID: 7B4DEB93212B3022
9 changed files with 240 additions and 167 deletions

10
.envrc Normal file
View File

@ -0,0 +1,10 @@
use_guix-shell() {
CHANNEL_FILE=channels.scm
if [ -f $CHANNEL_FILE ]; then
eval "$(guix time-machine -C $CHANNEL_FILE -- shell "$@" --search-paths)"
else
eval "$(guix shell "$@" --search-paths)"
fi
}
use guix-shell -f guix.scm -Df guix.scm

3
.ghci Normal file
View File

@ -0,0 +1,3 @@
:set -fwarn-unused-binds -fwarn-unused-imports
:set -isrc
:load src/site.hs

4
.gitignore vendored
View File

@ -17,3 +17,7 @@ infra/.terraform
infra/terraform.tfstate.d
infra/*.local.tfvars
infra/*.plan
# Generated by ./bootstrap
Makefile
scripts

View File

@ -45,6 +45,60 @@ Various licenses ([[https://www.gnu.org/licenses/gpl.html][GPLv3]], [[http://cre
License]]) are deployed dependent on which part of the site is in question. Please see the
[[./LICENSE][LICENSE]] file for full details.
* TODO Repository Structure
#+begin_src text
.
├── blog-rekahsoft-ca.cabal
├── bootstrap.sh
├── channels.scm --
├── clay
│ └── *.hs -- clay source files
├── css
├── drafts
│ └── *.md -- draft posts
├── files
│ ├── images
│ └── source
├── fonts
│ └── *.{ttf,woff,...} -- font files
├── guix.scm
├── images
│ └── ...
├── images-src
│ └── ...
├── infra
│ ├── channels.scm -> ../channels.scm
│ ├── main.tf
│ ├── Makefile
│ ├── manifest.scm
│ ├── outputs.tf
│ ├── production.tfvars
│ ├── staging.tfvars
│ └── variables.tf
├── js
│ └── *.js -- javascript files
├── lib
│ └── ... -- javascript libraries
├── LICENSE
├── manifest.scm
├── pages
│ └── ...
├── posts
│ └── ...
├── README.org
├── robots.txt
├── Setup.hs
├── site
├── src
│ └── site.hs
└── templates
├── default.html
├── pages
├── partials
└── tag-page.html
#+end_src
* TODO Guix Development Environment
:PROPERTIES:
:header-args: :session *vterm blog-rekahsoft-ca* :results none
@ -53,8 +107,32 @@ License]]) are deployed dependent on which part of the site is in question. Plea
[[https://guix.gnu.org/][Gnu Guix]] is used to package this project and manage its dependencies, as well as to provide
reproducible development environments.
A simple wrapper script [[./site][site]] is provided that wraps the hakyll powered site builder to
offer some additional functionality.
** Prerequisites
The only prerequisite for starting a development environment for this project is [[https://guix.gnu.org/][GNU Guix]].
Optionally, [[https://direnv.net/][direnv]] can be used to enable a non-containerized development environment that is
abridged with your existing shell.
** Quick Start
First run the bootstrap script, which uses this documentation to generate a ~Makefile~ that
can be used for development.
#+name: bootstrap
#+begin_src sh
./bootstrap.sh
#+end_src
Then run the development 'auto-watching' environment:
#+begin_src sh
make
#+end_src
This starts a containerized local development environment that uses [[https://github.com/ndmitchell/ghcid/][ghcid]] to watch haskell
sources and restart hakyll's [[*Watch][site watch]] feature when changes occur. The site will be
available at http://localhost:3000, and will be automatically rebuild as site files change
(templates, post, pages, etc..).
** Start Development Environment
@ -65,8 +143,8 @@ The development environment is defined by the following files:
To start a development environment, run the following:
#+begin_src sh
guix time-machine -C channels.scm -- shell -CN -E '^LANG$' -E '^TERM$' -E '^PS1$' -f guix.scm -Df guix.scm
#+begin_src sh :mkdirp yes :tangle ./scripts/start-development-environment.sh :tangle-mode (identity #o555)
guix time-machine -C channels.scm -- shell -CN -E '^LANG$' -E '^TERM$' -f guix.scm -Df guix.scm
#+end_src
This uses the [[info:guix#Invoking guix time-machine][guix time-machine]] feature to ensure the development environment is reproducible
@ -75,11 +153,11 @@ shell]] command is used within the time-machine to start a development environme
container (~-C~), which shares the hosts network namespace (~-N~). The environment variable
~LANG~ is passed into the container to ensure locales work as expected; without this, site
building will fail! Additionally, the environment variable ~TERM~ is passed into the
container to ensure the development shell behaves correctly. Finally, ~-f guix.scm~ loads the
~blog-rekahsoft-ca~ package, and ~-Df guix.scm~ indicates that development dependencies of
the ~blog-rekahsoft-ca~ package should be included in the environment.
container to ensure the development shell behaves correctly. The option ~-f guix.scm~ loads
the ~blog-rekahsoft-ca~ package, and ~-Df guix.scm~ indicates that development dependencies
of the ~blog-rekahsoft-ca~ package should be included in the environment.
*** Deployment Development Environment
*** Deployment Environment
[[https://guix.gnu.org/][Gnu Guix]] is used, similar to in the [[*Start Development Environment][previous section]], to create environments with all tools
necessary for deployments, with a notable difference being a ~guix.scm~ file is not provided
@ -93,33 +171,33 @@ or needed, as the deployment environment is used solely for its side effects.
including the development and deployment environment use the same set of Guix channels.
- [[./infra/manifest.scm][infra/manifest.scm]] :: Defines packages required for deployment of this site
To start a deployment development environment, run the following:
To start a deployment environment, run the following:
#+begin_src sh
#+begin_src sh :mkdirp yes :tangle ./scripts/start-deployment-environment.sh :tangle-mode (identity #o555)
cd infra
guix time-machine -C channels.scm -- shell -CN -E '^LANG$' -E '^TERM$' -E '^PS1$' -E '^AWS.*$'
guix time-machine -C channels.scm -- shell -CN -E '^LANG$' -E '^TERM$' -E '^AWS.*$'
#+end_src
*** Composing Site Development and Deployment Environments
#+begin_src sh
guix time-machine -C channels.scm -- shell -CN -E '^LANG$' -E '^TERM$' -E '^PS1$' -E '^AWS.*$' -f guix.scm -Df guix.scm -m infra/manifest.scm
#+begin_src sh :mkdirp yes :tangle ./scripts/start-development-and-deployment-environment.sh :tangle-mode (identity #o555)
guix time-machine -C channels.scm -- shell -CN -E '^LANG$' -E '^TERM$' -E '^AWS.*$' -f guix.scm -Df guix.scm -m infra/manifest.scm
#+end_src
** Build Site
** Hakyll Site Commands
*** Build Site
This website is built from a collection of markdown files and templates that are processed by
pandoc and are stitched together using Hakyll. To build the html/css/jss and all other assets
required to deploy and distribute the site, the hakyll derived site-builder,
~blog-rekahsoft-ca~ must be invoked. Under usual conditions it is not invoked directly, but
instead via the ~site~ wrapper script. For example, this is how within a development
environment the site can be built.
~blog-rekahsoft-ca~ must be invoked. For convenience, an alias ~site~ is provided for the
site builder as part of its guix package. Here is it being used to build the site:
#+begin_src sh
site build
#+end_src
** Clean Site
*** Clean Site
[[*Build Site][Building the site]] has the side effect of writing a couple files/directories to disk as a
result of the build process. In some cases, its useful to start of with a clean slate and
@ -130,13 +208,63 @@ used:
site clean
#+end_src
** Watch
*** Watch
During development of new content or adjustments to the site, it is useful to autocompile
upon changes to any site files (templates, pages, posts, etc..). This functionality is
provided by Hakyll.
#+begin_src sh
site watch
#+end_src
** Deploy Site
*** TODO ~site deploy~ command
#+begin_src sh
site deploy
#+end_src
** Clean up Guix Store
#+begin_src sh :mkdirp yes :tangle ./scripts/clean-guix-store.sh :tangle-mode (identity #o555)
guix gc --list-dead | grep -e '^/gnu/store/.*-blog-rekahsoft-ca-.*' | xargs guix gc -D
#+end_src
** Enhanced Watch
When making adjustments to the site builder itself, it is useful to have functionality
similar to the site content watching feature of Hakyll, but for haskell source files.
Luckily, [[https://github.com/ndmitchell/ghcid/][ghcid]] can be used for this, and is included in the projects development
~manifest.scm~ file.
#+begin_src sh :mkdirp yes :tangle ./scripts/watch-all.sh :tangle-mode (identity #o555)
ghcid --test _devWatch
#+end_src
* Building a Release
The software built that itself builds this blog is released as a Guix package. It is
currently not, and is not ever expected to be distributed via a channel, as it provides
little benefit to anyone except myself, and is meant to operate along with stateful data,
including the site templates, content, pages, posts, etc..
To build a release, run the following command:
#+begin_src sh :mkdirp yes :tangle ./scripts/build-release.sh :tangle-mode (identity #o555)
guix time-machine -C channels.scm -- build -f guix.scm
#+end_src
This will produce a guix package with the following three outputs:
- ~out~ :: The ~blog-rekahsoft-ca~ site builder (also available as ~site~), and ~gencss~ css
generator binaries
- ~site~ :: A build of the website made with the site builder, etc.. in the ~out~ output of
this package, using the content at the same version
- ~static~ :: License file and any other file that should be distributed (eg manual)
** TODO What is done with the release?
* Deploying the Site
Terraform is used to deploy this site. Its configuration files are located in ~./infra~.
@ -145,14 +273,8 @@ that the deployment process is reliable, repeatable and quick. However, in the c
development and emergency deployments, clear documentation surrounding the deployment process
is necessary.
*** TODO ~site deploy~ command
#+begin_src sh
site deploy
#+end_src
*** Start [[*Deployment Development Environment][Deployment Development Environment]]
*** Setup a Particular Environment
** Start [[*Deployment Environment][Deployment Environment]]
** Setup a Particular Environment
Three environments (terraform workspaces) are currently available, including:
@ -167,7 +289,7 @@ Three environments (terraform workspaces) are currently available, including:
From this point onward, any ~make~ target run will operate on the selected environment,
unless its switched with the ~workspace~ or ~setup~ targets, or manually with ~terraform~.
*** See What Infrastructure Will Change
** See What Infrastructure Will Change
Run a terraform plan to see how the selected environments infrastructure will change.
@ -175,7 +297,7 @@ Run a terraform plan to see how the selected environments infrastructure will ch
make plan
#+end_src
*** Deploy the Site
** Deploy the Site
Run a terraform apply to deploy to the selected environment.
@ -183,50 +305,76 @@ Run a terraform apply to deploy to the selected environment.
make deploy
#+end_src
*** Working with Terraform Directly
** Working with Terraform Directly
Within a development environment, ~terraform~, its providers and all other dependencies are
available. As such, its possible to directly leverage ~terraform~ and its various operations.
This is particularly useful when debugging or adding make targets.
** Clean up Guix Store
* TODO Writing a Blog Post
The most natural way to edit and preview a post is to use [[https://direnv.net/][direnv]] along with this repository,
which uses ~guix shell~ to transparently provide all necessary tools, including [[*Hakyll Site Commands][Hakyll Site
Commands]]. When using direnv, a containerized environment will not be used, however for
content development, this is not a concern.
#+begin_src sh
guix gc --list-dead | grep -e '^/gnu/store/.*-blog-rekahsoft-ca-.*' | xargs guix gc -D
guix time-machine -C channels.scm -- shell -CN -E LANG -E TERM -f guix.scm
#+end_src
* Building a Release
* TODO Makefile
The software built that itself builds this blog is released as a Guix package. It is
currently not, and is not ever expected to be distributed via a channel, as it provides
little benefit to anyone except myself, and is meant to operate along with stateful data,
including the site templates, content, pages, posts, etc..
#+begin_src makefile :noweb yes :tangle Makefile :tangle-mode (identity #o444)
# THIS IS A GENERATED FILE, DO NOT EDIT!
# Instead modify README.org appropriately
To build a release, run the following command:
.DEFAULT_GOAL := watch-all
#+begin_src sh
guix time-machine -C channels.scm -- build -f guix.scm
.PHONY: bootstrap
bootstrap:
<<bootstrap>>
.PHONY: dev
dev:
./scripts/start-development-environment.sh
.PHONY: dev-deploy
dev-deploy:
./scripts/start-deployment-environment.sh
.PHONY: dev-all
dev-all:
./scripts/start-development-and-deployment-environment.sh
.PHONY: watch-all
watch-all:
./scripts/watch-all.sh
.PHONY: build
build-release:
./scripts/build-release.sh
.PHONY: clean
clean:
./scripts/clean-guix-store.sh
#+end_src
This will produce a guix package with the following three outputs:
* Continuous Integration & Delivery
- ~out~ :: The ~blog-rekahsoft-ca~ site builder and ~gencss~ css generator binaries, as well
as ~site~ user script
- ~site~ :: A build of the website made with the site builder, etc.. in the ~out~ output of
this package, using the content at the same version
- ~static~ :: License file and any other file that should be distributed (eg manual)
** TODO What is done with the release?
* Writing a Blog Post
** TODO Generate ~.drone.yaml~
#+begin_src sh
guix time-machine -C channels.scm -- shell -CN -E LANG -E TERM -E PS1 -f guix.scm -- site watch
drone jsonnet --stream --format
#+end_src
*Note:* currently ~drone-cli~ is not packaged for Guix, so for the time being, it can be run
with docker as follows, where ~<version>~ is the drone-cli version.
#+begin_src shell
docker run -v ${PWD}:/tmp/app -w /tmp/app --rm -it drone/cli:<versin> jsonnet --stream --format
#+end_src
* Known Issues
If you have an issue while browsing [[http://www.blog.rekahsoft.ca][my blog]] please file a issue in the [[https://git.rekahsoft.ca/rekahsoft/blog-rekahsoft-ca/issues][blog-rekahsoft-ca]]
issue tracker.
To see a list of already known issues, see [[./TODO.org][TODO.org]].

3
bootstrap.sh Executable file
View File

@ -0,0 +1,3 @@
#!/usr/bin/env bash
guix time-machine -C channels.scm -- shell -C emacs git -- emacs -q README.org --batch --eval '(org-babel-tangle)'

View File

@ -12,7 +12,7 @@
(name 'rekahsoft-guix)
(url "https://git.rekahsoft.ca/rekahsoft/rekahsoft-guix.git")
(commit
"58deed23414b5c7e82fc415ad06d15f39a2dfa56")
"e016a7e7a9eb3d27a4d6368861222e9916e27a47")
(introduction
(make-channel-introduction
"191cdaa0947657e0c85fe89ebbb8e7b1e7a8e0a4"

View File

@ -24,11 +24,10 @@
(guix git-download)
(guix gexp)
(gnu packages base)
(gnu packages haskell-apps)
(rekahsoft-gnu packages haskell-web)
(git))
(setenv "PS1" "\\W [env]\\$ ")
(define %srcdir
(dirname (current-filename)))
@ -44,7 +43,9 @@
#:recursive? #t
#:select? (git-predicate %srcdir)))
(build-system haskell-build-system)
(native-inputs `(("glibc-utf8-locales" ,glibc-utf8-locales)))
(native-inputs `(("glibc-utf8-locales" ,glibc-utf8-locales)
("make" ,gnu-make)
("ghcid" ,ghcid)))
(inputs `(("ghc-hakyll" ,ghc-hakyll)
("ghc-clay" ,ghc-clay)))
(outputs '("out" "site" "static"))
@ -55,7 +56,7 @@
(lambda* (#:key outputs #:allow-other-keys)
(let ((out (assoc-ref outputs "out")))
(setenv "PATH" (string-append out "/bin:" (getenv "PATH")))
(install-file "site" (string-append out "/bin/"))
(symlink (string-append out "/bin/blog-rekahsoft-ca") (string-append out "/bin/site"))
#t)))
(add-after 'install-site-script 'build-site
(lambda* (#:key outputs #:allow-other-keys)

93
site
View File

@ -1,93 +0,0 @@
#!/bin/sh
function run_site_only() {
[ "$SITE_ONLY" == "true" ]
}
function run_override_only() {
[ "$OVERRIDE_ONLY" == "true" ]
}
case "$1" in
-)
OVERRIDE_ONLY=true
shift
;;
--)
SITE_ONLY=true
shift
;;
esac
case "$1" in
gencss)
shift
gencss -- "$@"
;;
# Override of hakyll site commands
-h|--help)
run_override_only || ! run_site_only && cat << EOF
Wraps hakyll's provided site tool to augment certain commands.
Usage:
./site [-|--] COMMAND
Available commands:
build*
clean*
deploy*
gencss
Hakyll site commands:
build
check
clean
deploy
preview
rebuild
server
watch
Starred (*) commands indicate a overridden hakyll site command. However once the override is
run, the corresponding hakyll command is then run. This can be disabled with by specifying '--'
as the first argument, which will then pass all remaining arguments to the hakyll site command.
Similarily, to only run the override, specify '-' as the first argument.
For more details about hakyll site commands and options, see './site -- --help'.
EOF
# Only run hakyll site --help command if override was not run
run_override_only && exit
# Only run hakyll site --help command if -- site only
run_site_only && blog-rekahsoft-ca -- --help | sed 's/\(Usage: \)blog-rekahsoft-ca/\1.\/site -/g'
;;
build)
run_override_only || ! run_site_only && stack build
run_override_only && exit $?
;;&
clean)
run_override_only || ! run_site_only && stack clean
run_override_only && exit $?
;;&
deploy)
pushd infra > /dev/null
# Only run hakyll site deploy command when site-only is given. Additionally, when
# neither site-only or override-only are given, run only the override. The deploy
# override uses terraform which is also setup to deploy the hakyll site static files
run_override_only || ! run_site_only && (
export PLAN=".plans/local-$(date +%F_%R).plan"
[ ! -d .plans ] && mkdir .plans
make plan deploy
) && exit $?
run_site_only && export S3_BUCKET="$(terraform output s3_bucket_static)"
popd > /dev/null
;;&
*)
blog-rekahsoft-ca -- "$@"
;;
esac

View File

@ -84,14 +84,14 @@ pandocWriterOptions = defaultHakyllWriterOptions
myConfig :: Configuration
myConfig = defaultConfiguration
{ deployCommand = "echo 'Deploying website...' && " ++
{ deployCommand = "echo 'TODO (what to do with this cmd) Deploying website...' && " ++
"aws s3 sync _site/ s3://$S3_BUCKET &&" ++
"echo 'Done!'"
, previewPort = 3000
}
main :: IO ()
main = hakyllWith myConfig $ do
siteRules :: Rules ()
siteRules = do
match ("action/**" .||. "files/**" .||. "images/**" .||. "fonts/**" .||. "robots.txt") $ do
route idRoute
compile copyFileCompiler
@ -116,23 +116,12 @@ main = hakyllWith myConfig $ do
("posts/**" .&&. hasNoVersion)
(\n -> fromCapture "blog*.html" (show n))
clayDeps <- makePatternDependency $ fromGlob "clay/**.hs"
clayDeps <- makePatternDependency $ fromGlob "clay/*.hs"
rulesExtraDependencies [clayDeps] $ create ["default.css"] $ do
route idRoute
compile $ makeItem =<< (unsafeCompiler $ readProcess "gencss" ["compact"] "")
match "css/**" $ do
route idRoute
compile compressCssCompiler
match "lib/Skeleton/*.css" $ do
route $ gsubRoute "Skeleton" (const "css")
compile compressCssCompiler
match "templates/**" $ compile $ getResourceBody >>= saveSnapshot "original"
>> templateCompiler
-- Generate tag pages
forM_ (tagsMap tags) $ \(tag, identifiers) -> do
paginatedTaggedPosts <- buildPaginateWith
@ -244,6 +233,14 @@ main = hakyllWith myConfig $ do
route r
compile $ copyFileCompiler
_devWatch :: IO ()
_devWatch = do
_ <- hakyllWithExitCodeAndArgs myConfig (Options { verbosity = False, optCommand = Rebuild }) $ siteRules
hakyllWithArgs myConfig (Options { verbosity = False, optCommand = Watch "localhost" 3000 False }) $ siteRules
main :: IO ()
main = hakyllWith myConfig $ siteRules
---------------------------------------------------------------------------------------------------------
-- Functions & Constants --------------------------------------------------------------------------------
feedConfiguration :: Maybe String -> FeedConfiguration