07.08.2023

Maintain a simplistic blog

What is nicer than just publishing blog posts? Right, figuring out how to get them published convenient.

My blog is a static build done by Jekyll which holds its post in the _posts folder. The articles themselves are written in Obsidian.md.

Frederik asked me about integrations or automations…

… I didn’t know about any that suits my idea.

Copy blog posts (Single source of truth)

We cannot directly copy the posts because the files are named by their title. Jekyll uses Front Matter to get some metadata into the posts. Luckily for me, also Obsidian supports this feature. In the beginning of a note, I can add for example

---
layout: post
file_name: blog_infrastructure
title: "Maintain a simplistic blog"
description: "Hacking around to be lazy later"
date: 2023-08-07
categories: git obsidian ios
---

I want to follow some basic steps here

Take everything in the Blog folder that has a md file extension
Copy and rename the file to (my) Jekyll pattern YEAR-MONTH-DAY-shortend-title.md

Scripting

And now the fun already begins. You can skip this section and directly jump to the bash block in the end if you are not interested in my thoughts during the process. To have a loose coupling between the physical file and the blog post, I have to extract the date from the metadata. For this I use yq.

cat Maintain\ a\ simplistic\ blog.md| yq --front-matter=extract ".date"
2023-07-10

Now let’s head over to the second part of the file name. Shorting it down to a fixed size would be enough, also replacing whitespaces with underscores (_) will be fine. Because I expect the input to come from “somewhere” (currently find), I am aware that I will get the whole path of the file in the first place. After playing with cut and awk I realized that it might be much easier and controllable to add the file_name to the FrontMatter metadata.

Images

Obsidian is pretty much lenient, when it comes to image references, but of course not as lenient as I thought. Having images in a separate folder, like the Jekyll standard structure suggests, allows to reference the images either just by name (myimage.png), or (partial) path (media/blogpost/myimage.png). As a consequence the rendered page expects the images to be either in the same folder domain.xyz/2023/08/15/myimage.png or as a sub-path domain.xyz/2023/08/15/media/blogpost/myimage.png. After trying to figure out if there is any built-in way to a) see the correct images in Obsidian and b) also getting the references rendered in the Jekyll site build, I decided to use a custom plugin https://github.com/nhoizey/jekyll-postfiles and bring them closer to the posts. For now, I cannot imagine getting problems by using this structure.

group :jekyll_plugins do
  gem "jekyll-feed", "~> 0.12"
  gem "jekyll-postfiles"
end

The only change will be now to create a folder for each post. In the end the script gets even a little bit shorter.

#!/usr/bin/env bash

export TARGET=<path/to/Jekyll/_posts>

cp -r * "$TARGET"
find "$TARGET" -name "*.md" -exec sh -c '
DATE_PART=$(cat "$1" |yq --front-matter=extract ".date");
FILE_NAME=$DATE_PART"-"$(cat "$1" | yq --front-matter=extract ".file_name");
DIRECTORY=$(dirname "$1")
mv "$1" "$DIRECTORY/$FILE_NAME.md"
' unused {} \;

Folder structure

The resulting folder structure in Obsidian looks like this. Not that it could have as many sub-folders as you want. The folder names are for pretending to be organized ;)

├── duckdb
│   └── First experiences with DuckDB (and SQL in general after a while).md
├── neo4j
│   └── Spring Data Neo4j GraphQL 1
│       ├── Spring Data Neo4j GraphQL 1.md
│       ├── example_flow.png
│       ├── graph_data_set.png
│       └── graphiql.png
└── random
    └── Maintain a simplistic blog
        └── Maintain a simplistic blog.md

Jekyll configuration

In the _posts folder used to be also the archived posts migrated from the my old blog. To have this folder exclusively available for the new posts imported from Obsidian, I created a new _archive folder and added is as a collection to my config.yml.

collections:
  archive:
    output: true

Every post in there is part of the Jekyll repository while the new posts are only in place during the build steps. Mainly because I don’t care enough to import them into Obsidian.

Automation

Because it is available to me, I use a friend’s GitLab setup. I am pretty sure that everything written here can be done with GitHub Actions, Codeberg Actions or the CI of your choice.

Blog sources repository

I tried various ways to not do this, but in the end, I needed to put (parts of) my Obsidian vault (which is shared via iCloud) into a git repository. After adding the script to the CI file, I ended up with:

package:
  image: alpine
  script: 
    - apk add yq
    - |-
        find ./ -name "*.md" -exec sh -c '
        DATE_PART=$(cat "$1" |yq --front-matter=extract ".date");
        FILE_NAME=$DATE_PART"-"$(cat "$1" | yq --front-matter=extract ".file_name");
        DIRECTORY=$(dirname "$1")
        mv "$1" "$DIRECTORY/$FILE_NAME.md"
        ' unused {} \;
  artifacts:
        paths:
            - ./*
        exclude:
          - ".git/**/*"
          - ".git"
          - ".git/"
          - ".gitlab-ci.yml"
deploy:
  trigger: meistermeier/Blog

A small alpine Docker image with just yq added to the mix, my script that can be boiled down to working in the current directory, and some exclusions to not push the whole git and config into the world later. After the packaging went through, it triggers the Jekyll repository to do it’s job.

The Jekyll repository

Here the journey of the blog posts continues. Since a pipeline script says more than thousand words, here it is:

stages:
  - build
  - upload

generate:
  stage: build
  image: jekyll/minimal
  script:
    - apk update
    - apk upgrade
    - apk add curl
    - 'curl --location --output artifacts.zip --header "PRIVATE-TOKEN: $ACCESS_TOKEN" "https://gitlab.server/api/v4/projects/meistermeier%2FObsidian-Sources/jobs/artifacts/main/download?job=package"'
    - unzip artifacts.zip -d _posts
    - bundle install
    - jekyll build
  artifacts:
    expire_in: 1 hours
    paths:
      - _site/
upload:
  stage: upload
  image: alpine
  script: 
    - apk add lftp
    - lftp -c "<send things over the wire>"

I have two stages (mostly to skip the upload part while experimenting with the pipeline): build and upload. The build stage fetches the latest artifacts archive from the upstream project containing the blog posts and puts it into the target Jekyll structure (_posts). To get the build support for the custom plugin, I mentioned above, it is necessary to invoke bundle install to have it in place when Jekyll needs it during processing. After this, the build command gets invoked and produces the site. After this, the site will get deployed (here by using lftp) in the upload stage to the web host.

Conclusion

This took by far a lot longer to get “right” than I have expected. In the end I hope to give you some hints on how you could also create a pipeline without hitting every trap on the way, as I did. At least for me this works as I wanted it do. Also, the two projects clearly show the separation of concerns. There is now the content repository with pure Obsidian markdown (and images) and the infrastructure repository that, besides the archived posts, contains only the Jekyll page itself.

Happy blogging.

Gerrit's Blog

Thoughts and ideas, mostly tech based. But I don't promise anything.